►
From YouTube: Keynote: OpenTracing and Containers: Depth, Breadth, and the Future of Tracing - Ben Sigelman
Description
Keynote: OpenTracing and Containers: Depth, Breadth, and the Future of Tracing - Ben Sigelman, Co-Founder, LightStep
"Those building microservices at scale understand the role and importance of distributed tracing: it’s the most direct way to understand how and why a system is misbehaving. But distributed tracing has long harbored a dirty secret: the source code instrumentation can been complex, fragile, expensive, and difficult to maintain.
The OpenTracing project addresses that integration problem through standardization and collaborations with other open-source software systems. In this talk, Ben will begin by describing OpenTracing and explaining why you should care about it. He will then show how OpenTracing will be able to deliver zero-touch, black-box instrumentation of distributed applications via orchestration systems like Kubernetes, and why that could change the way we all reason about distributed computation."
About Ben Sigelman
Ben is a cofounder at LightStep, a company that makes complex microservice applications more transparent and reliable. Previously, Ben spent nine years at Google where he ate lots of snacks and designed several large (~1M-process) distributed systems. The most significant of these were Dapper, an always-on distributed tracing system; and Monarch, a high-availability timeseries collection, storage, and query system.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
unless
it's
good
and
I
won't
say
anything
about
political
leanings,
but
you
can
take
a
guess:
I
guess,
anyhow,
yeah,
thanks
for
for
being
here,
I'm
here
to
talk
about
open
tracing
and
I'm
excited
to
to
tell
you
a
little
more
about
it,
so
I
think
a
lot
of
people,
don't
even
it
aside
from
open
tracing
or
like
well
what's
tracing.
Why
do
I
care
about
this?
So
let's
talk
about
that
first,
so
we
all
know
about
micro-services
I
think
it's
received
enough
buzz
word.
A
Actually,
it's
fun
go
on,
go
on
Google,
Trends
and
search
for
micro
services
and
then
serverless
computing.
It
just
like
this
total
and
sanity
hype,
curve,
stuff
and
they're
here
to
stay.
They
actually
deliver
a
lot
of
value
and
I
believe
in
that
value.
That
said,
they
break
your
monitoring
tools.
Monitoring
is
supposed
to
tell
stories
about
distributed
about
systems,
whether
they're,
distributed
or
not,
and
as
soon
as
your
as
your
monolith,
in
this
case,
this
rectangle
that
is
divided
into
libraries
as
soon
as
that
monolith
is
decomposed
into
microservices.
A
If
you
previously
told
a
story
about
what
happened
in
your
process
in
that
way,
with
this
squiggly
line
going
through
a
single
process,
that
story
is
gone
and
what
we
have
found
in
our
conversations
with
numerous
companies
that
have
adopted
this
sort
of
technology.
What
they've
been
telling
us
is
that
as
they
D
couple
their
systems
and
their
transactions,
you
know
lo
and
behold
are
no
longer
on
any
single
process.
A
They
are
literally
unable
to
answer
the
most
basic
questions
about
what's
happening,
and
if
this
doesn't
sound
painful,
then
you
haven't
gotten
there
yet,
but
it's
extraordinarily
painful
and
the
solution
historically,
for
this
has
been
distributed
tracing.
It
still
is
a
solution,
and
it's
wonderful.
So
the
question
is:
why
isn't
you
Vic
witness
if
tracing
is
so
great
and
it's
the
only
way
to
tell
a
good
concrete
story
about
transactions
and
distributed
systems?
Why
isn't
ubiquitous?
And
that
is
what
open
tracing
is
here,
for
the
reason
is
that
instrument
has
been
too
difficult.
A
It's
required
you
to
instrument
not
just
across
processes
but
across
library
boundaries
in
a
way
that
often
couples
you
to
poorly
engineered
libraries
that
you
know
we're
written
in
an
afternoon
or
a
weekend
by
someone
you're
locked
into
vendors.
In
many
cases
you
can
try
and
monkey
patch
things
in
Python
or
JavaScript
or
something,
and
then
you
know
the
version
that
your
monkey
patching
gets
upgraded
and
it
breaks
your
production
code
or
something
like
that.
This
happens
frequently.
This
is
another
common
failure.
Mode
for
distributed
tracing
API
is
aren't
consistent
from
one
language
to
another.
A
Even
a
nouns
and
verbs
aren't
necessarily
consistent
from
one
language
to
another
and
handing
off
the
context,
which
is
the
whole
point
of
tracing
from
one
library
to
another
or
for
one
process
to
another,
is
also
something
that
requires
some
amount
of
rigor
and
standardization.
So
this
is
why
tracing
instrumentation
has
been
too
hard.
Open
tracing
specifically
exists
to
solve
that
problem.
Unlike
a
lot
of
open
source
technologies,
open
tracing,
isn't
something
you
run
it's
a
standard.
It's
an
API
standard,
it's
not
like
it's,
not
something
that
you
deploy.
A
It's
something
that
you
program
against,
and
there
are
three
constituencies
that
care
about
open
tracing.
One
is
developers
who
are
working
on
cloud
native
or
micro
service
based
applications
as
I've
been
talking
about.
Those
are
typically
people
who
work
at
large
software
engineering
organizations,
but
it's
it
can
also
be
at
this
point.
A
We
see
it's
easy
enough
to
build
micro
services,
that
you
have
three-person
companies
that
have
dozens
of
services,
hundreds
of
services,
and
so
they
care
about
this
open
source
frameworks,
I'm
thinking,
especially
about
things
like
G,
RPC
or
finagle,
or
anything
having
to
do
with
control
flow
like
promises
and
future
libraries.
Those
things
have
a
lot
to
say
about
tracing
and
they
have
a
role
here-
oh
sorry,
boy
and
and
also
tracing
or
monitoring
systems.
A
So
that
should
be
obvious,
but
it's
beneficial
if
you're
building
a
tracing
system
to
bind
yourself
to
a
standard
and
rather
than
trying
to
conquer
the
world
with
your
particular
instrumentation
library.
So
this
is
open
tracing
of
constituencies.
The
architectures
is
pretty
straightforward
within
a
process.
A
There's
actually
just
since
I
made
this
slide
ocular,
which
is
a
something
coming
out
of
Red
Hat,
is
another
distributed
tracing
system
that
supports
open
tracing,
but
there
are
many
places
the
data
can
go
beyond
that.
Even
I
was
one
who
came
up
with
the
name.
Open
tracing
I,
sometimes
regret
it.
It's
it's
really
more
like
open,
instrumentation,
it's
an
API
that
could
be
used
for
many
things.
You
can
easily
bind
open
tracing
to
an
existing
metrics
system
like
a
stab
steer
from
UTS
base
system.
A
You
can
do
vanilla
logging
either
out
of
it
or
into
it.
It
supports
something
called
baggage
which
I'll
show
later
in
a
demo,
just
actually
very
powerful,
based
on
a
s
OSP
best
paper
two
years
ago
or
last
year.
I
can't
remember
called
pivot
tracing
and
it's
a
very
young
project
that
baby
is
the
same
age
as
I've
been
tracing
it's
ten
months
old
and
it's
it's
been
adopted
by
many
companies
that
have
I
think
just
experience.
A
A
lot
of
pain
in
this
area
and
open
tracing
address
as
many
of
those
pain
points
and-
and
it's
been
really
exciting
to
to
see
the
community
grow
around
it.
Okay,
great
so
I
tried
to
speed
through
the
intro
stuff.
I
I
now
need
to
talk
a
little
bit
about
the
history
of
tracing
before
I
get
some
demos
which
will
be
interactive
by
the
way
you
all
can
participate.
So
this
is
one
of
the
earlier
distributed
tracing
meetups
in
1755.
This
is
a
salon
in
France.
A
They
didn't
have
Donuts
yet,
but
you'll
get
to
that
later-
and
my
point
here
is-
is
partly
that
I,
just
like
talking
about
distributed
tracing
salon
which
we're
having
tomorrow
as
well,
but
also
that
this
has
been
going
on
for
a
long
time
like
tracing,
is
not
a
new
thing.
I
think
that
in
our
industry,
history
repeats
itself
every
five
or
ten
years.
That's
certainly
the
case
here.
Supercomputers
had
things
like
this
too
in
the
70s.
It's
not
like.
A
This
is
a
new
idea,
but
every
time
we
have
a
new
platform
to
build
it
on
new
questions,
come
up
and
there's
a
lot
of
discussion
to
be
had
and
open
tracing
being
as
young
as
it
is,
and
this
audience
being
as
skilled
as
you
are.
I
was
actually
hoping
that
I
could
recruit
some
of
you
to
show
up
tomorrow
morning,
at
9:50
to
11:45
anywhere
in
that
window
we're
having
kind
of
a
drop
in
thing
with
free,
donuts
and
coffee.
A
It's
a
distributed,
tracing
salon
with
donuts,
and
there
will
be
things
for
absolute
beginners
where
you
can.
You
know
learn
the
ropes
of
using
open
tracing
with
Zipkin,
as
well
as
people
who
are
a
little
more
sophisticated
up
to
experts
who
are
building
tracing
systems
and
want
to
you
know,
exchange
ideas
and
so
on
it'll
be
fun.
But
you
might
wonder
with
all
these
donuts
where
they
come
from.
A
That's
a
good
question,
so
they
come
from
donut
salon.com,
which
I
built
last
weekend
and
is
taking
off
like
very
quickly
it's
donuts
as
a
service
and
it's
blitzscaling
through
hyper-growth,
it's
micro,
service-oriented
it
has
concurrency
driven
latency
problems
and
thankfully
that's
built
open
tracing,
so
we
can
all
try
it
out.
So
what
I
was
hoping
you
all
can
do
is
maybe
just
take
a
minute
to
not
look
at
the
lecture
results
get
out.
Your
mobile
phones
go
to
donut
salon,
calm
on
your
mobile
device
and
order
some
donuts
I'm
going
to
do
it
here.
A
Let's
see
so
yeah
right.
So
this
is
donut
flan
calm.
We
have
six
donuts,
you
can
buy
it's
free.
Actually,
as
you
select
them.
Oh
wow.
This
is
good,
so
it
might
take
a
while-
and
that's
means
you
guys
are
actually
doing
it.
So
donut
salon
I
did
it
last
weekend
it
does
take.
It
does
have
some
concurrency
related
issues.
It
seems
like
people
are
really
doing
their
homework.
A
I
actually
applaud
you
all
for
for
actually
following
my
directions,
but
there
have
this
profound
problem
and
that
I've
built
dona
is
a
service
and
it's
not
working
we're
having
a
successes
aster,
because
lo
just
went
from
one
user
to
you
know
many
more
than
one
user.
So
I
think
there's
a
there's
a
question
here.
You
know
like
what's
happening.
So
let's
talk
about
donut,
salons
architecture.
For
a
moment
before
I
look
at
some
traces,
so
it
is
bootstrapped
as
a
start-up
with
a
single
donut
mixer,
a
single
donut
fryer
and
a
single
donut
topper.
A
So
these
are
all
services.
You've
got
your
web
client,
which
is
on
your
phones
right
now
and
unfortunately,
I'm
realizing
now
that
I'm
seeing
what's
happened
to
the
site.
That
is
probably
wrong
to
only
have
one
donut
at
a
time
going
through
these
queues,
they're
enforced
with
a
mutex,
thankfully
I've
instrumented
the
mutex,
with
open
tracing.
So
we
might
have
some
hope
of
understanding
what
happened
here
so
yeah.
Let's
look
at
some
traces,
I'll
start
and
being
that
it's
open
tracing.
This
is
we're
not
tied
to
a
particular
vendor
I'm
I
promise.
A
This
is
not
supposed
to
be
light
step
pitch,
but
I
am
going
to
use
light
step
just
because
I'm
more
familiar
with
it
yeah
you
can
see.
This
is
actually
pretty
funny.
So
when
I
was
pretending
when
I
was
going
through,
this
demo
with
earlier
I
was
trying
to
generate
real
load,
but
I
couldn't
top
what
you
guys
have
done
here
so
anyway.
This
is
load
over
the
last
six
hours.
A
Just
in
the
last
couple
of
seconds
really
there's
been
significant
regression
in
the
P
99.9
latency
and
some
of
these
traces
are
taking
1.2
minutes
I'm
sure
there
are
others
that
haven't
come
through
I'll
open
up
a
few
of
these.
Just
to
give
you
a
sense
yeah,
so
it
looks
like
we've
got
some
serious
problems.
This
is
a
typical
distributed
trace.
We
have
the
beginning
here,
you're,
actually
monitoring
from
the
browsers
themselves,
and
so
this
took
about
46
seconds.
It
talked
to
our
web
server
as
I
showed
you
in
the
this
diagram
here.
A
Talk
to
our
web
server
and,
and
then
the
web
server
has
a
serialized
process
like
any
donut
would
have
to
go
through.
You
have
to
mix
the
batter
you
have
to
fry
it.
You
have
to
top.
It
looks
like
topping,
is
going
pretty
well,
but
I'm
frying
is
really.
You
know,
chewing
up
a
lot
of
our
critical
path,
so
I'm
going
to
expand
that
and
yeah.
A
It
turns
out
that
this
instrumented
lock
has
told
us
that
we're
waiting
behind
141
actions
now,
if
I
go
back
to
donut
salon,
I
really
want
to
illustrate
this
feature
of
baggage.
I,
actually
think
I
mean.
Obviously
this
is
all
sort
of
a
joke,
but
it's
actually
not
totally
a
joke
in
that.
This
is
I,
think
pretty
powerful,
so
I've
assigned
an
ID
to
a
random
ID
to
every
web
client.
A
Here,
it's
just
a
four
digit
number:
nothing
interesting
that
that
ID
has
been
added
in
open
tracing
to
what's
called
the
baggage
which
is
carried
along
with
the
request
across
the
distributed
system
in
a
transparent
way.
So
it's
really
important
to
note
that
the
application
code
never
cares
about
that
baggage
it.
It
doesn't.
Even
really
have
to
know
about
it
and
open
tracing
is
able
to
propagate
this
context
through
through
the
entire
system,
and
that
allows
us
to
do
some
pretty
exciting
things
in
that
in
a
mutex
lock
way
down
the
stack.
A
Obviously
you
can't
have
a
mutex
lock
that
is
going
to
know
about
Donuts,
but
you
can't
have
a
mutex
lock.
That
knows
about
baggage,
so
I
can
say
well.
This
mutex
lock
is
instrumented
that
if
a
mutex
takes
too
long
to
acquire
it's
going
to
dump
out
some
additional
information,
and
so
in
this
case
it's
going
to
dump
out
the
141,
oh
darn,
it
that's
the
bummer.
A
This
state
that
we're
dealing
with
here
didn't
come
from
the
mutex
that
came
from
several
layers
up
in
the
stack
and
that's
pretty
important.
I
just
wanted
to
also
illustrate
that
this
works
just
as
well
with
Zipkin,
which
is
awesome,
iced
tea,
Adrian,
:
audience
actually
zipping
maintainer
I
can
find
traces
same
instrumentation.
The
only
difference
is
that
we
we
we
bound
the
the
open
tracing
API
to
Zipkin
instead
of
to
this
other
tracer
at
initialization
time.
So
this
is
the
idea
with
open
tracing
is
that
you
can
get
pretty
detailed.
A
High
fidelity
instrumentation
just
system
get
distributed
transactions
with
some
powerful
features
and
decouple
yourself
from
a
provider.
So
with
that
I'm
going
to
cease
giving
my
demo
yeah
so
vendor
neutrality
is
really
important.
It's
not
just
important
because
you
don't
want
to
lock
yourself
into
a
vendor.
I
mean
that's
always
true,
but
it's
particularly
important
in
this
case,
because
this
decision
about
which
vendor
you
use,
isn't
just
used
in
your
own
application
code.
It
also
needs
to
be
made
by
well.
A
The
non
decision
needs
made
by
every
aspect
of
your
system,
including
open
source
libraries
that
you
depend
on
so
a
vendor
neutrality
is
really
important,
because
if
you
use
a
web
services
framework
that
hasn't
chosen
the
same
provider
as
you
that
would
break
your
traces
and
with
open
tracing,
you
don't
have
to
make
that
decision
up
front
and
yeah
I.
Think
the
the
traces
reveal
a
lot
about
system
behavior.
A
So
I
wanted
to
get
into
talking
about
kubernetes
a
little
bit
since
this
is
so
many
of
you
here
care
about
kubernetes,
so
getting
quality
tracing
as
a
matter
of
getting
enough
breadth
of
what
we
call
spans
in
the
sort
of
open
tracing,
dapper,
Zipkin
model
and
and
links
between
traces
called
references.
If
you
only
have
spans,
it's
basically
just
logging,
it's
useful,
but
it's
not
sufficient
to
do
the
sorts
of
things
I'm
talking
about.
A
If
you
only
have
references
that
would
be
sort
of
like
just
tracing,
not
even
tracing
with
just
monitoring,
Network
links
or
something
like
that,
but
not
being
able
to
actually
say
what
those
messages
are
about,
so
it'd
be
at
level
3.
Networking
monitoring,
which
is
valuable
as
well
but
still
not
tracing
spans,
are
actually
very
easy
in
in
kubernetes
environment
or
something
similar
to
it.
You
can
incorporate
something
like
a
level
7,
client,
client
proxy,
like
linker,
D
or
Envoy.
You
can
easily
generate
one
span
per
request.
A
That's
again,
it's
cool,
but
it's
also
just
logging.
What's
more
difficult
is
to
get
the
references
to
work,
so
that's
to
connect
the
dots
between
clients
and
servers
and
ingress
and
egress
in
a
process,
and-
and
this
is
something
that
I
actually
I,
have
been
thinking
a
lot
about
and
I
think
open
tracing
in
2017.
A
big
piece
of
our
our
development
as
a
project
will
be
to
have
get
as
close
as
we
can
to
zero
touch
integration,
and
this
is
a
big
piece
of
that.
A
So
if
anyone
thinks
this
is
fun
I'm
going
to
replug
the
idea
of
coming
tomorrow
morning
and
talking
as
me
about
X
I'm
actually
really
want
to
solicit
ideas.
The
basic
idea
of
how
this
would
work-
and
we
have
seen
this
in
practice
and
in
large
scaled
out
commercial
installations
that
I'm
not
allowed
to
cite
the
specific
companies,
but
it
can
be
done,
but
I
want
to
generalize
it.
What
they
did
wasn't
I
in
my
mind,
totally
generalized.
You
have
this
process
where
an
HP
request
comes
in
enters
an
app
process.
A
Open
tracing
has
something
called
extract,
which
is
a
way
to
to
get
the
tracing
context
out
of
that
thing
in
a
way,
that's
that's
a
pretty
abstract
in
vendor-neutral.
Three
is
just
your
winding
path
of
whatever
application
code.
Does
your
business
logic
etc,
which
I
have
no
say
in
the
application,
eventually
makes
an
outbound
HTTP
request,
and
this
is
where
it
gets
interesting
that
it's
possible
and
I
want
to
talk
more
about
how
to
do
the
specifics
of
this
I
can't
go
into
it
in
detail.
A
With
the
time
allotted
the
application
makes
or
an
h-2b
request
via
a
local
proxy,
which
would
run
either
as
a
daemon
set,
or
you
know,
within
a
pod
or
so
on,
that
proxy
can
do
what's
called
an
open
tracing
in
jet
call,
which
is
just
a
compliment
to
extract.
It
will
add
the
tracing
context
to
that
bound
request,
wherever
it's
it's
bound
for
and
and
then
the
proxy
forwards
it
to
its
peer,
and
this
entire
process
starts
over
again.
This
basic
flow
is
is
something
that
I
think
is
very
exciting.
A
If
we
can
figure
out
how
to
do
it
properly,
it
would
be
possible
to
have
a
non.
You
know
non
built
on
a
weekend.
Donut
services
like
real,
honest,
Scot
services.
We
can
do
integration
with
distributed
tracing
with
almost
no
application.
Modification
we've
seen
this
at
companies
that
happen
to
have
build
something
like
this
and
I
really
want
to
generalize
it.
So
I
think
this
is
really
exciting
and
would
take
it
well.
It
would
would
take
good
advantage
of
many
of
the
features
that
the
features
within
kubernetes
yeah.
A
If
that
didn't
make
any
sense,
you
can
have
your
cake
and
you
can
eat
it
too,
like
this
wonderful
child,
I,
think
it's
possible
to
get
good
quality
tracing
and
avoid
the
pain
and
suffering
of
adding
a
lot
of
instrumentation
or
even
really
change
your
application
in
meaningful
ways.
If
we
can
add
the
existing
API
standardization
of
open
tracing
to
a
little
bit
of
magic
between
applications,
client
proxies
and
and
then
the
network
that
connects
containers
to
each
other
final
plug.
Just
for
some
of
the
topics
we'll
be
talking
about
tomorrow
morning.
A
It's
definitely
a
drop
in
kind
of
thing.
We
have
you
know
some
cool
t-shirts
and
doughnuts
and
stuff
like
that,
but
I
really
sad
to
me
to
come
in
at
a
conference
like
this
and
just
talk
to
talk
at
everyone
in
the
audience.
I
would
really
much
rather
talk
with
everyone
about
this
stuff.
So
please
come
by.
If
this
strikes
your
fancy
or
if
you
just
want
to
get
started
with
with
the
basics
and
yeah
I,
think
that's
that's
it
for
me.
You
can
be
in
touch
with
me.