►
From YouTube: OpenTracing Isn't Just Tracing: Measure Twice, Instrument Once [B] - Priyanka Sharma & Ted Young
Description
OpenTracing Isn't Just Tracing: Measure Twice, Instrument Once [B] - Priyanka Sharma & Ted Young, LightStep
Those building microservices at scale understand the role and importance of distributed tracing: it’s the most direct way to understand how and why a system is misbehaving. However, often this information has been hard to procure, given the challenges of explicit instrumentation, and once obtained, it is siloed from other relevant data such as logging and other monitoring.
The OpenTracing project provides a standard, portable API for distributed tracing instrumentation and changes that. In this talk, Priyanka and Ted will begin by describing OpenTracing and explaining why anyone who monitors microservices should care about it. Having laid that groundwork, the talk will step back to examine the historical role of operational logging and metrics in distributed system monitoring, then illustrate how the OpenTracing API maps to these tried-and-true abstractions. There will also be a demo involving donuts, distributed traces, and prometheus monitoring (all via OpenTracing).
About Ted Young
Ted has spent the last 15 years building distributed systems in a variety of environments: computer animation pipelines for VFX, national elections, and elastic compute platforms. Previously, he helped design the Diego Container Runtime for Cloud Foundry. Currently focused on OpenTracing and tools for root cause analysis.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So,
first
of
all,
why
should
you
care
about
tracing
at
all?
What
is
it
and
what
problem
is
it
trying
to
solve?
The
problem
is,
if
you're
running
an
application,
that
is
a
single
process
running
on
a
single
computer.
We
have
a
lot
of
great
monitoring
tools
in
that
situation,
but,
as
you
start
to
break
your
application
up
into
multiple
processes
running
on
multiple
computers,
you
start
to
fragment
the
information
collection.
A
A
There
isn't
a
lot
of
new
research
really
because,
as
a
research
subject,
it's
pretty
much
done
and
microservices.
Well,
that's
a
new,
like
buzzword,
we've
been
doing
service
architecture.
Since
you
know,
dinosaurs
ruled
the
internet
over
dial-up,
so
none
of
this
is
new.
So
why
aren't
we
doing
it?
The
problem
has
mostly
been
a
problem
of
the
API,
the
degree
to
which
that
we're
tracing
API
is.
They
were
tied
to
a
specific
implementation.
You
built
a
tracing
system.
You
built
an
API
for
that.
That
meant,
if
I
use
that
I'm
locked
into
that
tracer.
A
If
there
were
tracers,
they
tended
to
be
language-specific,
so
there
would
be
a
tracer
for
the
Java
Runtime.
It
may
be
a
tracer
for
the
Ruby
runtime,
but
there
was
no
way
to
get
these
talking
to
each
other
very
easily
and
then
the
third
problem
which
you'll
see
later
in
this
talk,
is
a
lot
of
the
things
you
want
to
trace.
A
A
lot
of
the
information
is
in
libraries
frameworks,
open
source
components
that
are
shared
amongst
many
applications,
and
so
those
libraries
really
have
a
problem
because
they
have
to
pick
a
winner
right,
they
could
trace,
but
they
would
have
to
pick
what
kind
of
tracing
you
are
going
to
do.
This
is
the
same
problem
with
logging
right
you
want
to
log,
you
have
to
pick
a
logging
format,
a
logging
framework,
and
that
might
not
be
the
one
that
the
application
running
your
code
is
going
to
use.
A
So
this
is
where
open
tracing
comes
in
open
tracing
is
a
standard
and
open
API
and
what
I
mean
by
open
is
not
just
open
source,
but
it's
not
tied
to
a
specific
tracing
implementation.
A
number
of
experts
in
the
field
who
had
already
built
previous
tracing
systems
got
together
and
developed
an
API
knowing
how
their
systems
work
to
ensure
that
it
would
work
across
multiple
systems
and
then
we've
recently
standardized
that
API
under
the
auspices
of
the
CMC
F
open
tracing
is
useful
for
a
wide
variety
of
instrumentation.
A
A
I'm
sure
people
have
been
in
this
situation
where
you're
asking
the
question
what
is
going
on
and
then
you
say,
turn
on
the
debug
logs
and
then
you
say,
turn
off
the
debug
logs,
and
so
you
really
want
to
separate
these
two
things
you
want
to
have
instrumentation
just
be
the
semantics
of
your
application,
not
really
worrying
about
the
overhead
so
much
and
then,
when
you
collect,
you
want
to
have
a
lot
of
choice
about
dynamically
scoping.
What
your!
A
There's
a
number
of
very
interesting
monitoring
and
tracing
solutions
starting
to
hit
the
market
Zipkin
from
Twitter
has
been
out
for
wire,
while
goober
will
be
releasing
Jaeger,
hopefully
in
about
a
month
ocular
from
Red.
Hat
is
another
open
source
solution,
there's
also
app
light
step,
which
is
where
I
work
and
a
number
of
other
things
is
starting
to
show
up.
So
it's
becoming
a
fairly
rich
environment,
even
though
it's
relatively
young,
so
real,
quick
sorrel
in
the
same
page.
This
is
basically
the
extent
of
the
API
from
a
conceptual
standpoint.
A
So
the
basic
idea
when
you're
tracing
a
system
is
you
have
spans
and
spans
are
just
operations,
so
a
span
might
be
an
HTTP
request.
It
might
be
a
database
call
it's.
It
might
be
just
a
made-up
concept
that
ties
together
a
bunch
of
things
for
you,
but
all
it
really
is
as
far
as
the
system
is
concerned
is
a
name,
a
tag
set
of
key
value
pairs.
A
start
time
and
a
duration,
and
from
that
you
can
start
linking
these
things
together.
So
a
span
can
spawn
child
spans.
A
As
you
link
your
operations
together
and
off
of
the
spans,
you
can
start
logging.
So
this
is
just
structured
data
that
you're
recording
on
to
the
span.
When
you
want
to
connect
systems
up
over
a
network,
you
can
serialize
the
span
into
what's
called
a
span
context
and
send
it
as
metadata
on
your
RPC
call.
That
span
context
contains
baggage,
which
is
again
key
value
pairs
that
you're
propagating
in
band.
A
So
this
is
things
like
user
IDs
request,
IDs
information
generated
from
one
component
that
you
might
want
to
use
later
down
your
stack
and
then
the
final
piece
of
the
API
is
the
tracer
API,
and
this
is
the
part
where
you
actually
plug
in
your
implementation.
So
you
can
write
tracers,
that
output
to
Zipkin
light
step,
but
also
as
we'll
see
later
in
this
talk
metric
solutions
such
as
Prometheus.
A
A
A
metric
you're
gonna
have
like
a
stack
of
instrumentation
code
like
five
times
the
size
of
the
actual
code,
and
a
lot
of
it
will
start
to
look
repeated
if
you
start
doing
this
so
after
a
bit,
you
might
want
to
consider
removing
your
logging
and
having
your
logger
just
pull.
The
information
coming
out
of
the
tracer,
metrics
and
alerting
also
is
very
useful,
because
all
the
information
coming
out
of
a
tracing
API
is
structured.
A
Data
is
very
easy
to
create
metrics
for
this
context,
propagation,
that's
baggage,
the
degree
to
which
you
are
already
trying
to
propagate
IDs
and
other
things.
This
is
a
great
mechanism
for
doing
that,
and
then
the
traditional
tracing
tools,
so
critical
path.
Analysis
is
really
trying
to
figure
out
where
exactly
the
latency
is
in
your
system.
A
There
are
tracing
tools
that
can
help
dig
into
those
kinds
of
problems,
and
so
you
get
all
these
different
ways
of
instrumenting
your
system
out
of
this
one
API,
which
you
can
kind
of
see
here.
So
this
is
the
sort
of
Venn
diagram
of
what
it
covers.
You'll
notice.
By
the
way,
there
are
some
things
that
are
not
covered
by
the
open
tracing,
API,
necessarily
I'm
sure
there
you
can
find
edge
cases
where
you
would
want
to
do
something
differently,
but
in
general
it's
it's
a
fairly
good
solution.
A
This
leads
me
to
turnkey
tracing
turnkey
tracing
is
the
fact
that
if
you
look
at
hooking
your
system
up
most
of
the
components
you
want
to
hook
up
to
link
all
of
your
components
together.
They're
your
servers,
your
HTTP
server,
your
G
RPC
server,
they're,
your
frameworks
right,
whether
it's
a
web
framework
like
Django
or
rails
or
runtime,
a
synchronous
framework,
for
example
like
G
event
or
the
Java
basic
Java,
runnable,
callable
kind
of
stuff,
and
then
you've
got
your
outbound
network
holes
in
your
service
clients,
so
G,
RPC
clients.
A
You
know
database
clients,
AWS
clients.
Those
are
the
things
that
you
need
to
instrument
with
open
tracing
in
order
to
propagate
these
fans
across
the
network.
What's
interesting
about
that
is
those
are
also
the
components
in
your
system
that
you
tend
not
to
write.
Those
are
the
off-the-shelf
components
that
you're
getting
from
the
open
source
community
and
those
are
the
components
we're
attacking
with
things
like
plug-in
and
native
open
tracing
integration.
A
So
you
hit
this
very
interesting
thing
now
now
that
we
can
instrument
those
libraries
if
you
want
to
come
to
an
instrument,
your
application,
rather
than
going
in
and
writing
a
bunch
of
logging
and
instrumentation
code
directly
into
your
system.
You
can
install
plug-ins
for
these
key
components
and
instantly
get
yours
I'm
connected
up
into
a
tracer
and
most
of
the
important
information
is
already
there
and
then
from
that
really
high
baseline
start
drilling
down
and
then
adding
application-specific
information.
A
But
you
can
really
get
up
to
speed
very
quickly.
The
degree
to
which
the
library
ecosystem
is
growing
to
include
the
libraries
that
you're
using
likewise
getting
information
out
of
your
load,
balancers
proxies
other
components
in
your
system
that
are
more
things
that
you
are
installing
and
running
rather
than
writing.
Engine
X
hot
proxy
stuff,
like
that
open
tracing,
is
a
good
solution
for
pumping
information
out
of
those
systems.
A
Okay.
So
let's
get
into
some
specific
examples.
So
this
talk,
we're
gonna,
focus
on
Prometheus
integration
and
showing
how
you
can
pump
metrics
out
of
open
tracing
into
Prometheus.
So
we're
gonna
do
a
quick
demo
of
this.
So
this
is
my
new
start
up.
It's
called
drone
Utz
its
drone
based
donut
delivery
right.
This
is
the
next
big
thing.
It's
gonna
be
amazing.
A
We
don't
have
the
drones
yet
that
will
be
in
the
next
funding
cycle,
but
we
do
have
a
web
app
so
and
unfortunately,
this
is
just
running
on
localhost,
so
we
can't
all
play
it
long.
But
if
you
are
interested
in
this
app,
we
are
doing
a
walkthrough
at
the
open
tracing
salon
tomorrow.
So
show
up
for
that.
If
you'd
like
to
dig
into
what's
going
on,
but
basically
I'm
going
to
click
on
some
donuts
I'm
gonna
order,
the
donuts
there
appears
to
be
some
things
happening,
I'm
waiting.
Why
are
these
donuts
taking
so
long?
A
I
thought
it's
supposed
to
be
fast
and
then
I
get
an
alert
because
we
don't
have
drones.
So
what's
cool
about
drone
'its?
Is
it's
instrumented
with
open
tracing
and
is
currently
piping
out
to
two
different
systems,
so
it's
piping
out
to
light
step,
which
is
a
tracer
and
it's
piping
out
to
Prometheus,
which
is
a
metric
system.
So
if
we
pop
into
light
step,
we
can
see
what
one
of
these
requests
looks
like
so
here
let
me.
A
So
we
can
see
the
draw
nuts
client
has
an
operation
called
buy
donuts
it
took
about
six
and
a
half
seconds
to
buy
these
donuts
and
we
can
see
if
we
pop
it
open.
What's
going
on
there
was
it
made
an
order
request
to
the
API,
followed
by
two
status
requests,
so
we
can
see
it
ordered
the
donuts
and
then
sat
there
and
pulled
the
server
until
the
donuts
were
ready
and
that's
how
this
wacky
app
works.
A
If
you
drill
into
the
order
span,
you
can
see
that
the
API
server
is
calling
this
kitchen
component
over
G
RPC
and
it's
first
calling
the
add
method
and
then
it's
calling
the
check
method
and
it
seems
like
for
some
reason
the
check
method
is
taking
way
longer
than
the
add
method.
If
we
dig
into
that,
we
don't
have
any
more
detail
in
the
kitchen,
but
we
can't
see
that's
where
all
the
time
is
going
and
if
we
pop
open
this
span,
we
can
see.
A
G
RPC
has
automatically
logged
all
of
the
request
information
onto
it,
some
structured
data.
So
you
can
start
using
this
information
to
figure
out.
What's
actually
going
on
in
this
system
now,
naturally,
if
you
wanted
to
dig
into
what
was
going
on
in
the
kitchen,
perhaps
you'd
want
to
add
more
granular
spans
into
that
service,
but
you
have
to
come
to
a
workshop
to
do
that.
A
At
the
same
time,
we're
pumping
this
information
into
Prometheus.
These
graphs
look
a
little
silly,
because
it's
just
me
standing
up
here,
but
have
faith
that
they're,
interesting
graphs
here
lives
show
this
so
this
first
one.
This
is
all
of
the
operations
in
the
system,
but
with
their
timing
divided
out
by
quantile.
So
we
have
the
50
percent,
ninety
percent
and
ninety
nine
percent
latency
distributions
for
all
the
different
things
that
are
going
on
in
our
system.
And
then
you
can
also
just
see
a
request.
Count
going
on
this.
A
And
so
that's
basically
it
right.
You
can
see
like
one
set
of
tracing
you're,
getting
multiple
things
out
of
it
and
just
to
be
clear,
you
don't
need
to
necessarily
read
all
of
this
or
no
go,
but
this
is
basically
all
of
the
code
that
I
wrote
to
connect
Prometheus
up
to
this
tracing
system.
So
every
time
a
span
comes
in
it
logs
the
latency
and
then
if
there
was
an
error
tag
on
the
span,
it
increments.
A
A
A
But
the
other
things
we're
really
interested
at
this
point
in
this
sort
of
life
cycle
of
open
tracing
is
instrumenting
the
ecosystem.
Now
that
the
API
is
really
starting
to
gel
up
we're
trying
as
quickly
as
possible
to
either
add
native
instrumentation
to
all
the
networking
and
service
clients
out
there,
as
well
as
the
frameworks,
basically
everything
in
that
turnkey
tracing
slide
and
if
we
can't
get
them
to
natively,
add
open
tracing
to
look
for
ways
to
add
plugins,
and
you
know
various
other
hooks.
A
A
We're
also,
of
course,
interested
in
instrumenting
larger
systems
such
as
kubernetes,
open
ship
things
like
that
Prometheus
we're
actually
just
talking
to
fluent
D
guys.
They
are
interested
in
this.
So
there's
a
lot
of
movement
there
and
finally,
we
have
a
distributed
tracing
salon
tomorrow
it
starts
at
2:00
p.m.
I'll,
be
doing
an
open
tracing,
walk
through
and
go.
You
can
actually
follow
along
in
your
own
language
if
you
like,
but
we've
got
an
actual
app.
A
B
A
A
Yes,
because
you
said
the
word
Linux
in
there
right
and
so
right,
so
how
do
I
add
tracing
to
my
JavaScript
client
in
the
browser
with
Linux
trace
points
right?
How
do
you
do
it
so
I
think
the
if
you
look
at
the
open
tracing
API,
it's
it's
based
on
these
things,
but
generalized
in
a
key
way,
so
that
it
works
across
platform
and
across
different
kinds
of
systems.
So
I
think
that's
the
reason
why
there's
a
need
for
a
new
API.
A
Yes,
so
there's
active
work
going
on
right
now
around
being
able
to
extract
information
from
existing
tracing
systems,
I
believe,
like
perf
of
n,
the
first
thing
that
people
have
added
I
think
there's
actually
another
talk
about
that.
But
Linux
trace
points
are
obviously
a
key
thing
being
able
to
read
log
output
and
generate
spans
after
the
fact
this
is
called.
You
know:
out-of-band
tracing,
that's
an
open
area
right
now
as
well,
but
yeah.
Definitely
that's
what
we're
looking
for
another.
A
A
Traditional
tracers
tend
to
focus
on
latency,
so
those
kind
of
like
bar
graphs,
showing
the
latency
and
where
it
is
in
the
critical
path
of
your
request.
There's
a
number
of
systems
that
supply
that,
at
light
step,
we're
really
focused
on
root,
cause
analysis,
so
being
able
to
add
SLA
x'
to
your
code
to
your
system,
dynamically
right
and
dynamically.
Do
a
lot
of
measurement
and
alerting
without
having
to
reboot
your
system
or
get
in
there.
A
So
that's
you
know
kind
of
where
our
focuses
and
you've
also
seen
I've
seen
examples
of
people
being
able
to
get
a
topological
view
of
their
system,
like
what
components
are
running,
what's
actually
going
on
like
what's
talking
to
what
out
of
just
getting
aggregate
information
out
of
traces.
So
that's
definitely
things
I've
seen
and
I've
also
seen
people
like
graphs
that
show
like
queue,
lengths
and
things
of
that
nature.
A
So
we
are
focused
exclusively
in
the
open
tracing
project
on
the
API
and
are
trying
to
avoid.
You
know,
to
a
certain
degree,
tapping
some
particular
thing
and
saying
this
is
an
official
implementation,
but
trying
to
just
you
know,
link
everyone
up
to
all
those
instrumentations,
as
they
show
up
yeah.
B
A
This
is
an
interesting
question.
Yeah
do
we
standardize
the
wire
protocol
so
far?
The
answer
has
been
an
emphatic.
No,
we
do
not
standardize
that
wire
protocol
I
do
believe
what
we
need.
As
far
as
a
standard
protocol
is
not
so
much
a
wire
protocol
as
an
interchange
format,
the
degree
to
which
it's
not
necessarily
the
format
you
would
be
using
to
send
over
the
wire.
Perhaps
that's
optimized,
for
you
know
a
particular
implementation,
but
the
degree
to
which
you
know
the
devops
world,
the
the
standard
token
of
currency
is
log
snippets
right
what's
wrong.
A
A
A
Yes,
so
tracers
in
general
have
to
deal
with
what
you
might
call
a
ragged
graph
right.
It's
you
might
not
get
an
end.
You
might
not
get
all
the
spans
right.
The
degree
to
which
all
these
fans
are
coming
from
different
systems
and
I
get
it
from
point.
A
and
I
get
it
from
Point
C.
But
point
B
is
having
some
trouble,
so
I
don't
get
that
so
in
general,
tracing
systems
have
to
deal
with
the
fact
that
it's
going
to
be
sort
of
ragged
information.
There's
going
to
trickle
in
right.