►
Description
Would You Like Some Tracing With Your Monitoring? - Yuri Shkuro, Uber Technologies
Understanding how your microservices based application is executing in a highly distributed and elastic cloud environment can be complicated. Distributed tracing has emerged as an invaluable technique that succeeds where traditional monitoring tools falter. Yet deploying it can be quite challenging, especially in the large scale, polyglot environments of modern companies that mix together many different technologies. In this talk we share what we have learned while building and rolling out Jaeger, our open source, OpenTracing-native distributed tracing system, to hundreds of microservices at Uber. We showcase new and exciting features that make it even more valuable to engineers.
About Yuri Shkuro
Yuri is a Staff engineer at Uber Technologies, working on distributed tracing, reliability, monitoring, and performance. He is a member of the CNCF OpenTracing Specification Council, and the founder of Jaeger, Uber's open source distributed tracing system.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Oh
in
this
talk,
I
will
try
to
explain
why,
as
cloud
developers,
we
should
care
about
distributed.
Tracing
I
will
show
a
demo
of
CN
CF
project
Jaeger,
which
is
a
distributed.
Tracing
system
and
I
will
speak
about
the
challenges
of
actually
rolling
out
distributed
tracing
in
a
large
organization
like
uber,
and
some
lessons
that
we've
learned
from
that.
Just
as
an
introduction,
why
you
should
care
about
anything
I
say
so:
I
been
doing
tracing
for
about
two
years.
A
So
this
is
a
diagram
which
Jager
can
present,
which
is
about
like
sort
of
snapshot
to
fubar
architecture,
business,
not
everything
but
some
of
it,
and
so,
when
you
use
uber
app
or
any
right
healing
app
really
every
second,
the
app
communicates
to
the
backend
right-
and
this
communication
might
look
like
this.
So,
instead
of
talking
to
just
one
single
server,
every
request,
like
spreads
out
across
different
number
of
micro
services,
potentially
hitting
hundreds
of
thousands
even
of
individual
service
instances,
and
at
least
at
our
scale,
that
happens
billions
time
today,
right
so
complex
system.
A
A
There
are
various
techniques
of
people
telling
you
what
you
should
be
monitoring
to
kind
of
know,
whether
your
components
are
healthy
or
not,
and
there
is
various
products
like
study,
premises,
graph
on
which
allow
you
to
collect
those
metrics
visualize
them
similar
store
and
login
I'm,
taking
about
application
events
log
in
and
also
like
errors
and
stack
traces
and
so
as
well.
There's
the
ecosystem
of
tools
that
allow
you
to
grab
all
the
data
and
sort
of
aggregated
and
present.
So
do
these
tools
actually
help?
A
A
Well?
I,
don't
know
like
the
code
in
that
demo
didn't
even
use
this
scanner
class
at
all
right,
and
so
yet
it
crashed
with
this
message
so
and
the
it's
very
unusual
for
program
to
crash
like
this,
because,
typically
you
get
a
stack
trace
in
in
any
program
right
and
this
one
crashed
without
x-rays,
which
is
surprising
to
me
anyway.
A
I
didn't
wanna
investigate
it,
but
the
point
here
is
that
when
we
are
talking
about
microservices
based
application,
metrics
and
logs
are
essentially
giving
you
this
one
line
about
one
instance
of
a
service
somewhere
right
and
how
do
you
debug
a
problem
without
stack
traces,
so
tracing
distributed?
Tracing
is
essentially
giving
you
what
you
couldn't
consider
a
distributed,
stack
traces.
What
what
happens
across
all
your
service
is
in
a
micro
C
in
in
architecture
and
so
really
when
we
trying
to
mordor
architecture
as
complex
as
uber.
A
We
want
to
modern
monetary
disability
transaction,
not
just
individual
instances
and
just
before
I
do
the
Jama
I,
just
wanna
give
a
basic
idea
of
how
distributed
tracing
works.
So
imagine
we
have
a
service
that
requests
coming
to
our
system
and
then,
when
the
request
comes,
what
we
do
is
we
assign
a
unique
ID
to
that
request,
simple
thing,
and
we
also
introduce
a
notion
of
a
context
and
the
context
is
something
that
we
want
to
keep
with
this
request
with
this
transaction
as
it
travels
or
the
rest
of
the
architecture.
A
A
As
long
as
we
sort
of
can
identify
all
the
instrumentation
within
those
service
by
this
unique
idea
that
we
assigned
at
the
top
right,
you
can
modify
context
who
else
record
some
causality
information
like
the
fact
that
actually
be
called
C,
and
not
just
someone
called
C
in
this
transaction
right,
and
so,
if
we
capture
all
that
data,
in
addition,
it
doesn't
have
to
be
in
the
context
but
captured
in
the
background
somewhere.
Then
we
can
build
a
trace.
A
The
timeline
which
you
see
on
the
right
here
so
and
then
next
thing
just
before
again
before
the
GMO,
just
a
quick
introduction.
What
Jaeger
is
a
disability
rating
system?
We
started
the
tuber
and
we
open
sourced
it
a
few
months
ago
now
it's
a
official
CN
CF
project
from
September
and
now
I
will
show
you
some
demo
of
of
the
tracing,
so
I
will
start
with
a
kind
of
show
you
showing
you
an
application
that
I
want
to
use
for
the
demo.
A
So
this
is
a
mock
application,
hot
rod,
that's
like
right
on
demands
and
what
you
can
do
with
this
one
is
you
can
pick
a
customer?
You
click
a
button
and
the
bug
back-end
just
goes
and
dispatches
a
car
pretend
car
to
you
and
it
gives
you
the
license
number.
It
also
says
when
this
car
is
arriving
and
there
is
some
for
our
purposes.
A
Debugging
information
say-
and
this
is
the
unique
request
ID
for
that
request
to
the
backend,
and
this
is
how
long
it
took
to
execute
on
the
back
and
from
the
point
of
view
of
the
of
the
front-end
right.
So
that's
all
I'm
going
to
tell
you
about
this
application
right
and
so
back
to
the
point
of
monitoring
tools
to
tell
you
the
stories
about
the
application.
So
what
can
I
get
about
this
application
actually
using
tracing
as
a
monitoring
tool
right
so
I'll
go
to
to
the
Yaeger
front-end?
A
Let
me
reload
it
so
the
instrumentation
in
that
application
already
sent
some
data
to
tracing
back
on,
and
so
one
thing
I
see
here
that
ow
I
suddenly
got
this
list
of
services
that
I
pretend
apparently
included
in
this
application
right.
So
but
let's
not
go
there.
First,
let's
go
to
this
tab
called
dependency
diagram,
there's
a
separate
view
so
purely
by
monitoring.
The
interactions
between
the
services,
of
course,
with
instrumentation,
we
got
this
diagram
which
actually
tell
a
lot
about
the
application
right
away
right.
A
We
can
kind
of
figure
out
what
its
architecture
we
can
see,
how
many
requests
go
to
which
services
we
can
see
that.
Apparently
there
are
two
back
and
database
backends
within
this
application,
so
we
have
need
to
do
with
envy
to
do
anything
about
like
to
get
that
just
to
run
the
application.
So
the
second
thing
is
what
you've
all
this
gives
us.
Turkey
T
actually
doesn't
give
us
the
actual
workflow
who
calls
whom,
what
the
business
logic
within
this
application
so
for
that
I
can
go
and
I
search
for
traces.
A
So
there
is
one
trace
here
at
the
top
notice.
It
says
seven
or
four
milliseconds.
This
is
a
bit
lower
than
this
one,
because
there's
a
this
is
from
the
client
point
of
view.
This
is
from
the
server
point
of
view,
so
obviously
it's
shorter,
but
when
I
click
on
the
trace.
So
there
are
a
lot
of
things
on
this
on
the
screen.
A
A
lot
of
information
so
most
important
one
is:
you
can
see
that
this,
the
time
sequence
diagram
that
I
showed
you
in
the
previous
slide
is
kind
of
now
for
the
real
service
right,
and
these
are
individual
operations
executed
by
individual
services
listed
on
the
left
and
this
evaporation
names
and
how
long
they
took.
So
one
thing
that
we
can
immediately
say
about
the
service
is
that
okay?
Well,
this
MySQL
select
operation
takes
almost
50%
of
the
time.
A
So
if
you
were
trying
to
understand
why,
if
your
recent
production
is
slow,
just
one
look
at
the
trace
as
well.
This
is
probably
well
at
least
a
good
place
to
investigate
what
what
is
it
doing
to
for
that
long
right
after
all,
so
Lorraine
on
my
local
machine?
How
much
the
second
time
you
can
spend
on
the
retrieving
date
from
MySQL?
What
you
can
do
as
well
is
you
can
actually
drill
down
into
this
and
say:
okay?
Well,
this!
This
is
the
actual
statement
that
was
executed.
A
So
this
is
something
that's
automatically
captured
by
tracing
instrumentation
and
you
can
see
that
within
this
individual
kind
of
span
right
span
as
an
iteration
within
within
a
trace,
so
very
rich
data.
It
also
says
that
there
is
a
log
statements
here,
acquiring
log
blah
blah
blah,
whatever
we'll
see
why
that
is
interesting
later
so
another
thing
we
can
see
here
is
if
I
click
at
the
top.
This
is
the
kind
of
the
entry
level
of
the
whole
application
right,
the
so-called
root
span,
which
spans
the
whole
request.
There
are
lots
of
logs
here.
A
If
we
look
at
the
logs,
then
if
you
read
it
all
of
it,
you
kinda
can
get
a
overall
idea
and
not
just
the
architecture
now,
but
what's
the
actual
business
flow
within
this
application
so
like
there
is
a
we
were
getting
a
customer,
then
we
were
finding
the
like
closest
drivers.
We
were
calculating
the
routes
to
each
driver
like
this
and
then
finally,
we
pick
the
best
driver
who
is
like
the
closest
or
the
shortest
to
reach
it,
and
they
in
the
given
point
that
we
wanted
and
they
will
turn
that
data.
A
So
why
am
I
showing
this?
You
could
clearly
you
could
get
the
same
thing
from
the
logs,
however,
note
that
these
all
these
applications
could
have
been
running
on
different
instances,
potentially
right
and
also
each
application.
If
it
was
a
production,
then
there'd
be
a
hundred
requests
going
through
a
single
instance.
So
if
you
try
to
look
at
the
logs
this,
this
18
entries
are
going
to
be
mixed
up
with
gazillion
of
other
entries
from
the
same
application
just
from
different
threads.
And
how
do
you
make
sense
of
any
of
that?
A
While
tracing
actually
gives
you
just
this
this
transaction?
It
ignores
everything
else,
because
it
can
correlate
all
the
logs
with
this
transaction
ID
or
the
trace
ID
and
that's
very
important.
Basically,
a
feature
of
the
of
the
tracing
is
that
it
can
give
you
a
highly
contextualized
information
about
all
the
things
that
your
application
is
doing,
and
it
also
like
again
note
that
these
logs
are
attached
to
this
span,
because
it's
really
the
separation
that
was
doing
all
these
requests
from
the
top
level
right.
A
A
A
The
red
button
here
says:
there's
an
error.
We
can
actually
look
at
it.
It
was
probably
like
radius
time
out
right.
So
fine,
the
service
retries,
it
increases
the
latency,
but
even
if
there
are
no
errors,
there's
a
very
clear
pattern,
staircase
pattern
here,
which
indicates
that
well
all
these
operations
are
done
sequentially
and
now,
since
we
didn't
write
the
application,
we
can't
really
say
whether
that's
the
right
thing
or
not
for
the
application.
Well,
I
wrote
it
so
I
do
know
that
it's
the
wrong
thing.
A
It
could
have
been
like
Fenian
out
all
these
requests
in
parallel,
because
all
the
detail
is
basically
it
loaded
all
the
drivers
first
and
then
getting
some
additional
information
for
other
drivers.
So
you
just
read:
pool
paralyze
it.
You
could
reduce
this
hundred
80
milliseconds
to
probably
what's
the
longest
like
10
milliseconds,
so
another
source
of
latency
right
away,
just
by
looking
at
the
trace
I,
didn't
have
to
do
any
sort
of
measurements
anywhere,
at
least
oh
and
the
final
one.
So
this
one
is
interesting
by
the
way.
A
There's
a
I
can
actually
do
this
zoom
in
onto
this
section.
So
so
here
we
have
a
whole
bunch
of
requests
to
the
route
service,
and
actually
there
are
ten
of
them
because
there
are
ten
drivers
loaded,
but
the
the
execution
part
is
a
bit
strange
like
we
see
here
that
there
is
some
parallelism,
the
three
requests
were
starting
to
execute
in
parallel,
but
then
it's
not
three
at
a
time.
A
Actually,
if
you,
if
you
look
at
the
vertical
lines,
it's
always
three
at
the
time
because
well
simply
there
is
a
thread
pool
behind
it
with
the
size
of
three
and
it's
it
limits.
How
much
work
you
can
do
on
doing
this
particular
iteration.
So
this
is
not
a
problem
in
this
specific
trace,
but
if
we
were
doing
a
lot
of
requests
and
I
can
easily
show
you
like,
if
I
go
here
and
let's
say,
do
this
right
notice
that
the
latency
keeps
climbing.
A
A
So
one
point
six
seconds
well
so
now
this
MySQL
is
not
just
it
got
even
worse,
read
with
with
the
load.
Obviously
my
SQL
server
could
have
scaled
is
it.
It
is
not
a
problem
here,
because
it's
a
simulated
series
really,
but
we
can
look
at
the
again
and
the
span
and
say:
oh,
you
know
what
I
was
actually
waiting
on
a
log
for
other
transactions
and
notice.
The
interesting
thing
it
gives
you,
the
transaction
I
just
remember
this
transaction
IDs
from
here,
so
this
service
somehow
knows
this
individual
request
like
we're.
A
Looking
at
one
trace
one
request,
but
it
knows
about
other
requests.
We
should
block
in
this
particular
request
on
a
particular
resource
contention
right,
so
that
information
is
actually
hard
to
come
by
if
without
certain
features
that
ll
Devon
in
in
a
second
India.
So
again,
speaking
of
optimizing
performance
application.
Clearly
this
is
one
of
the
best
thing
like
this.
I
was
actually
not
able
to
show
you
the
this.
A
This
impact
on
this
thing,
because
we
need
to
solve
the
my
skill
problem
first,
but
if
we
solve
it
to
make
it
fast,
then
this
thing
becomes
a
bottleneck,
because
your
thread
pool
of
three
obviously
is
blocking
now
a
bunch
of
requests.
So
you
can't
even
two
three
equations
per
like
in
parallel
for
one
transaction
you're
going
to
be
waiting
on
this
contention.
A
So
the
whole
point
of
that
is
like
I
I'm,
just
tweaking
one
single
trace
about
the
application
and
suddenly
I
know
so
much
about
performance
profile
of
this
whole
application
and
I'm,
not
thinking
about
one
single
service
right
that
you
could
potentially,
who
could
have
done
with
some
profiling
tool.
But
I
am
talking
about
the
application
as
a
distributed
application
with
multiple
micro
services,
which
you
could
run
as
multiple
instances.
So.
A
One
one
one
last
thing
I
want
to
demonstrate
here
is
so
this
application
also
needs
metrics
and
I.
Have
this
this
metric
here,
which
is
kind
of
interesting,
so
this
metric
says
this
is
a
how
much
time
we
spent
calculating
the
route
by
customer
and
what's
interesting
about
it,
was
like
kind
of
easy
to
calculate,
but
the
issue
here
is
that
if
we
look
at
the
at
the
diagram
here
right,
so
this
is
the
the
route
service,
the
route
service
actually
doesn't
know
anything
about
the
customer.
It
doesn't
need
to
it's
all--it's.
A
If
you
store
the
customer
in
the
context,
then
that
context
is
available
to
every
single
node
within
your
application,
and
then
they
can
do
additional
statistic
gathering
based
on
that
information,
even
though
they
don't
really
get
that
information
in
there
a
direct
API
call
or
anything,
because
they
really
don't
care
and
that
provides
like
very
powerful
features.
If
you
wanted
to
cost
attribution
like
at
Google.
Most
of
the
requests.
Google
has
also
very
deep
stack.
So
if
you
have
a
like
a
gmail
eventually
it
might
reach
some
storage
like
a
big
table.
A
So
the
request
actually
carried
the
fact
that
this
is
request
coming
from
Gmail
all
the
way
to
the
storage,
because
then
they
can
attribute
that
cost
to
something
that
makes
sense
for
the
business
right.
So
in
this
case
yeah
sure
I
can
attribute
cost
to
let's
say
like
this
service.
That's
calling
me
that's
easy,
but
really
we
want
to
attribute
the
cost
of
doing
the
work
to
some
business
concept.
That
makes
sense
for
our
business.
For
example,
if
it
was
over,
we
could
say
okay.
This
is
a
write
request
like
for
the
for
the
ride-sharing.
A
Oh,
this
is
a
uber
is
delivery,
so
this
is
a
high
level
business
and
then
we
can
say:
ok
well
we're
spending
this
many
dollars
per
year
on
this
kind
of
business.
By
doing
this
attribution
right,
so
it's
very
powerful
technique
and
I
will
I'll
speak
about
it
in
a
bit
more.
But
this
is
this
kind
of
DM
application
demonstrates
how
you
can
get
it
and
just
to
prove
you
that
I'm
not
lying
about
the
fact
that
route
service
doesn't
know
anything
about
the
customer.
A
We
can
go
to
the
root
server
and
we
can
look
at
the
URL
request
that
it
gets.
This
is
all
it
gets
right.
Let's
just
start
pick-up
and
drop-off
point,
that's
all
it
has
right.
So
how
does
it
get
the
customer
information?
Well,
customer
information
comes
from
the
context
and
Kyle
context
is
propagated
automatically
by
tracing
instrumentation
transparently
to
the
service.
You
don't
need
to
change
the
API
all
series
or
anything
else.
Imagine
if
you
want
to
pass
this
data
like
for
you,
so
that
services
actually
know
about
that.
A
You
would
have
to
go
and
change
like
a
lot
of
services
and
change
their
API.
So
it's
very
expensive.
So
let
me
stop
here
and
go
back
to
my
slides
and
by
the
way,
this
demo
there
is
a
walkthrough.
If
you
interested
in
in
this
link,
the
slides
will
be
shared
afterwards.
So
it
goes
in
a
lot
more
details
about
how
you
can
actually
troubleshoot
these
things
using
tracing
and
it
it
has.
Two
aspects
is
like
talks
about
open
tracing
and
the
jäger,
specifically,
and
and
really
what
you
can
do
with
that.
A
So
to
summarize,
one
thing
that
opened
that
the
tracing
systems
provide
is
the
ability
to
monitor
distributed
transactions.
The
other
thing
is
to
do
root,
cause
analysis
right.
So
if
you
can
find
a
trace
which
is
looking
suspicious,
you
can
actually
easily
drill
down
into
all
various
components
that
participated
in
the
transaction
and
figure
out
what
was
going
wrong.
You
can
also
do
performance
and
latest
optimizations
using
the
same
tool
and
that's
not
just
with
individual
traces.
You
can.
A
If
you
build
some
aggregation
of
the
traces,
then
you
can
see
patterns
within
the
applications
and
that's
okay,
I'm
going
to
be
talking
more
about
this
and
finally
service
dependency
analysis.
So
far,
it
was
fairly
simple,
but
like
it's
just
a
simple
diagram:
I'll
show
you
a
more
advanced
versions
of
it
and
all
of
that
functionality
is
fundamentally
built
on
distributed
context
propagation,
and
this
is
something
that
I
want
to
stress
that
context.
Propagation
in
micro
service
is
an
extremely
important
concept
concepts.
A
Many
people
not
doing
this
and
you'll
regret
if
you
reach
a
certain
maturity
with
an
organization
but
you're
not
doing
this.
So
it's
easier
to
start
from
the
from
the
beginning
to
have
it
so
like
who
likes
tracing
now
like
after
I've,
given
this
presentation
right
so
quick
poll,
how
many
people
here
in
your
company
organization
have
distributed
tracing
system
deployed
and
then
use
all
right,
pretty
good
percentage
more
than
I
expected.
A
So
if
the
trace
eNOS
was
so
fun,
as
we
saw
before,
how
come
it's,
not
everyone
raising
their
hands
right
and
well.
The
question
is
kind
of
embarrassing.
A
bit
for
for
the
industry
is
that
the
instrumentation
has
been
too
hard
so
far
to
do
tracing
it's
like
with
logging
and
metrics
it's
easier.
It's
also
work
but
we're
stressing
it's
it's
a
bit
more
work
and
I
want
to
explain
why
it's
a
bit
more
work.
A
So
imagine
you
have
a
service
and
it
has
like
a
server
end
point
and
then
there
is
a
downstream
call
that
you
make
and
right-
and
so
presumably
we
add
some
instrumentation
around
these
things
to
actually
do
the
tracing
at
the
entry
and
exit
points,
and
presumably
some
upstream
server
is
already
also
instrumented
and
sent
in
as
the
trace
ID
in
the
request
headers
right.
So
what
the
very
first
thing
we
do
in
this
application
with
the
instrumentation.
A
Really
what
it
does
it
says:
okay,
take
the
headers
extract,
the
trace
context,
create
a
context,
object
in
memory.
That
kind
of
you
get
that
almost
for
free
with
the
standard
frameworks
today.
The
second
thing
you
need
to
do
is
like,
as
your
application
doing
its
work.
You
need
to
keep
that
context
around
so
that,
if
you
do
happen
to
make
a
downstream
call,
you
can
pass
that
context
downstream
right,
because
if
you
lose
it,
then
the
trace
is
broken.
A
You
can't
follow
the
transaction
anymore,
so
number
two
and
then
the
third
one
is
when
you
actually
do
to
make
a
call
you
take
that
context.
The
instrumentation
encodes
it
again
and
to
the
trace,
headers
and
passes
on
to
the
next
request,
and
meanwhile
there's
a
another
library
which
kind
of
gets
the
callbacks
from
all
this
instrumentation
as
though
I'm
collecting
tracing
data
I'm
gonna
submit
it
to
the
tracing
back
end
somewhere
for
for
actual
tracing
aggregation.
A
So
so
this
the
this
number
two
thing
is
what
what's
known
as
in
process
context
propagation
different
from
distributed
context
propagation,
and
this
is
actually
the
the
thing
that
was
preventing
tracing
from
being
wild
mainstream
these
days,
because
it's
actually
not
straightforward
to
do
so.
It
sort
of
depends
on
the
language
as
well.
If
you
take
languages,
we
support
thread
locals
then
well,
you
can
actually
store
the
context
in
thread
local
and
sort
of
get
it
in
the
next
phase
or
like
in
the
next
layer
of
your
application.
A
Let's
say
in
the
client
in
the
previous
example,
and
so
it
seems
like
it's
a
KP
case
you,
you
can
do
instrumentation
almost
without
change
in
your
application
at
all
right,
it's
almost
KP
case,
except
that
we
don't
do
software
these
days
anymore.
Like
this
right,
we
don't
write,
Java
applications.
We
request
this
process
by
a
single
thread.
Most
applications
now
become
my
synchronous.
They
use
queues
inside
the
kamati,
threaded,
pools,
etc,
and
so
there's
actually
complicates
a
lot.
This
whole
process
of
yeah
what
seems
simpler
with
the
thread-local.
A
Now
it's
not
as
simple
there's
another
set
of
languages
where
thread-local
isn't
even
anything
that
can
go,
you
can't
identify
your
routine
and
so
what
you
do
in
that
case.
Well,
you
have
to
pass
context
explicitly
and
that's
like
a
sad
story
here,
because
in
many
cases
it's
if
you
didn't
write
your
application
from
the
start.
With
passing
the
context,
then
we'll
calculate
or
you
have
to
rewrite
a
lot
of
API
calls
internally.
So
it's
not
so
happy
I
mean
fortunately
in
go.
The
context.
A
Object
is
a
standard
language
function
like
object
to
the
class
so
goal
language
encourages
you
to
use
that
all
over
the
application,
so
hopefully,
as
the
community
will
start
doing
that.
But
it's
actually,
if
you
do
do
that,
then
tracing
becomes
almost
easy,
so
like
in
in
a
tuber
tracing.
Go
applications
was
actually
not
that
hard,
because
most
people
didn't
know
how
to
pass
contacts
around
right
and
contacts
exist
for
the
purposes
as
well
like
timeouts
and
then
conciliation
and
stuff.
So
it
was
already
a
mechanism
which
we
can
piggyback
on.
A
So
is
there
a
zero
trace
instrumentation
does
it
exist
even
as
a
concept
so
like
I,
said
fundamentally,
it's
not
impossible
in
some
languages
like
go,
but
if
you
do
pass
context
that
it
kind
of
becomes
almost
easier,
almost
like
almost
free
and
then
with
the
thread
locals
as
I
said
it's
it's
a
double-edged
sword.
It's
you
get
some
benefits.
It's
sometimes
it's
easy,
but
sometimes
very
hard,
especially
if
you
work
in
with
the
customer
we're
going
to
do
some
custom,
a
synchronous
processing,
so
we're
cube
con.
A
A
So
very
nice
concept
and
they
do
monitoring
as
well,
so
they'd
clearly
concerned
metrics,
but
they
can
also
do
tracing
for
you
like
the
example
lyft
I
can
lift
95
percent
of
services,
don't
do
tracing,
it's
all
comes
from
from
envoi
right
and
the
the
fine
print
over
there
is
that
to
enable
tracing
you
just
need
to
pass.
The
header
with
an
application.
Well,
ironically,
pass
in
the
header
is
the
exact
same
program,
open
process,
context,
propagation
right.
A
If
you
run
in
the
situation
with
again
the
thread
locals
and
multi-threaded,
it
becomes
the
SS
challenging
as
otherwise
and
so
like
open
tracing
as
an
API,
now
provides
a
primitives
for
doing
context,
propagation
and
because
open
tracing
API
is
being
integrated
into
a
lot
of
framers
like
akka,
which
is
like
actor
frameworks
with
very
high
I
synchronous
II.
It
can
do
the
things
for
you
without
you
change
in
application.
But
if
you
write
your
custom
thread
pools,
then
you
sort
of
have
to
do
a
bit
of
work
for
that.
A
So
now,
what
about
kind
of
what
we
learnt
at
uber
from
doing
these
things
like
it
actually
felt
like
this
guy
we
have
by
now
close
to
3,000
micro
services
and
about
half
of
them
are
instrumented
for
tracing
and
that
half
has
been
a
percentage
for
like
a
year
and
a
half
as
far
as
I
remember,
even
though
the
number
of
micro
service
keeps
growing.
So
it's
a
tough
and
it
doesn't
help
with
the
fact
that
we
have
like
four
languages
at
uber.
A
So
it
makes
it
even
harder
on
my
team
to
actually
provide
instrumentations
for
languages
and
frameworks
and
write
the
client
libraries
so,
but
if
you
are
going
to
do
this
in
your
canoes
Asian,
so
what
what
you
should
do
like
one
thing
I,
would
strongly
advise
using
open
tracing
because
it's
actually
decouples
you
from
the
actual
tracing
system.
If
you
don't
like
Jaeger,
if
you
want
to
switch
to
commercial
vendor
which
does
a
lot
of
work
potential
Danny
ager,
then
you
don't
need
to
change
your
applications
that
open,
trace,
instrumentation
stays
the
same.
A
You
should
use
infrastructure
libraries
in
the
organization
which
are
shared
across
teams,
so
that,
like
you,
don't
reinvent
the
wheel
in
every
service,
and
if
you
do
that,
then
it
becomes
a
bit
easier
to
enable
tracing,
because
that's
the
only
place
you
had
kind
of
have
to
go
in
instrument
things
and
a
good
thing
is
many
of
them
already
instrumented
open
tracing,
because
it's
an
open
source
API
and
it's
like
still
vendor
independent
and
one
important
thing
is
like
don't
make
configurations
for
tracing
right.
It
should
come
enabled
by
default.
A
You
can
have
a
configuration
to
disable
it,
but
we
we
made
a
mistake
originally
with
our
Python
clients
off
like
you,
we
actually
had
to
go
and
enable
like
put
a
boolean
flag
and
the
config,
and
that
just
like
was
completely
unnecessary.
Friction
to
to
roll
in
this
out
education
is
very
important,
distributed
context.
Propagation
I
mean
the
upper
came
out.
What
in
2008
10
years
ago,
the
concept
is
still
new
to
many
developers.
A
So
definitely
given
talks
internally
in
the
company
and
explaining
why
it's
important
and
showing
some
use
cases
helps
so
I
was
mentioning
this.
The
feature
where
the
customer
ID
was
passed
around
this
is
in
open
tracing.
It's
called
baggage,
because
it's
something
that
you
carry
with
the
request:
it's
not
the
actual
trace
ID,
but
it
it's
an
additional
key
value
where
and
so
what
we
use
it
for
is
several
things
that
a
tuber.
A
So
one
thing
is
the
we
have
various
sources
of
synthetic
traffic,
a
tuber,
let's
say:
there's
a
black
box
system
which
keeps
like
pinging
the
api's
and
saying:
is
my
services
working
correctly
right
and
there
could
be
some
performance
testing
or
like
capacity
testing.
That
increases
a
lot
of
like
create
a
lot
of
loads
and
the
services
as
well.
A
So
if
you,
if
you
cannot
distinguish
the
traffic
by
these
sources,
then
a
new
monetary
metric,
suddenly
metrics
goes
wild
and
your
alerts
start
firing
all
because
someone
ran
a
performance
test
somewhere
else
like
not
even
on
your
service
potential
upstream
right.
So
that's
bad.
So,
by
using
the
baggage
and
propagating
this
sort
of
the
type
of
traffic
that's
getting
into
your
service,
you
can
separate
metrics
regionals
at
the
dimension,
and
then
you
put
alerts
on
the
real
production.
A
A
Is
you
pass
it
around
and
you
can
do
a
lot
of
cost
attribution
using
that
and
calcining
is
another
aspect
like
if
we
have
3000
micro
services
and
you
do
the
cows
monkey
approach
you're
going
to
like
keep
killing
things
forever
and
not
find
anything,
because,
there's
too
many
permutations
of
things
that
you
can
kill
and
and
figure
out
what
affects
the
actual
reliability
of
the
service
so
with
with
tracing
you
can
actually
do
targeted
sort
of
cows
introductions
into
the
architecture
based
on,
like
you
say,
no
queue.
You
know
where
the
request
is
going.
A
So
one
other
thing
is
kind
of
useful.
We
measure
the
adoption
and
the
trace
quality,
so
we
wrote
the
process,
which
kind
of
looks
at
all
the
traces
come
in
from
various
application
and
says:
okay.
Well,
does
this
actually
trace?
Look
correctly,
the
instrumentation
makes
sense,
and
if
it's
not,
we
sort
of
erase
certain
things
like
you
know.
If
you
can
see
that,
but
this
is
like
a
dashboard
you
can
get
by
services
just
oh.
These
are
all
the
metrics
specifically
for
tracing
quality,
and
these
metrics
like
this
is
good.
A
But
this
is
not
so
good.
You
can
improve
it,
and
this
is
how
right
so
we
service
this
information
as
part
of
like
a
standard
like
a
quality
metric
for
or
quality
dashboard
for
a
service
and
that
every
service
guess
integration
with
other
tools
is
supremely
helpful
for
rolling
out
racing,
because
those
tools
are
your
additional
sort
of
people
who
can
go
and
and
and
and
harass,
other
people
to
implement
tracing
instead
of
you
doing
this.
A
So
I'm
writing
to
the
time
but
like
black
box
testing
I
mentioned,
is
like
external
testing
tool,
but
because
it's
a
low
traffic,
you
can
actually
force
tracing
and
make
sure
that
every
request
from
the
black
box
tool
is
actually
traced,
and
then
you
get
to
trace
ID
and
suddenly
something
fails.
You
can
raise
an
alert
and
say,
oh
and
by
the
way,
this
felt
right
so
as
opposed
to,
if
you
don't
have
that
information
all
you
know
that
this
end
point
to
your
total
API
layers
failed
with
trace.
You
can
pinpoint
some
downstream
service.
A
That's
responsible,
developer
studio
is
kind
of
an
IDE
for
developers.
Tuber.
We
have
there
where
you
can
drag
your
position
on
the
map.
Put
the
driver
here
and
simulate
the
trip
and
all
this
fun.
It
captures
all
kinds
of
requests
going
on
between
the
api's
and
it
also
does
tracing
so
again.
Very
useful
people
can
actually
get
used
to
using
tracing
I
mean
I'm,
not
a
showman
or
sorry
I'm,
not
a
sales
person.
So
this
is
sort
of
kind
of
obvious
point,
but
tracing
is
a
product
you
have
to
sell
it.
A
A
And
finally,
well
I
can't
really,
unfortunately,
go
the
service
dependence
diagram.
So
I
mentioned
that
this
is
very
powerful
tool
for
actually
understanding
what
your
application
is
doing
right
or
how
the
system
is
organized,
and
so
you
can
see
there
are
a
lot
of
questions.
Like
is
my
service
critical
for
overall
request
flow?
What
workflows
business
workflows
my
service
is
participating
in
right?
Will
my
service
arrive
Halloween?
A
The
big
thing
for
bird
like
Halloween
is
super
high
traffic
for
us,
and
then
we
always
do
this
capacity
planning
to
make
sure
we
have
enough
capacity
for
services
and
then,
but
you
don't
really
know
without
tracing,
because
just
because
our
business
number
of
trips
increases
by
two
times
doesn't
mean
you
service
needs
two
times
capacity,
it
could
be
ten
times.
It
looks
like
the
factor
is
not
clear,
actually
so
with
tracing.
You
can
get
that
factor
right
and
another
thing
here
is
like
your
sample
dependency
diagram.
A
So
the
previous
one,
you
remember
that
was
kind
of
a
simple
version,
because
it
just
measured
pairwise
connections
between
services.
This
one
actually
looks
at
the
path,
and
so
when
we
say
this
dingoo
at
the
top
left
is
:
cursor
with
shrimp.
This
is
not
real
names
and
it's
calling
service
dog.
So
my
question
is:
is
dingo
service
actually
dependent
on
the
door
not
no
way
to
tell
here
right?
A
We
have
a
tool
which
I
will
demo
if
you
come
to
the
Jager
dip
dive
session
later
tomorrow,
because
I'm
running
the
time,
but
you
can
actually
tell
by
by
this
new
tool
where
you
wear,
which
services
depends
like
at
any
depth
of
the
dependency
residence,
the
pairwise.
And
finally,
my
closing
thought
is
that
monitoring
traditionally
has
been
a
lot
about.
Firefighting,
like
our
measure,
is
something
the
fire
alerts.
I
do
something
really
tracing
can
do
that
as
well.
A
It
can
help
you
root
cause
and
troubleshoot
problems,
but
Tracy
also
provides
a
very
vast
amount
of
data
to
do
better
than
that
to
do
like
fire
prevention,
to
figure
out
what
your
capacity
constraints,
how
you
should
optimize
the
performance
and
which
components
in
the
architects
you
need
to
be
optimized
because
they
are
the
actual
bottlenecks
etc.
So
an
improving
reliability.
Basically
of
the
service,
so
quick
call
out.