►
From YouTube: A Picture is Worth 1,000 Traces - Steve Flanders, Splunk & Yuri Shkuro, Uber Technologies
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
A Picture is Worth 1,000 Traces - Steve Flanders, Splunk & Yuri Shkuro, Uber Technologies
Distributed tracing has emerged as the go-to solution for understanding what’s going on in the ever-changing cloud native architectures. A single trace can reveal many things: network latencies, time spent in databases, a service spinning idly, etc. but finding the right trace among billions that demonstrates a problem in a large distributed application is very hard. By looking at traces in aggregate, we can eliminate the need to state and validate hypotheses and instead answers start to emerge naturally. Especially when we use creative visualizations that put our visual cortex to work without overloading it with useless information. This talk will present the power of aggregate analysis of distributed traces by highlight its applications beyond performance troubleshooting.
A
All
right,
hello,
everyone
welcome
thanks
for
joining
us
on
this
talk
where
we're
gonna
talk
a
lot
about
the
complexities
of
micro
service
based
architectures
and
how
information
that
can
be
extracted
from
Detroit
distributed
tracing
information.
Specifically,
the
aggregate
information
can
be
very
useful
in
troubleshooting
these
types
of
environments,
as
the
title
of
the
slide
indicates
we're
actually
going
to
be
using
visualization
extensively
throughout
this
talk
to
show
some
of
the
power
of
the
eye
or
get
information
that
you
can
extract.
A
Now,
the
claim
that
we're
going
to
make
is
that
systems
that
are
moving
to
more
micro
services
based
and
some
of
the
biggest
cloud
providers
and
vendors
in
the
market.
Today,
the
really
dealing
with
the
issue
of
the
complexity
of
their
back-end
infrastructure
and
they've
really
reached
the
the
level
where
it's
really
hard
to
understand
all
the
nuances
of
how
the
architecture
works
end
to
end.
In
fact,
I'm
sure
URI
will
tell
you
at
uber.
A
You
probably
wouldn't
be
very
happy
about
that
and,
as
you
think,
about
SAS
based
applications
or
cloud
providers
providing
cloud
services
natively
to
end-users,
there's
kind
of
an
expectation
that
that
service
is
always
available.
Like
downtime,
really
is
not
an
option
and
when
downtime
happens,
it
really
impacts
the
business
as
well
as
end
users
experience.
A
So
developers
spend
a
lot
of
time
making
sure
that
their
application
is
highly
available.
There's
been
a
lot
of
time
testing
it
before
they
release
the
production.
They
go
through
fully
automated
CIC
IDIC,
D
delivery
pipelines
and
they
follow
really
good
DevOps
culture
and
best
practices
to
try
to
minimize
the
likelihood
of
an
incident
or
an
outage
occurring
button
at
the
end
of
the
day.
Outages
are
inevitable.
A
Big
names
companies
that
you
would
expect
to
have
this
kind
of
solved,
but
because
they're
dealing
with
really
complex,
architectures,
really
complex,
backends
that
dynamically
change
the
dependencies
between
them
change
the
communication
paths
change
it
can
be
really
difficult
to
ensure
that
things
are
behaving.
The
way
that
you
expect
now,
if
you're
on
call,
you
have
to
deal
with
some
of
these
issues,
I
guess
quick
show
of
hands
how
my
feel
we're
on
call
today
percentage,
maybe
40,
50
percent,
so
a
fair
number
of
people.
A
This
may
resonate
with
you
as
part
of
the
on
call
one
of
the
most
important
things
that
you
need
to
do
actually
mitigate
the
issue
as
quickly
as
possible.
Sure
you
want
to
know
why
it
happens.
You
want
to
prevent
it
from
happening
again.
You
want
to
learn
lessons
to
improve
your
processes
so
that
you
can
ensure
that
your
system
is
available,
but
knowing
that
an
outage
is
going
to
be
inevitable
anyway,
as
an
on-call
person,
you
want
to
mitigate
as
quickly
as
possible.
A
The
reason
for
this
is
because
it's
impacting
your
business
could
be
costing
the
business
a
lot
of
money
and,
of
course,
it's
impacting
end-users
they're,
trying
to
leverage
the
services
that
you
are
providing
and
they
have
an
expectation
that
is
available
all
the
time.
Now,
if
you're,
following
some
good
practices
like,
for
example,
you
deployed
your
application,
multiple
different
availability
zones,
you
might
be
able
to
do
something
like
failover
traffic
to
a
different
availability
zone
to
mitigate
the
issue.
A
But
can
you
really
do
that
without
understanding
the
implications
or
the
blast
radius
of
the
current
incident?
That's
going
on
what
happens
if
that
incidents
impacting
the
entire
region?
Well,
then,
that
mitigation
tactics
not
going
to
help,
so
at
the
end
of
the
day,
you
need
to
be
able
to
investigate.
You
need
to
be
able
to
do
it
quickly,
so
you
can
figure
out
what
happened
and
where
and
then
you
can
get
to
the.
A
Why,
after
the
fact
now,
that's
why
we're
here
today
to
talk
about
observability,
because
observability
allows
you
to
ask
questions
of
your
system
to
figure
out.
What's
going
on,
we're
gonna
spend
a
lot
of
time
showing
you
visually.
How,
through
traces
and
aggregate
information,
observability
can
really
lay
the
groundwork
for
you
to
deal
with
issues
in
these
complex
environment.
B
Thanks
Steve,
so
as
a
as
an
agenda
for
this
talk,
we
will
give
a
little
bit
more
introduction
into
why
distributive
tracing
and
most
people
are
familiar
with
it
and
we'll
go
quickly.
Just
really
just
to
make
a
point.
Rather
than
give
you
an
in-depth
introduction
to
tracing
I
will
talk
about
our
sorry.
B
Well,
I
will
talk
about
why
so
white
race
is
a
very
good
tool
to
tell
you
a
story
about
what's
going
on
in
your
system
right
and
then
will
tell
you
why
they
are
not
a
good
tool
and
what
we
can
do
better,
and
especially
with
the
visualizations
and
we'll
have
a
time
for
Q&A.
Hopefully
so
I
skipped
you
the
slide.
So
there's
an
introduction.
I
work
at
uber
for
about
four
four,
four,
four:
four
and
a
half
years.
All
this
time
on
observability
team
and
my
primary
focus
is
distributed
tracing
at
uber.
B
We
started
the
project
Jager,
which
is
now
agency
of
graduated
project.
We
graduated
in
October
I,
also
like
very
much
involved
in
open
tracing
and
open
telemetry
and
so
happens.
I
wrote
a
book
earlier
this
year
about
the
tracing,
so
you
can
find
it
on
my
website.
If
you
interested
it
talks
about
a
lot
of
things
related
to
tracing
on
how
you
deploy
it,
how
Iran
and
production,
etc
and
Steve
will
give
you
a
bit
more
intro
for
himself.
Thank.
A
You
my
name's
Steve
Flanders
I,
was
previously
at
a
company
called
a
mission
where
I
was
head
of
products.
Recently
we
required
by
Splunk
so
now,
director
of
engineering
at
Splunk.
We
actually
helped
co-create,
what's
known
as
the
open
census.
Collector
are
open
sent
to
service
and
the
opens
limit
tree
collector.
These
are
open
source
projects
that
make
it
really
easy
to
get
telemetry
data
both
metrics
and
distributed
traces
in
to
process
that
data
and
to
send
it
to
the
back
end
or
back
ends
of
your
choice.
A
It
actually
supports
a
broad
range
of
open
source
projects,
including
Jaeger
and
Prometheus.
It
has
the
ability
of
performing
some
advanced
features
on
top
of
the
data.
It's
collecting,
including
tag
addition
as
well
as
redaction
compression
retry.
Encryption
features
like
tail
based
sampling
are
all
natively
built-in,
and
then
it
has
the
ability
of
sending
to
a
variety
of
different
backends,
both
open
source
and
commercial.
So
it
serves
as
a
vendor
agnostic
way
to
help
collect
limitary
information
in
your
environments,
and
then
you
can
process
it
in
the
backend.
A
If
you're
interested
in
learning
more
about
me,
I
have
a
blog
and
Twitter
information
on
this.
Alright.
So
with
that
we're
gonna
kick
off
the.
Why
of
distributed?
Tracing
I,
guess
I'll.
Ask
another
question:
how
many
people
leverage
micro
services
or
moving
to
a
micro
services
based
architecture
in
their
environment.
Today,
a
8090
percent,
not
too
surprising
how
many
of
you
are
leveraging
distributed.
A
Tracing
information
to
observe
those
environments,
Wow
good
number
of
hands,
even
more
than
I
expected
at
least
at
least
half
that
percentage
raised
sprays
hands
now
distributed
tracing
is
very,
very
powerful
because
it
can
help
you
figure
out.
What's
going
on
in
your
environment,
where
other
telemetry
points
they
may
not
be
able
to.
As
you
are
kind
of
mentioned,
we're
not
going
to
cover
some
of
the
fundamental
pieces
of
like
what
distributed
tracing
is,
or
some
of
the
core
concepts
like
definitions
of
spans.
A
If
you're
interested
in
that
information,
both
you
and
I
have
done
previous
talks
at
cube
cons.
You
can
look
that
up.
So
what
I
do
want
to
talk
about
is
why
you
should
care
about
distributed
tracing
information,
and
really
it
can
be
summarized
in
in
these
few
words,
as
we
start
dealing
with
more
complex
micro
services
based
architectures
those
deep
systems,
if
you
will,
it
can
be
very
difficult
to
figure
out
what's
going
on,
and
if
you
read
any
paper
on
distributed
trait
on
distributed
systems,
you
probably
seen
the
assumption.
A
That's
communication
between
nodes
is
unreliable
and
in
the
microservices
world,
the
answer
to
that
is
to
actually
double
down
on
those
unreliable
connections.
So,
as
you
move
from
a
monolithic
application
to
a
microservices
based
application,
the
number
of
failure,
modes
and
failure
possibilities
grow
exponentially,
and
you
need
to
be
able
to
deal
with
these
scenarios
to
ensure
that
your
system
is
highly
available.
Now
many
of
you
have
probably
seen
this
slide
before.
A
Yearing
I
actually
talked
about
potentially
swapping
it
out
with
another
image,
but
it
really
does
capture
the
problem
statement
pretty
well,
so
we
decided
to
keep
it
in.
This
is
a
visual
representation
of
ubers
back-end
microservice
is
based.
They
have
north
of
3000
microservices
today.
This
is
actually
a
visual
representation
depicted
from
the
Jaeger
UI.
Every
circle
represents
a
service.
The
size
of
that
circle
represents
the
amount
of
traffic
that
is
receiving
and
then
the
lines
connecting
them
as
the
services
service
communication.
A
So
you
can
mitigate
that
when
issues
arise,
as
you
can
see,
distribute
transactions
can
be
very
complex.
This
was
one
example,
but
there's
hundreds
of
thousands
of
these
happening
every
second.
In
this
environments,
an
investigation
for
actual
incidents
usually
starts
with
a
high-level
business
metric
like,
for
example,
let's
say
the
number
of
ubers
being
booked
in
New
York
City
is
currently
down.
Why
well
I
need
to
go.
A
Look
at
my
observability
information
to
really
answer
those
questions
and
when
the
art
architecture
is
so
complex,
it's
actually
hard
to
troubleshoot
the
outage
and
get
to
root
cause
quickly.
At
the
end
of
the
day,
you
want
to
know
what
caused
it,
but,
arguably,
that's
usually
more
that's
usually
easier
than
determining
what
happened
and
where
it
happened
in
the
environments.
Think
of
it
this
way,
I'm
a
service
owner.
You
tell
me,
there's
something
wrong
with
my
service:
you
can
reproduce
the
issue
quickly.
A
I
probably
have
ways
of
investigating
that
particular
issue
and
mitigating
it
very
quickly.
But
how
did
we
get
to
that
state
where
I
knew
that
this
service
and
this
particular
path
was
causing
all
the
problems?
As
an
on-call
engineer,
you
typically
don't
have
that
insights.
So
it's
your
responsibility
to
quickly
figure
out
what
and
where,
in
the
environment
this
is
happening.
This
can
be
extremely
difficult
and
typically
takes
more
time
and
practice
than
determining
the.
A
Why
and
the
root
cause
and
remediating
that
for
the
long
term
now
this
is
where
observability
comes
in
and
in
a
way
it's
a
word
that
captures
the
system's
ability
to
really
answer
questions
about
the
particular
environment.
This
should
not
be
confused
with
monitoring.
Now,
there's
many
different
ways
to
look
at
the
differences
between
monitoring
observability,
but
one
particular
way
that
I,
like
is
monitoring,
is
typically
from
more
of
the
known
knowns
I
know.
What's
going
to
happen,
I
know
the
behavior
I
have
created
processes
to
address
these
I
have
alerts
in
place.
A
To
tell
me
maybe
proactively
before
they
happen.
I
have
run
books
that
articulate
how
to
remediate
it
or
I
have
Auto
remediation
in
place.
Observability
on
the
other
side
is
really
for
the
unknown
unknowns.
I,
don't
know
how
the
system
is
going
to
behave.
The
system
is
dynamically.
Changing
I
need
to
be
able
to
ask
questions
of
this
telemetry
data
to
figure
out
what's
going
on
and
to
actually
mitigate
issues.
A
So
observability
is
trying
to
solve
a
couple
key
things.
What
went
wrong
now,
if
you
think
about
it
from
a
tracing
perspective,
this
typically
includes
your
red
metrics
or
requests
errors
and
duration.
So
this
could
be
issues
with
increase
the
number
of
requests,
some
sort
of
timeout
going
on
or
an
error.
That's
propagating
through
your
system,
the.
Where
did
it
go
wrong?
It
could
be
service
to
service
communication.
It
could
be
a
dependency
on
a
third
party
service
that
you're
leveraging.
It
could
be
infrastructure
related
things
and
then,
of
course,
why
did
it
go
wrong?
A
Is
this
a
network
contention
issue?
Is
it
a
deploy
issue?
It
is
a
configuration
issue
in
my
environment
and
how
can
I
remediate
that
very
quickly
now,
as
it
turns
out
distributed
tracing,
is
actually
a
unique
position
to
address
all
of
these
problems
and
that's
because
it
has
context
and
correlation.
So
if
you
look
at
other
telemetry
data
points
like
metrics
and
logs,
they
can
tell
you
that
something
happens,
but
can't
necessarily
stick
stitch
together.
The
entire
picture
in
a
microservices
based
architecture
distributed
tracing
actually
relies
on
context
propagation.
So
it
can.
A
It
understands
a
request
through
your
system,
as
it's
calling
other
micro
services
it
understands
where
errors
originated
from
and
how
they
propagate.
Through
your
stack.
This
is
extremely
powerful
because
it
can
actually
enhance
metrics
and
logs
and
help
you
in
investigating
incidents,
doing
performance
analysis
dealing
with
deployment
issues
and
the
like.
So
next
Yuri
will
walk
you
through
the
pros
and
cons
of
individual
traces,
as
well
as
looking
at
traces
in
aggregate
and
give
you
some
rather
detailed
examples
of
how
you
can
leverage
the
power
of
distributed
tracing
information.
A
B
You
so,
as
I
promised
I
will
go
through
this
section
fairly
quickly,
because
I
assume
most
people
are
familiar
with
it,
but
the
main
strength
of
tracing
comes
from
its
ability
to
reconstruct
the
path
of
the
request.
That
is,
it
takes
through
the
micro
services
architecture
and
visualize
it
in
some
this
complex
transaction,
so
that
we
can
try
to
reason
about
this
and
and
the
ants
the
questions
that
we
posed
in
the
previous
slide.
B
You
can
actually
try
and
get
to
those
answers
fairly
quickly
from
just
a
single
trace,
and
so
for
that
I
want
to
show.
This
is
the
very
classic
representation
of
a
distributed
tracing
use,
the
Gantt
chart,
which
is
the
over
time
sequel
diagram,
where
you
basically
have
a
time
line,
left
right
services
vertically,
and
then
you
plot
the
operations
that
every
service
is
doing
right
and
there
are
a
few
things
that
are
very
obvious
like
we.
B
We
can
immediately
reason
about
what
was
the
overall
shape
of
the
Cole
graph
like
how
services
are
calling
each
other,
because
on
the
Left
we
see
there's
a
hierarchy
of
coat
depicted
right.
The
other
thing
is
that
also
very
obvious
is
latency
is
very
easy
to
spot
in
this
case,
because
clearly,
this
is
a
blocking
operation
that
takes
like
three
like
60%
of
the
overall
trade
duration.
So
if
you
were
to
optimize
it
that
well
not
even
optimize
it,
maybe
it's
an
outage
even
investigating
it.
So
it
draws
your
attention
immediately.
B
You
don't
need
to
understand
much
to
figure
out
that
this
is
a
bottleneck
right.
It
may
be
another
one
on
a
staircase
pattern
on
the
right
below
that
as
well.
There's
a
bunch
of
sequential
iterations
could
be
done
in
parallel,
perhaps
so
another
option.
And
finally,
when
there
are
errors,
if
you
have
good
instrumentation,
then
the
errors
will
be
captured
in
the
traces
and
this
same
Gantt
chart.
You
can
surface
them
with
some
additional
annotations
and
Nokia.
There's
an
error
in
this
operation.
B
B
It
allows
you
to
sort
of
drill
down
into
your
individual
details
of
every
operation
and
and
reason
about
them.
And
finally,
if
you
are
like
used
to
debugging
something
in
monitors
applications,
then
you
probably
always
look
for
stacktrace
like
now.
Imagine
you
have
Micro
Services
and
you
don't
have
a
stack
trace
so
distributed
tracing,
actually
acts
as
a
stack
trace
so
that
single
trace
you
is
certain
you
can
think
of
it
as
a
step
trace
right.
B
However,
that
same
view
has
a
couple
of
challenges
which
are
actually
what
sometimes
make
tracing
not
as
effective
as
sumantra
VISTA
mentioned
in
his
talk
right
and
one
of
them
is
well.
You
know
that
you
troubleshoot
the
problem
and
production.
You
need
to
have
a
representative
trace
like
something
that
actually
happens
and
did
when
you
start
looking
at
the
trace.
It
has
to
represent
correctly
to
what
would
the
behavior
or
the
erroneous
behavior
the
system,
but
how
do
you
get
to
that?
B
Maybe
the
trace
that
you
picked
up,
even
if
you
looked
it
to
some
some
parameters
today.
That
could
still
be
anomaly
rather
than
the
steady
state
behavior,
whether
good
or
bad,
and
the
second
issue
is
in
the
more
complex
and
deep
systems.
The
traces
themselves
are
actually
becoming
so
complex
that
the
Gantt
chart
is
becoming
just
too
unwieldy
to
use
by
itself.
I
will
in
this
example.
This
is
a
real
production
trace
from
uber.
B
It's
probably
like
one
of
the
front
end
requests
like
either
pickup
or
maybe
up
start,
so
there
are
30
more
or
more
services
involved
in
that
trace
model.
Condit
RPC
calls-
and
this
is
a
concise
representation
of
that
call
graph
a
bit
condensed.
So
imagine
if
you
go
back
to
the
Gantt
chart
of
that
thing.
B
So
if
you
know
which
specific
region
you
want
to
zoom
in,
you
can
obviously
zoom
in
and
get
rich
details,
but
on
the
top
surface,
if
you
are
in
the
outer
Shiva
tation
need
to
find
out
quickly
what's
going
on
wrong,
it's
very
hard
still
and-
and
this
is
not
like
the
most
complex
trades
like
some
of
the
Amazon
homepage.
Traces
have
like
several
hundred
services
involved
right,
so
so
what
what
we're
trying
to
solve
in
it
with
the
solutions
in
the
stock
is
really
trying
to
achieve
it?
B
The
failure
to
a
specific
place
in
the
architecture,
because
we
said
this
is
one
of
the
most
challenging
problems.
And
how
do
we
do
that?
Well,
looking
at
one
thing
is
not
the
best
thing
like
if
you
get
a
pool
request
or
a
patch
you're,
not
going
to
apply
it
to
your
source
code
and
then
start
reading
source
code
front
layout
right,
which
is
what
we're
doing
with
a
single
trace.
B
Instead
you're
going
to
look
at
the
differences
and
the
same
thing,
if
you
perform
an
optimization,
let's
say
memory
leak,
you
take
a
snapshot
before
Schnauzer
separate
after
you
compare
them.
You
look
at
the
differences,
so
we
want
to
do
the
same
thing
with
traces
looks
at
the
differences
because
that
will
highlight
us
a
lot
quicker,
some
of
the
things
that
may
be
going
wrong
and
I
presented
last
year.
So
again,
I'm
not
going
to
go
very
long
on
this
one.
B
But
one
of
the
thing
that
we've
built
and
Jaeger
and
open
source
is
the
comparison
of
traces
where
you
can
structure
and
compared
to
two
traces
and
using
the
typical,
like
code,
diff
color
code
in
red
missing.
Now
it's
green
and
extra
nodes
right,
so
your
attention
is
immediately
drawn
to
the
bottom
part
of
this
saying.
Oh
there's
something
going
on
with
that
execution
right,
like
a
bunch
of
calls
that
normally
happens
on
the
left
side
didn't
happen
on
the
right
side.
B
So
this
is
the
kind
of
simple
visualization
allows
you
to
to
droop
down
into
a
problem.
Much
quicker
and
color
coding
could
be
different.
You
could,
for
example,
if
you
investigate
in
a
latency,
then
again
we
can
use
a
heat
map
color
code
in
here,
and
you
can
see
immediately
the
red
path
of
the
nodes.
B
We
again,
this
is
a
differences
right,
so
one
of
those
nodes
which
are
in
gray
could
be
actually
taking
80%
of
your
total
transaction,
but
that
80%
may
be
normal
everyday,
whereas
the
things
that
in
red
are
different
between
these
two
things
and
that's
what
you
want
to
focus
on
when
you
investigation
things,
and
so
nonetheless,
these
two
differences
different
views.
They
still
have
kind
of
the
same
challenges
that
individual
traces
could
be
outliers.
B
Now
we
have
two
of
them,
but
we
still
have
to
pick
them
somehow
and
it's
up
to
the
user
to
actually
pick
the
right
one,
which
is
not
great.
So
the
question
is:
can
we
do
better?
What
if
the
system
kind
of
did
that
for
us
and
we
didn't
have
to
guess,
and
so
that's
the
approach
that
I
want
to
talk
about
here,
so
we
want
to
use,
and
we
built
that
the
tuber,
by
the
way
it's
an
introduction.
B
So
we
want
to
aggregate
a
bunch
of
traces
that
will
now
represent
a
good
behavior
of
the
system
and
we'll
learn
from
them
a
statistical
model
of
that
behavior.
And
then,
when
something
bad
happens,
we
can
take
that
bad
request
and
overlay
it
with
the
good
model
and
compare
them
and
see
what
the
differences
are
right.
But
when
we
create
this
good
model
first,
we
doing
that
in
a
business
specific
way.
B
This
is
by
the
way
this
tool
is
not
open
source
because
it's
really
tied
to
the
way
the
uber
does
business
and
I'll
talk
about
this
in
a
bit
more.
So
the
way
this
monetary
solution
works,
it's
called
black
box
money
generated,
so
we
have
uber
users,
drivers
and
riders.
They
have
their
phones,
their
phones,
talking
to
doober
back-end
all
the
time
right.
B
And
so
we
have
traces
immediately
from
that
that
we
can
use
for
for
the
approach
that
I'm
discussing
and
by
the
way
there
isn't
the
uber
built.
This
is
not
specifically
for
tracing
or
even
or
outage
resolution.
It
was
actually
for
outage.
Detection
uber
is
a
very
sent
City
centric
business
like
if
you
go
to
another
city,
there's
probably
going
to
be
hundreds
of
different
experiments,
runyan,
which
are
not
like
in
New
York
and
there's
tons
of
features
that
may
be
enabled
on
a
procedure
level.
B
So
if
you
think
about
how
all
those
features
affect
the
call
graph
in
the
backend
you,
you
may
have
very
different
called
call
graphs
or
disability
traces
between
New
York
and
San
Francisco,
for
example,
right
and
so
the
proper
is,
is
specific
to
the
city.
So
it
executes
the
execution
one
city.
So
it
knows
what
to
expect
within
that
city
and
that
way
it
can
build
a
logical
model
of
what
the
normal
execution
looks
like
gradually,
I
mean
it's
the
stunt
I'm
sliding
windows.
So
if
something
changes
it
will
readjust
itself.
B
And
so,
but
when
this
black
monitor,
so
it
was
very
helpful
for
us
in
isolating
the
outages.
Sometimes
we
couldn't
even
detect
an
outage
from
the
usual
metrics,
because
if
it
happens
in
one
of
the
cities
in
uber
support
like
a
thousand
cities,
then
you
may
not
notice
the
blip
on
the
metrics.
But
with
the
specific
like
test
within
a
city
it
fails
and
we
can.
We
can
raise
an
alert
to
people
right,
but
once
that
alert
goes
off,
we
are
back
to
that
same
problem.
B
It
tags
the
trace
with
a
special
signal
that
we
can
detect
in
the
back
and
it
it
indicates
an
a
failure.
So
there
now
we
have
two
basically
pieces
that
we
can
compare
fully
automatically.
We
didn't
have
human
involved
in
choosing
either
of
those
right,
and
so
we
do
the
difference,
and
then
we
can
investigate.
Let
the
human
investigate
the
difference
only
and
the
visualization,
for
the
difference
is
very
similar
to
what
what
I
showed
before.
But
again,
there
is
no
human
choice
involved
in
selecting
the
pieces
to
diff.
B
B
B
Where,
because
again,
we
will
have
a
bad
trace
that
we
can
pair
in
with
a
model,
we
can
actually
go
and
look
into
that
bad
trace
and
get
all
the
rich
information
that
we
typically
get
from
a
single
trace,
especially
if
you
know
where
to
look
an
island
here.
We
do
know
where
to
look
for
it,
because
every
node
and
this
graph
can
be
deep
linked
into
the
trace
because
they're
all
like
connected
so
the
nodes
themselves.
This
is
again.
This
is
just
a
call
graph
with
nodes
chronologically
sorted.
B
Some
of
the
nodes
are
collapsed
because
if
you
have
a
service
called
another
series
multiple
times,
we
don't
want
to
show
that
it
can
they
get
too
noisy
again,
as
I
mentioned,
like
hundreds
RPC
calls,
there's
fewer
knows
definitely
here
than
in
the
actual
trace,
so
that
that's
what
makes
this
visualization
actually
consumable
by
human
and
finally,
with
the
color
coding
white
means,
looks
okay.
Well,
black
means
the
nodes
are
missing
in
in
the
failed
execution
right.
B
Each
of
them
had
errors
in
them
and
on
the
right
side,
we
can
kind
of
drill
down
without
leaving
the
tool
to
see
what
specific
error
messages
were
recorded
in
those
steps.
And
so
again,
if
you
look
at
visually
at
this
graph,
you
kinda
see
again,
okay,
well,
the
black
top
section
at
the
top
is
certain
some
things
that
didn't
go
well
right
away.
There
are
some
error,
so
there's
you
know
that
you
might
click
around
and
figure
out,
okay.
Well,
this
is
the
problematic
area
that
I
want
to
investigate.
B
We
rule
out
outliers
from
this
comparison
right
because
we
don't
have
to
worry
about
them.
And
finally,
this
tool
was
specifically
done
to
attribute
failures.
Much
quicker
than
normal
production
would
allow
us
to
do
because
it
used
to
take
our
operations
like
30
minutes,
to
figure
out
which
services
is
behaving
now
they're
down
to
2
minutes
with
this
tool,
so
we
kinda
run
out
of
time.
Yeah.
A
All
right
close
the
time
so
I
kind
of
go
through
this
information
very
quickly.
I
want
to
show
some
aggregate
analysis
leveraging
visualizations
all
powered
by
distributed
tracing
information,
so
as
I
mentioned
in
microservices
based
architecture
is
complexity,
makes
it
very
hard
to
troubleshoot
from
an
on
call
perspective,
but
one
of
the
main
problems
you're
trying
to
solve
for
is
what
I
like
to
call
the
who
to
wake
up
problem.
A
So
here's
an
example
where
every
circle
represents
a
service.
The
size
of
the
service
again
represents
the
amount
of
traffic
it's
receiving
and
the
arrows
indicate
the
directionality
of
traffic.
You
can
clearly
see
here
that
the
currency
service
is
experiencing
a
large
number
of
errors.
You
can
also
see
that
the
check
out
service
is
seeing
errors,
but
not
actually
originating
them,
they're
actually
propagating
from
the
currency
service.
So
in
this
particular
case,
if
we
were
to
page
an
on
call,
it's
really
the
currency
service
that
we
want
to
get
involved
to
figure
out.
A
What's
going
on
now
in
order
to
figure
out
what's
going
on,
we
have
to
get
to
problem
isolation,
and
so,
in
addition
to
the
typical
red
metrics
that
you
can
get
are
distributed
tracing
information
like
there
Quest's
and
the
errors
in
the
duration.
You
can
actually
tag
rich
metadata
on
to
your
spans
as
well.
This
could
be
build
information,
infrastructure
environments.
A
It's
deployed
on
whatever
is
applicable
to
the
application
that
you're
running,
and
then
you
can
start
asking
questions
of
your
data
like,
for
example,
if
I
just
look
at
the
currency
service
and
its
dependencies
and
I
break
down
the
currency
service.
I
can
see
that
it's
running
in
two
different
regions,
and
only
one
region
is
having
an
issue.
This
is
helping
me
isolate
the
problem
and
focus
my
efforts
really
where
the
problem
is
happening
from.
A
If
I
want
to
take
that
one
step
further,
there
are
actually
nested
levels
of
metadata,
so
maybe
I
then
want
to
know
for
that,
given
region
which
crew
brandies
pods,
is
this
service
deployed
on
and
are
any
of
them
being
impacted
this
nested
analysis
or
going
like
multiple
layers?
Deep,
is
also
critical
from
an
analysis
perspective
to
help
isolate
the
problem,
and
here
we
can
quickly
see,
there's
that
actually
only
a
single
pod
in
the
US
west
to
region.
That's
experiencing
issues
now
as
an
on-call
person.
A
If
my
responsibility
is
really
to
mitigate
the
issue,
I
now
have
some
options:
I
could
terminate
that
pod.
I
can
quarantine
that
host
I
can
try
to
get
the
system
back
online
and
then
I
can
take
the.
Why
of
why
this
happened
in
the
first
place
offline
and
get
to
my
root
cause
after
the
effect.
So
it's
really
critical
that
you
have
the
ability
to
ask
any
question
of
your
data,
and
that
means
you
have
to
enrich
it
with
things
like
things
that
are
applicable
to
your
particular
environments.
A
This
could
be
like
tag
analysis
of
breaking
it
down
by
all
the
additional
additional
metadata
that
you
add
to
the
spans.
This
could
be
from
a
performance
standpoint,
looking
at
the
performance
across
its
different
dimensions,
as
well
as
also
looking
at
things
like
metric
comparisons,
where
I'm
actually
looking
at
metrics
that
are
being
extracted
from
the
rich
distributed
tracing
data
that
I'm
collecting.
A
A
The
canary
released
that
you're
rolling
out
so
from
a
takeaways
perspective,
really
distributed
tracing,
makes
it
possible
to
deal
with
the
complexities
of
a
microservices
based
architecture.
It
provides
the
missing
link
of
context
and
correlation,
that's
critical
from
a
durability
standpoint
and
can
help
highlight
and
enhance
the
metrics
and
logs
and
you're
likely
leveraging
today.
A
So
you
can
actually
ask
the
questions
that
you
care
about,
so
definitely
take
a
look
at
distribute
tracing
options
that
are
out
there
as
you
move
to
micro
services,
you'll
probably
see
that
it
really
provides
some
rich
insights
and
curious
to
see
what
others
come
up
with
visualization
standpoint
in
order
to
really
enhance
the
data
that
they're
collecting
with
that
I
think
we're
at
time.
But
thank
you
so
much.