►
From YouTube: Tracing is More Than Traces: The Insights in Trace Aggregates - Daniela Miao, LightStep
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Tracing is More Than Traces: The Insights in Trace Aggregates - Daniela Miao, LightStep
The value proposition for distributed tracing is well-understood: assembling and visualizing end-to-end transactions helps to identify latency bottlenecks and provides a head-start on problem diagnosis. Yet the data from traces – when aggregated – can reveal much more, and can do so with greater precision and certainty. This talk presents the profound insights trace aggregates help unlock, including sources of resource contention, latency anomalies in the context of service infrastructure, and correlations of metrics with high-cardinality characteristics of the distributed system. The talk demonstrates, using concrete examples, how novel applications of aggregated traces reveal new opportunities for performance improvements. However, aggregation is not possible without a standardized tracing output format, as well as a proliferation of traces via cloud-native service mesh integration.
https://sched.co/MPbg
A
Okay,
so
let's
get
started
and
just
let
people
trickle
in
I
am
Daniella
I
work
currently
at
a
distributed
tracing
startup
in
San,
Francisco,
California,
called
lights
up
and
I'm
here
today
to
talk
to
you,
you
all
about
Chase
aggregates
before
we
get
into
trace
aggregates
I
wanted
to
give
everybody
an
overview.
Let
me
grab
my
pointer
here,
a
quick
overview
of
the
types
of
data
primitives
that
we
work
with
in
observability.
We
like
to
call
these
the
three
data
pipes
of
observability.
A
This
is
a
term
that
our
CEO
bunge
Siegelman
came
up
with
recently,
you
might
have
heard
about
the
three
pillars
of
observability,
which
is
a
little
bit
different
because
we
think
about
them
as
data
pipes
that
carry
raw
data
to
developers
to
software
they're,
just
raw
data
without
processing
without
further
analysis.
You
cannot
get
useful
insights
into
your
system
and
I
put
a
picture
of
him
up
here
so
that
you
can
ask
me
questions
afterwards,
but
mainly
I
got
a
lot
of
ideas.
A
He
helped
me
a
ton
with
this
talk,
so
it's
more
of
an
attribute
back
to
him
to
his
credit
and
also
I
think
he
might
be
around
here
afterwards
somewhere.
So,
if
you
don't
disagree
with
me,
go
yell
at
him
all
right,
so
a
very
quick
overview,
just
so
we're
all
on
the
same
page,
I'm
gonna
move
very
quickly
through
this
overview
since
I
think
most
of
you
know
the
value
of
distributed
tracing
I.
Think
that's
well
understood.
A
Now,
on
the
left,
you
basically
have
a
monolithic
service
as
more
and
more
modern-day
architecture
moves
towards
a
micro,
microservices
architecture.
You
basically
need
to
follow
the
end-to-end
transaction,
the
end-to-end
trace,
which
is
basically
highlighted
in
red
up
there
in
both
cases.
Now,
in
the
second
case,
in
a
micro
services,
architecture
tracing
is
the
only
way
you
can
get
a
coherent
picture
at
the
from
the
perspective
of
a
request.
This
used
to
be
easy
when
everything
was
in
one
instance
or
one
pod,
using
metrics
or
Logs
with
microservices.
A
A
trace
is
really
the
best
way
to
understand
how
your
requests
are
going
through
the
system.
Okay.
So
what
does
a
trace?
Look
like
again
very
simple
I
think
most
of
you
probably
have
seen
other
zatia
of
this.
We
have
typically
some
kind
of
a
client,
a
web
framework
that
makes
RPC
calls
to
a
back-end
server,
an
API
server
and
that
API
server
intern
is
you
get
deeper
and
deeper
into
the
stack?
This
is
a
very
well
instrumented
trace.
A
In
this
case,
you
see,
you
know
a
blue,
a
red,
RPC,
agreeing
RPC
a
bunch
of
gray
RPCs,
maybe
an
external
service,
a
third
party
vendor
and
basically,
as
you
move
from
left
to
right,
we're
seeing
a
progression
in
time
now
with
this
progression,
each
single
work
unit.
So
each
line
each
block
in
tracing
terminology,
that's
called
a
span,
it's
a
very
context,
rich
set
of
data
primitives
that
we
can
then
use
later
to
form
all
kinds
of
interesting
analysis
which
is
covered
in
today's
presentation.
A
Ok,
so
that's
just
one
trace,
though,
and
then
really,
if
you
squint
your
eyes
a
little
bit,
this
is
sort
of
the
shape
of
many
traces.
You
see,
you
know
kind
of
a
waterfall
eview
a
bunch
of
different
spans,
and
then
you
look
at
another.
One
kind
of
similar
a
little
bit
different,
but
the
same
transaction
and
then
another
one
and
then
another
one
and
another
one
now.
The
problem
here
is:
how
many
do
you
look
at?
How
do
you
know
that
your
trace
is
representative?
A
The
key
here
is:
we
cannot
feasibly
capture
a
hundred
percent
of
all
transactions.
That's
just
not
cost
effective
right.
You
for
a
typical
distributed
system,
probably
handling
at
least
thousands
of
transactions
per
second,
so
sampling
is
inevitable
with
sampling.
The
problem
you
have
is
that
you
don't
know
whether
or
not
you're
looking
at
a
trace.
That
represents
the
total
population,
and
if
you're
not
careful
about
it,
it
might
lead
you
down
the
wrong
path
to
optimize
the
wrong
thing,
so
sort
of
leads
to
leads
us
to
a
very
difficult
problem
as
humans.
A
We
can't
process
large
amounts
of
data
easily.
So
it's
like
you
know
you
open
10
tabs
each
with
a
trace,
and
then
you
look
at
maybe
10
of
them
and
you
try
to
deduce
ok.
What
is
the?
What
is
the
10
represent,
but
really
what
we
want
to
look
at
is
the
aggregate
analysis
over
thousands
of
traces
right
at
least
thousands
of
traces.
You
can
probably
do
tons
of
thousands
we're
gonna
with
just
a
number:
that's
not
so
humanly
possible.
A
So
let's
start
with
something
simple
like
correlations,
so
this
is
a
symptom
of
something
going
wrong.
Maybe
you
got
paged,
maybe
you
deployed
something
just
staring
at
your
metrics
dashboard.
This
is
a
real
dashboard.
I
just
took
a
screenshot
of
our
internal
dashboard.
Clearly,
there's
something
wrong
here:
something
is
out
of
whack
there's
a
little
bump.
We
don't
know
what
happened.
That's
around
12
o'clock
on
a
Monday,
so
this
is
where
the
investigation
starts.
This
is
your
symptom,
ultimately
it
a
bit
of
a
spoiler
here.
This
is
this
was
a
root
cause
right.
A
We
found
the
magic
graph
if
you
segment
this
particular
metric
by
customer.
You
see
that
one
single
customers
inadvertently
in
this
case
needling
one
of
our
servers,
basically
rendering
the
Box
unusable
it's
very,
not
responsive,
but
in
the
metrics
dashboard.
That's
not
clear.
We
know
something
is
wrong.
We
don't
know
which
one
to
start
investigating,
because
they
all
look
very
similar.
They
all
lead
to
the
same
problem.
So
how
long
does
it
take
to
get
from
your
symptom
to
the
root
cause?
In
this
case?
It
took
too
long.
A
I,
don't
remember
the
exact
number
at
least
an
hour
I
think
it
took
at
least
an
hour.
The
problem
is
to
get
to
this
graph.
You
need
to
know
what
two
segments
lie
right.
There's
a
lot
of
tag,
values
that
you
associate
with
your
metrics
and
knowing
that
this
was
the
magic
value
you
had
to
segment
by
was
the
first
problem
there's
a
second
problem,
a
much
harder
one,
which
is
that
with
low
cardinality,
in
this
case,
some
kind
of
a
very
fake
case
that
I
I
made
with
just
a
select
set
of
customer
there's.
A
What
maybe
five
lines
on
here.
That's
just
not
realistic
for
most
of
your
business
and
services,
you're
going
to
be
serving
at
least
hundreds,
if
not
thousands
of
customers-
and
it
starts
to
look
like
this-
it's
very
very
difficult
for
the
human
brain
to
process
and
to
find
the
anomaly
in
a
graph
like
this-
and
this
is
assuming
you
know
what
to
segment
by
and
the
tooling
that
you're
using
allows
you
to
segment
by
something
that
has
so
many
distinct
values
in
the
thousands
and
most
of
the
case.
This
is
really
a
problem.
A
It's
it's
just
not
cost-effective.
Most
of
the
cases
not
even
feasible.
The
tools
that
we
use
a
lot
of
metrics
products
literally
cannot
support,
cannot
visualize
thousands
and
ten
thousands
of
lines
on
a
graph.
Like
this,
so
cardinality
is
really
your
your
fatal
flaw
with
using
metrics
for
for
root,
causing
a
problem
like
this.
So
what
can
you
do?
Instead,
you
can
slice
the
data
a
slightly
different
way
and
really
solve
the
cardinality
problem
using
aggregate
traces.
A
So
this
is
a
latency
histogram
for
those
of
you
familiar
with
histograms,
you
probably
know
what
this
really
means,
but
on
the
x-axis
you
basically
have
these
latency
buckets
one.
They
each
bucket
represents
a
range
of
latency,
so
the
one
I'm
pointing
at
here
is
between
five
hundred
and
fifty
milliseconds
and
600
milliseconds,
and
when
you
have
on
the
y-axis,
is
the
number
of
traces
I
put
in
the
number
of
requests
here.
A
You
can
think
of
it
as
number
of
spans
number
of
traces
doesn't
matter
the
number
of
units
of
work
that
was
done
in
your
system
that
fell
in
this
latency
bucket,
so
you're.
Basically,
your
fast
operations
over
here
and
you're,
really
slow
operations
are
all
over
here.
This
is
just
a
simple
count:
right,
okay,
so
then
I
drew
a
line.
Approximately
your
p99
is
here.
So
these
these
three
bars
fall
within
the
slowest.
One
percent
of
all
of
your
requests
for
your
system
now
notice
that
a
basically
segmented
might
take
data
into
two
I.
A
Have
the
slowest
1%
and
now
I
have
the
rest
99%.
What
can
we
do
here
at
this
point?
We
can
actually
run
statistical
analysis
and
try
to
find
defining
characteristics
of
the
top
ones,
top
1%
slowest
requests.
What
stands
out
about
this
top
1%
versus
the
rest
and
you're?
Basically
doing
this
ad
hoc
right
you're,
looking
at
all
of
your
data,
you're
saying
well,
what's
causing
my
tail
latency
is
to
be
slow
and
in
this
case
again
it's
a
pretty
artificial
example.
In
this
case
you
have
a
particular
customer.
A
In
the
previous
case,
you
have
to
segment
by
customer
ID.
In
this
case,
your
what
you
basically
have
is
a
the
defining
characteristic
is
a
certain
tag,
value
that
says.
Customer
ID
equals
the
food
bar
and
those
requests
fall
largely
so
they
make
a
large
part
of
your
top
1%
of
requests,
not
all
of
it.
You
have
some
over
here,
so
it's
not
a
perfect
correlation,
but
that's
not
what
we're.
Looking
for
we're
just
looking
for
the
first
hint
towards
where
you
should
go
with
your
investigation,
remember
what
you
started
with
is
basically
nothing.
A
You
know
something
is
wrong.
You
got
paged
or
you
saw
something's
wrong
with
your
metrics
dashboard.
You
come
into
a
view
like
this,
and
you
just
want
the
system
to
run
these
analysis
and
suggest
to
you
the
characteristics,
that's
abnormal
the
characteristics
that
point
at
something
that
might
be
wrong.
Ok,
how
does
this
actually
apply
in
a
real
life
scenario?
So
here
is
a
more
real-life
example,
one
faulty
network
card
on
some
random
rack
and
some
random
data
center
right.
A
You
don't
hit
it
very
often,
but
once
in
a
while,
a
request
reaches
that
box
and
it's
just
completely
non-responsive
and
that
causes
your
tail
latency
in
this
case,
I
actually
changed
it
to
P
99.9.
So
this
allows
you
to
really
explain
what's
happening
with
your
tailing
season.
The
point
1
percent
of
your
slowest
requests
again
I've
segmented
here.
This
is
another
latency
histogram
I've,
segmented,
here
right
at
the
P
99.9
mark,
so
we're
basically
looking
for
defining
characteristics
of
that
top
0.1%
and
I've
overlaid.
A
Here
again
in
black
and
circled
here
and
as
you
can
see,
there's
actually
a
strong
correlation.
This
is
what
we
call
a
correlation
coefficient,
it's
sort
of
the
mathematical
representation
of
how
strong
of
a
correlation
and
because
almost
everything
falls
inside
this
bucket,
we
have
a
very
strong
correlation.
The
best
a
perfect
correlation
would
be
one
with
a
single
server
with
a
unique
server
ID
in
u.s.-
and
this
is
this-
is
just
really
difficult-
I
mean
imagine
having
this
on
a
metrics
dashboard
you.
A
First
of
all,
you
have
to
know
that
this
is
the
server
ID
you
wanted
to
segment
by
and,
secondly,
how
many
servers
are
running,
I
think
easily
tens
of
thousands
of
servers
in
any
of
your
of
distributed
systems.
Imagine
looking
at
a
dashboard
with
tens
of
thousands
of
lines
and
picking
out
the
top
one
percent,
that's
slightly
above
the
rest,
or
you
know,
timing
out
whenever
these
requests.
A
Okay,
so
this
is
sort
of
where
I
want
to
focus
a
lot
of
my
energy
on.
So
this
is
a
different
type
of
analysis
on
critical
path
by
looking
at
many
many
traces,
so
we're
back
to
this
single
trace
view
and
another
one
and
another
one.
This
is
a
bit
of
a
theme
today,
looking
at
many
many
traces.
So
again,
if
you
look
at
it
from
a
30,000
foot
view,
the
commonality
that
you
will
notice
here
is
that
you
have
this
red
bar
I've
highlighted
it
to
have
it
really
pop.
A
That
seems
to
be
the
longest
in
all
three
of
these,
and
it's
probably
the
case.
If
you
look
at
more,
but
you
can
you
can
let
the
computer
do
the
work,
you
can
look
at
thousands
of
them
and
let
the
computer
tell
you
what
your
aggregate
critical
path
is
and
a
critical
path
is
basically
the
single
operation
sometimes
they're,
multiple.
A
But
let's
stick
with
a
simple
example
of
a
single
operation
that
basically
contributes
that
makes
up
the
duration
of
your
entire
transaction,
meaning
that,
if
you
shrink,
if
you're
able
to
optimize
that
transaction
that
operation,
if
you're
able
to
shrink
the
the
length
of
the
red
bar
you're
in
our
transaction
becomes
faster.
So
this
is
really
where
you
want
to
focus
a
lot
of
your
energy
and
engineering
resources
to
optimize
and
that's
where
the
aggregate
critical
path
is
and
by
looking
at
thousands
of
traces.
A
The
system
can
just
automatically
tell
you
here's
your
critical
path
and
remember
these
spans
might
belong
to
different
services.
So
up
here
might
be
a
service
that
you
own
and
you
end
up,
making
a
bunch
of
RPC
calls,
and
this
might
be
a
few
hops
away.
Some
team
that
you
don't
even
talk
to
and
that's
what's
impacting
your
service
I'll
go
more
into
this
in
the
demo
later.
Okay,
so
once
you
have
your
critical
path,
the
system
tells
you
you
need
to
explain
it
all
right.
A
You
need
to
explain
why
it's
a
critical
path,
why
it's
really
slow
and
that's
actually
the
hard
part
most
of
the
time
I
think
we
generally
have
an
idea
where
the
critical
path
is
actually
so
that's
not
the
most
valuable
thing
that
the
system
can
tell
you.
The
system
can
tell
you
where
the
critical
path
is
the
harder
question
that
requires
often
time
human
intervention
to
figure
out
is
why
rarely
do
we
know
why
the
critical
path
is
a
critical
path.
Sometimes
it's
sometimes
it's
easy.
A
Sometimes
you
know
that
there's
a
network
halt
or
database
call
that
you
simply
cannot
optimize
further
a
lot
of
the
time
there
is
resource
contention
and
there's
a
bold
statement
here
that
I'm
basically
making.
In
order
to
make
my
my
demo
and
my
example
work.
Is
that
and
I
think
I
think
it's
true
for
most
of
the
time
is
that
nearly
all
latency
issues
remember
we're
focused
on
studying
latency
in
this
performance
investigation.
Nearly
all
latency
issues
are
due
to
some
sort
of
contention.
In
this
example,
I'm
gonna
go
into
it
very
soon.
A
It's
about
resource
contention,
contending
for
a
common
resource,
a
mutex,
a
lock.
Basically,
sometimes
it's
contending
for
machine
resources.
Cpu
network
sometimes
is
contending
for
you
know,
maybe
a
cache,
maybe
a
database
query,
but
nearly
all
latency
issues
boils
down
to
resource
contention,
different
processes,
different
threads,
wanting
access
to
the
same
thing,
oops
so
back
to
the
example
I'm,
basically
putting
in
a
very
specific
example
here
already
previewed
it
a
little
earlier,
a
mutex
acquire
operation.
A
So
what
this
is
is
an
operation
representing
how
much
time
this
process
this
transaction
had
to
wait
for
a
mutex
to
become
available,
and
it's
capturing
it
here
as
a
critical
path.
So
generally
I
would
say
in
a
typical
investigation
without
aggregate
analysis,
you
can
find
a
trace
like
this
and
you're
kind
of
at
a
dead
end.
Here
I
mean
you
know
that
it
there's
a
contention
on
the
lock.
You
understand,
maybe
where
it
is
in
the
code.
A
Maybe
you
have
to
go
back
to
the
code
and
try
to
understand
how
to
break
the
lock
of
how
to
you
know,
basically
optimize
it
further.
I
would
say
this
is
a
dead
end
and
generally
without
aggregate
trace
analysis.
This
is
where
you
stop,
but
with
contention
resource
contention,
alice
is
what
you
really
want
to
understand
is
what
the
jam
is,
why
it's
so
contended
for,
and
sometimes
because
many
developers
contribute
to
the
same
code.
Sometimes
you
lose
sight
of
this.
It's
not
always
easy
to
understand
why
a
lock
is
so.
A
It
takes
so
long
to
acquire
a
lock
after
you've
developed
on
the
complicated
program
for
a
little
while.
So
really
here,
maybe
a
car
back
here.
You
know,
we've
all
been
in
a
situation
and
you
want
to
know
what's
holding
up
all
that
traffic.
What
what's
what's
wrong
with
this
and
what's
missing
in
a
distributed
monitoring
environment
is
a
view
like
this.
It's
a
bird's-eye
view
a
holistic
view
of
what
is
happening
to
that
contending
resource.
We
can
make
educated
guesses
and
often
people
do
when
they
reach
that
dead.
End
I
was
talking
about.
A
Maybe
it's
you
know
in
this
case.
Maybe
it's
an
accident.
It's
a
construction.
In
the
lock
case
you
can,
if
you
have
knowledge
of
the
code,
you
can
make
guesses
but
they're
after
all,
still
guesses.
We
need
data
to
to
prove
to
ourselves
that
our
theories
are
correct,
and
in
this
case,
thankfully
there
is
a
solution.
I'm
gonna
stay
on
theme
here,
trace
aggregates
right,
trace
aggregate
analysis
to
get
to
explaining
why
the
resources
being
contended
for
this
is
an
important
slide.
It's
basically
the
whole
idea
around.
A
What
opening
in
the
live
demo
that
I'm
gonna
show
in
a
few
minutes
is
remember:
I,
have
thousands
of
these
of
these
traces
and
for
each
of
those
chases
I've
used
some
statistical
analysis
to
find
the
critical
path
that
critical
path
in
my
case
is
mutex
acquire
time
spent
waiting
for
that
lock.
All
I
need
during
instrumentation
when
I'm
acquiring.
That
lock
is
to
assign
a
unique
ID
to
that
lock
it
just
as
a
tag
value
so
you're
taking,
remember
spans
or
contacts
rich
information
that
you
can
take
arbitrarily
with
any
number
of
characteristics.
A
So
in
this
case,
I
have
a
unique
ID
associated
with
that
critical
path
and
I
have
many
many
traces.
So
these
are
all
the
critical
paths
from
all
the
other
traces
and
what
you
can
do
is
you
can
actually
do
a
back
trace
because
you
have
the
entire
transaction.
You
can
figure
out
who
the
parent
color
is.
So
what
is
the
process?
What
is
the
operation?
What
is
the
function
in
code?
A
That's
trying
to
acquire
this
lock
and
how
long
did
they
spend
waiting
for
it
and
maybe
how
long
did
they
spend
holding
onto
the
lock?
So
if
you
look
at,
let's
say
thousands
of
these
you're
starting
to
basically
get
a
group,
an
aggregate
analysis
of
what's
contending
for
this
you're,
focusing
your
energy
on
alum
analyzing,
the
resource
that's
been
contended
for,
and
in
this
case
before
I
go
into
the
demo.
A
I
want
to
walk
through
quickly
some
terminology
that
I'm
gonna
be
using
as
a
part
of
the
demo,
it's
kind
of
made-up,
but
just
for
the
for
the
purpose
of
this
talk,
we
have
a
very
simple
concept
of
the
waiters,
so
you
know
a
few
threads
they're
trying
to
acquire
the
lock,
maybe
there's
a
queue
so
they're
in
queue
waiting
for
this
lock,
and
this
is
the
time
spent.
This
is
what
shows
up
as
that
critical
path
in
your
trace,
the
red,
critical
path
as
waiters,
and
maybe
at
some
point
thread.
A
So
you
have
context
for
the
different
operations
that
are
happening,
so
this
is
my
donut
as
a
service
architecture.
Have
this
really
awesome
bakery,
where
I'm
selling,
Donuts
and
I
have
an
API
order
service,
so
what's
happening
in
the
demo,
is
I
have
basically
have
a
bunch
of
robotic
a
bunch
of
background
orders.
That's
ordering
Donuts
non-stop.
In
the
meantime,
they
make
a
payment
for
the
donut
and
they
come
to
the
fryer
service
to
fry
the
dough
and
the
fryer
service
has
a
giant
mutex
on
it.
A
You
grab
the
mutex
you
fry
the
donut,
then
you
release
the
mutex,
and
this
is
done
serially,
one
by
one.
After
that
it
needs
to
be
topped.
The
topping
needs
to
be
added,
there's
three
different
types
of
donuts.
We
have
sprinkles
chocolate
and
cinnamon,
and
in
this
case,
before
you're
able
to
apply
the
topping,
you
also
have
to
grab
the
topper
lock,
though
different,
lock,
a
second
lock,
you
apply
the
topping,
and
then
you
release
the
lock
to
let
the
next
donut
through.
So
this
is
the
this.
A
Alright,
it's
a
little
tiny
here,
so
let
me
zoom
in
that's
a
bit
better.
Okay,
so
I
have
these
traces.
It
looks
very
similar
to
the
ones
I've
shown
before,
except
that
now
it's
with
somewhat
real
data,
at
least
the
data
that
I'm
generating.
So
you
have
this
background
order
and
you
can
see
I
can
add
a
tag
here
for
what
topping
it
is.
So
this
one
is
a
cinnamon
and
it
makes
a
request
to
process
a
payment.
Then
it
fries.
A
This
is
a
fryer
operation,
and
here
you
can
see
that
it
eventually
waits
on
the
fryer
lock.
If
I
click
on
this
span,
you
can
see
that
I've
tagged
this
with
a
fire
lock.
Remember
that
the
key
to
resource
contention
analysis
is
that
unique
ID,
which
is
fire
lock
in
this
case,
and
then
it
held
on
to
it
for
some
period
of
time
and
eventually
passed
it
off
to
the
topper.
It's
very
similarly
has
to
sprinkle
the
topping
on.
So,
if
you,
if
I
click
on
this
lock,
you
can
see
now
it's
a
different
lock.
A
It's
the
same
operation
mutex
acquire,
but
now
two
different
lock
is
the
topper
lock
and
we'll
see
this
show
up
in
the
analysis
later
and
I
have
a
few
of
these
opened
just
in
this
example.
So
this
one
is
a
chocolate
one
and
this
one
is
sprinkles.
So
you
know
if
we're
just
processing
this
visually,
it's
very
different.
To
really
see
the
differences.
I
mean
there's,
there's
definitely
some
duration
differences,
but
really
how
many
do
you
look
at
before
you
make
a
conclusion
about
what's
happening,
so
the
point
is
to
use
aggregate
analysis.
A
So
let
me
just
go
to
my
local
running
instance.
Here,
give
it
a
minute
like
Wi-Fi
through
okay
and
then
I
want
to
be
looking
at
my
donut
web
server
service.
So
I
run
a
query
for
that,
and,
and
here
at
the
top
you
can
see
the
latency
histogram
concept.
That
I
was
talking
about
earlier,
where
you
can
see
where
your
p95
is.
Actually
you
can
even
run
this
on
like
a
smaller
set
of
data?
Sorry,
my
computer
is
pretty
slow,
there's
the
p99
and
what
we
can
actually
look
at
is
a
diagram.
A
This
is,
it
looks
very
similar
to
the
one
in
my
slides,
where
you
have
the
donut
server
and
that's
making
requests
to
the
fryer
and
a
payment
service
and
a
topper
in
in
that
order
in
payment,
fryer
and
then
topper
order.
The
interesting
thing
here
is
that
it's
already
highlighting
the
two
services
of
slowing
me
down
so
the
Halos,
the
yellow
halos.
It's
already
telling
me
that
that's
what
I
should
be
looking
at
the
payment
took
no
time
which
we
you
know.
We
could
see
that
though,
and
then
in
the
trace.
A
It's
a
very
small
amount
of
payment.
Time
spent
okay,
let's
go
back
to
this
view
and
basically
what
the
correlation
remember.
The
correlation
I
was
talking
about
in
terms
of
looking
at
many
traces
is
that
you
can
already
see
it
telling
me
that
there
is
a
critical
pack
here
for
donut
top
her
operation
mutex
acquire
so
it's
it's
look
at
all
these
traces,
all
the
ones
that's
showing
the
histogram,
and
it
knows
what
operation
is
holding
up.
What
is
my
critical
path,
which
is
straighter
earlier?
A
So
what
we
really
want
to
see
is
a
mutex
view,
so
I
click
on
this
tab.
Ok,
so
here's
what
we
got.
We
have
on
the
top,
the
sprinkle
lock
and
then
there's
two
graphs.
There's
a
waiter
graph.
Remember
the
waiter
how
long
the
process
is
spent
waiting
and
we
have
a
holder
graph
because
there's
how
long
the
process
he
spent
waiting,
I'm,
sorry
holding
on
to
the
lock
and
then
on
the
bottom.
A
The
same
thing,
waiter
and
holder,
except
for
the
fryer
lock,
so
I
think,
let's
I
think
I
have
enough
time
to
go
through
the
the
sprinkle
case.
So
in
this
case
the
legend
might
not
be
clear
to
those
sitting
in
the
back,
but
basically
we
have
chocolate,
cinnamon
and
sprinkle.
So
in
this
case,
if
you
look
at
the
waiting
time
on
average,
this
is
divided
by
weighted
by
the
number
of
threads
or
the
number
of
instances
in
a
number
of
orders.
A
On
average,
they
have
the
same
waiting
time
in
total,
though
the
with
what
is
this
one,
the
cinnamon,
the
cinnamon
Dona,
spends
a
lot
more
time
waiting,
and
this
actually
makes
a
lot
of
sense
because
I
artificially
added
a
lot
more
cinnamon
orders,
so
in
total
its
spending
more
time,
the
system
is
many
more
time
waiting
on
for
don't
for
cinnamon
doughnut
orders,
just
because
there's
more
orders.
So
nothing
too
surprising
here,
but
here
it's
kind
of
interesting
here.
A
Despite
cinnamon
being
sort
of
the
the
biggest
order
in
terms
of
count,
so
let's
go
back
and
review
basically
what's
happening
here,
so
you
basically
have
cinnamon.
Maybe
this
is
your
largest
customer
sending
the
most
data
to
you
spending
a
lot
of
time,
waiting
on
chocolate
orders,
and
these
might
be
orthogonal
right.
A
You
can
totally
split
out
the
lock,
and
this
is
a
very
simple
conclusion
you
can
draw
by
just
looking
at
the
analysis
for
about
you
know
30
seconds
and
you
can
say:
okay,
let
me
split
up
a
lock
and
have
chocolate
and
cinnamon
acquire
separate
locks,
so
that
cinnamon
is
not
impacted
by
how
long
chocolate
spends
topping.
Even
though
they're
you
know,
they're,
basically
different
orders.
Okay,
so
I
need
to
wrap
up.
My
talk.
A
I
was
making
this
slide
and
I've
gotten
a
really
good
advice
in
the
past
about
how
I
should
schedule
my
my
talk
and
I
had
that
slide
at
the
beginning,
where
you
should
always
tell
your
audience
what
you're
gonna
be
talking
about,
and
then
the
advice
I
got
was
that
you
should
always
remind
your
audience.
The
things
that
you
talked
about
on
the
talk
so
I
was
in
the
middle
of
making
this
slide.
A
It's
very
simple
as
its
gonna
copy
over
you
know
the
correlation
stuff,
and
then
I
said
you
know
it's
a
at
the
end
of
a
very
long
day.
You're
gonna
fly
home
you're,
not
gonna.
Remember
any
of
this,
so
my
request
to
you
is
I'm.
Not
taking
that
advice,
but
all
I'm
doing
is
please
remember,
Trace
aggregates
as
the
keywords
it's
in
the
title
of
my
talk.
A
It's
the
theme
of
my
talk
the
next
time
anybody
talks
to
you
about
distributed
tracing,
remember
to
ask
them
about
what
insight
you
can
draw
by
looking
at
trace
aggregates.
You
know
if
you're
evaluating
products,
if
you're
just
exploring
tracing,
think
about
that.
Getting
to
that
single
trace
is
not
the
end
of
the
journey.
That's
just
the
beginning.
A
There
was
a
very
exciting
merger
yesterday
announced
for
open
telemetry,
which
is
the
next
new
version,
the
next
major
new
version
for
open
tracing
and
census.
So
if
you're
thinking
about
distributed
tracing
now
get
started
with,
one
of
them
get
started
with
open
tracing
and
it's
backwards
compatible
whenever
open
too
much
telemetry
comes
out
and
open
telemetry
enables
you
to
basically
have
built-in
telemetry
features
in
your
cloud
native
application
and
that's
the
end.
I
think
I
have
enough
time
for
a
few
questions.
C
Hi,
in
this
case,
it
seems
you
can
only
after
the
fact
it's
kind
of
when
things
are
slow.
You
cannot
need
information
for
things
that
you
have
been
tracing
for
and
since
even
your
code,
if
you
are
not
been
tracing
for
mutex,
then
you
would
not
know
why
you
spend
your
time
yeah
and
acquiring
that
mutex
is
just
a
part
of
a
cold.
A
I
mean
I,
think
I.
Think
instrumentation
is
sort
of
a
separate
talk
altogether,
but
in
this
case
we
basically
use
a
integrated
chased
block.
So
whenever
you
acquire
a
trace,
it's
a
it's,
a
third
party
library
and
it's
very
easy
I
in
this
case,
I
wrote
it
myself
whenever
a
acquire
a
trace,
I
start
a
span
and
whenever
I
get
that
get
the
mutex
I
end
this
ban,
so
that
I
have
contacts.
I
mean
this
is
the
thing
about
tracing.
Is
that
the
primitive
is
context
rich?
A
And
the
point
is
that
as
developers
we
need
to
get
used
to
adding
contacts
to
your
spans,
which,
with
four
metrics
and
logs
it's
a
little
bit
different.
You
know
metrics
typically,
is
you
have
a
key
and
you
have
a
value
and
that's
it
for
for
tracing
I.
Think
adding
as
much
contacts
at
developing
tab
development
time
into
your
traces
can
never
hurt
you
right
because
there
is
no
limit
on
cardinality.
So
you
might
as
well
add
all
the
tags
you
can
think
of,
and
this
is
what
we
internally
do
as
well
at
light
stuff.
B
D
Hey
thanks
for
the
talk.
My
question
is
when
looking
at
the
trace
aggregates
right.
We
had
that
particular
operation
that
that
particular
span
standout.
Do
you
have
a
way
to
aggregate
for
the
similar
span
happening
a
lot
of
time,
so
let
let's
assume,
for
instance,
the
services
multiplexing
calls
and
would
make
a
lot
of
calls
to
another
service.
D
So,
instead
of
like
having
one
time,
one
call
to
topping,
we
would
across
the
production
of
a
donut
top
like
27
times,
though
they
would
be
represented,
separate
spans,
but
in
in
a
total
time
taken
in
in
the
whole
process,
they
would
aggregate
to
the
to
a
lot
of
time.
Isn't
it
do
trace
aggregates,
have
any
capacity
to
depict
that
where
this
cooperation,
that
is
kind
of
sliced
up
into
multiple
pieces
kind
of
comes
together,
yeah.
A
So,
for
the
purposes
of
my
demo,
I
didn't
demonstrate
that
I
kind
of
the
total
time
is
kind
of
like
total
CPU
time.
If
you
will
it's
not
total
wall
clock
time,
you
could,
though,
slice
it
up.
You
could
put
a
restriction
also
on
the
time
that
you're
studying
the
lock
for
such
that
you're
only
looking
at
the
lock,
that's
literally
spending
time
waiting
for
other
locks,
because
they're
they're
contending
for
this
same
resource.
A
So
in
this
case
it's
just
a
matter
of
changing
the
view
of
your
analysis,
such
that
you're
aggregating,
like
parallel
tasks
and
you're,
not
counting
that
as
like
a
ballooning,
you
know
wall
clock
time,
but
it
is.
You
know
it
is
still
CPU
time,
even
if
it's
being
processed
in
parallel
in
process
yeah,
of
course,.
E
In
your
example
beats
at
the
end,
like
it
was
one
specific
server
which
was
slowing
down,
so
it
was
amount.
Tunneled
of
several
bits
was
one
tag.
You
were
looking
at
what,
if
you
have
like
thousands
of
different
like
different
different
inputs
and
that's
combination
of
five
ten
different
tag,
which
is
actually
the
culprits?
How
do
you
manage
to
actually
find
the
right
combination,
which
is
slowing
down
the
wall
system,
yeah.
A
I
think
I
think
there's
maybe
two
questions
two-part
question
and
I'll
answer
the
first
one,
which
is
you
know
the
different
values
which
is
you
know
if
there's
thousands
and
thousands
of
servers
and
you
were
diligent
about
instrumentation
and
you
have
the
unique
values
the
system
is
very
fast
at
picking
up
the
one.
That's
wrong
and
I
think
your
real
question
is
what
happens
when
there's
a
combination
of
tags?
A
That's
really
the
problem,
not
just
a
single
one,
but
maybe
this
server
combined
with
this
time
or
this
server
combined
with
this
customer
and
that's
sort
of
a
different,
that's
actually
called
multi
multivariate
correlation,
which
I
didn't
show
in
this
case
that
we
can
do
that
as
well.
It's
just
I
think
requires
more
processing,
time
and
statistical
analysis.
A
You
basically,
then
have
to
look
for
combinations
of
things,
that's
causing
issues
and,
of
course
it
can't
do
infinite
combinations,
and
that
would
take
a
really
long
time
so
there's
optimizations
associated
with
which
variables
make
sense
to
be
combined
together.
But
that's
a
that's
a
really
enhanced
level
of
analysis
that
this
can
definitely
support
as
well.