►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Distributed Tracing Without the Pain! - Sachin Ashok & Vipul Harsh, University of Illinois at Urbana-Champaign (UIUC)
Monitoring and debugging modern cloud-based applications is challenging due to their highly distributed nature. End-to-end distributed tracing (tracing individual calls through a request's lifecycle) has emerged to be essential in a developer's toolbox as they're critical to describing a request's flow through a microservice. Unfortunately, to enable distributed tracing, each service along the request tree needs to propagate a global request ID to help link the related requests (i.e., link incoming requests with spawned backend requests). Enabling such header propagation can be non-trivial for large microservices with 100+ independent services or for legacy apps where instrumentation is hard. This talk explores whether distributed tracing can be made more accessible by eliminating the need for application instrumentation. We describe a method that combines observations external to the app (using an envoy-based service mesh) with timing analysis of the requests to construct end-to-end traces. In an evaluation with a simple microservice, this preliminary method boosts trace reconstruction accuracy to 96% (compared to 77% for a baseline), and can help answer useful developer queries.
A
Okay,
hello,
everyone,
I'm
satina,
show,
along
with
my
teammate,
with
Bluehost
today,
I'm
going
to
be
presenting
our
recent
work
on
distributed
tracing
and
specifically
on
how
to
enable
distributed
tracing
without
the
pain
where
a
couple
of
PhD
students
from
the
University
of
Illinois
I've
heard
about
our
shopping,
and
this
is
work
in
collaboration
with
our
advisors
at
University
of
a
lot
as
well
as
a
few
researchers
and
idea.
Research.
A
Observation
that
debugging
microservices
is
quite
hard,
I'm
sure
many
of
the
audience
differently,
but
your
library
consider
this
sample
microservice
system,
as
shown
here
where
we
press
flow
from
left
to
right
and
suppose
you
want
to
service
many
requests
that
all
right,
after
maybe
I
gave
in
such
a
system,
requests
typically
take
a
very
contributed
product
through
the
smart
service
system
in
order
to
get
a
single
response,
and
this
can
be
a
nightmare
for
visibility.
This
is
because
many
complex
systems
must
be
dragged
to
produce
even
a
single
and
direct
request.
A
Red
and
yellow,
so
all
red
and
yellow
red
candles
here
correspond
to
me.
Requests
that
propagate
within
the
system,
or
they
call
stands
in
this
terminology
and
the
job
of
distributed
tracing
is
to
put
together
these
tasks
in
such
a
way
that
you
can
obtain
the
cholera
of
a
single
Loop
rest
after
this
flowing
through
this
microservice.
C
A
A
A
A
Is
that,
let's
say
you
have
a
scenario
where
you
have
him
coming
to
refresh
RMS?
Okay,
the
area
might
generate
different
Equity
requests
to
the
bank
in
order
to
Services
these
client
requests.
The
main
challenge
lies
in
the
confusion
that
there
are
many
options
available
for
this
match.
For
example,
is
this
the
right
mapping
between
the
interview
request
and
the
backup
request,
or
is
this
the
right
number
without
thinking
into
application
code?
A
A
A
B
B
D
B
Is
because
these
mappings
are
buried
inside
the
application
logic,
so
without
peaking
inside
the
application,
it's
not
possible
to
decide
which
request
was
sent
in
response
to
what
incoming
request-
and
this
is
really
the
Crux
of
the
problem.
How
do
we
map
incoming
requests
to
the
backend
request,
and
this
and
solving
this
problem
would
enable
us
to
stitch
together
these
mappings
and
form
an
end-to-end
Trace.
B
B
B
B
The
second
part
of
the
input
are
a
set
of
constraints
which
tell
us
which
mappings
are
possible
and
which
mappings
are
not
possible.
So
in
this
example,
you
can
see
that
this
mapping,
if
you
look
carefully,
you
can
see
that
this
mapping
is
not
possible,
and
this
is
because
the
response
of
the
back
end
could
not
have
been
sent
out
after
the
response
was
sent
out
to
the
user.
B
So
using
these
three
inputs,
we
feed
the
we
feed
that
to
a
trace
reconstruction
algorithm,
which
outputs
the
most
likely
mapping
given
his
given
a
historical
distribution
of
delays
at
each
micro
service.
So
specifically
for
our
previous
example,
where
we
had
three
requests:
three
incoming
requests
and
three
back-end
requests.
It
outputs
a
specific
mapping
for
this
for
the
backend
request
and
the
incoming
requests
at
this
node
and
similarly,
it
outputs
the
mapping
for
every
service
node,
and
then
we
can
stitch
them
up
to
get
the
end
to
entries.
B
In
contrast,
the
algorithm
that
we
have
generates
these
end-to-end
faces
with
high
accuracy
even
at
high
loads.
So
we
asked
the
question:
how
can
this
be
useful
to
a
developer?
So
let's
say
the
developer
is
interested
in
answering
the
following
question,
which
service
contributes
most
to
the
delay
experienced
by
the
slowest
two
percent
of
the
requests.
B
So
here
is
a
latency
profile
of
all
those
services.
When
you
only
have
span
level
information
which
you
can
get
today
in
non-voice
Via
a
simple
flag.
So
in
the
absence
of
end-to-end
traces,
the
developer
would
look
at
the
98
percentile
latency
of
each
service,
and
the
developer
might
conclude
that
some
of
these
services
are
contributing
to
the
delay.
B
Now
this
benchmarking
thing
that
we
did
was
was
it
does
not
cover
the
entire
breadth
of
you
know:
different
kinds
of
micro
service
applications
that
exist
out
there,
but
this
is
an
ongoing
project
and
we
are
still
we
are
looking
into
how
we
can
incorporate
the
various
idiosyncrasies
of
you
know:
complex
microservice
applications
that
exists
out
there,
which
includes
things
like
asynchronous
execution.
Caching
of
responses,
or
you
know,
batching
of
requests.
B
B
E
C
A
So
we'd
be
piggybacking
on
existing
sort
of
logging,
about
Envoy
dots
for
requests
and
request
response
mappings.
What
we
would
not
be
doing
is
actually
going
inside
the
code
and
adding
like
code
there,
which
can
be
really
time
consuming
for
when
you're
dealing
with
like
thousands
or
hundreds
of
microservices
in
terms
of
the
cost.
If
you
need
to
store
so
end-to-end
tracing
does
require
you
to
store
the
log
so
that
we
can
do
the
mapping.
D
F
A
Right
right,
that's
a
great
question,
so
it
is
a
tough
problem
to
solve.
So,
given
that
we
cannot
actually
instrument
the
application,
what
we
can
do
is
try
to
provide
the
best
accuracy
possible
and
then
for
the
end-to-end
traces
that
you
you
do
make
available
to
the
application
developer
you
we
are
able
to
assign
a
confidence
score,
which
we
can
say
which
can
like
you
know,
relate
to
the
developer,
that
we
are
like
sort
of
99
sure
that
this
mapping
is
correct
and
in
cases
of
low
confidence
developer,
rather
not
look
at
those
traces.
A
That's
one.
The
other
is
that
there
is
a
chance
to
augment
information
which
is
other
than
timing.
So
if
we
can
use,
if
you
have
like
a
partially
instrumented
application,
where
some
of
the
services
are
easier
to
modify,
but
the
Legacy
applications
are
not
or
the
proprietary
applications
system
are
not
amenable
to
changing
their
code,
then
we
could
do
timing
based
analysis
or
our
technique
for
just
those
thoughts,
but
you
could
still
get
the
entire
end-to-end
Trace
by
leveraging
the
existing
sort
of
instrumentation.
So.
B
Just
to
add
to
that
there
are,
for
example,
if
you
have
aggregate
queries,
if
you
want
to
know
about
you
know
some
latency
distribution
for
some
subset
of
queries,
then
we
believe
that
you
know
good
approximate
accuracy
should
be
fine
for
that.
So
those
are
the
kinds
of
queries.
That
would
definitely
be
the
use
case,
and
if
you
provide
the
confidence
score,
then
the
individual
requests
traces
can
be
useful
to.
E
Yeah
I
think
I
have
the
same
question
a
little
bit.
What
what
Matt
was
asking
in
these
large-scale
production
systems
like
not
having
something
that's
100
is
is,
and
we
cannot
trust
it.
It's
very
hard
because
you,
you
know
you're,
always
optimizing
the
bottom
or
top
point
one
percent
anyway.
So
it's
very
hard.
My
question
is:
does
your
prototype
include
the
ability
to
only
report
on
spans
where
you
have
a
hundred
percent
confidence
and
just
disregard
the
rest.
A
A
A
To
answer
your
earlier
question,
so
if
you
do
have
a
large
microservice
application,
where
you
know
you
do
care
about
that,
one
request:
that's
sort
of
seemingly
failing
it's
a
really
hard
problem
to
solve,
but
what
I
want
to
highlight
is
that
you
could
have
scenarios
where,
in
those
in
that
large
application,
where
you
have
systems
that
just
not
just
do
not
cooperate,
you
don't
have
like
you.
Let's
say
you
have
a
legacy
application
that
does
not
propagate
the
header.
You
in
that
scenario,
you
just
do
not
have
the
end-to-end
trace
right.
A
B
A
D
A
Right,
so
that's
a
good
question,
so
we
do
so.
We
do
consider
those
cases
and
that's
ongoing
work
in
that,
like,
for
example,
you
could
have
request.
Caching,
where,
like
you
know,
a
response
doesn't
go
out
to
a
microservice
and
they
just
cached.
Then
it
goes
out
or
you
could
have
like
request,
retries
or
patching
of
requests.
So
what
we
can
do
is
that,
so
what
we
do
is
right
now
we
use
a
developer
environment
to
try
out
these
very
cases.
A
All
these
cases
to
figure
out
what
is
the
scope
or
what's
the
space
of
possibilities
and
then
based
off
the
real-time
measurements
we
get.
We
can
use
that
combination
of
that
information
to
figure
out
whether
oh
this
looks
like
caching
has
happened,
or
this
scenario
looks
like
matching
has
occurred
and
that's
why
the
timings
are
so
are
the
spans
collected
also
and
then
combine
that
information
to
give
the
right
answer.
A
But
this
space
is
quite
open
because
there
could
be
like
a
Marathi
pattern,
so
you
could
say
every
seventh
response:
I'm
going
to
generate
a
like
a
request
to
a
different
service
and
those
kinds
of
things
are
harder,
but
we
think
that
this
solution
can
apply
to
a
broad
class
of
applications
that
do
not
have
very
crazy
patterns.
So
it's
still
using
by
a
wide
variety
of
applications.
Yeah.
A
A
A
Right
so
so
the
problem
is
that
Envoy
can
pass
the
request
onto
the
application,
but
it's
the
application
that
decides
which
other
service
I
want
to
talk
to.
What's
the
semantic
of
this
application
flow,
which
are
the
service,
do
I
talk
to,
and
that's
the
one
that
generates
a
new
HTTP
connection
and
sends
it
so
Envoy
even
from
online
there's,
no
way
to
know
which
outgoing
request
corresponds
to
which
incoming
request
it
can
hand
over
the
request
of
the
service,
but
it
cannot
do
the
matching
by
itself.