►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Keynote: Distributed Tracing - The Struggle is Real - Ian Smith, Chronosphere
In 2010, Google published a paper describing how they implemented distributed tracing internally. 12 years later, we’ve made limited progress on widespread implementation of this technology, despite near universal agreement that distributed tracing has the potential to solve many of the challenges brought on by cloud native. Despite our best efforts, few companies have fully implemented distributed tracing and even fewer engineering organizations utilize it with confidence. In this session, we’ll explore where organizations are getting stuck with distributed tracing and how they can overcome these obstacles. To drive broader adoption of distributed tracing, we need a new approach.
A
Most
people
don't
okay,
one
person
all
right.
It
doesn't
matter
we're
not
going
to
walk
through
the
whole
room,
but
in
any
case,
for
me,
March
of
2014
was
when
I
started.
Working
on
monetary
observability
I
started
working
at
a
company
that
I'm
sure
a
lot
of
you
know
that
was
working
on
commoditizing
APM
for
the
masses
and
since
then,
I've
had
the
fortune
of
working
for
a
bunch
of
vendors
that
you
love
and
some
vendors.
A
Quite
literally
so
for
those
of
you
who
unaware
the
Segway
was
revealed
to
the
world
around
the
time
that
a
bunch
of
us
were
still
worried
about
Y2K,
there's
a
lot
of
hype
and
Fanfare,
and
really
it
was
meant
to
change
the
world.
If
you
believed
every
bit
of
hype
and
media
that
you
saw
out
there,
you
know
things
like
the
way
that
we
move
the
way
that
our
towns
and
cities
were
going
to
be
built
was
all
going
to
be
affected
by
this
brand
new
product
and
Technology
lots
of
flashy
features.
A
So
chances
are,
if
you're
here,
you've,
probably
seen
or
heard
some
issues
around
the
use
of
distributed
tracing
and
much
in
sort
of
the
same
way
that
the
Segway
is
was
really
definitely
didn't,
live
up
to
its
hype.
There's
definitely
a
potential
for
distributed
racing
to
not
have
the
impact
that
we
all
hope
that
it's
going
to
have,
particularly
if
we
keep
treating
it
the
way
that
we
have
up
until
now,
as
we've
gone
from
monoliths
and
VMS
to
microservices
and
containers,
our
engineering
teams
have
really
struggled
with
the
amount
of
data.
A
The
complexity
of
the
observability
problems
that
they're
dealing
with
and
distributed
tracing
is
sort
of
generally
seen
as
this
okay
cool
for
distributed
environments,
we've
got
distributed.
Tracing
makes
sense,
it's
all
in
the
name
really
just
need
to
load
in
some,
some
tracing
instrumentation
load
up
a
tracing
back
end
and
you
know
throw
as
many
complex
analytical
features
of
the
problem
as
possible,
but
despite
all
of
these
huge
Investments,
both
the
open
source
Community
covered
some
of
those
earlier,
as
well
as
on
the
vendor
side.
A
A
So
I
want
to
talk
a
little
bit
about
the
kinds
of
problems
that
our
engineering
teams
are
facing
in
using
distributed
tracing.
But
importantly,
we
really
need
to
look
at
the
organizational
problems
that
we're
seeing
not
just
the
technical
problems,
not
just
the
hey.
What
can
I
build?
You
know
a
new
query
engine.
A
Your
adoption
never
really
hits
that
critical
mass
and
it
starts
to
fall
off
rapidly
after
day,
one
right,
it's
the
most
valuable
it
ever
gets
the
most
engaged
it
ever
gets
in
day,
one.
It's
also
fall
off
and
a
lot
of
it
tends
to
focus
on
four
key
problems.
Four
key
themes
that
I
hear
continually
the
tools
are
really
only
useful
for
power
users.
You
know
the
smartest
the
brightest,
the
most
long
tenured
Engineers
are
the
ones
who
can
operate
these
tools.
A
They
know
their
technology,
they
know
the
data
set,
they
know
the
architecture
and
they
can
make
use
of
it,
but
oftentimes
it'll
bottleneck
for
the
rest
of
their
organization,
and
then
part
of
the
thing
is
that
it's
not
easy
to
go
into
this
oftentimes
tracing,
tooling.
Sort
of
slapped
on
on
top
of
everything
else
is
happening.
A
So
you
run
into
issues
where
you
really
need
data
from
someone
else
and
they
haven't
done
their
instrumentation
and
vice
versa,
and
this
ends
up
with
this
sort
of
sorry
State
of
Affairs,
where
you're
tracing
for
those
who
have
invested
heavily
and
have
utilized
it
for
a
period
of
time,
really
see.
There's
a
high
promise,
High
effort,
but
sort
of
low
outcome,
low
value,
sort
of
story
that
they've
they've
spent
this
time
and
effort
on
so.
A
How
do
we
move
away
from
sort
of
that
potential
future
of
the
the
lonely,
little
segue
in
the
desert
lost,
and
how
do
we
think
about
really
taking
a
step
back
and
moving
away
from
the
sort
of
features
and
functions
and
all
that
sort
of
like
technical
Focus
that
we've
really
applied
to
distributed
tracing
up
until
now
and
really
think
about
the
outcomes
and
think
about?
Well?
How
do
we
be
purposeful
about
this?
A
There's?
No
real
silver
bullets.
I'm
not
like
up
here
to
say:
Hey,
you
know,
chronosphere
or
anything
else
is
about
to
announce.
Some
new
magic
feature
is
really
making
sure
that
we
start
thinking
about
this
properly,
particularly
as
a
mass
number
of
organizations
are
going
to
start
adopting
distributed
tracing.
A
So,
let's
think
about
maybe
the
the
number
one
thing
that
most
people
think
about
when
they
think
about
the
use
of
tracing
is
sort
of
instance
and
responding
to
alerts.
Sorry,
there's
going
to
be
lots
of
different
variations
of
this
diagram,
but
in
terms
of
what
I
hear
continually
from
from
organizations,
how
are
they
using
it
today?
As
I
said,
a
lot
of
the
times
distributed
tracing
sort
of
gets
slapped
on
existing
processes,
but
particularly
as
you've
moved
into
more
complex
architectures,
more
microservices,
more
containers,
more
Cloud
native
I've
got
all
these
alerts
going
off.
A
Everyone
has
alerts
going
off,
but
it
sort
of
leads
into
this.
Your
big
question,
puddle
of
like
okay:
where
do
we
look
who's
responsible?
How
many
teams
do
we
go
get
involved?
Maybe
you
look
at
logging?
First,
maybe
you
look
at
your
legacy.
Apm
first,
maybe
look
at
metrics
dashboards
first,
but
almost
no
one
looks
at
distributed
tracing
first
I'm,
not
necessarily
saying
you
should,
but
oftentimes
distributed
tracing
is
used
by
that.
You
know
one.
You
know
principal
engineer
at
the
back
of
the
room,
going
wait,
I
have
a
theory.
A
A
So
if
we
take
another
step
back
and
think
less
about
the
the
incident
workflow
and
more
a
little
bit
about
well,
what
should
it
actually
do
for
us
think
about?
Well,
we
had
all
of
this
sort
of
microservices
complexity.
We
deliberately
added
this.
Why
we
did
this,
because
Engineers
should
be
able
to
go
and
move
and
operate
independently
of
each
other
sort
of
think
about
your
individual
team.
Think
about
your
your
coordination,
only
one
tier
away
from
you
but
you've
introduced
all
this
additional
complexity,
and
now
you
really
can't
handle
it.
A
Can
we
really
make
it
so
that
a
human
being
can
think
about
the
wear
of
what's
going
on
and
then
start
applying
all
the
other
tools
that
we've
we've
used
before
it's
not
about
distributed
tracing,
replacing
everything,
but
it's
making
the
the
haystack
that
we've
got
to
dig
into
a
little
smaller.
Then
we
can
have
sort
of
a
much
more
focused
use
of
it
and
we
can
think
about
okay.
We
can
be
purposeful
as
how
we
use
it
in
those
incidents
and
other
use
cases.
A
So
if
we
think
about
a
slightly
nicer
diagram,
you've
got
your
alerts,
those
alerts
they
can
lead
into
dashboards.
Let
you
know
about
things
like
your
symptoms.
Justify
your
investigation,
your
symptoms
and
your
impact.
You
can
use
distributed
tracing
to
figure
out
where
so
you
can
go
and
narrow
that
scope
the
the
size
of
that
Haystack.
You
can
better
use
your
logging,
your
API,
your
metrics,
whatever
it
is
events,
exceptions
all
those
wonderful
things
that
we
want
to
use,
but
we
also
need
to
use
and
feed
it
back
into
things.
A
So,
for
example,
everyone
focuses
on
slos
tracing
is
a
great
data
set
to
be
able
to
improve
our
slos?
It's
a
great
data
set
to
be
able
to
help
illustrate
what
symptoms
we're
having
and
the
impact
that
we're
having
on
our
customers
and
so
being
able
to
sort
of
tie
this
together
and
make
it
a
loop
means
that
you
don't
have
that
oh
distributed
tracing
is
only
as
valuable
as
it
was
on
day.
One
type
situation
so
ultimately
really
quickly.
A
What
we
want
to
do
is
drive
the
adoption
and
realize
the
value
of
distributed
tracing,
but
to
do
so,
we
need
to
be
purposeful
about
what
we're
doing.
We
can't
just
you
know,
throw
up
a
distributed,
tracing
back
end
and
have
one
engineer:
who's
really
good.
At
fiddling
around
with
Advanced
query
language
we
need
to
integrating
workflows,
we
need
to
focus
on
showing
where
problems
exist,
because
that's
allowing
us
to
negate
the
complexity.
We
introduce
purposefully
into
this
situation
and
make
sure
it's
useful
for
every
engineer.