►
From YouTube: OpenTracing and Containers: Depth, Breadth, and the Future of Tracing [I] - Priyanka Sharma
Description
OpenTracing and Containers: Depth, Breadth, and the Future of Tracing [I] - Priyanka Sharma & Ted Young, LightStep
Those building microservices at scale understand the role and importance of distributed tracing: it’s the most direct way to understand how and why a system is misbehaving. But distributed tracing has long harbored a dirty secret: the source code instrumentation can been complex, fragile, expensive, and difficult to maintain.
The OpenTracing project addresses that integration problem through standardization and collaborations with other open-source software systems. In this talk, Ted will begin by describing OpenTracing and explaining why you should care about it. He will then show how OpenTracing will be able to deliver zero-touch, black-box instrumentation of distributed applications via orchestration systems like Kubernetes, and why that could change the way we all reason about distributed computation.
About Priyanka Sharma
Head of Product Marketing, Partnerships, & Open Source
Priyanka Sharma is an entrepreneur with a passion for building developer products and growing them through open source communities. She heads Marketing and Partnerships at LightStep and also works on the OpenTracing project, an instrumentation standard for distributed tracing. In her copious spare time she advises startups at HeavyBit industries, an accelerator for developer products. Priyanka cofounded WakaTime, an open source time-tracker for developers and holds a BA in Political Science from Stanford University.
About Ted Young
Ted has spent the last 15 years building distributed systems in a variety of environments: computer animation pipelines for VFX, national elections, and elastic compute platforms. Previously, he helped design the Diego Container Runtime for Cloud Foundry. Currently focused on OpenTracing and tools for root cause analysis.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Buddy
I
am
Priyanka
and
I
work
on
the
open
tracing
project.
I
also
had
open
source
and
partnerships
at
life
step.
So
some
of
you
might
have
heard
my
keynote
yesterday.
How
many
people
were
there?
Okay,
a
good
number,
that's
cool,
so
you
already
have
a
little
bit
of
an
intro
to
distributed
tracing
and
open
tracing.
However,
for
those
who
are
not
there,
I'd
like
to
go
over
a
brief
overview
and
discuss
the
need
for
distribute
tracing
where
open,
tracing
steps
in
and
then
follow
that
up
with
the
donor
tool
on
demo.
A
A
So
this
is
the
recap
part
what
is
tracing
and
why
care
at
the
core
of
the
concept
distribute
tracing,
helps.
You
know
where
transactions
are
going
throughout
your
system
across
process
boundaries,
and
this
helps
you
create
a
visual
or
mental
map
of
what's
going
on
in
your
distributed
system,
you
can
think
of
it
as
an
action
replay.
From
the
perspective
of
a
request.
The
question
is
what
happened
to
the
requests
in
each
service
and
how
that
connected
across
different
services?
A
So
distribution
really
comes
into
focus
when
you're
dealing
with
a
lot
of
concurrency.
When
somebody
starts
an
MVP
basic
project,
they
might
not
see
anywhere
they're
doing
the
simple
and
efficient
saying
for
the
hackathon
to
get
that
project
out,
ASAP
right
slowly.
Let's
say
that
app.
Like
my
donut
salon,
apps
that
started
getting
popular,
then
you
start
introducing
a
little
bit
of
concurrent
processes
because
you
want
to
move
faster.
You
have
more
of
a
user,
a
user
load
at
Sara
and
then,
as
things
get
more
complicated,
you
get
two
async
and
then
finally
distribute
concurrency.
A
But
you
see
on
the
on
my
left,
so
I
guess
you're
right
yeah
on
your
right,
so
the
red
dotted
line
and
dashes
is
the
path
of
a
single
request
as
it's
going
through,
and
this
is
the
visual
that
a
distributed
tracing
can
afford
you
now
all
that's
great.
But
why
is
this
useful,
as
we
discussed
in
the
keynote
developer?
A
So
if
you
had
complete
observability
into
a
monolithic
structure
where
you
could
quickly
go
and
pinpoint
a
problem,
suddenly,
that's
not
true
anymore,
mean
time
to
resolution
is
rising,
because
how
do
you
resolve
something
when
you
don't
know
what
it
is
and
where
it
is
and
that's
what
the
same
story
is
root,
causing
an
issue,
and
we
all
know
it's
much
harder
when
it's
2
a.m.
at
night.
So
that's
the
problem
part
and
that's
where
this
is
where
distributed
tracing
becomes
useful
and
so
tracing
is
super.
A
Valuable
companies
have
done
it
for
a
while
now,
but
it's
not
been
ubiquitous,
and
we
touched
on
this
in
the
keynote
as
well,
but
to
go
a
little
bit
deeper,
which
is
to
me
the
value
of
the
session
there.
There
are
specific
issues
for
companies
looking
into
a
technology
like
tracing
by
the
way.
In
this
talk,
I
used
the
word
tracing
so
many
times,
I
apologize
for
that
there
I
couldn't
find
a
good
synonym.
It
would
change
the
meaning
cetera
anyway,
so
yeah.
A
So
first
off
is
the
auto
auto
instrumentation
option
right
ironically,
I'll
be
talking
about
that
in
a
kubernetes
environment
towards
the
end
of
this
talk,
but
at
the
moment
that
doesn't
give
you
the
kind
of
visibility
you
need
in
a
custom
system.
Each
distributor
system
is
a
special
snowflake
and
to
the
developers
working
on
it,
it's
extra
extra
special
and
you
every
business
is
also
different.
A
So
the
business,
metrics
and
values
are
not
the
same,
so
Auto
instrumentation,
just
it's
too
magical
and
it
doesn't
give
it
doesn't,
exhibit
the
exact
things
you
need
and
I
hear
this
every
day
from
people
who
use
the
our
API
spec.
The
other
thing
is
there
are
api
is
for
tracing
in
different
languages
as
like
log4j
and
all
of
that,
but
the
value
of
tracing
is
when
you
can
see
across
your
entire
system
across
processes.
A
How
many
of
you
here
have
a
distribution
system
in
one
language,
only
okay,
a
few
hands
and
how
many
of
you
have
more
than
one
language
in
your
system,
most
of
their
own?
Exactly
so.
If
you
only
saw
part
of
the
system
again
the
problem,
it's
it's
a
moot
point
right
then
there's
the
excuse
me
handoff
pose
so
most
how
many
people
here
in
their
system,
there
is
open
source
code
right.
A
All
of
you,
that's
the
that's
the
nature
of
software
development
now
and
if
each
open
source
project
is
a
black
box
or
has
its
own
special
kind
of
tracing,
you
again
lose
that
whole
purpose
of
having
end-to-end
and
instrumentation
and
understanding
of.
What's
going
on
so
this
was
another
reason
why
a
tracing
effort
would
be
very
onerous
both
in
terms
of
time,
money,
effort
for
a
company
and
then,
finally,
even
if
some
standards
were
there
or
instrumentation
best
practices,
they
were
all
very
vendor
specific,
which
is
a
huge
risk
for
a
large
organization.
A
We've
been
hearing
from
all
the
big
company
people
in
this
conference
and
they
talk
about
how
hard
it
is
do,
as
someone
was
saying,
shift
the
elephant
now,
if
you're,
shifting
an
elephant
and
you're
tied
to
the
person
who's
shifting
it,
what
if
the
elephant
gets
mad
and
you
can't
get
rid
of
the
person
who's
shifting
it.
That's
really
scary
right,
I
grew
up
in
India
and
my
father
is
a
forest
officer.
So
I
think
this
analogy
makes
more
sense
to
me
than
you.
A
But
anyway,
the
point
I'm
trying
to
make
is
that,
when
you're
making
such
a
big
effort,
if
you
have
no
way
out
of
a
specific
strategy,
then
it's
very
risky.
What
you
ideally
want
is
that,
if
you
instrument
you
can
then
try
out
different
ways
to
make
things
work
for
you
and
arrive
at
the
best
solution,
which
kind
of
goes
back
to
the
first
Auto
instrumentation
point:
that
every
system
is
a
special
snowflake
and
needs
its
own
solution.
A
So
this
is
where
open
tracing
enters
the
story
like
before
we
get
to
the
open
tracing
spec.
It
will
be
helpful,
tarnish
and
a
little
bit
of
what
led
to
it.
We
discussed
the
problems
that
companies
face
already,
but
if
you
think
about
the
past,
this
is
not
a
new
concept
right.
Tracing
research
has
happened
since
the
70s
I
know.
Brown
University
just
does
a
lot
of
work
on
this
or
two
dates.
A
They
come
to
all
the
distributed,
tracing
workshops
that
we
do
and
then
Google
faced
the
problems
of
scale
with
microservices
much
before
most
other
companies,
because
they
had
they
were
seeing
web
traffic
at
this
unprecedented
level.
So
they
broke
out
into
services,
and
then
they
instantly
started
feeling
that
tooling
problem
and
some
people,
including
been
single
men,
who's.
A
co-author
of
the
open
tracing
spec
built
out
dapper
at
Google,
which
is
Google's
always-on,
distributed
tracing
system.
A
Then
they
wrote
a
paper
about
this
that
got
very
popular
and
based
on
it.
All.
The
forthcoming
solutions
for
tracing
were
bills,
including
Zipkin,
which
was
built
for
Twitter
and
then
open
sourced
in
around
2009.
I
am
NOT,
her
percent
sure
on
the
exact
year
and
that
became
sort
of
the
de
facto
open
source
go-to
for
most
people
and
that's
built
exactly
on
the
principles
of
dapper.
A
So
that's
a
little
brief
history
of
where
open
tracing
comes
from.
It
didn't
come
out
of
vacuum.
It
came
from
years
of
experience
and
years
of
pain
that
eventually
led
to
this.
That
by
no
means
means
that
this
is
like
a
perfect
dance
back.
We
released
1.0,
but
its
ever
incremental.
There
are
some
big
changes
coming
up
in
the
API
that
will
be
soon
announced,
which
I
think
you
will
be
excited
by
anyway.
A
So
open
dressing,
in
a
nutshell,
is
a
standardization
spec
for
people
who
are
looking
at
tracing
and
it
helps
developers,
of
course,
but
it
helps
people
who
are
managing
running
applications
that
are
in
production,
because
it's
provides
vendor
neutrality.
You
can
go
to
any
different
solution,
build
your
own,
whatever
works,
and
you
have
that
flexibility.
It
helps
open
source
framework
owners
because
suddenly
they
can
help
their
user
base
have
more
visibility
into
the
the
framework
itself,
debug
their
own
problems
without
having
to
pick
any
sides
open
tracing
is
a
vendor-neutral
standard.
A
A
So
when
we
think
of
the
open
tracing
architecture,
it's
important
to
know
it's
an
API
that
sits
in
the
middle
that
is
talking
on
what
that
is
on
one
side.
There
is
the
actual
application,
where
there's
the
logic,
the
RPC
framework,
the
other
things
existing
instrumentation
and
then
on.
The
other
side
is
actually
the
tracing
systems.
Such
as
Zipkin
light
step
Jaeger
by
uber,
which
is
just
going
to
be
open
sourced.
A
So
the
important
thing
here
that
if
the
API
sits
in
the
middle
and
provides
the
plumbing
sort
of
the
pipes
and
the
drains
and
the
roads
for
data
to
be
taken
out
of
your
system,
but
it
does
nothing
with
it
all
the
analysis.
All
the
insights
that
you're
going
to
get
are
going
to
come
from
the
tracing
solution.
Open
tracing
has
no
hand
in
that.
So
that's
something
important
to
remember:
it's
a
no
hablan
by
itself
very
glamorous,
unfortunately,
but
it
is
what
it
is
and
what
it
does
it.
A
It
ends
up
menteng
traditional
tools,
because
the
kind
of
data,
as
you
can
imagine,
is
very
helpful
when
you're
looking
at
logs,
metrics
or
doing
critical
path.
Analysis
now,
I'm
not
saying
that
this
replaces
all
your
logging
tools
or
a
metrics
tool.
It
means
that
you
can
prove
send
better
data
to
these
tools
and
analyze
wherever
you
want
to
analyze
sort
of
your
choice
and
that's
kind
of
why
the
open-source
community
has
really
gotten
behind
this
in
the
one
short
year
of
existence.
A
Our
PC
is
so
great
for
anybody
who
has
inter
process
communication
and
Ruby
is
new
support,
but
these
are
just
some
of
the
examples,
but
there's
a
lot
more
and,
as
I
said
before,
there's
tons
of
vendors
who
are
joining
the
bandwagon
as
well,
because
this
is
just
good
for
everybody
there's
a
ways
to
go,
but
the
starting
point
is
feeling
right
to
a
lot
of
people
and
that's
the
same
reason
why
companies
are
risk
free
using
it
and
we've
been
through
this.
So
whatever
now,
how?
A
Many
of
you
remember
the
donut
salon
demo
from
yesterday?
Okay
about
half
so
for
those
who
don't?
Do
you
guys
mind
if
we
just
go
and
go
over
it
again
to
demonstrate
this?
We're
tracing
cool
awesome
so
for
that
I
would
like
your
help.
Please
take
out
your
mobile
phones
and
go
to
donut
salon.com,
which
is
donuts
as
a
service.
This
is
my
startup
that
I
have
built
it's
experiencing
hyper
growth,
going
growing,
really
fast,
it's
built
with
micro
services,
but
we're
having
concurrency
issues,
but
we
build
it
with
open
tracing
was
really
smart.
A
A
This
will
not
be
a
disaster
if
you
don't
help
so
click
a
lot
I'm
relying
on
you.
Okay,
we've
done
this
perfect
okay.
So
now,
let's
see
what
is
the
architecture
of
this
amazing
startup
mind
of
a
billionaire
tomorrow?
So
basically
it's
all
bootstrapped,
so
very
simple
system.
We
have
a
client,
we
have
a
server
and
then
we
have
a
mixer
fryer
topper.
All
of
these
are
built
with
open
tracing.
Now.
A
The
one
thing
I
have
done,
which
you
might
remember
from
yesterday,
is
that
I've
added
these
open
tracing
baggage
tags
and
what
baggage
does
is
like
it
fall
like
if
the
tag
just
follows
through
the
entire
trace
so
and
I
have
done
these
tags
for
the
type
of
doughnuts
that
we
have
like
sprinkles,
cinnamon,
glazed,
etc.
So
this
will
tell
me
if
there
is
an
issue
with
a
particular
kind
of
doughnut.
A
So
now,
let's
see
what
the
traces
are
going
to
show
us
about,
like
you
might
have
seen
that
the
response
was
kind
of
slow
and
somebody
said
it
crashed,
which
is
exactly
what
I
expected
so
I'm
gonna
look
in
Zipkin,
which,
as
I
discussed,
was
the
you
know
open
source
solution
of
choice.
What
is
the
date
today?
Can
anybody
tell
me
sorry.
B
A
Search,
okay,
so
we're
seeing
traces
here.
So
this
is
what
traces
look
like
for
those
who
have
never
seen
them
before
surprise,
so
you
will
notice
here
if
people
can
see
that
the
fryer
is
causing
is
taking
up
most
of
the
time
right.
It's
saying:
65
percent
60
to
69.
So
this
tells
me
that
the
issue
the
slowness
is
in
the
fryer,
so
I
click
this
and
here
I
can
see
more
detail.
This
is
Zipkin
guys
by
the
way
and
I
see
what's
going
on
and
this
Naoto
like.
A
So
the
point
is
that
in
one
click,
I
need.
This
is
the
longest
time
that's
happening
in
this
transaction,
just
to
see
that
portability
of
the
system
I'm
going
to
go
into
another
vendor
light
step,
I'm,
gonna,
okay,
this
is
what
you
guys
did
for
the
success
disaster,
so
great
job
people,
it's
like
going
all
the
way
up.
So
I
really
appreciate
it.
Let
me
find
a
trace
that
I
can
actually
click
on
refresh.
A
A
So,
as
you
can
tell
like
what
I'm
trying
to
convey
over
here,
is
that
I
clicked
once
I
got
to
this
view
and
instantly
I
knew
the
friar
was
the
problem
and
then
had
more
detail
should
I
need
it.
That's
a
lot
different
than
knowing
there's
an
SLA
violation
and
then
first
like
seeing
okay.
Where
is
that
still
a
violation?
Okay,
then?
What
system
do
I
go
check
out?
Okay?
Well,
there
is
the
problem
zactly.
This
is,
like
click,
click
click,
so
problem
solved.
Well,
not
problem
solve
problem
identified.
A
A
Since
what
we
saw
here
with
open
tracing
was
we
saw
vendor
neutrality
right.
You
saw
traces
at
exactly
the
same
time
for
exactly
the
same
thing
and
both
like
step
and
Zipkin.
The
the
view
is
different,
but
that's
a
you
I
think
not
like
vendor
neutrality
thing
then
we
you
also
saw
like
excuse
me
the
how
the
micro
services
are
interacting.
What
comes
first,
what
comes
next?
What's
the
what's
the
story
and
then
the
baggage
tidbit
that
I
added
was
able
to
tell
us
why
there
was
trouble.
A
A
If
you
want
to
get
hands,
this
is
a
plug
shameless
by
the
way.
If
you
want
to
get
more
serious
about
tracing
and
do
a
tutorial,
please
come
tomorrow
to
the
distribute
tracing
workshop.
It's
a
oh
eight!
So
across
the
halt
at
two
o'clock
there
bring
your
laptop
we'll,
be
helping
each
and
every
person
go
through
a
go
tutorial
or
a
project
of
your
choice
in
a
different
language
and
for
those
who
are
already
you
know
further
along
there'll,
be
like
kind
of
an
unconference
going
on
where
we
can
talk
about
specific
issues.
A
You're
facing
or
just
you
know,
thought
exercises.
So
hopefully
some
of
you
will
come
there
and
we
owe
and
real
donuts
big
selling
point
I've
been
talking
Donuts.
So
much
people
are
coming
up
to
me
in
the
conference
being
like
hey.
Where
am
I
donuts,
but
tomorrow
I'll
be
able
to
actually
offer
real
donuts,
so
it'll
be
good
for
me
anyway.
So
now
we
move
on
to
sort
of
the
futuristic
experimental
part
of
this
talk,
so
everything
they've
talked
about
has
been
about
how
you
know.
Auto
instrumentation
doesn't
work
you.
A
You
need
to
instrument
your
system
to
what
you
need
it
to
be
and
then
kind
of
take
it
from
there
and
that
that
is
the
best
approach
in
today's
environment.
But
there
is
a
thought
exercise
about
potentially
using
kubernetes
to
take
away
a
lot
of
the
work
and
do
other
instrumentation
across
kubernetes.
So
this
is
a
really
bad
diagram.
I
made
literally
today
super
so
sorry.
A
So
that
would
be
ideal
now
in
order
to
get
that
we
need
spans,
which
is
tracing
within
the
pod
and
that's
fairly
straightforward,
because
even
if
your
application
is
not
instrumented,
if
you
use
a
proxy
like
a
Nelson
proxy
such
as
Envoy
link
or
D,
these
are
sort
of
the
popular
ones.
Linker
D
isn't
CNC
a
project.
Then
you
can
get
one
span
per
request
and
it's
like
something
entered
and
like
went
through
the
whole
pod
and
like
because
there
was
a
proxy
right
there.
A
It
wouldn't
be
able
to
spit
out
one
span
for
the
entire
request,
and
this
is
without
instrumentation
right,
so
that
works
straightforward,
but
then,
once
that
spans
come
out,
how
do
you
connect
it
to
the
next
destination
and
the
next
so
that
you
have
the
full
visibility
we've
seen
actually
in
the
open
tracing
or
we
have
seen
large
companies
make
this
happen
by
doing
their
own
instrumentation
and
figuring
things
out
so
like
on?
Why
does
something?
A
That's
actually
an
example
of
this
feel
free
to
check
it
out
at
your
time,
but
this
can
be
standardized
and
that's
the
future
we're
talking
about
where
you
can
use
said,
l7
proxy
and
which
is
built
with
HTTP
client
library
to
actually
do
the
context
propagation
with
the
help
of
Oban
tracing.
So
let
me
kind
of
try
to
help
visualize
this
room,
there's
a
bunch
of
words.
A
So
basically,
the
way
it
will
work
here
is
that
this
diagram
hopefully
helps
explain
where
you
have
an
HTTP
request
to
come
in
and
then
the
open
tracing
context,
propagation
piece
starts
with
like
extract
context.
Then
the
application
does
whatever
and
sends
out
an
HTTP
request
through
the
proxy.
Now
that
created
one
span
right.
A
We
remember
that
part,
but
because
we
did
extract
ready,
the
proxy
injects
the
context
and
then
it
forwards
it
to
the
next
spot,
and
so
you
can
use
it
in
daemon
sets
or
pods
whatever,
and
then
you
kind
of
rinse
and
repeat
the
process,
and
suddenly
you
have
everything
visible
across
your
entire
kubernetes
cluster
so
effectively.
All
you
will
be
doing.
This
is
like
zero
instrumentation.
Basically,
you
would
add
an
else
on
proxy
which
has
a
HTTP,
client
library,
I'm
paired
with
it.
A
You
make
and
receive
a
HTTP
calls,
because
the
l7
proxy
has
instrumentation
for
open
tracing.
You
suddenly
get
the
depth
as
well
as
the
breadth,
so
that
would
be
the
entire
picture.
This
is
not
true
right
now
so,
but
you
could
go
home
and
try
this.
This
is
safe,
but
it's
something
that
can
be
standardized.
People
are
doing
this
in
their
individualized
cases.
I
think
the
next
step
is
to
make
it
a
standard
for
everybody
which
would
make
people's
lives
a
lot
easier.
A
It
is
something
that
open
tracing
is
looking
at
very
seriously
in
2017
and
early
eighteen,
but
if
people
are
excited
about,
this
would
definitely
love
to
talk
to
you,
either
at
the
workshop
or
hereafter,
because
this
will
move
the
project
and
our
mission
of
being
providing
observability.
That
can
then
lead
to
controllability
much
forward,
much
farther
and
another
shameless
plug.
If
you
want
to
get
more
details,
please
come
to
our
workshop
there'll
be
mentors
there,
such
as
your
I'd
say
your
name
wrong
every
time.
A
Second
geraci,
as
well
as
Ted
and
a
few
other
folks
who
are
veterans
in
the
tracing
world
as
well
as
Tom
Loki,
who
was
here
before
us,
I,
think
so
yeah.
You
should
definitely
check
us
out
if
you
want
to
get
serious
about
tracing,
but
if
not
I
hope.
This
information
was
useful
and
feel
free
to
check
us
out
on
github.
We
have
a
website
and
like
git
er,
where
people
go
and
talk
about
their
problems,
etc
and
you'll
find
folks
from
all
walks
of
the
three
types
of
developers.
A
I
talked
about,
such
as
providers
of
tracing
solutions,
open
source
projects,
as
well
as
application
guys
so
you'll
find
your
ilk
quickly
anyway.
That
was
that,
thank
you
so
much
for
your
time.
Let's
definitely
stay
in
touch
and
I
would
like
to
invite
Ted
now
on
stage
to
help
me
with
answer
answering
your
questions,
because
he
knows
a
lot
more
than
me
so.
B
A
B
A
So,
as
I
mentioned
in
the
talk
openly,
Singh
is
a
no
op
by
itself.
It's
a
way
to
port
data
into
whatever
situation.
Your
system,
you
want
to
take
it
in
that,
could
be
your
homegrown
solution
or
it
could
be
Zipkin,
in
which
case
you
would
need
to
run
a
server
by
yourself,
because
that's
how
Zipkin
works
and
then.
B
C
B
A
Yeah,
a
simple
test
would
be:
if
you
are
open
to
using
golang,
then
there
is
a
open
source,
tracer
called
app
dash,
which
is
really
simple,
and
there
is
a
like
docker
images
available
for
it.
If
you
want
like
a
quick
test,
I
would
recommend
that
and
tomorrow
in
the
tracing
workshop,
I
think
I
should
stand
farther
from
that
mic
in
the
tracing
workshop.
I
think
you'll
be
helping.
People
connect
to
a
tracer
as
well
right,
yeah.
C
B
A
C
A
C
To
take
that
yeah,
so
on
his
face,
you
can
do
these
things
totally
separately
right,
the
degree
to
which
you're
just
logging
with
log4j
one
potential
thing.
If
you
are
trying
to
not
touch
your
system,
there's
active
work
looking
into
doing
out
of
fan
tracing,
which
means
you
know
ingesting
logs
and
turning
them
into
spans
and
traces.
The
degree
to
which
that
information
is
in
there.
C
The
other
direction
is,
as
you
start,
instrumenting
application
with
open
tracing
you'll
notice
that
you're
doing
all
the
same
logging
against
your
spans
that
you
would
be
doing
in
log4j,
and
so
you
start
doubling
up
your
log
points.
So
what
I
suggest
to
people
at
that
stage
is
to
make
a
open
tracing
log4j
bridge.
So
you
rip
out
your
log4j
logs
in
your
application
and
do
your
login
with
open
tracing
and
then
have
a
log4j
tracer.
That's
just
teeing
off
and
outputting
it
that
way.
If
that
makes
sense,
yeah.
C
D
D
C
C
Right,
yes,
so
that's
very
straightforward
to
do
so
open
tracing.
It
is
just
an
API
right.
So
there's
an
API
you
using
to
instrument
your
code
and
then
separately
you're
installing
a
tracer
like
when
you
start
your
application.
So
it's
pretty
straightforward
to
create
a
tracer
that
is
taking
that
information
in
and
then
creating
metrics
out
and
and
pumping
them
out
into
the
system
of
your
choice.
D
A
A
A
C
Yeah
and
I
would
say
one
thing
you
discover
when
you
start
doing
it,
where
it's
all
of
your
logging
and
other
stuff
is
coming
out
of
these
open
tracing
API
endpoints
is
now
everything
is
sort
of
collated
in
lockstep,
the
degree
to
which
you
were
sort
of
measuring
something
with
metrics
and
sort
of
logging,
the
same
thing,
but
maybe
not
exactly
in
the
same
spot
right
now.
That
is
all
perfectly
lined
up
right
with
the
same
names
for
things,
and
so
it
becomes
a
lot
easier
to
be.
D
D
D
C
A
Because
what
it
has
done
for
you
is
basically
pick
out
the
needle
in
the
haystack
we're
okay,
there
were
all
these
potential
issues.
This
is
the
one
then
the
developer,
DevOps
person,
whoever
goes
in
and
fixes
the
problem,
so
it
helped
you
to
discovery
and
then
the
fix
can
is
going
to
come
from
you,
but
what
will
impact?
It,
though,
is
your
understanding
of
how
are
all
these
services
interacting.
The
donut
example
was
a
little
too
simplistic,
but,
for
example,
like
let's
say
one
one
service
is
gumming
up
the
resources
right.
A
That
is
it
because
of
that
service,
or
is
it
because
five
services
before
that
there
is
some
block
there
right
that
knowing
the
interaction
of
all
the
services
is
going
to
help
you
to
the
solution
faster?
But
ultimately
it's
gonna
come
from.
You
add
the
more
at
least
it
to
at
least
all
the
systems
we
know
of
or
like
that.
Maybe
there's
some
special
one
that
I
haven't
heard
of
yet.