►
From YouTube: A Survey of the OSS Tracing Ecosystem - Ben Sigelman & Ted Young, LightStep (Beginner Skill Level)
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
A Survey of the OSS Tracing Ecosystem - Ben Sigelman & Ted Young, LightStep (Beginner Skill Level)
There has been great movement over the past two years in the tracing landscape. In this relatively short period of time, projects such as OpenTracing, Jaeger, and OpenCensus have joined Zipkin in the open source world. Additionally, there are multiple adjacent projects from observability and infrastructure that help proliferate the need and set up for tracing. Examples are Prometheus, Envoy, and a new w3c trace context standard. This talk will explain the complex landscape. We will cover: * The need for distributed tracing * The changes in tracing open source over the past two years * The core open-source tracing projects, their respective value propositions, and how they fit together and relate to one another * How other projects in infrastructure help reduce time to value for tracing projects * Best practices for today's software practitioners
About Ben
Ben Sigelman is the CEO and Cofounder of LightStep. He is an expert in diagnostics and performance analysis for complex software systems. He spent nine years at Google where he led the design and development of several global-scale monitoring systems. The most significant of these were Dapper, an always-on distributed tracing system, and Monarch, a high-availability timeseries collection, storage, and query system. Both systems continuously observe hundreds of millions of distinct services at Google. Ben is also the co-creator of the OpenTracing API standard and a frequent speaker about tracing and observability in production software at scale. He graduated with honors from Brown University with a Bachelor of Science in Mathematics and Computer Science.
About Ted
Ted Young is a software engineer at LightStep. He has spent the last 15 years building distributed systems in a variety of environments: computer animation pipelines for VFX, national elections, and elastic compute platforms. Previously, he helped design the Diego Container Runtime for Cloud Foundry. Currently, he is focused on OpenTracing and tools for root cause analysis.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Separate
from
this,
but
really
about
the
ecosystem
in
general,
with
tracing
tracing
is
always
a
complicated
process,
because
you
need
to
integrate
across
your
entire
stack
in
order
to
see
it
all
and
it's
even
more
complicated
in
the
open-source
world,
because
the
positioning
of
the
various
projects
can
be
unclear
and
there's
a
lot
of
overlap
and
terminology.
So
in
this
talk,
I'm
hoping
to
demystify
a
lot
of
that
for
everyone
here.
A
It
goes
without
saying,
but
I
really
encourage
people
to
interrupt
me
and
ask
questions
I
love
that
it's
fun
to
be
interrupted
in
this
context
anyway,
and
I'll
try
and
pause
for
for
that
here
and
there.
But
if
not,
we
can
wait
till
the
end.
So
these
are
the
sorts
of
things.
I
was
hoping
we
could
address.
I
want
to
define
what
distributed
tracing
is
for
people,
hopefully
without
taking
too
much
time
as
I.
A
Imagine
if
you're
here,
you
probably
already
have
a
sense
I'd
like
to
compare
it
to
the
rest
of
the
universe
of
monitoring
and
observability
technology,
talked
about
the
moving
parts
and
then
get
down
to
business
and
figure
out
why
there
are
so
many
different
projects
that
talk
about
tracing
and
where
the
boundaries
are
and
how
they
compare
and
contrast,
and
why
it's
not
an
either/or,
and
it's
probably
a
met
for
anyone
in
this
room.
Alright,
so
first
things:
first,
how
does
this
shivudu
tracing
fit
into
the
monitoring
system?
So
I
made
this
slide
up?
A
This
is
not
I.
Think
the
conventional
talk
track
is
that
you
need
logging,
metrics
and
tracing
I.
Personally,
don't
subscribe
to
that
I!
Think
of
it
as
being
an
odd
way
to
segment
things.
I
think
that
there
are
fundamentally
two
activities
and
fundamentally
two
types
of
data
for
everything
related
to
observing
a
system.
There's
the
activities
are
the
rows.
You
either
are
measuring
a
symptom
of
interest
to
you
or
you're
explaining
why
it
doesn't
look
the
way
it
should.
A
Those
are
the
only
two
things
you're
doing
if
you're
trying
to
observe
your
system
so
root,
cause
analysis
and
debugging
as
clearly
in
explaining
measuring
symptoms
will
be
things
like
I
want
my
p99
latency
to
be
less
than
950
milliseconds
or
what
have
you?
But
there
are
symptoms
that
are
important
to
you,
usually
for
business
reasons,
and
then,
of
course,
the
explanation
can
involve
many
different
tools
and
technologies.
Now,
in
a
column,
is
there
really
only
two
types
of
data?
A
A
A
Alright,
so
this
is
a
diagram
that
I
like
to
use
in
these
talks
on
the
top
is
a
is
an
illustration
of
what
a
transaction
might
look
like
in
a
monolith
the
oh.
This
has
a
little
laser
pointer.
Look
at
that
Wow
amazing.
This
isn't
even
digital,
so
this
this
transaction
goes
across
package
boundaries
in
your
monolith
and
then,
roughly
speaking,
when
you
move
to
micro-services,
your
packages
become
microservices,
but
your
transaction
still
goes
through
them.
So,
in
order
to
understand
the
basic
story
of
an
individual
transaction,
you
need
some
new
technology.
A
You
can't
just
look
at
your
logs
or
whatever
you
did
in
the
monolith
and
so
distributed
tracing
is
about
taking
requests
and
telling
a
clear
story
about
them
as
they
go
through
a
system,
and
it's
really
nothing
more
than
gluing
events
together.
That's
all
it
is
it's
very
simple
at
a
conceptual
level,
but
there
are
a
lot
of
complexities
to
making
that
work.
You
need
to
glue
them
not
just
in
time,
but
also
structurally.
A
You
need
to
know
what
caused
what
because
micro
services
involve
huge
amounts
of
concurrence,
the
in
parallelism,
which
I'll
get
to
you
in
the
next
slide.
Here,
oh
yeah,
this
is
a
on.
The
left
is
a
kind
of
system
diagram
where
a
client
calls
a
server,
does
authentication
then
billing
and
it
hits
a
database.
This
is
one
way
of
thinking
about
a
transaction
structurally,
and
this
is
the
kind
of
thing
that
we
typically
draw
in
our
whiteboards
right
and
then
over.
A
Here
is
a
timing
diagram
showing
the
same
transaction
with
a
shared
time
axis
most
distributed
tracing
systems
have
a
visualization
that
looks
more
like
this.
Without
naming
names,
this
is
an
eval
you
done
or
anything
like
that,
but
APM
systems
that
are
pre
distributed
systems,
things
that
deal
with
monoliths
typically
represent
transactions
like
this,
because
if
you
don't
have
a
lot
of
concurrency
you
can.
You
can
say
a
lot
here.
However,
if
you're
dealing
with
tons
of
concurrency,
you
really
need
to
have
a
shared
time
access
in
order
to
understand
the
transactions.
A
So
I
think
this
transition
from
a
conceptual
standpoint,
to
seeing
things
as
some
shared
waterfall
is
indicative
of
the
complexity,
light
step.
We
see
a
lot
of
production
traces
that
have
you
know
many
thousands
of
these
timed
events
in
them
just
for
single
user
facing
transactions
that
get
back
to
the
end-user
within
a
couple
of
hundred
milliseconds.
So
you
do
need
to
have
a
mental
model
that
can
scale
with
that
sort
of
complexity.
A
This
is
the
basic
mental
and
data
model
that
we
use
in
open,
tracing
I'm,
not
suggesting.
This
is
the
only
way
to
look
at
it.
There's
a
great
academic
paper
called
X
Trace
that
came
out
in
the
early
2000s
and
another
great
academic
paper
called
pivot
tracing
that
was
published
in
2015
and
won
best
paper
at
s,
OSP
that
has
a
totally
different
set
of
names.
The
concepts
are
very
similar.
A
It's
all
event,
data
in
some
fashion,
in
the
open
tracing
model
which
borrows
from
the
dapper
terminology
which
I
worked
on
at
Google
in
the
early
2000s,
it
refers
to
I'll
go
back
to
this
slide.
It
refers
to
each
each.
One
of
these
things
is
a
span
span
is
something
that
has
a
start
time
and
an
end
time
and
a
parent,
and
you
can
use
the
parent
references
to
construct
a
tree
of
causality.
So
what
you
have
here
are
a
series
of
spans
one
causing
the
other
and
each
span
can
have
tags
on
it.
A
That
are
key
values
as
well
as
logs,
which
potentially
is
a
misnomer,
I
think
in
the
Zipkin
world,
they're
called
annotations,
but
the
idea
is
that
it's
just
a
time
stamp
with
a
bunch
of
structured
data.
That's
attached
to
a
span,
so
an
example
of
a
login,
a
trace
might
be
something
like
I'm
gonna
have
a
span
for
this
RPC
call
and
I'm
going
to
log
when
the
bytes
actually
hit
the
wire
and
I'm
going
to
log
when
they're
received
on
the
other
side,
and
things
like
that.
A
So
little
moments
in
the
life
time
of
a
span
are
called
logs
in
the
open
tracing
world,
and
then
there
are
a
number
of
identifiers
that
are
attached
to
these
things
as
well
to
figure
out,
which
is
the
parent,
which
is
the
child.
What
the
unique
ID
is
for
the
transaction
in
general,
these
sorts
of
things
I,
the
slides,
are
going
to
be
attached
to
the
sched.
Comm
is
site
for
this
talk,
so
if
this
doesn't
make
sense-
or
if
you
don't
feel
like
you
can
read
all
this-
don't
sweat
it.
A
The
the
only
important
thing
to
understand
for
the
purpose
of
this
talk
is
that
traces
are
structured,
they're
made
out
of
these
little
spins
that
have
a
start
and
an
end
time
and
there's
a
lot
of
structured
data
attached
to
them.
The
rest
of
it
is
level
of
detail.
You
don't
need
for
this
time.
Good,
ok,
see
people
don't
seem
confused.
This
is
great
all
right.
So
now
we
get
to
the
meat
of
the
talk.
So
let's
talk
about
tracing
tracing
tracing
and
tracing.
There
are
four
words
that
are
used
to
describe
different
things.
A
Unfortunately,
it's
the
same
word,
which
is
causing
a
lot
of
confusion,
as
you
might
imagine
so,
okay,
so
what
is
tracing
tracing
is
about
analyzing
transactions.
These
are
six
screenshots
from
six
tracing
solutions.
Some
are
open
source,
some
are
not
I
was
a
good
boy
and
I
didn't
include
my
company's
product
in
this
as
much
as
I
was
tempted
to,
but
it's
about
analyzing
transactions.
If
you
talk
about
doing
tracing
what
you're
saying
is
that
I'm
gonna
go
as
an
operator
and
look
at
a
tool
like
one
of
these
and
deduce
something
about
my
system.
A
So
that's
what
tracing
is
it's
about?
Analyzing
transactions
and
and
when
I'm
saying
this
I'm,
usually
in
the
mode
of
an
operator
who's
trying
to
get
to
the
bottom
of
some
kind
of
latency
or
production
issue?
No,
actually,
that's
wrong.
So
tracing
is
about
recording
transactions.
So
what
we're
doing
here
is
we
have
some
kind
of
service
as
a
micro
service
or
maybe
as
a
side
car
like
envoy
or
linker
to
you
or
something
like
that.
There's
some
kind
of
runtime,
that's
sitting
in
there.
A
That's
taking
data,
recording
it
tracing
it
and
sending
it
out
of
the
process
to
a
back-end
system.
So
this
is
what
tracing
this
you're
and
you're
thinking.
When
the
perspective
of
someone
who
needs
to
get
this
data
into
a
central
place,
either
it's
your
own
company's
ELQ
stock.
You
need
to
get
it
into
your
in
into
some
kind
of
database
or
maybe
you're
a
tracing
vendor
or
someone
developing
an
open-source
tracing
system
and
so
tracing
is
the
act
of
recording
transactions.
So
that's
a
valid
interpretation
as
well.
A
Unfortunately,
it's
complete
be
different
than
the
first
one,
but
it's
valid
another
way
to
think
of
it
is
that
tracing
is
about
federating
transactions.
So
now
we
have
everyone,
you
know
moving
to
the
cloud
you'll
use
whatever
Amazon,
s3
or
Google
Cloud
spanner,
or
something
like
that.
These
are
not
open-source
projects,
there's
absolutely
no
chance
that
Amazon
is
going
to
open
source,
s3
and
I
would
say
the
same
thing
about
spanner,
so
you
trace
into
these
things
in
a
sense
that
you
call
them
you
depend
on
them.
A
Maybe
Kinesis
is
an
even
better
example,
because
it's
not
just
a
leaf
in
your
stack.
It's
actually
in
the
middle
of
your
stack
and
data
is
going
in
and
out
of
it
over
and
over
again,
you
need
to
see
into
it.
You
need
to
trace
it,
but
unfortunately
they're
not
going
to
open-source
that
code.
So
what
are
you
going
to
do
so?
There's
this
movement
in
the
tracing
world
right
now
to
federate
transactions
and
some
people
when
they
think
about
tracing
what
they're
talking
about
doing
is
getting
from.
A
You
know
the
code
that
you
run
and
deploy
yourself
into
cloud
services
and
then
into
something
like
X
ray
and
then
have
all
this
data
flow
into
the
same
place.
So
when
you're,
using
your
open
source
tracing
tool
or
your
APM
solution,
you
can
see
the
transactions
that
go
through
your
business
logic
in
and
out
of
cloud
services
and
then
back
to
the
same
place
again.
So
Federation
is
an
activity
that
matters
deeply
to
people
who
are
concerned
with
observability
in
their
systems
and
also
matters
to
cloud
providers.
A
So
this
is
a
major
area
of
investment
right
now
from
a
standards
standpoint
and
then,
finally,
last
but
not
least,
tracing
as
the
act
of
describing
a
transaction,
and
so
what
I
mean
there
is
that
you,
as
someone
who
owns
an
application,
you
own
huge
amounts
of
code
and
much
of
it
is
code.
You
didn't
write
its
Cody
brought
in
from
github
that
you
depend
on
or
that
came
in
from
NP
or
go
DAPs
or
something
like
that
and
you
deploy
this
code
and
you
need
to
understand
how
it
works.
A
So
some
might
say
that
tracing
this
code
is
a
matter
of
instrumenting.
It
I
would
argue
that
from
the
standpoint
of
a
developer
of
someone
who's
writing
code.
This
is
the
part
of
tracing
that
is
presently
the
most
painful,
but
it's
not
the
same
as
the
other
three
tracings.
It's
not
the
same
thing
as
as
federating
the
transactions,
it's
not
the
same
thing
as
recording
them,
and
it's
not
the
same
thing
as
observing
them
or
understanding
them
in
an
in
a
tool.
So
these
are
the
four
tracings
I
hope
it
made
sense.
A
I
haven't
shown
this
deck
before
so
I
have
no
idea
if
people
think
it
makes
sense
or
not
I
don't
see.
People
sleeping
see
one
thumbs
up
suggests
some
sort
of
compliance
with
ideas
I'm
putting
on
the
screen.
This
is
great
yeah,
so
these
are
the
four
things
now.
This
is
a
matter
where
I'm
going
to
provide
a
personal
opinion,
I'm
allowed
to
do
that
as
long
as
I
say
so.
I
think
that
these
all
have
their
place
I
feel
very
strongly.
They
should
be
decoupled.
A
The
tracing
world
should
be
clear
about
the
position
in
these
projects
and
make
sure
the
projects
solve
one
of
these
things,
ideally
or
if
they
don't,
if
they
solve
three
at
least
make
it
really
clear
that
you
can
depend
on
the
three
separately
when
open
source
projects
have
a
scope
creep.
It
always
leads
to
bad
things.
I
think
it
creates
a
lot
of
competing
interests
and
you
can't
get
one
without
the
other
and
so
on
and
so
forth.
A
So
a
big
part
of
my
talk
here
is
to
hopefully
indoctrinate
that
idea
in
some
of
you
out
there
in
the
audience.
So
let's
look
at
some
tracing
projects
how
they
fit
together.
How
am
I
doing
for
time
not
too
bad?
So
all
right,
oh
whoops,
I,
I,
messed
up
my
animation,
that's
all
right!
Well,
here's
an
example
of
a
transaction
it
comes
in
to
service
and
then
goes
into
a
micro
service
and
then
goes
into
a
cloud
provider.
A
There's
a
small
amount
of
context,
data
that's
sent
in
over
the
wire,
and
then
there
are
tracing
systems
that
will
take
data
and
get
it
into
the
tracing
tool.
This
this
data
right
here
is
much
larger
than
this
data.
Here,
that's
sent
in
over
the
HTTP
request.
This
is
usually
in
headers.
This
is
something
that's
buffered
and
ends
up
being
you
know,
megabytes
of
data
and
not
you
know
single
bytes
of
data
like
here.
The
cloud
service
generates
data
as
well.
A
That
might
go
into
something
like
x-ray
and
then
you
want
it
all
to
be
in
the
same
place,
people
who
write
tracing
systems-
that's
like
you
know
me
or
the
people
working
on
Jaeger
or
Zipkin
or
whatever
we
care
deeply
about
these
things.
I'll
get
to
that
later.
The
people
actually
use
the
tools,
the
operators,
the
only
thing
they
really
care
about
is
that
the
tool
works.
So
they
care
deeply
about
this.
This
is
where
the
person
analyzed
in
the
transactions.
A
It's
there,
the
people
who
care
about
recording
the
data
this
incorporates
the
vendors
who
create
tracing
tools,
as
well
as
people
who
want
to
kind
of
get
in
and
hack
their
system
to
send
data
to
a
login
platform
as
well,
and
then
there's
a
fourth
class
of
users
who
instrument
the
systems.
These
are
usually
developers
either
who
work
at
some
company
and
need
to
get
visibility
into
their
services
or
who,
in
their
spare
time
or
if,
as
part
of
their
job
work
on
open
source
software.
A
There
are
some
nice
examples
of
tracing
getting
built
directly
into
things
like
Couchbase
and
cockroach,
and
things
like
that
right
now
that
work
is
done
by
instrumentals,
so
I've
talked
before
about
there
being
four
basic
use
cases
or
four
basic
problems
that
tracing
concerns
itself
with
those
are
reflected
by
these
four
different
colors
people,
instrumenting
stuff
people,
recording
stuff
people
who
write
tracing
systems
and
then
finally,
people
who
actually
use
them
and
derive
operational
efficiencies
through
them.
So
now
we
can
talk
about
how
different
projects
map
on
this
here.
A
Zipkin
I
started
with
this
one
because
it's
it's
like
the
OG,
the
Original
Gangster
tracing
system
and
has
a
little
bit
of
everything
because
it's
it
it
was
when
it
started.
There
was
nothing.
Zipkin
is
an
implementation
of
dapper
that
was
started
at
Twitter
and
has
now
incubated
out
of
pivotal
and
it's
run
by
Adrian
Cole,
and
it
has
really
a
great
community
built
around
it
and
a
lot
of
momentum.
As
a
result,
they
have
a
little
bit
of
everything
for
everyone
this
when
something
is
grey.
A
A
They
have
this
thing
called
b3,
which
is
their
own
context,
propagation
format,
although
they're
part
of
a
later
slide
to
standardize
that
more
they
have
de
facto
standard
way
of
describing
data
that
is
accepted
by
other
cloud
providers
like
Google
stackdriver
trace,
accepts
Zipkin
formatted
data
Jaeger
accepts
Zipkin,
formatted
data,
so
they
have
a
de
facto
standard
here
and
then,
of
course,
they
have
the
the
actual
tool
itself
that
people
run
and
use
to
understand
their
systems.
So
they
cover
a
lot
of
these
bases.
A
Jagr,
which
is
a
younger
project,
but
a
similar
type
of
value
proposition
to
Zipkin
is
actually
a
little
bit
tighter
and
that
it
it
uses
open
tracing
natively
to
handle
this
so
its
scope
is
concerned.
It
is
the
context
it
has
its
own
recording
system,
it
has
its
own
data
format
and
it
has
a
sound
tool.
A
I
think
it's
an
interesting
comparison
to
look
at
Yeager
and
Zipkin.
They
they
have
again.
They
have
similar
feature,
sets
similar
value,
propositions
and
I'm,
not
here
to
say
one
is
better
than
the
other
by
any.
You
should
look
at
them
both.
It
is
interesting
that
Jaeger
doesn't
need
to
concern
itself
with
instrumentation,
because
it's
deliberately
decoupled
that
piece
of
the
puzzle
out
open
tracing
also
has
some
compatibility
of
the
Zipkin.
A
A
A
But
if
your
open
source
software
depends
on
s3
or
a
Kinesis,
you
can't
see
it
unless
you
can
get
into
the
cloud
providers,
and
so
there
is
a
need
for
Amazon
to
cooperate
in
that
regard
and
they
do
actually
Abhishek
who
runs
that
project
is
very
collaborative
in
this
type
of
space,
but
they
have
a
role
to
play
and
there
are
standards
that
need
to
be
created
to
get
the
data
from
x-ray,
which
is
this
box,
basically
into
whatever
tracing
or
APM
tool.
You
want
to
use
I.
A
A
X-Ray
also
has
its
own
separate
way
of
instrumenting
software.
To
be
honest,
I
wish
they
didn't
I,
think
it
it.
This
is
a
sort
of
fragmentation
that
I,
don't
think,
is
particularly
helpful,
but
they
do,
and
they
also
have
Amazon,
has
their
own
context
format
up
here
that
that
they've
announced
and
have
guarantees
around
the
stability.
So
you
see
a
project
like
x-ray
and
it
actually
encompasses
three,
maybe
four,
depending
on
how
you
count
different
areas
of
standardization
for
tracing
and
I.
A
Think
having
a
mental
model
like
this
help
you,
when
you're
analyzing
these
things
for
your
own,
your
own
solutions
within
your
organization's
help
you
analyze,
you
know
which
parts
of
this
you
should
really
be
taking
for
which
moving
into
w3c,
so
w3c
I'm,
just
checking
my
time
here
still
doing
pretty
well.
W3C,
of
course,
is
the
World
Wide
Web
Consortium.
They
generally
do
standards
around
things
like
web
browsers.
They
also
have
a
really
interesting
project
right
now.
A
This
guy
Louise,
who
works
as
a
CTO
at
dynaTrace,
has
kicked
off
an
effort
involving
many
different
constituencies.
I
have
a
dedicated
slide
about
this
later,
as
well
just
to
focus
on
this
and
nothing
else.
I
love
this.
This
makes
me
very
warm
in
my
heart.
It's
like
a
narrow
scope,
a
clear
value
proposition.
If
everyone
can
agree
about
how
to
do
this,
then
fine
we
can
like
interchange
data.
We
can
have
traces
go
from
vendor
to
vendor
and
everyone
wins
and
keeping
that
thing
scoped
to
just
this
little
box.
A
It's
like
a
massive
win
for
that
project
and
I
would
argue
for
everyone
who
depends
on
it.
As
you
know,
you're
not
going
to
pull
in
a
bunch
of
other
garbage.
Basically
and
there's
been
plenty
of
pressure
to
broaden
the
scope
of
that
discussion
and
they've
been
very
disciplined
about
keeping
it
tight
and
focused
and
I
think
you
can
see
the
momentum
coming
out
of
that.
A
As
a
result,
this
benefits
mainly
people
who
authored
tracing
systems
frankly,
but
it
will
benefit
everyone
in
this
room
and
that
once
people
can
actually
agree
about
this
stuff,
progress
will
move
much
more
rapidly
towards
actual
interoperability
between
these
systems,
and
so,
if
this
does
have
general
benefits
as
well.
The
last
thing
I'll
say
about
this
is:
is
that
it's
pretty
funny
I
mean
if
you
look
at
the
initial
proposals
for
this.
Let's
see
the
first
time
I
saw
a
national
proposal
was
over
a
year
ago,
as
March
of
2017.
A
It's
the
kind
of
thing
where,
if
I
went
and
took
anyone
here
for
half
an
hour
and
described
the
data
model
and
said
you
know,
come
up
with
a
format,
let
just
propose
something.
If
you
have
a
technical
background,
you'd
probably
propose
something,
that's
very
similar
to
what
this
is
and
it's
been.
You
know,
bike
shedded
and
argued
into
oblivion.
I
mean
they've
spent,
like
25
P
spent
over
a
year,
getting
the
thing
totally
perfect
and
it
turns
out
like
actually
getting
it
perfect.
Getting
that
last
10%
is
really
really
difficult.
A
You
need
to
maintain
for
its
compatibility.
You
need
to
make
sure
that
all
the
vendors
who
depend
on
this
will
actually
have
their
needs
met.
You
need
to
have
compatibility
tests
and
reference
implementations
and
all
this
w3c
stuff
and
they're
doing
all
that
work.
It's
actually
it
would
be
pretty
thankless,
except
that
I'm
sitting
here
saying.
Thank
you
to
them
for
doing
this,
but
it's
awesome
work
that
they're
doing
this
stuff
and
everyn
benefits
from
it.
So
the
technical
stuff
looks
pretty
basic
when
you
take
a
look
at
it,
but
it's
actually
quite
impactful.
A
They
also
have
a
project.
That's
starting!
That's
talking
about
standardizing
this
piece.
This
is
the
missing
link
to
make
interoperability
of
systems
like
x-ray,
and
you
know
your
favorite
tracing
tool
really
work
in
a
standards-compliant
way,
also
very
important
and
overlaps
a
bit
with
some
of
the
instrumentation
pieces.
This
is
where
you
would
say
things
like
if
you're
going
to
make
an
HTTP
call.
These
are
the
five
tags
you
must
have.
You
must
have
path.
A
A
If
you
care
about
it,
let
me
know-
and
I
can
put
you
in
touch
with
them,
but
but
they're
trying
to
get
all
this
stuff
sorted
out,
so
that
when
you
export
data
from
even
things
that
are
like
hardware,
routers
and
stuff
like
that
and
the
future
will
emit
data
in
formats
that
can
be
consumed
by
any
system,
and
that
is
also
quite
valuable
and
is
also
a
very
narrow
scope.
So
I'm
excited
about
it.
Open
census
is
a
project
that
was
announced
by
Google
in
the
last
three
or
four
months.
A
It's
it
basically
handles
everything
that
sits
inside
your
process.
That's
part
of
a
tracing
system,
so
it
handles
instrumentation.
It
also
handles
the
collection
of
data
within
the
process,
getting
it
into
some
kind
of
format,
potentially
that
format
I
just
described
and
then
exporting
it
into
whatever
tool
you
are
using
open
census
has
some
internally
coupling
between
the
instrumentation
and
the
recording
I.
A
It's
it's
based
on
the
the
work
they
did
inside
Google
on
the
same
stuff
and
it's
very
high
throughput
and
then
finally,
open
tracing
also
has
a
very
narrow
scope.
It's
really
designed
with
these
people
in
mind.
The
idea
is,
if
you're
going
to
adopt
tracing
within
the
organization
you're
going
to
hit
a
brick
wall,
which
is
that
you
probably
depend
on
dozens
of
open-source
projects
and
you
need
to
instrument
them,
and
you
don't
want
to
do
that.
Work
yourself.
A
The
value
of
open
tracing
is
that
it
brings
that
instrumentation
in
from
an
open
source
community
that
wrote
it
for
you,
so
you
don't
need
to
do
that
work
and
it
doesn't
do
anything
else.
That's
the
only
thing
it
does.
It
then
allows
you
to
make
all
other
decisions
about
your
tracing
system
independently,
again
I.
Think
of
that
as
a
feature
of
the
project
and
I
like
that
narrowness
and
it
interoperates
with
you
know
virtually
everything
that
we've
seen
so
far.
A
There's
also
some
overlap
with
this
effort
to
standardize
the
fields
that
go
into
things
like
HTTP
calls,
database
calls
etc,
because
that's
part
of
instrumentation,
so
there
needs
to
be
some
coordination
there.
So
that's
a
summary
of
like
the
major
ecosystem
efforts.
There
are
certainly
other
things
out
there
that
I
didn't
cover,
but
I
got
through
most
of
it.
I'll
spend
my
last
couple
of
minutes
just
making
a
quick
pitch
again
for
having
narrow
scope
for
these
sorts
of
things.
A
Everyone
was
just
complaining
about
the
fact
that
we
couldn't
get
instrumentation
deployed
and
that
was
preventing
any
of
the
cool
features
from
delivering
value,
and
so
a
bunch
of
us
banded
together
and
created
open
tracing
in
an
effort
to
basically
band.
You
know,
band
together
and
solve
that
particular
instrumentation
problem
once
and
for
all.
So
no
one
else
has
to
do
it,
and
I
would
argue
that
if
you're
developer,
as
opposed
to
an
operator,
the
main
pain
point
for
tracing
this
instrumentation
there's
a
little
bit
of
stuff
for
an
SDKs
and
back
ends.
A
But
the
instrumentation
is
just
like
a
massive
massive
headache.
Unless
you
have
something
that
factors
that
out
so
open
tracing
is
designed
to
address
that
pain,
that
specific
pain
point
it
I
think
there
are
nine
sort
of
a
standard
languages
and
there's
a
couple
more
that
are
languages.
I,
don't
even
really
know,
there's
some
like
Haskell
stuff
going
on,
and
things
like
that.
There's
over
a
hundred
packages
that
support
open
tracing
now
and
many
more
being
added
kind
of
month-over-month
and
there's
also
a
diversity
of
places.
You
can
send
this
data.
A
It's
tempting.
We
actually
have
literally
reduced
the
scope
of
an
tracing
over
time,
we'd
like
remove
stuff
from
it,
to
make
it
as
small
as
possible
and
I
think,
especially
in
open
source
sort
of
standards.
Work
with
a
lowercase
s
like
things
we're
trying
to
get
many
constituents
to
agree
the
narrowest
scope
is
the
best
scope,
also
again
give
a
plug
to
the
w3c
work
very
similar
conceptually.
A
They
have
over
25
people
participating
this
discussion,
trying
to
sort
out
exactly
how
the
standard
is
going
to
work
that
will
affect
everything
from
hardware
to
you
know
the
coolest
newest
service
mesh
stuff,
that's
going
on,
and
they
have
everyone
in
there
solving
this
one
very
narrow
problem.
It
helps
keep
the
discussion
super
focused
and
encourages
as
this
best-of-breed
approach,
and
so
I
mean
I'm
really
enthusiastic
about
this
kind
of
stuff.
So,
in
terms
of
the
overall
message
for
this
talk,
I
mean
tracing
as
table
stakes.
I,
don't
even
need
to
say
that
anymore.
A
A
Unfortunately,
it
involved
many
constituents
and
I
think
that
the
ecosystem
around
tracing
is
starting
to
get
really
complex.
There
are
many
projects
that
have
overlapping
scope
and
overlapping
focus,
so
I
hope
that
this
talk
provides
a
mental
model
for
people
to
think
about
which
problem
you're
really
trying
to
solve,
with
which
piece
and
that
you
can
choose
the
solution
that
works
the
best
for
each
piece.
A
A
I'm
here
for
three
days
come
and
talk
to
me,
even
if
it's
not
about
this
I'd
love
to
meet
you
and
get
your
feedback
about
your
own
experiences,
getting
tracing
into
your
particular
organisations
and
it
can
be
about
open
tracing,
can
be
about
anything
in
this
deck.
But
I'd
love
to
talk
to
you
and
I
hope.
This
is
useful,
and
with
that,
are
there
any
questions?
A
A
Yes,
so
the
question
was:
what's
the
difference
between
recording,
oh
by
the
way,
do
I
have
to
stop
right
at
in
two
minutes
or
can
I
go
a
few
minutes
over
I
have
to
stop
okay,
great
okay,
so
recording
the
difference
between
recording
and
logging
I
think,
like
logging,
as
a
word
is
so
it's
almost
worse
than
tracing
in
terms
of
being
overloaded.
It's
really
no
different.
A
If
you
mean
logging
in
terms
of
getting
the
data
into
some
sort
of
event,
stream
I
think
that
the
type
of
data
people,
usually
log,
is
much
coarser
grained
like
if
you're
using
elk
or
something
like
that.
The
amount
of
data
you'll
put
into
an
elk
stock
is
probably
a
lot
smaller
than
the
total
amount
of
data.
The
recorder
would
see.
Recording
often
involves
things
like
sampling
strategies
and,
and
things
like
that,
which
reduced
the
volume
of
tracing
data
in
order
to
make
it
more
tractable
from
a
cost
standpoint.
A
So
the
questions,
if
you're,
moving
from
a
monolith
like
a
rails,
app
into
micro-services
world,
that
has
a
bunch
of
go
line
services.
How
do
you
do
that
from
an
adoption
standpoint?
It's
a
really
really
good
question
and
not
not
that's
a
great
talk
like
how
do
you
actually
deploy
tracing
during
a
model?
Lifter
Mike
reverses
transition.
I
don't
have
time
for
the
full
answer,
but
the
short
version
is
that
you
start
where
your
business
cares
about
it.
Don't
try
to
do
it
all
at
once.
A
If
there's
one
thing
you
learn
from
this
talk
an
entire
talk.
It's
this
answer:
don't
you
tracing
all
at
once?
Please,
for
your
own
sake.
Just
do
the
part
that
your
business
really
depends
on
start
at
the
core.
So
in
your
monolith
just
do
the
core
pieces
and
then,
when
you
add
new
microservices,
make
sure
it's
built
in
from
scratch.
That's
pretty
sane,
not
too
bad.
A
A
Yes,
yes,
so
the
question
is:
if
you
depend
on
a
mic,
reverse
you
didn't
right
and
it
doesn't
use
the
same
tracing
scheme
that
you
do.
What
do
you
do
now?
That's
a
good
question.
It
depends
on
what
the
service
is,
if
there's
really
no
way
to
contribute
to
upstream,
it's
really
tough,
frankly,
I,
don't
I
think
this
is
the
vision
of
open
tracing
for
what
it's
worth
is
that
those
sort
of
sins
are
built
in
like
Couchbase
recently
built
open
tracing
into
Couchbase
I,
don't
mean
into
like
the
wire
protocol,
but
into
cowface
itself.
A
So
if
you
use
open
tracing,
you
can
see
right
through
into
the
end
of
the
meet
of
the
database
and
back
up.
That's
the
model
I'd
like
to
see,
but
it
does
require
standardized
methods
of
describing
transactions
and
that's
the
only
path,
unfortunately,
and
if
you're,
using
a
micro
service
that
you
didn't
write
and
it
doesn't
support
tracing
that
is
a
black
box
and
as
a
huge
bummer,
and
we
just
need
more
standards,
all
right,
I'm
done.
Thank
you.
Everyone
please
come
up
and
talk
to
me.
If
you
want.