►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
OpenTracing Project Deep Dive – Priyanka Sharma & Ted Young, LightStep (Intermediate Skill Level)
About Priyanka
Priyanka Sharma is an entrepreneur with a passion for building developer products and growing them through open source communities. She heads Marketing and Partnerships at LightStep and also works on the OpenTracing project, an instrumentation standard for distributed tracing. In her copious spare time she advises startups at HeavyBit industries, an accelerator for developer products. Priyanka cofounded WakaTime, an open source time-tracker for developers and holds a BA in Political Science from Stanford University.
About Ted
Ted Young is a software engineer at LightStep. He has spent the last 15 years building distributed systems in a variety of environments: computer animation pipelines for VFX, national elections, and elastic compute platforms. Previously, he helped design the Diego Container Runtime for Cloud Foundry. Currently, he is focused on OpenTracing and tools for root cause analysis.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Cool
okay,
we're
on
all
right.
Everybody
welcome.
My
name
is
Ted
young
I
work
on
the
open
tracing
project,
and
this
is
supposedly
a
deep
dive
into
open
tracing.
And
what
that
means
to
me
is
that
we're
gonna
get
all
of
your
questions
answered.
So
first
of
all,
I
want
to
figure
out
where
people
are
kind
of
at
in
the
audience.
As
far
as
all
of
this
stuff,
so
we're
going
to
do
sort
of
a
series
of
hand,
races
so
get
your
hand
ready.
A
First
of
all,
a
easy
one
who
here
writes
code:
okay,
good,
the
right
audience
for
this.
Who
here
has
used
open,
tracing,
okay,
good
good
mix,
who
here,
writes
open-source
libraries
written
a
library,
something
all
right:
good
mix
of
people
who
here
has
wanted
to
pump
out
some
kind
of
logging
or
other
diagnostic
information
from
the
open
source
library
that
they
wrote
got
some
hands?
Who
here
found
that
to
be
like
really
frustrating?
A
It's
really
hard
right
like?
How
are
you
supposed
to
do
it?
There's
a
bunch
of
questions
you
need
answered,
and
you
can't
seem
to
pick
the
right
thing.
So,
oh
and
final
question
who
here
saw
bins,
talk
about
open
tracing
earlier
on
Monday,
okay,
good
too
many.
So
we
can
do
a
bit
of
repeat
so
what
I
would
like
to
do
with
this
talk
is
sort
of
I
have
a
bunch
of
slides,
but
we're
gonna
sort
of
very
quickly
run
through
them
just
to
give
people
a
baseline.
A
But
then
we're
gonna
really
turn
this
into
a
question
and
answer
session,
because
to
me
a
deep
dive
is
really
about
an
opportunity
to
ask
whatever
burning
question
you
have
because
whatever
question
you
have
is
probably
similar
to
questions
that
other
people
have
and
I
would
like
to
spend
most
of
this
half
hour
really
kind
of
discussing
those
things.
So
I'm
gonna
run
through
the
slides,
real
quick.
A
So
don't
necessarily
expect
to
keep
up
and
think
about
questions
you
want
to
ask
and
then
we're
going
to
start
passing
the
microphone
around
and
in
addition
to
me,
we've
also
got
Ben
Siegelman
here
who
started
the
open
tracing
project.
We
have
Carlos
up,
though,
who
works
on
the
Python
and
Java
API,
so
we
can
really
get
into
some
detailed
Q&A.
A
A
We
want
to
know
what
they're
doing
and
that's
really
hard
to
observe
from
the
outside,
so
you
need
to
get
on
the
inside
your
your
never
doing
anything
more
than
this
I
would
argue
when
you're
monitoring,
there's
a
sequence
of
events
that
you're
trying
to
put
together
you're
trying
to
analyze
those
events
in
aggregate
which
is
basically
metrics,
and
then
you
want
to
set
alerts
and
SLA
Xand
and
other
things
based
on
those
aggregated
events.
So
we're
tracing
comes
into
this
picture.
A
Is
that
used
to
be
really
easy
when
everything
was
running
on
one
computer
right
like
this
computer
starts
to
heat
up
I,
just
pop
open,
Activity
Monitor
and
find
out
the
thing:
that's
going
wrong
and
restart
it.
Damn
you
Photoshop,
but
once
everything
gets
distributed,
trying
to
understand
the
sequence
of
events
in
any
particular
transaction
becomes
harder,
because
you
know
you've
got
some
logs
on
this
machine.
A
If
you
haven't
used
a
tracing
system
before
this
is
tends
to
be
the
kind
of
display
that
you'll
see
at
first
so
audience
is
left.
You've
got
a
sort
of
standard,
looking
architecture.
Let's
imagine
this
is
a
sort
of
web
checkout,
a
transaction
that
we
want
to
look
at,
so
you've
got
a
web
client
you've
got
a
web
server
and
that
web
server
is
talking
to
three
backends,
an
authentication
service,
a
billing
service
and
a
database,
and
so
when
the
user
on
the
client
hits
purchase.
A
That
request
goes
to
the
web
server.
The
web
server
first
authorizes
you.
It
then
goes
to
the
billing
system.
Inside
the
billing
system,
you've
got
two
steps
verifying
so
your
address
and
processing
the
payment,
and
when
that
succeeds,
you
record
the
transaction
in
your
database.
So
this
kind
of
visualization
on
the
right
shows
you
some
interesting
things
right.
You
can
see
the
latency
how
long
things
took
and
you
can
see
concurrency
and
parallelism.
So
you
can
see
this
is
a
very
serialize
set
of
transactions.
A
A
If
you
look
at
this
view
here,
this
graph
on
the
top
is
the
same
kind
of
thing:
you've
got
a
service,
a
making
a
network
call
to
service
B
and
inside
of
service
a
there's,
a
set
of
operations
that
are
occurring
that
are
related
to
each
other,
so
each
operation
in
tracing
lingo,
we
tend
to
call
a
span.
So
a
span
is
just
something
your
computer
did.
That
has
a
name.
It
has
a
start
time.
A
It
has
a
duration
or
an
end
time,
and
then
it
has
a
bunch
of
tags
that
are
attached
to
it,
and
you
can
think
of
these
tags
is
basically
indices,
that
you're
going
to
use
in
your
database
later
in
order
to
look
up
these
transactions
or
aggregate
them
into
metrics.
In
order
to
then
glue
these
things
together,
there's
some
bit
of
information
that
has
to
go
in
band
over
the
wire,
so
in
service,
a
talks
to
service
B,
let's
say
you're,
making
an
HTTP
call.
A
A
Yet,
but
that's
the
thing
we're
trying
to
work
towards
another
reason:
why
there's
some
amount
of
confusion
is
depending
on
who
you
are
in
this
ecosystem?
You
may
care
about
one
aspect
of
the
standardization:
sorry
over-over
the
other
one,
and
so
because
people
might
care
about
one
part
of
this
more
than
they
care
about
another
part
of
it.
You
know,
sometimes
these
conversations
can
kind
of
move
past
each
other.
So
part
of
this
talk
in
this
deep
dive
is
to
really
get
people
on
the
same
page
about
what
these
different
pieces
are.
A
It
really
doesn't
get
much
more
complicated
than
this,
even
though
it
might
look
complicated
because
you
have
a
big
system
at
the
end
of
the
day,
you
have
your
services
that
you're
running
your
services,
for
the
most
part,
are
probably
made
up
of
your
application
code
in
your
application
logic,
but
then
you're
leveraging
a
whole
bunch
of
shared
libraries
right
so
and
in
particular,
things
like
frameworks
and
RPC
calls
that
shared
code
that
you're,
not
writing
yourself.
You
don't
say:
okay,
I'm,
going
to
write.
A
The
down
the
wire,
so
in
addition
to
your
services,
you're
also
talking
to
other
people's
services,
so
you're,
not
just
leveraging
shared
libraries,
you're,
also
leveraging
shared
services
running
on
shared
infrastructure
and
as
soon
as
you
do
that
those
things
are
kind
of
like
black
boxes.
You
can
see
like
inside
your
service.
You
can
install
some
kind
of
tracing
API
or
a
tracing
system
and
pump
data
out
into
some
kind
of
analysis
system,
but
you
can't
do
that
with
other
people's
services,
but
increasingly
they're
all
providing
some
form
of
tracing
analysis.
A
You
know
you've
got
x-ray
and
stackdriver
and
all
these
other
things.
So
if
you
want
to
get
a
complete
trace
of
everything
that
includes
not
just
your
application
code
and
your
services,
but
also
maybe
include
some
details
about
how
they're,
interacting
with
these
databases
and
things
that
other
people
are
running,
there's
some
bit
on
the
back
end,
where
you're
going
to
end
up
with
potentially
multiple
different
analysis
systems,
and
if
you
want
to
get
a
complete
trace.
A
You're
gonna
have
to
take
part
of
that
tracing
data
out
of
one
of
these
systems
and
pump
it
into
another
one.
I
would
say
for
a
lot
of
application
developers
you're-
probably
not
needing
to
do
too
much
of
that
yet,
especially
if
you're
not
really
pushing
the
boundaries
of
the
services
that
you're
using,
but
more
and
more
people
are
building
bigger
systems
that
that
really
are
pushing
those
boundaries,
and
so
that's
a
problem.
That's
going
to
need
to
get
solved
more
and
more
in
the
future.
A
So,
in
order
to
make
that
whole
system
work,
you're
going
to
need
four
pieces
and
three
of
them
ideally
should
be
standardized
or
shared
in
some
way.
If
you
have
a
tracing
API,
that's
standardized,
then
you
can
put
it
into
your
libraries
that
then
gets
shared
with
other
people
right
because
you're
not
making
a
decision
for
them
about
whether
they're
using
Jaeger
or
Zipkin
or
some
other
system.
A
You
can't
you
wouldn't
want
to
say,
put
like
you
a
New,
Relic,
API
or
something
you
know,
vendor
proprietary
into
your
open
source
library,
but
someone
who's
using
your
open
source
library
may
want
to
pump
information
out
into
one
of
those
systems,
so
some
kind
of
API
that
kind
of
abstracts.
That
problem
would
be
very
useful
in
terms
of
sending
data
over
the
wire.
A
It
would
be
great
to
have
some
kind
of
standard
data
format
that
that
information
was
coming
out
in
rather
than
having
to
like
write
adapters
for
all
of
these
different
data
formats
that
are
out
there
and
then
last
but
not
least,
you
have
some
kind
of
system
that
you're
analyzing.
This
data
in
I'd
argue
that's
the
place
where
you
don't
need
Standardization,
because
everyone's
just
running
their
own
system
and
that's
also
the
place
where
we're
trying
to
sort
of
move
monitoring
forward
and
coming
up
with
better
and
better
ways
of
analyzing.
This
data.
A
So
I
would
argue
that
really
what
you
want
is
the
part
where
you're
getting
the
data
out
of
the
system.
We
should
all
agree
on
kind
of
what
that
looks
like
and
have
some
standards
there
and
then
really
focus
in
terms
of
you
know
competing
with
each
other
and
trying
to
push
the
boundaries
just
just
in
the
analysis.
Part.
That's
not
happening
inside
of
these
systems.
A
When
you
go
to
instrument
your
own
application
code,
though
I
would
say,
you're
gonna
hear
people
say:
oh
you
really
got
to
use
this.
You
really
got
to
use
that
you
have
to
use
open
tracing.
You
have
to
use
census.
You
have
to
use
something
rather
I.
Don't
think
you
have
to
use
anything
when
it's
just
your
application
code,
you
can
kind
of
do
whatever
you
want.
That's
your
code,
you're,
not
sharing
it
with
anybody
else.
A
So
as
soon
as
you
do
that
you're
talking
about
instrumenting,
shared
libraries
and
as
soon
as
you
want
to
instrument
shared
libraries,
you
run
right
into
this
problem
of
not
wanting
to
make
a
bunch
of
these
decisions.
So
you
need
some
kind
of
abstracted
API
in
order
to
help
be
able
to
actually
bake
instrumentation
directly
into
your
library
rather
than
people
writing
a
bunch
of
plugins
and
adapters
for
each
kind
of
tracing
or
monitoring
system.
You
might
want
to
attach
shared
services.
A
So
what
are
the
solutions
that
I
would
suggest
to
solve
these
problems
or
for
the
tracing,
API
I?
Think
open
tracing?
Is
the
project
that's
really
trying
to
move
API
standardization
forward.
So
what
we
do
there
is.
We
have
an
API
that
if
you
look
at
the
API
in
detail,
you
notice
it
really
goes
out
of
its
way
to
avoid
exposing
certain
kinds
of
decisions
that
you
would
want
to
make.
For
example,
it
doesn't
expose
what
kind
of
wire
protocol
you're
using
because
there's
a
variety
of
wire
protocols
out
there.
A
A
There's
nothing
externally
visible
as
part
of
that
API
that
describes
what
HTTP
header
you're
going
to
put
it
into,
but
we
would
like
to
have
a
standard
wire
protocol
that
works
currently
being
done
in
a
w3c
group
called
the
trace
context,
working
group
and
we're
starting
with
HTTP
headers
there.
But
you
could
imagine
other
standards
for
other
kinds
of
message.
Queues
and
things
coming
out
of
that.
A
I
would
love
to
see
a
sort
of
standard
data
format
that
doesn't
exist
yet
at
all,
but
this
same
w3c
group
is
starting
to
kind
of
dig
into
that
problem
and
then
analysis
systems
who
cares.
So
this
is
just
that
same
graph
again,
showing
this
thing
from
kind
of
with
these
solutions
baked
in
and
where
I
think
they
need
to
be.
So
you
would
need
open
tracing
inside
of
your
service
as
the
API
that
you're
using,
and
that
the
API
that
your
shared
libraries
are
using.
A
You
would
need
the
standardized
trace
context
in
order
to
propagate
that
tray
state
to
somebody
else's
service.
That
service
can
kind
of
pump
information
out
in
whatever
data
format
it
wants.
But
as
soon
as
you
want
to
pull
that
back
in
to
your
tracing
system,
some
kind
of
standardized
trace
data
format,
that's
the
place
where
you
would
want
to
see
that
so
and
then
normally
want
to
give
this
talk.
I
do
a
demo
at
this
point,
but
rather
than
doing
that,
I
want
to
turn
this
now
into
a
QA.
A
So
that
was
a
big
rush
through
the
landscape,
so
I'm.
Imagine,
there's
a
lot
of
questions
and
I
know
also
I've
been
talking
about
the
general
tracing
world,
but
this
is
kind
of
an
open
tracing
focus,
deep
dive,
but
feel
free
to
ask
questions
about
some
of
this
w3c
stuff
as
well.
If
you
have
a
question
on
that
front
and
with
that.
A
So
there's,
oh
right,
sorry!
So
the
question
is
what
about
asynchronous
workflows?
How
do
you
trace
asynchronous
workflows
and
there
there's
two
kinds
of
asynchronous
workflows:
one
is
very
short-lived,
asynchronous
workflows
or
what
you
might
call
asynchronous
programming
where
you're
doing
some
kind
of
non-blocking,
I/o
and
tracing
here.
Let's
see
which
slide
that
I
have.
That
path
shows
this.
If
you
go
back
to
here
the
same
way,
you
have
these
parent-child
relationships.
A
When
you
make
that
reference,
there's
a
couple
kinds
of
references
you
can
use
the
primary
one
is
called
a
child
of
relationship
that
says
the
this
child
operation
is
going
to
return
to
the
parent
operation.
So
there's
some
work
it
was
doing
was
on
behalf
of
the
parent
operation
and
then
there's
another
reference
type
called
follows
from
and
the
follows
from
reference
says:
hey
I
just
kicked
off
this
work,
but
I'm
not
waiting
around
for
it
to
finish
so.
A
I
just
did
some
stuff
and
then
go
send
an
email
in
the
background,
and
then
that
stunned,
so
you
can
use
those
reference
types
to
actually
collect.
All
of
that
into
a
single
graph
where
this
becomes
problematic
are
very
long-running
asynchronous
flows
and
I.
Think
that
might
be
what
you're
you're
asking
about.
A
Yeah
can
I
jump
on
it.
So
let
me
just
repeat
the
question:
what
about
something?
That's
very
long-running,
so
you
have
some
like
scheduled,
cron
job
that
happens
monthly.
Another
example
I
would
give
is
like,
rather
than
a
checkout
like
where
I'm
like
Boop
I'm,
going
to
purchase
this
item.
What
about
the
whole
shopping?
Cart
experience
where
you're
browsing
around
and
picking
things,
and
you
want
to
see
all
of
that
in
a
single
graph,
yeah.
B
I
was
gonna,
say
the
the
challenge
of
tracing
in
general,
it's
being
met,
kind
of
in
order
of
greatest
need
and
I
think
that
tracing
latency-sensitive
workloads
has
been
such
a
priority
that
the
sort
of
thing
you're
describing
has
been
getting
short
shrift.
That's
my
honest
opinion.
There's
two
questions
really:
there's
modeling
that
type
of
transaction
you're
describing
and
there's
a
problem
there
and
then
actually
using
tools
that
were
designed
to
analyze
and
visualize
those
transactions.
I'm
unaware
of
a
system.
B
That's
out
there
right
now
that
you
can
use
that
works
well
for
transactions
that
take
like
100,
milliseconds
and
transactions
that
take
multiple
days.
I've.
There's
this
guy
Demetrius
who's
involved,
it's
open
tracing
who
works
at
Bloomberg
Bloomberg,
and
they
have
some
workloads
like
the
ones
you're
describing
where
they
do
a
lot
of
computation
over
the
course
of
many
hours.
There's
also
some
people
at
she
Grail
it's
like
they're
trying
to
cure
cancer,
or
some
other
insignificant
task
like
that,
but
they're
doing
analysis
of
human
genomes.
B
It's
this
intensely
complicated,
multi-hour,
MapReduce
and
similarly
like
they
have
these
traces
they
want
to
generate
and
they
had
to
write
their
own.
It
analytical
tools,
the
thing
that
that
I
do
like
about
the
open
tracing
approach.
This
is
that
Ted
is
correct
in
that
we
have
like
a
follows
from
relationship,
but
we
also
have
the
idea
of
multiple
parents
has
the
other
thing
that
comes
up
in
asynchronous
workloads.
B
You
often
have
Forks
and
joins
and
the
data,
and
if
you
have
many
different
pieces
of
data
that
are
all
colliding
into
one
joined
thing,
you
can
model
that
well
and
open
tracing.
The
irony
is
that
you
can
go
and
use
any
open,
tracing
off-the-shelf
thing
and
none
of
them
properly.
Visualize
that
you
know,
and
so
I
think,
there's
a
question
of
how
you
model
it
in
the
code.
I
think
open
tracing
actually
does
that
quite
well.
B
Unfortunately,
I
can't
tell
you
and
then
you
should
use
this
tool
that
binds
to
open
tracing
to
analyze
the
data.
The
only
thing
I
can
tell
you
is
what
I
told
the
people
at
Grail,
which
is
what
they're
doing
already,
which
is
to
write
your
own
system
to
visualize
the
thing.
Hopefully
you
open-source
it
and
then
other
people
can
use
it
as
well,
because
I
think
the
task
of
analyzing
long-running
MapReduce
jobs
and
things
like
that.
It's
a
different
problem,
even
if
the
the
instrumentation
is
the
same
yeah.
A
A
So
the
question
is:
if
tracing
isn't
just
about
logging,
you
know
what
is
it
about
and
I
would
say
my
own
personal
opinion-
and
this
is
weirdly
contentious
is
tracing,
is
just
logging.
Tracing
is
logging
in
a
world
where
you
need
some
kind
of
index
like
this
X
Trace
ID
header
that
you're
propagating
in
order
to
get
your
logs
into
a
logging
system.
A
That's
then
indexing
them
based
on
that
header,
so
that
you
can
look
them
up
again
later
right,
so
you're
tracing
right
now,
you're
tracing
and
where
I
would
say,
open
tracing
comes
into
play
is
right
now,
you're,
probably
having
to
write.
You
know
some
glue
code
yourself
to
actually
shove
that
header.
A
You
know
that
request
ID,
both
in
and
out
of
your
HTTP
headers,
and
then
you
have
this
sort
of
passed
the
baton
where,
like
through
your
code,
you
need
to
somehow
keep
track
of
that
ID
somewhere
somehow
and
just
doing
that
sort
of
in-house
on
your
own.
You
end
up.
Writing
some
code
to
deal
with
that
and
all
open
tracing
is,
is
a
sort
of
standardized
way
of
doing
that.
B
Disagreement
disagree,
no
I,
I,
don't
I
would
say
that
not
all
tracing
is
logging
or
sorry.
There's
there's
some
distinctions
here.
Tracing
typically
has
some
notion
of
causal
information
as
well,
so
you
know
which
log
line
cause
which
log
line,
if
that
makes
sense,
that's
these
references
which
are
usually
not
in
a
in
a
distributed
logging
system,
and
that's
pretty
important
in
that.
B
If
you
have
a
transaction
in
a
microservices
architecture,
it
will
typically
involve
a
lot
of
concurrency,
and
so,
if
you
just
have
a
correlation
ID
in
the
log,
it's
really
difficult
to
figure
out.
If
you
just
chronologically
order
all
the
statements
it's
hard
to
figure
out,
which
ones
are
actually
on
the
critical
path
of
the
transaction,
which
is
the
thing
that's
holding
up
the
end
user
and
a
tracing
system,
because
it
has
the
information
about
what
caused.
What
will
allow
you
to
focus
just
on
the
part
of
the
system?
B
That's
actually
slowing
down
the
end
user,
so
the
causal
information
is
important.
You
could
certainly
put
that
in
your
logs
as
well,
but
then
you'd
have
to
go
and
and
reconstruct
destroy
yourself.
The
other
thing
that
tracing
systems
typically
involve
some
form
of
sampling.
The
amount
of
data
that's
generated
at
scale
by
a
detailed
logging
system
is
sometimes
a
little
overwhelming,
at
least
from
an
ROI
standpoint,
and
then
the
last
thing
I
was
going
to
say
is
related
yeah
related
to
this.
B
Instead
of
what
you're
doing
already
you
mean
I
I
think
Ted
sort
of
spoke
to
that,
but
it's
just
the
advantage
of
using
open
tracing.
Is
that
there's
like
hundreds
of
projects
that
already
support
it?
And
so,
if
your
developers
depend
on
those
projects,
they
don't
need
to
instrument
those
by
hand.
That's
it
and
honestly
that
that
may
not
be
enough
of
a
value
proposition.
B
Maybe
you
shouldn't
I'm
serious,
but
but
if,
if
it's
beneficial
to
have
that
integration,
there
already
that's
great
and
then
also
if
down
the
line,
you
decide
to
send
the
data
somewhere
else
or
to
multiple
places.
That's
a
nice
multiplexer
and
you
know
there
could,
depending
how
you
factored
your
code.
B
You
might
have
already
accomplished
that
and
I'm
not
suggesting
you
didn't,
but
if
that's,
if
that's
an
advantage
that
that
might
be
compelling
as
well,
but
it's
ok
to
you
know
and
that's
it
I,
don't
think
open
tracing
should
be
used
by
everyone
for
all
things.
I
think
it
makes
sense
if
you're
trying
to
decouple
you
instrumentation
from
where
it
gets
sent
and
for
greenfield
stuff
I
think
it
makes
a
ton
of
sense.
B
But
if
you
already
have
a
correlation,
ID
and
you're
not
planning
on
adding
a
lot
more
to
the
application
in
the
future,
and
it's
just
in
maintenance
mode,
it
really
might
not
be
worth
it.
It
wasn't
a
rhetorical
question:
I,
it's
it's
a
it's
reasonable
to
keep
it
where
it
is.
If
you're
not
going
to
add
a
lot
of
code
to
it.
Yes,.
A
And
I
want
to
just
just
add
two
little
writers
onto
that.
One
is
just
to
put
it
in
concrete
terms.
You
have
one
ID,
which
is
the
trace
ID,
so
you
have
X
trace
ID.
If
you
wanted
to
turn
this
more
into
tracing,
you
would
be
adding
an
additional
ID
that
would
be
you
know,
X
span,
ID
right
and
that
span
ID
would
be
the
thing
that
would
be
changing
with
each
operation,
along
with
a
reference
type.
You
know
to
say
so
that
you
can
figure
out.
A
This
is
one
stack
and
this
is
another
stack
trace,
and
this
is
another
stack
trace
and
the
other
thing
I
would
say
is
you
were
saying
for
my
application
code?
Why
should
I
change
I'm
using
Cabana
and
it's
fine?
But
if
you
take
some
part
of
that
application
code
and
turn
it
into
a
library-
and
you
want
to
hand
it
to
somebody
else
and
when
they
plug
it
into
their
system,
it
automatically
binds
to
any
kind
of
monitoring
they
might
be
running.
A
Then
you
can't
just
put
cabana
directly
into
there
right
because
you'd
be
making
a
decision
that
your
library
only
uses
Cabana
and
rather
than
saying
that
you'd
like
to
say,
hey
my
library,
just
omits
event
data.
It
will,
if
it's
a
RPC
library
it
will
inject
and
extract
these
correlation
IDs,
whether
it's
X
rays,
ID
or
something
else.
It
doesn't
matter
and
then
other
people
can
use
it.
A
A
B
So
open
tracing
as
an
instrumentation
project
gets
to
avoid
this
issue,
because
the
instrumentation
is
the
same.
You
say
I'm
starting
this
I'm
stopping
this.
The
implementation
has
a
real
pickle
on
its
hands.
I
mean
I,
know
from
our
work
at
light
step.
It's
one
thing
on
the
server
side,
but
when
you
do
tracing
across
mobile
clients,
where
those
clocks
are
off
by
a
couple
of
seconds
and
the
network
latency
is
so
high,
you
can't
run
something
like
ntp.
B
It's
pretty
painful
that
one
nice
thing
about
tracing
is
that
you
do
have
some
well-defined
ordering
events.
So
you
know
that
if
you
have
a
child
of
relationship
here,
what
kind
of
saying
earlier
you
can
use
the
invariants
expressed
in
the
data
model
to
say
that
the
child
start
time
has
to
be
after
the
parents
start
time
and
the
child's
finish
time
has
to
be
before
the
parents
finish
time,
and
if
you
apply
those
across
the
trace,
you
can
start
to
line
up
timestamps,
but
I,
don't
know
of
a
very
satisfying
answer.
B
The
spanner
paper
goes
into
some
detail
about
trying
to
reify
uncertainty
in
clocks
and
blah-blah-blah-blah-blah
and
I
mean
at
some
point.
It
all
just
falls
apart,
though,
if
your
clocks,
if
your
network,
latency
czar
too
large,
there's
really
I,
don't
know
of
a
really
satisfying
answer
to
the
time.
Synchronization
problem
for
tracing
implementations,
yeah
I
mean
yeah,
so
the
Lamport
clock
type
of
type
of
approaches
is
the
the
kind
of
thing
I
was
getting
at
with.
B
A
But
yeah
and
just
to
re-emphasize
one
reason
why
it
doesn't
matter
as
much
as
you're,
not
using
time
or
clocks,
to
order
these
you
know
trace
events
you're
using
a
graph,
so
you're
connecting
them
into
a
graph.
So
even
if
the
clocks
are
skewing,
you
still
know
what
led
to
what
led
to
what,
because
the
time
doesn't
matter
and
then
when,
if
you're
doing
latency
analysis,
assuming
your
clock
isn't
skewing
between
when
you
started
the
span
locally
and
ending
it,
which
does
sometimes
happen.
A
A
Yes,
so
the
question
is:
how
are
errors
and
exceptions
handled
in
open
tracing
and
tracing
and
I
would
say?
The
answer
there
is:
is
tagging
so
on
each
span
you
can
set
tags
and
some
of
those
tags
are
used
for
indexing.
Other
tags
are
used
to
adds
like
semantic
information
to
your
span
and
the
most
common
one
is
called
the
error
tag.
So
there's
a
set
of
error,
values
and
there's.
B
Also
yeah,
there's
sort
of
a
span
can
be
in
an
error
State,
which
is
well
defined,
as
also
the
specification
has
a
particular
way
of
representing
exceptions
and
stack
traces
and
things
like
that
in
the
in
the
each
span
has
a
list
of
time-stamped
events
which
are
called
logs,
which
may
be
confusing
sorry
but
there's
yeah.
This
is
the
span
level
air
and
then
below.
There's
another
specification
for
for
the
stack
trace,
type
information
right.
A
So
here's
an
example
of
like
what
we're
trying
to
say
is:
if
you
have
an
error
here,
are
some
some
standard
tag
names
so
that
when
you're
looking
in
these
tracing
systems,
they're
they're
expecting
these
these
particular
identifiers
to
be
there.
So
if
you
see
a
tag,
that's
named
error
and
it's
a
boolean
value.
That
system
knows.
That
means
this.
This
ban
had
a
logical
error.
A
So
the
question
is:
is:
is
the
open
tracing
project
also
involved
in
storage
or
any
kind
of
back-end
solutions,
and
so
I
would
say
very
emphatically?
No,
the
the
purpose
of
the
project
is
to
have
an
abstracted
API
that
makes
no
decisions
about
what
you
might
do
with
the
data.
It
doesn't
make
any
decisions
about
where
you
would
put
it,
how
you
would
export
it,
and
that's
very
intentional,
to
allow
you
to
pick
your
own
back-end.
A
B
B
A
The
the
big
ass
that
we're
asking
for
an
open
tracing,
if
you
get
involved,
is
to
consider
as
the
API
stabilized,
if
you're,
making
open
source
libraries
rather
than
having
people
include
the
the
open
tracing
API
calls
as
a
plug-in.
You
just
make
it
directly
into
the
library.
So
Couchbase,
for
example,
is
a
database
and
their
Java
SDK
now
comes
with
open
tracing
baked
in
and
along
with
that,
they've
also
shipped
a
playbook,
so
they're,
like
here's
all
the
event,
data
that
comes
out
of
our
client,
here's
what
it
means.
A
So,
if
you're,
seeing
like
retries
or
timeouts
or
if
you
see
you
know
this
event
point,
this
is
what
you
should
do.
This
is
how
you
should
tune
this
thing
so
that
that's
actually
the
main
focus
of
the
project
is
to
enable
that,
because
that's
the
thing,
if
you
go,
write
these
libraries
right
now,
that's
like
really
hard
to
give
people
that
you
end
up
having
to
come
up
with
some
made-up
event
thing
and
some
hooks
for
them
to
figure
out
how
they're
gonna
put
it
into
their
system.
Definitely.
B
I've
had
some
conversations
with
some
of
the
people
in
the
lambda
team,
and
there
is
some
implementation
concerns
with
doing
tracing
in
an
environment
where
you
aren't
allowed
to
retain
state
from
call
to
call
cause.
You
have
to
flush
with
every
single
call
which
are
kind
of
annoying,
but
nothing
for
family
difficult
about
it.
The
open
tracing
piece
is
not
so
bad,
it's
actually.
The
implementation
of
the
tracing
system.
X-Ray
is
probably
the
only
thing
that
has
a
good
shot
at
it
and
lambda
because
they
can
cheat,
but
I'm
I've
asked
them
with
my.