►
From YouTube: OpenTracing Monthly Meeting - 2018-02-09
Description
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
B
A
C
A
A
So
here's
the
agenda
for
this
talk
we'll
start
with
the
pin
trace
infrastructure,
basically
how
our
distributed
tracing
works.
Then
I'll
go
over
some
data,
visualization
tools
that
we
have
failed
to
look
at
this
data
and
then
some
more
tools
that
we've
built
to
do
deeper
analysis
and
then
some
future
plans
for
for
for
this
project.
A
This
is
how
our
tracing
pipeline
looks
like
we
have
instrumented
most
of
our
services,
starting
from
the
clients
like
iOS,
Android
and
web,
including
CDN,
the
front
end
API
and
all
of
the
backend
services
and
all
these
services.
When
a
request
passes
through
these
services,
they
log
trace
data
to
our
Kafka
pipeline.
In
this
diagram,
we
can
see
the
black
arrows
are
the
requests,
the
user
requests
path
and
the
dotted
arrows
are
or
the
trace
data
path.
So
all
this
data
from
all
these
services
goes
to
Kafka
from
Kafka.
A
We
have
a
spark
collector
that
picks
up
this
data
and
pushes
it
to
elastic
search
from
elastic
search.
We
have
built
bunch
of
tools
to
to
view
this
data
to
search
this
data
and
and
then
to
do
deeper
analysis
on
this
data
and
in
this
talk,
I'll
quickly,
go
over
all
these
tools
and
explain
why
we
and
what
are
different,
use
cases
of
these
tools
before
I
move
forward.
I
just
want
to
highlight
few
things.
A
The
first
thing
is
end
to
end,
so
our
tracing
is
enabled
from
from
the
very
start
from
the
client
to
all
the
way
to
the
back
end.
So
we
capture
we
capture
all
steps
of
this
process.
This
includes
timestamp,
is
spent
on
the
network
between
the
user
and
CDN
time
spent
on
CDN
and
the
time
spent
by
all
back-end
services.
A
A
Third
thing
is
we
we
built
spark
collector.
This
is
basically
to
to
consume
data
from
Kafka
and
push
it
to
elasticsearch,
but
we
added
some
additional
functionality
in
this
collector
as
well
to
do
some
kind
of
post
processing
so
that
we
can
do
some
feature
extraction
or
data
cleaning
or
data
blacklisting,
or
these
kind
of
things
just
want
to
highlight
one
more
thing
here
that
this
spark
collector
does
all
the
processing
at
span
level
and
where
we're
planning
to
extend
it.
A
To
add,
when
doing
in
sparks
so
that
we
can
do
some
post-processing
at
trace
level
and
again,
we
need
this
ven
doing
functionality
so
that
different
different
events,
different
spans
of
our
trace,
can
reach
at
different
time.
So
unless
we
have
this
window
of
one
minute
or
something
where
we
can
get
all
the
spans
for
a
given
trace
and
then
process
them,
we
cannot
do
trace
level
processing
and
the
last
thing
is
scalability.
A
So
our
current
pipeline
is
very
scalable
right
now
we
handle
around
20
million
data
events
per
day
and
we
have
seen
10x
spikes
in
the
data
volume
and
so
far
our
system
scaling
pretty
well
for
these
spikes
towards
the
end
of
the
presentation.
I'll
circle
back
to
this
point
and
explain
why
why
having
a
scalable
system
is
very
important,
so
we'll
start
with
how
we
view
our
traces
to
view
our
traces.
We
use
the
kanji.
Why
it's
a
really
nice
tool
to
look
at
the
details
of
a
single
trace.
A
A
A
I'll,
just
quickly
show
a
couple
of
screenshots
from
how
we
use
use
it
in
UI,
so
basically,
in
this
screenshot
you
can
see
that
we
highlight
errors
in
our
traces.
We
also
add
in
process
fans,
for
example.
Here
we
can
see,
there's
a
sub
spent
for
authentication,
there's
a
sub
spent
for
request,
handling
and
processing.
This
really
helps
us
in
in
performance
tuning
our
services.
We
exactly
know
how
much
time
is
spent
in
user
authentication
and
how
much
time
is
spent
in
request,
processing
and
then
encoding.
The
the
response.
A
A
A
A
Ok,
so
now
we
know
how
to
how
to
look
at
a
trace.
The
next
problem
that
we
wanted
to
solve
is
how
to
find
the
most
relevant
traces
for
a
given
issue.
Zipkin,
do
ID
gives
you
some
functionality
to
to
find
these
traces,
but,
but
still
you
don't
get
a
visual
representation.
So
first
thing
we
did
was
we
integrated
our
tracing
data
with
our
metrics
metrics
visualization?
A
This
trace
data
on
top
of
our
metrics
data
and
from
here
user,
get
a
really
good
visual
represent
of
spans,
for
example,
in
this
case,
we're
looking
at
the
metrics
for
for
one
endpoint
for
for
a
service
and
we're
overlaying,
the
say
the
corresponding
spans
for
the
same
endpoint
and
here
user
can
see
that
with
which
which
traces
correspond
to
the
p99
latency
of
that
endpoint,
and
they
can
quickly
see
that
okay
at
this
time
there
was
a
spike.
Let's
look
at
these
set
of
traces
and
they
they
would
be
good.
Representative
of
that
spike.
A
What
so
next
thing
that
we
we
want
you
to
look
at
is
trends
in
our
in
our
traces
prints
like
is
it's
a
latency
increasing?
Is
the
error
rate
increasing
for
a
service
or
not
so
for
that?
What
we
use
is
Cubana
dashboard.
So,
since
our
data
is
in
elasticsearch,
it's
aligned,
elasticsearch
comes
with
this
Cabana
plugin.
This.
These
dashboards
were
really
helpful.
To
give
us
a
high
level
trend
in
our
traces.
We
have
three
main
dashboards.
One
dashboard
is
for
services
where
service
owners
can
look
at
their
trace
trends.
A
The
other
dashboard
is
for
clients
like
iOS,
Android
and
web,
and
the
dashboard
is
for
for
the
Pinterest
developers
and
operation
people
so
that
we
can
see
how
Pinterest
pipeline
is
doing
and
if,
if
there
are
any
latency
or
delays
in
in
our
data,
so
for
example,
here
we
can
see
the
dashboard
for
for
a
service
and
the
service
owners
can
see
how
much
spans
are
coming
for
their
service.
What's
the
breakdown
per
endpoint?
A
A
But
without
a
tool
like
this,
it
would
have
been
really
hard
to
figure
out
where
what's
the
source
of
this,
this
bug.
Similarly,
on
the
right
side,
we
see
a
spike
in
spans
coming
from
our
dev
dev
servers.
So
again,
someone
had
some
problem
on
their
dev
server
and
it
was
it
started,
sending
too
many
spans
and
we
were
able
to
catch
it
quickly.
A
Okay,
so
now
now
we
have
a
way
to
look
at
traces
and
a
way
to
look
at
trends.
Next
thing
that
we
wanted
to
do
was
to
get
a
deeper
analysis
of
our
traces
and
answer
more
complex
questions
like
how
much
time
is
spent
on
a
single
service
for
for
for
a
for
a
given
user
operation
or
what's
the
difference
between
performance
between
two
different
versions
of
software.
So
for
this
we
built
a
trace
analyzer,
and
this
is
the
tool
that
we
explained
in
our
blog
as
well.
A
So
the
requirement
for
for
this
tool
was
that
we
want
you
to
get
an
aggregated
view
of
traces.
Looking
at
one
trace
might
not
be
the
right
way
to
go
because
it
might
not
be
the
representative
trace
for
for
any
service,
so
so
the
first
requirement
meant
was
to
have
a
way
to
look
at
the
aggregated
view.
Second
requirement
was
to
compare
two
sets
of
crises.
For
example,
you
want
to
compare
trace
from
last
from
last
week
to
this
week
or
between
two
versions
or
between
two
devices.
A
You
want
to
compare
traces
from
Android
and
iOS.
Then
we
also
wanted
to
provide
a
way
to
do
root-cause
analysis.
Let's
say:
there's
a
regression
in
your
performance
and
now
because
we
have
80
services
this
in
our
back
and
it's
really
hard
to
find
out
which
service
is
causing
that
regression.
So
we
wanted
an
automated
way
to
surface
up
the
the
root
cause
for
any
any
any
performance
problem,
and
then
we
wanted
it
to
be
scalable.
A
So
the
we
build
here,
here's
the
basically
architecture
of
the
tool
that
we
build
for
the
UI
part.
We
used
Jupiter
notebooks,
it's
a
tool,
that's
used
here
at
Pinterest,
and
people
are
used
to
this
tool.
This
UI
from
this
UI
users
can
provide
their
input
their
parameters
for
the
analysis.
This
notebook,
then
triggers
a
spark
job.
This
bar
job
reads
spans
from
elasticsearch.
A
Does
lots
of
processing
and
analysis
builds
a
report
and
writes
it
back
to
elasticsearch
and
then
again
the
Jupiter
notebook
reads
the
report
from
elasticsearch
and
displays
it
to
the
user,
and
this
whole
process
can
take
from
seconds
to
minutes.
So
once
the
report
is
done,
the
user
gets
an
email
and
then
they
can
go
and
look
at
this
report.
A
Here's
the
UI
for
for
running
this
report.
Basically
you-you-you
can
specify
the
time
period
that
you
want
to
look
at
the
service
endpoint
and
some
other
binary
annotations
and
you
can
provide
it
for
two
different
sets
of
phrases.
If
you
want
to
compare
these
traces
and
here's
the
sample
report
summary
for
one
of
these
reports-
and
in
this
summary,
you
can
see
things
like
what
was
the
average
latency,
what
were
the
calls
portrays?
A
What
total
number
of
calls
made
the
most
interesting
part
here
is
if
we
look
at
the
three
graphs
in
the
bottom,
the
first
graph.
This
report
is
by
the
way
for
ng
API,
so
the
first
graph
shows
the
self
latency
of
ng
API
and
it's
a
histogram,
and
we
can
see
that
in
batch
one
and
batch
to
the
two
sets
of
majid
batches
that
we
were
comparing
there.
A
There
wasn't
much
difference,
but
in
the
second
graph,
the
overall
latency
we
we
see
a
clear
shift
of
the
histogram
towards
the
right,
and
this
kind
of
highlights
that
viola,
ng
API,
did
not
add
any
performance
delay
in
in
the
in
the
request.
The
backend
services
did
add
some
performance
delay
because
the
overall
latency
has
increased.
So
this
kind
of
shows
you
how
this
tool
can
be
used
to
do,
root,
cause
analysis
and
fight
the
root
cause
that
where
this
extra
latency
is
coming
from
and
from
here
we
can
again.
A
This
report
has
lots
of
other
things.
One
one
more
thing
is
downstream:
Service
latency,
so
this
this
table
lists
down
all
the
latency
that
was
interest
introduced
by
all
downstream
services,
and
this
is
their
self
latency.
So
this
does
not
include
the
latency
that
was
incurred
by
any
further
downstream
calls
made
by
those
services.
A
And
again
we
have
a
table
for
number
of
downstream
calls
as
well
and
again
in
these
tables
the
the
two
kind
of
highlights,
the
services
that
that
have
the
highest
impact
and
then
the
engineers
they
can
quickly
look
at
other
dashboard
for
those
services
and
see
how
they're
performing
this
last
table
is
for
for
looking
at
per
endpoint
per
and
point
latency
impact.
So
it
has.
Basically
each
row
shows
you
a
span
name,
client
and
server.
A
So
this
gives
you
a
further
detailed
overview
of
what
endpoint
added
the
highest
latency
in
the
set
of
traces
that
we
are
looking
at
and
again
from
here
then
users,
the
engineer's.
They
can
see
that
okay,
this
service,
this
endpoint
of
this
service,
had
the
most
impact
and
they
can
go
and
look
at
the
the
core
changes
that
they
made
for
that
endpoint
and
see
what
what
broke
the?
What
added
this
extra
latency
okay.
So
we
are
kind
of
running
out
of
time.
A
I'll
quickly
go
over
a
few
more
slides,
I'll
quickly
go
over
them,
this
tool
that
I
explained
earlier
the
trace
analyzer,
it's
not
very
flexible,
so
some
of
the
engineers
they
requested
some
more
a
library
where,
where
they
can
just
pull
all
the
traces
and
do
some
ad
hoc
analysis,
so
we
built
a
Python
client
for
them
that
has
all
the
helper
functions
to
do
the
analysis.
Here's
the
sample
code
for
this
client.
A
A
So
this
I
don't
know
if
you
guys
are
familiar
with
Netflix
visceral
too,
but
this
tool
lets
you
view
real-time
state
of
your
services
and
we
are
using
this
tool
to
basically
get
to
display
the
real-time
view
of
our
trace
data
and
the
output
of
these
analysis
tools.
So
let's
say
if
one
of
the
service
is
having
some
issue
some
errors
or
some
latency
spike,
this
will
clearly
show
you
where
the
spike
is
coming
from,
and
you
can
see
the
root
cause
of
that
spike.
A
C
D
C
A
A
B
A
So
far,
we
we
did
not
have
any
trouble
with
Kafka
and
again
we
have
other
services
at
Pinterest
that
that
use
basically
that
have
high
requirements
for
Kaufmann.
Even
for
them,
it's
working
pretty
good
and
our
service
is
not
one
of
those
services
that
pushes
Kafka
to
its
limits.
So
so
far
we
didn't
see
any
problem
with
alpha
great.
F
I
have
a
couple
of
questions
on
try
make
expect.
First
of
all,
thank
so
much
awesome
sensation.
Second
of
all,
I'm
serious,
like
almost
work
from
a
product
standpoint
and
from
a
technology
standpoint.
How
do
you
most
people
who
use
the
we
have
various
entities
shown
get
into
it,
that
they
they
go
in
their
browser
and
navigate
to
an
internal
URL,
or
is
it
that
they're
being
they
were
diverting
through
some
other
like
a
metrics
tool
or
something
else?
F
And
then
my
second
question
as
a
follow-up
is,
is
you
know
I
last
time
I
supposed
to
assume
on
about
this
was
probably
a
year
and
possibly
go,
and
he
hadn't
managed
to
get
the
kind
of
instrumentation
coverage
that
you
you've
gotten
at
this
point
and
I
was
curious
like
from
a
organizational
standpoint,
what
what
you
all
did
to
get
such
high
quality
coverage
of
your
system.
It's
a
challenge
for
a
lot
of
companies.
I
was
curious
to
hear
about
that.
Both
thanks
so
much
sure.
A
For
the
second
question,
how
did
we
get
this
instrumentation
coverage
so
good?
So
what
we
did
was
we
we
kind
of
delegated
this
instrumentation
work
to
the
service
framework
owners
for
each
language.
So
it's
not
I
mean
if,
if
just
me
or
my
team
is
doing
all
the
instrumentation,
it
won't
be
scalable
and
again
we
don't
have
skillsets
in
iOS
and
Android
and
CDN
and
all
those
places.
A
So
so
we
worked
with
other
teams
and
gave
the
ownership
for
every
language
and
framework
to
the
right
team,
and
since
they
have
all
the
skill
sets
for
them,
it
wasn't
too
hard
to
instrument
their
code
and
instrument
their
frameworks
and
that's
how
we
were.
We
were
able
to
to
get
such
a
such
a
good
coverage
and
again
now
moving
forward
those
teams
they
maintain
their
own
instrumentation.
A
So,
from
our
perspective,
we
just
give
them
guidelines
that
okay,
let's
use
this
standard,
let's
use
this
schema
and
if
they're-
and
we
help
them
debug
issues
as
well-
that
when
they
start
implementing
these
instrumentations,
usually
they
run
into
some
problems,
some
encoding
problem
or
those
kind
of
things,
but
what
we
helped
them
and
we
worked
with
them.
But
in
the
end
those
teams
are
the
owners
of
of
the
instrumentation
and
that's
why
we
were
able
to
scale.
So
else
did
I
answer
your
question.
A
F
You
did
thank
you,
it
seems
I
mean
some
companies,
don't
have
the
benefit
of
having
a
central
framework
scheme
for
his
language.
I
think
that's,
that's
probably
something
that
helps
a
lot
for
Pinterest
is
like
you
actually
captured
that
into
your
org
chart
in
some
way,
which
is
which
is
really
smart.
So
so
that's
great.
Thank
you.
So
much
sure.
G
A
We
I
mean
I,
I,
haven't
looked
at
how
many
traces
are
viewed,
but
once
in
a
while
I
go
and
see
how
many
traces
are
archived
because
they
are
in
a
different
index.
So
it's
really
easy
to
go
and
view
them.
So
we
don't
have
a
whole
lot,
but
everyday
we
have
few
traces
and
again,
the
the
biggest
use
case
is
that
let's
say
someone
finds
a
bug
in
their
code
or
they
they
see
some
interesting
trace.
A
They
want
to
save
it,
because
the
timeline
of
fixing
that
trace
and
then
the
point
that
code
can
take
more
than
seven
or
ten
days.
So
our
retention
policy
for
regular
traces
as
10
days,
but
for
any
such
use
case.
We
want
to
keep
those
traces
for
longer
and
and
again
people
are
using
it
a
lot.
It's
like
I
would
say
a
few
traces
a
day,
but
even
then,
though,
those
are
really
critical
traces
that
we
really
want
to
save.
For
that.
H
B
B
Were
there
I'll
just
give
a
high-level
report
back
and
then
maybe
Erika
and
Ben
and
others
can
chime
in
with
their
thoughts
the
the
main
thing
that
was
discussed
there
was
the
trace
context,
specification
that
really
has
two
parts
trace
context
which
is
information
being
propagated
about
the
trace
itself,
from
one
tracing
system
to
another,
potentially
and
then
there's
another
header
called
correlation
context,
which
is
sort
of
like
a
baggage
propagation,
a
way
to
just
transport
a
set
of
key
value
pairs
down
the
stack
through
some
header.
That's
been
whitelisted
by
the
various
proxies.
B
There
was
a
lot
of
discussion
about
these
two
things
in
general,
it
seems
like
more
and
more
agreement
is
being
reached
with
the
Tres
context.
In
trace
context,
extension,
headers
I
would
encourage
people
to
go.
Look
at
the
repo
to
read
up
on
on
where
that's
currently
at,
but
it
felt
to
me
like
that
was
was
really
getting
in
the
hair-splitting
territory
and
as
far
as
the
open
tracing
project
is
concerned,
I
think
I
think
it's
it's
reaching
some
point.
B
That
means
we
should
start
thinking
about
it
in
particular.
Do
we
want
to
expose
some
of
these
fields
on
the
span
context
API,
so
there's
going
to
be
a
span
ID
and
a
trace
ID
that
come
with
this
new
header
and
people
have
been
asking
for
the
ability
to
correlate
spans.
You
know
doing
span,
observers
and
things
like
that
and
not
having
an
exposed
span.
B
Id
or
trace
ID
has
kind
of
been
a
blocker
for
us
to
be
able
to
do
some
useful
things
so
I
think
to
me
that
was
the
most
important
thing
that
came
out
of
that.
The
second
most
important
thing,
which
relates
to
the
correlation
context,
header.
It
was
a
discussion
on
security
and
when
you
add
baggage
and
your
program,
and
then
you
have
some
third
party
piece
of
instrumentation-
that's
serializing
that
baggage
onto
your
HTTP
calls
or
message
queues
or
whatnot,
but
in
particular,
HTTP
calls.
B
There's
not
really
any
API
mechanism
for
indicating
whether
or
not
a
request
is
outbound
to
another
third-party
system
or
staying
in
bound
within
your
own
system.
And
that
means
this
is
a
security
issue
without
some
clear
way
of
identifying.
You
know
when
that
correlation
contacts,
header
should
be
populated
and
without
any
kind
of
encryption
on
the
data
inside
of
there.
B
It's
it's
really
kind
of
handing
a
foot
gun
out
to
application
developers,
in
fact,
in
the
open
tracing
API
right
now,
there's
literally
no
mechanism
for
deleting
or
indicating
not
to
propagate
baggage,
so
I
think
that's
a
real
problem
that
we
should
think
about.
There
are
some
other
things
that
happened
there.
Open
census
showed
up.
It
was
nice
to
see
them,
but
I
think
those
two
points
that
I
just
mentioned:
exposing
the
trace
context,
fields
on
span
context
and
really
thinking
hard
about
baggage
in
terms
of
security.
H
Somehow
I
volunteered
myself
to
write
trés
context,
implementation,
it's
a
basic
tracer.
My
agent
was
going
around
the
room
after
you
guys,
left
and
and
I
was
like.
He
was
like
who's
gonna
write
reference,
implementations
and
someone
thought
you'd
for
I.
Don't
know
what
for
Zipkin
someone
volunteered
for
this
I
think
I
said
I.
Do
it's
the
basic
triskin,
okay.
H
You
know
who
cares
if
there's
a
trace
ID
or
not,
that
I
thought
was
sort
of
interesting
and
fundamentally
challenging
some
of
the
models,
but
people
in
all
trades
vendors
have
been
doing
yeah.
That's.
B
I
think
I
was
convincing
specifically
that
you
know
we
were
trying
to
solve
lower-level
problems
at
the
higher
level
by
not
having
some
form
of
you
know
some
form
of
standardized
context,
propagation,
because
that
doesn't
really
exist.
We
end
up,
you
know
we
need
it
for
tracing,
and
so
then
we
baked
it
in,
and
then
everyone
comes
over
to
the
tracing
system,
which
is
an
observability
system
and
says,
can
I
please
ride
shotgun
on
this
thing?
B
H
Right
yeah,
because
you're
gonna
do
the
work
at
the
same
time,
and
it's
maybe
a
nice
carrot
to
provide
the
instrument
or,
if
they're,
having
to
propagate
context
for
tracing
to
tell
them.
Oh
also,
you
can
use
this
channel
for
all
this
other
wacky
stuff
that
you
probably
shouldn't
rely
on
for
your
application.
It's.
H
I
was
often
thinking
of
trace
linux
as
a
way
to
we
could
somehow
productize
the
propagation
of
context
on
its
own
and
then
it
seemed
like
a
bad
idea
because
we
would
be
at
fog,
keep
providing
automatic
instrumentation
if
they
missed
a
spot,
and
then
they
will
be
mad
at
us.
So
it's
I
think
it's
much
safer
to
give
that
to
them
to
the
manual
instrument
or
the
engineer
themselves
that
their
own
company,
rather
than
have
them
automatic,
instrumentation
vendors
to
try
and
you
know,
promise
to
hit
every
context
propagation
opportunity
perfectly.
H
E
Say
in
a
previous
life,
when
a
very
large
scale
micro-service
this
deployment,
with
probably
on
the
order
of
2,000
microservices
and
where
engineers
put
a
lot
of
stuff
into
context
and
we
kind
of
let
it
be
like
freeform
like
if
you
want
to
use
it,
go
ahead
and
use
it
and
there
wasn't
really
much
governance
around
what
go.
What
should
or
should
not
go
into
context,
I.
E
Think
at
one
point
we
incurred
like
I,
think
we
had
like
50%
extra
latency,
because
we
were
sending
that
much
more
unnecessary
context
through
a
micro
service
architecture
because
of
it.
So
we
had
to
actually
dial
it
back
because
we
had
so
much.
There
was
just
so
much
in
there,
but
that's
a
it's
an
interesting
problem
to
hit
and
have
at
that
point.
B
B
There's
you
can
tag
spans
and
you
can
attach
baggage
to
a
tracing
system
and
baggage
is
sort
of
like
trace
level
context
in
a
way
right,
you're
saying
this,
this
trace
has
this
project
ID
associated
with
it
things
of
that
nature,
but
because
there's
getters
on
baggage.
The
semantics
is
this
a
way
of
propagating
in-band
information,
then
that
implies
there's
this
cost
to
to
adding
you
know
baggage
to
your
system,
because
you're
going
to
be
propagating
it.
So
you
have
to
think
about.
B
You
know
the
the
size
of
that
and
something
various
tracing
vendors
were
bringing
up
at
this
workshop.
Was
you
know
if
you're
just
doing
it
for
the
purposes
of
monitoring
within
your
tracing
system?
There's
no
need
to
be
propagating
that
information
in
band
right
like
it
would
be
much
better
to
be
telling
your
tracing
system-
hey.
Here's
I'm
just
tagging
this
trace
with
project
ID
and
an
out-of-band.
It's
indexing
it
or
doing
whatever
it
does
with
that
and
you're,
not
worrying
so
much
about.
B
B
It's
baggage
seemed
to
be
doing
double
duty
right,
like
if
the
tracing
system
is
trying
to
use
that
as
information
and
you're
trying
to
provide
users
with
indexing
based
on
that,
then
you've
got
this
sort
of
spurious
overhead,
that's
associated
with
them
versus
baggages,
for
not
for
the
tracing
system
at
all,
and
it's
just
it's
just
the
mailman
delivering
this
to
some
other
system,
in
which
case
tracing
vendors
were
like.
That
is
not
my
job.
B
G
Agree
I
enjoyed
talking
to
vendors,
but
as
a
vendor,
I
enjoy
talking
to
customers
and
trying
to
really
nail
down
why
they
wanted
baggage,
because
I
was
hoping
that
the
answer
would
be.
They
just
really
want
to
tag
their
traces.
Not
they
actually
need
to
use
it
at
every
single
hop
because
I'd
make
my
life
a
lot
easier.
B
Yeah,
it
seems
like
like
that
API
is
really
doing
double
duty
and
when
you
throw
in
the
extra
added
fund
of
security
and
propagating
these
things
out
of
your
system
by
accident,
without
any
kind
of
encryption
on
the
data
or
adding
encryption
to
the
data
and
thus
potentially
ensuring
a
lot
of
overhead
into
this
whole
affair.
I,
don't
know
I,
think
I.
Think
baggage
needs
needs
to
get
reviewed
in
a
very
serious
way.
B
All
right,
possibly
some
of
these
things
may
show
up
in
this
specification
issue.
Backlog
I
think
a
couple
things
that
we
should
start
thinking
about.
Based
on
that
workshop
are
exposing
span,
ID
and
trace
ID,
you
know,
just
assuming
trace
context
is
going
to
come
through
the
pipe.
What
would
we
like
that
API
to
look
like
an
open
tracing?
Would
it
be
problematic,
so
I
think
that's
one
issue
we're
gonna
raise
again.
B
It
was
sort
of
called
correlation,
ID
or
debug
correlation
ID
in
the
past,
but
we're
gonna
bring
that
back
up
and
then
I
think
an
issue
around
baggage
review
around
security,
and
some
of
these
other
things
will
get
brought
up
and
I
bring
it
up
on
this
call
because
the
OTS
C
is
nominally
supposed
to
be
kind
of
reviewing
and
driving
that
stuff,
because
we
haven't
really
been
doing
a
lot
of
it
because
we've
been
focused
on
in
process
context
propagation.
There
hasn't
been
a
lot
of
movement
on
the
speck
outside
of
that.
B
So
now
that
we've
sort
of
passed
that
particular
gallstone
I
think
the
time
is
right
to
sort
of
pick
up.
Some
of
these
other
issues
and
start
moving
on
them
quickly,
so
I
would
ask
that
when
those
things
pop
up
in
the
specification
issue
backlog
people
who
are
members
of
the
OTS
C,
you
know
pay
attention
to
them
and
make
sure
we're
not
I,
don't
think
we
have
an
official
SLA
on.
B
All
right
so
I
think
that
gets
through
the
next
thing
on
our
item
here
was
sort
of
bagged
in
Spain
context,
relationship
where
the
getters
live
in
the
interest
of
time.
I
think
I
want
to
just
hop
over
that
for
the
time
being,
I
think
we
can
discuss
those
nuances
on
github
I.
Don't
think
we
need
to
talk
about
them
here,
but
just
basically
you
know
where
do
the
getters
live?
B
You
know
around
whether
span
context
is
immutable
or
not,
but
I
think
we
should
discuss
that
on
github,
since
it's
very
dry
moving
down
the
list.
I
think
this
next
one
is
a
a
fairly
non-controversial
issue.
The
CMC
f
has
requested
that
we
switch
the
licenses
for
all
of
our
repos
to
the
Apache
I
believe
Apache,
you
know
v2
or
whatever
the
standard
apache
licenses.
We
have
a
number
of
them
that
are
currently
the
MIT
license.
The
rest
are
already
apache.
B
B
F
E
F
Make
sense
to
license
it,
but
but
yeah
it's
not
well
I,
don't
think
it'll
be
controversial.
Oh
that
the
reason
for
a
people
if
they're
curious,
it's
just
that
MIT
license,
has
some
loopholes
in
it.
That
can
become
problematic
was
like
submarine.
Ip
is
long
story
that,
but
it
it's
it's
just
like
super
wonky
legal
stuff,
but
it
for
love
interesting.
It
makes
like
literally
no
difference
the
only
reason
anyone
at
MIT
is
that
I
think
MIT
was
more
popular
into
some
of
the
languages.
F
D
We
went
with
this
probe
for
this
through
this
process
with
Yaeger
when
we
joined
since
EF
and
the
way
we
approached
it
and
I
consulted
with
our
legal.
They
said
like
open
an
issue
in
a
particularly
poor
and
just
being
every
contributor
who
was
who
is
in
the
history
of
github.
For
that
repo
and
just
let
it
sit
for
like
a
couple
weeks,
if
no
one
objects
done
feel
free
to
upgrade
to
a
budget.
D
You
we
also
adopted
by
the
way
so
like
a
girl
code
used
to
be
on
uber
repo,
which
had
a
CLA.
So
we
killed
that
when
we
moved
to
independent
work,
but
we
also
adopted
the
certificate
of
origin
from
CN
CF,
like
the
one
that
Linux
uses.
So
every
commits
that
happens
in
the
repo
has
to
be
signed,
which
is
like
a
like
a
yes
switch
to
the
git
commit.
This
is
not
much,
but
there
is
a
checker
which
verifies
that
all
commits
in
the
P
are
assigned.
B
B
B
So
in
order
to
kind
of
keep
the
OTS
see,
focus
on
sort
of
a
steering
committee
specification
level
stuff
and
not
fill
this
time
up
with
a
lot
of
nuts
and
bolts,
we
still
need
to
get
those
nuts
and
bolts
put
into
place
so
created
a
new
set
of
working
groups.
One
is
called
the
cross
language
working
group,
which
is
tasked
with
figuring
out
some
of
the
day-to-day
project
management
around
these
various
backlogs,
so
the
language
API
backlogs
in
ot
core
and
then
the
backlogs
for
all
of
the
contributed.
B
Instrumentation
there's
just
a
certain
amount
of
process.
We
want
to
make
sure
we
have
in
place
around
just
processing
issues
in
PRS
that
currently
is
not
really
nailed
down.
You
know
things
like
you
know,
templates,
making
sure
when
people
make
a
PR
or
an
issue,
it's
it's
just
focused
on
a
single
subject.
B
Who
should
be
a
sign
just
reviewers
to
these
things,
not
that
other
people
can't
comment,
but
who
should
be
assigned
to
ensure
that
these
things
are
getting
shepherded
through
to
completion
if
they
can't
reach
a
completed
state
and
have
to
be
set
aside?
Where
do
we
put
them
a
bunch
of
that
kind
of
day-to-day
project?
Management
needs
to
get
sorted
out.
So
this
is
a
group
focusing
on
that
and
as
far
as
API
decisions
is
concerned,
also
focused
on
making
sure
all
of
the
language
api's
are
implementing
this
spec.
B
So,
for
example,
we
had
a
contact
propagation
as
a
concept
making
sure
that
actually
goes
into
all
the
languages
that
would
need
it.
The
API
is
that
result
feel
coherent
as
a
whole
if
you're
doing
cross-language
open
tracing.
So
that's
sort
of
the
Mandate
of
that
particular
group
and
I
would
welcome
people
to
join
it.
There's
a
new
getter
channel
under
open
tracing
called
cross
language
and
we'll
be
having
sort
of
regular
workshop
meetings
to
kind
of
get
this
stuff
moving
any
questions
or
comments
about
that
working
group.
B
Cool
feel
free
to
ask
questions
on
git
er
related
to
that
there's
also
a
new
documentation.
Working
group
that's
been
kicked
up
and
that's
shock
to
overhaul
or
documentation.
Basically
we're
missing
a
bunch
of
styles
of
documentation.
That
would
be
really
helpful.
There's
a
lot
of
you
know
consistent
questions.
People
have
when
they
come
to
the
project
that
we
could
clear
up
so
I
think
making
that
a
lot
cleaner
would
be
helpful.
B
You
can
kind
of
dig
around
that.
You
know
ot.
You
can
trib
github
organization
right
now
and
find
things,
but
I
think
that's
just
too
opaque
enough
that
it's
not
doing
good
ecosystem
justice.
So
that's
the
other
project.
I.
Think
that
group's
going
to
work
on
by
the
docks
working
group,
I
would
HIGHLY
encourage
members
open
tracing
members
who
come
from
organizations
large
enough
to
employ
technical
writers.
B
To
consider
having
your
technical
writers
join
this
group
and
donate
some
of
their
time
and
to
also
think
about
how
the
open
tracing
documentation
could
get
worked
back
into
your
own
vendor
specific
documentation,
because
a
certain
amount
of
your
API
is,
of
course,
the
open
tracing
API.
So
there's
some
other
reason
to
come
to
this
group.
B
B
Yeah
so
open-floor
anyone
have
any
other
questions
or
topic
discussion.
I.
F
F
It
has
a
fancy
title
there
a
couple
of
days
ago
in
San,
Francisco
and
I
think
that
we're
going
to
try
and
find
some
it's
unfortunate
that
the
nest
distributed
tracing
workshop
where
the
trace
context,
stuff
is
being
discussed,
and
that's
one
of
those
is
concurrent
with
pucon
in
Europe
in
May.
Second,
third
I
think
so
he
was
talking
about
trying
to
get
that
workshop
moved
a
couple
of
days
or
to
maybe
have
some
separate
thing
that
happens
just
on
either
just
before.
F
Just
after
the
coop
fun
thing
try
and
get
together
and
talk
a
little
bit
more
about
how
to
consolidate
a
bunch
of
API
is
dynaTrace
is
also
pretty
interested
in
being
involved
up
and
tracing
and
the
capacity
of
contributing
like
higher
level
higher
level
API.
Is
that
build
on
top
of
open
tracing?
Like
they
have
a
bunch
of
API
as
that
track,
individual
users
and
things
like
that
that
wouldn't
make
sense
that
open
tracing
this
layer
of
abstraction
would
make
a
lot
of
sense
from
level
up
from
that.
F
B
But
there
is
a
request
to
sort
of
bake
that
into
the
api's,
the
other
high
level.
Api's
might
be
useful.
Now
that
the
you
know,
lower
level
ones
are
sort
of
settling
into
place.
So
that's
that's
something
worth
worth
considering.
We
haven't
spent
much
time
focusing
on
tags,
since
we've
been
so
focused
on
the
api's
but
tags.
How
should
things
be
tagged?
What
counts
as
an
official
you
know,
HTTP
request
span
or
something
like
that.
D
B
Yeah
or
you
can,
but
maybe
they
I
don't
know
like
they're,
not
gonna,
help
you
automatically
slep
the
information
out
of
the
request.
Object
right.
It
would
be
more
just
shaped
in
such
a
way
that
there
was.
You
were
forced
to
extract
all
the
information.
This
thing
needed
to
tag
things
correctly,
yeah.
D
H
B
B
I
I
would
be
just
as
satisfied
with
very
clear
guidelines
and
then
ensuring
that
for
stuff
that
that's
in
ot
contribs,
like
a
review
to
prove
that
these
things
you
know
it's
just
like.
If
you
do
a
CD
B,
you
have
to
have
a
test
that
proves
that
you
know
you're
assigning
these
these
various
fields.
You
know
various
tags
and
things
to
it
and
hope
you
satisfied
with
that.
Other
people
actually
tests.
B
You
could
have
a
testing
tracer.
The
whole
job
in
life
was,
to
you
know,
help
verify
these
things
right.
You
know
it
seems
like
the
kind
of
thing
that
would
be
easy
to
to
maybe
make
some
standardized
tests
their
test
harness
runs
on
this
stuff,
like
no
it's
worth
thinking
about
where
we
just
have
a
review
process
of
being
like
things
can't,
you
know,
come
in
without
you
know,
reviewing
it
saying:
do
you
have
a
test,
as
the
tests
show
you've
tagged
this
properly.