►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Jaeger - Prithvi Raj Venkat Raj & Won Jun Jang, Uber
This session is dedicated to an in-depth understanding of the Jaeger project. We will give a short demo of the recently added features, talk about various topics including the architecture, adaptive sampling, multi-tenancy, and configuration, and review the roadmap for the upcoming year. After this session the attendees should better understand the Jaeger architecture and be ready to make contributions to the project.
To learn more click here: https://sched.co/FuLa
A
Good,
okay,
hi
hi,
everyone
thanks
for
coming.
My
name
is
one
and
down
there.
This
is
the
Jaeger
deep
dive
session,
so
both
me
and
prefeer,
both
and
I
are
software
engineers
at
uber
and,
more
importantly,
we're
both
maintainer
of
the
Jaeger
project.
So
before
we
get
started,
how
many
people
here
have
actually
used
here?
A
Okay,
okay,
I'm,
not
sure
why
there
are
so
many
people
at
the
deep
dive.
Okay
go!
So
here's
a
brief
overview
of
the
agenda.
So
hopefully
a
majority
of
you
on
attended
the
introduction
to
Jaeger
yesterday,
so
we'll
go
ahead
and
skip
the
demo
and
start
with
the
architecture.
Then
we'll
look
at
how
you
can
tune
the
performance
of
Jaeger
and
gain
observability
into
it.
Then
pretty
we'll
go
into
detail
over
the
new
Jaeger
features,
look
at
our
roadmap
and
then
cover
some
administrative
topics.
A
We'll
try
to
get
through
these
as
quickly
as
possible
to
leave
time
for
a
lot
of
questions.
Okay,
so
let's
just
jog
our
memory
of
and
recall
why
we
need
this
retracing
in
the
first
place.
So,
firstly,
rather
than
monitoring
individual
components
of
the
architecture
like
you
can,
with
metrics
and
logs,
we
can
use
distributed
tracing
to
monitor
distributed
transactions
across
multiple
components.
A
We
can
also
use
traces
to
detect
the
critical
path
and
do
performance
and
latency
optimizations,
so
the
critical
path
is
composed
of
the
shortest
path
from
the
start
of
the
trace
to
the
end
of
the
trace.
So
if
we
were
to
optimize
and
speed
up
all
the
spans
along
this
critical
path,
we
can
reduce
the
overall
latency
of
the
entire
distributed
transaction.
So
if
you
recall
the
demo
that
proof
he
gave
yesterday,
if
there
was
a
my
sequel
span
in
our
trace,
we're
about
90%
of
our
time
was
being
wasted.
A
So
using
that
information,
you
could
easily
have
gone
in
there
and
optimized
that
my
sequel
read
by
adding
like
the
bigger
thread,
pool
or
perhaps
even
adding
a
cache
in
front
of
it
and
reducing
the
overall
time
spent
on
that
read
so
I
recall
that
there
was
a
study
at
Amazon.
That
said,
if
you
were
to
increase
latency
by
100
milliseconds,
it
would
cost
them
1%
of
profit,
so
think
of
all
the
money
your
company
could
save
by
using
distributed
tracing
and
figuring
out
where
to
optimize
correctly.
A
So
getting
back
on
topic
with
some
post-processing
we
can
use.
This
should
be
tracing
to
generate
dependency
diagrams
that
allow
engineers
to
better
understand
the
system
that
they're
working
on
and
finally
distributed
tracing
provides
a
generic
distributed
context,
propagation
functionality,
which
everything
else
is
built
upon.
Ok,
but
you
guys
knew
all
that,
so,
let's
just
take
a
deeper
look
at
Jaeger,
so
I'm
gonna
fly
through
the
sidecut.
It
was
here
yesterday,
so
Jaeger
is
primarily
written
in
golang
and
we
have
a
couple
new
components.
A
In
Java
we
allow
pluggable
storage,
like
Cassandra
elasticsearch
and
in
memory
storage
to
persist
traces.
There
is
also
an
ongoing
effort
to
add
support
for
other
storage
engines.
The
UI
is
written
and
react
and
we
support
many
different
languages
with
different
our
clients.
We
currently
support
the
8
that
you
see
at
the
bottom
of
the
screen,
so
here's
what
the
architecture
looked
like
in
2017
when
we
first
open
sourced
the
project,
so
the
light
gray
box
on
the
left
side
is
the
host
your
application
is
running
in.
A
So
your
application
is
instrumented
with
the
open
tracing
API
and
you
use
the
Jaeger
client
library
as
a
concrete
implementation
of
that
API.
So
the
Jaeger
client
batches
bans
and
then
force
the
spans
to
the
local
Jaeger
agent
over
UDP.
The
Jaeger
agent
also
runs
on
the
same
host
as
your
application,
but
as
a
separate
process,
the
agent
then
buffers
the
spans
for
all
the
other
application,
also
running
on
the
same
host,
and
then
it
forwards
all
of
them
to
the
Jager
collector,
which
is
the
blue
box
in
the
middle
over
TCP.
A
So
we
chose
to
have
an
agent
per
host
rather
than
having
every
client
emit
spans
directly
to
the
Jager
collectors,
to
reduce
the
number
of
incoming
collect
connections
to
the
collectors,
since
the
number
of
applications
far
outnumber
the
number
of
physical
hosts.
Also,
we
want
to
take
advantage
of
using
you
to
reduce
the
amount
of
overhead
that
the
clients
have
on
your
application,
so
speaking
of
overhead,
so
this
is
a
common
question
that
I
get
all
the
time
so
I'm
just
gonna
address
it
here,
so
how
much
overhead
does
tracing
actually
add
to
your
application?
A
So
the
answer
is,
unfortunately,
it
depends.
So
if
your
application
has
a
lot
of
complex
business
logic
and
takes
in
the
order
of
hundreds
of
milliseconds
to
complete,
then
the
overhead
from
tracing
will
be
negligible
compared
to
the
amount
of
work
being
done
by
your
application.
However,
if
your
application
is
super
simple
and
only
takes
microseconds
to
finish,
then
tracing
will
add
a
lot
added,
a
lot
more
overhead
relative
to
your
application.
So
essentially,
what
I'm
trying
to
get
at
is
that
the
overhead
for
tracing
is
relative
to
the
application.
A
A
So
the
collector,
which
is
the
blue
box
in
the
center
of
the
screen,
is
in
charge
of
buffering
the
spans
in
memory,
sanitizing
them
and
then
storing
the
spans
in
a
data
store
like
Cassandra,
we
have
a
separate
Yaeger
query
service,
which
is
on
the
right
side,
and
that
is
used
to
read
traces
from
the
data
store
and
then
present
them
to
the
user
through
the
Yagyu.
I
additionally,
Jager
collectors
are
in
charge
of
configuration
distribution.
A
This
means
that
the
sampling,
probabilities
and
all
other
configurations
for
applications
are
all
stored
in
the
collector
and
then
forwarded
to
the
Jager
clients
via
the
Jager
agent.
So
this
is
shown
as
the
red
arrows
in
the
diagram.
So
yeah,
the
red
arrows,
show
the
controls
laughter.
Okay,
so
we
also
had
an
apache
spark
job
which
did
very
basic
post-processing
of
our
traces
and
it
generated
dependency
diagrams.
A
Okay-
and
this
is
what
the
architecture
looks
like
today-
there
hasn't
been
much
change
on
the
clients
and
agents.
Most
of
the
changes
have
occurred
in
the
backend,
so,
first
of
all,
our
architecture
back
in
2017,
just
gone
back
was
all
based
on
was
everything
was
a
sari.
Everything
was
a
synchronous,
the
meant
that,
during
a
load
spikes
we
were
not
able
to
keep
up
with
the
traffic
and
therefore
we
had
to
drop
and
lose
data
right.
So
we
now
have
an
asynchronous
ingestion
pipeline
that
allows
us
to
better
deal
with
these
load
spikes.
A
So
essentially,
we
traded
data
loss
for
increase
the
ingestion
latency,
so
we
now
buffer
spans
in
Kafka
and
have
a
separate
and
jester
service
that
handles
writing
the
actual
spans
to
Cassandra.
So,
in
addition,
since
now
we're
buffering
spans
in
Kafka
internally,
over
at
least
we've
been
able
to
run
a
span
processing
job
in
apache
flink
that
generates
dependency,
diagrams
and
extracts.
Other
features
pretty
we'll
be
going
into
more
detail
about
this
new
enzyme,
a
synchronous
ingestion
pipeline
and
the
new
Jaeger
streaming
pipeline
as
well
in
a
bit
sorry.
A
So,
additionally,
we
now
added
support
for
Jaeger
clients
to
directly
submit
spans
to
the
collectors
over
HTTP,
rather
than
going
through
the
Jaeger
agents.
So
this
feature
is
super
useful
for
applications
that
may
be
deployed
to
hosts
where
you
can't
run
the
Jaeger
agent
and
you
still
need
a
way
to
get
the
spans
to
the
collectors.
So,
for
example,
if
your
application
is
running
on
a
user's
desktop
emitting
spans
over
HTTP
is
the
only
option
you
have,
because
you
can't
install
the
Jaeger
agent
and
run
it
on
their
host.
Ok!
A
So
now,
let's
take
a
brief
look
at
the
model
for
Jaeger.
This
is
the
current
data
model
and
it's
based
on
Apache
thrift.
It's
written
in
Apache
first,
but
there's
currently
ongoing
work
to
move
it
to
a
protobuf.
So
the
most
basic
unit
of
data
is
the
span,
which
is
the
big
blob
right
in
the
middle.
It
contains
a
start
time,
duration,
logs
and
tags.
Each
span
belongs
to
a
trace
and
is
identified
by
the
trace
ID.
Additionally,
each
span
kind
of
references
which
can
be
used
to
model
causality
between
spans.
A
So,
for
example,
if
one
span
could
be
a
child
of
another
span,
this
simply
means
that
the
child
span
was
initially
initiated
by
the
parents
man.
So
we
then
combine
all
these
spans
into
something
called
a
batch,
which
is
the
structure
you
see
on
the
left
side
of
the
screen.
A
batch
is
just
the
simple
collection
of
spans
for
the
same
process,
so
you
can
think
of
a
process
as
a
server
or
an
application.
The
batch
data
structure
is
not
exposed
to
the
users.
A
It's
just
used
internally
as
a
normalization
tool
so
that
we
can
duplicate
the
amount
of
data
we
send
over
the
wire
so
to
break
that
down
a
bit.
Each
span
in
the
same
batch
all
are
associated
with
the
same
process
so,
rather
than
duplicating
a
process
for
each
span.
We
just
have
a
batch
that
just
contains
one
process
so
great
so
now,
let's
say
you
want
to
check
out
Jaeger
and
run
it
for
yourself.
How
do
you
go
about
configuring,
the
Collector's,
the
in
gestures
and
other
components?
A
So
we
use
a
open
source
project
called
Viper
which
allows
us
to
configure
Jaeger
with
both
command
line.
Flags
and
environment
variables,
we
also
support
kubernetes
config
Maps,
for
those
of
you
guys
deploying
with
Cuban
Ares.
So,
additionally,
the
UI
is
highly
customizable
and
it's
all
driven
by
configuration
you
can
easily
adjust
the
items
shown
in
the
UI
header,
add
Google
Analytics
token
to
get
usage
information
and
also
configure
spans
with
contextual
links.
So
contextual
links
are
a
new
feature
that
was
contributed
by
one
of
our
open
source
users.
A
A
Another
example
is
perhaps
you
could
add
the
add,
a
tag
to
your
span
with
the
build
version
that
you're
using
and
have
the
contextual
link
directly
link
you
to
the
actual
build.
So
you
can
see
the
latest
changes
or
from
the
UI.
So
this
feature
has
a
lot
of
potential
and
we're
very
interested
in
seeing
what
the
community
can
do
with
it.
A
Ok,
cool
and
so
Yaeger
also
provides
some
knobs
for
configuration
for
configuring
performance.
So
if
you
believe
that
your
Cassandra
can
take
more
writes
and
your
because
collectors
are
actually
your
bottleneck,
you
can
either
add
more
collectors
or
you
can
configure
the
number
of
workers
and
the
queue
size
of
the
individual
collectors
and
yet
expect
and
extract
more
performance
out
of
each
one
of
them
rather
than
adding
more
hosts
and
then
increase
the
throughput
this
way.
Additionally,
you
can
do
the
same
thing
at
the
agent.
A
These
are
all
configurations,
so
we
have
configurations
at
every
level
of
the
architecture,
so
you
can
easily
find
tune
the
performance
to
fit
your
hardware
specifications.
These
configurations
here
that
we
listed
I'll
just
just
example-
number
of
them,
but
we
also
have
a
lot
more
and
our
documentation
provides
more
knobs
at
your
disposal.
So
let's
say
you're
now
running
Jaeger
and
you
fine-tuned
everything
for
your
exact
needs
and
have
it
running.
How
do
you
observe
that
it's
actually
doing
the
right
thing?
A
B
Thank
you
one
for
that
introduction,
so
my
part
of
the
talk
would
be
composed
of
the
new
features
and
then
we'd
go
over
the
roadmap
and
then
we'd
have
time
for
Q&A
at
the
end.
So
here
we
go
so
one
of
the
things
that
we've
introduced.
Asou
of
these
new
features
are
out
and
released.
They've
been
at
use
at
uber
for
about
an
year.
Some
some
such
and
the
first
one
is
adaptive
sampling.
Now
in
any
given
API
any
service,
you're
running,
you
have
differences
in
terms
of
what
traffic
hits
water
endpoint.
B
What
that
means
is
a
get
endpoint
for
a
user
might
get
a
lot
more
traffic
than
a
post
endpoint.
Now
the
problem
with
this
is
that
sampling
works
like
this.
Something
sampling
is
a
decision
to
steal
the
trace
and
it
is
taken
at
the
first
edge.
So
whenever
the
request
comes
to
your
application,
a
sampling
decision
is
made
and
this
sampling
decision
is
propagated
throughout
your
application.
So
this
is
made
at
the
head
and
so
far
it's
been
made
as
a
call
in
probability.
You
might
do
it.
B
You
might
say
that
sample
one
in
every
10,000,
traces
or
10,000
requests.
So
when
you
are
seeing
a
distribution
like
this
meaning,
one
endpoint
has
lot
more
traffic
than
another
endpoint.
What
ends
up
happening
is
that
you
get
lot
more
traces
for
that.
Endpoint
that
receives
more
traffic
and
the
endpoint
that
receives
less
traffic
has
less
traces,
so
you're
collecting
different
amounts
of
observability
data
based
on
the
traffic.
Your
application
is
receiving.
Adaptive
sampling
helps
with
that.
So
adaptive
sampling
works
by
setting
a
trial
target
GPS.
B
So
we
do
some
magic
talk
to
us
like
do
you
figure
out
what
that
magic?
Is
it's
essentially
a
proportional,
integral
differential
controller
that
takes
in
the
target
key
PS
and
the
sample
QPS
and
dynamically
figures
out
the
sampling
probability,
and
this
is
done
fairly
quickly,
like
it's
updated
every
minute
or
so,
and
it's
tunable
how
quickly
you
want
this
up.
These
updates,
and
this
will
lead
to
a
constant
sampling
rate
so
that
you
observability
data
per
endpoint
is
roughly
the
same.
B
The
next
thing
that
we
have
introduced
this
asynchronous
injection,
so
this
feature
essentially
puts
Kafka
queue
between
the
collectors
and
the
writers.
So
we
are
focusing
on
this
part
of
the
architecture
right
in
the
middle,
so
we've
added
calf
kind
between.
So
what
this
allows
us
to
do
is
when
we
get
load
spikes
of
this
nature.
We
just
put
it
into
Kafka
and
we
read
out
at
nearly
a
constant
rate.
B
So
if
you're
running
a
storage
engine
like
Cassandra,
which
can
only
take
a
limited
amount
of
QPS
in
the
sense,
the
storage
engine
itself
doesn't
have
any
protection
from
load
spikes.
This
architecture
enables
you
to
control
the
rate
at
which
you
are
actually
writing
to
your
storage.
You
can
control
the
amount
of
ingest
instances.
You
can
also
separate
it
out
so
that
you
write
only
some
of
the
spans
into
Cassandra
or,
like
you,
can
do
lots
of
advanced
logic
on
the
receiving
end.
B
B
Other
services
that
have
been
removed
and
green
are
the
services
that
have
been
added
services
and
endpoints.
So,
at
a
glance
when
you
take
two
traces,
you
can
compare
the
shapes
of
these
traces
to
see
that
whether
they've
been
any
new
requests
made
by
your
architecture.
So
in
this
particular
example,
we
see
a
lot
of
green
at
the
bottom,
which
says
that
this
trace
contains
new
calls
that
were
not
made
in
the
previous
trace,
and
this
view
allows
you
to
visualize
that
in
a
really
user-friendly
manner,.
B
Also,
the
other
thing
you
could
do
is
you
could
look
at
trace
durations.
So,
instead
of
looking
at
just
the
number
of
calls
that
have
been
added
or
removed,
we
can
look
at
the
duration
differences
like
in
this
particular
case.
The
highlighted
node
has
an
additional
815
milliseconds
and
we
also
do
a
percentage
duration
of
that.
So
these
are
comparison
tools
for
traces.
B
So
next
we
go
on
to
the
roadmap.
So
one
of
the
things
that
I
mentioned
earlier
with
adaptive
sampling
is
that
the
sampling
decision
is
made
when
a
request
hits
your
service.
So
we
want
to
make
it
such
that
this
sampling
decision
is
not
made
at
the
beginning,
but
it's
made
at
the
end.
So
what
this
allows
us
to
do
is
collect
all
traces
in
all
spans,
forgiven
trees
and
then
decide
whether
we
want
to
stow
the
trace
a
lot.
B
So,
instead
of
making
the
decision
at
the
beginning
and
throwing
away
spans
that
we
do
not
need
like.
Oh
here
we
collect
all
the
spans
and
then
we
figure
out
based
on
any
sort
of
heuristics.
So
this
is
on
the
roadmap.
Data
pipeline
is
something
we
are
using.
We
haven't
yet
open
sourced
it.
We
use
it
internally,
so
data
pipeline
is
just
reading
spans
from
Kafka.
We
create
recession
eyes
to
spans
into
traces,
and
then
we
run
algorithms
on
it
to
figure
out
interesting
data.
B
B
Another
thing
we
are
working
on:
we
have
an
open
pool
request
for
this.
An
open
issue
is
the
ability
to
store
and
retrieve
incomplete
spans,
so
in
a
this
architecture
today,
spans
are
only
written
to
storage
when
they
are
complete.
So
if
an
application
is
processing
something
for
a
very
long
time,
we
do
not
get
any
observability
information
from
that
application
until
that
process
is
completed.
Imagine
something
that
imagine
an
application
that
runs
over
multiple
days
and
mate
requests
over
multiple
days.
B
Finally,
if
we
go
into
this
section
about
project
and
community,
it's
mainly
a
bunch
of
links.
It's
it's
fairly,
simple.
Most
of
our
code
isn't
go.
All
of
the
clients
are
in
the
specific
client
languages.
The
data
pipeline
is
in
Java,
so
we
look
for
100%,
test
coverage
and
signed
commits.
All
of
this
is
on
our
github
page,
so
feel
free,
and
these
are.
These
are
useful
links.
Finally,
QA.
B
C
C
D
C
A
D
A
So
in
the
future,
we're
gonna
provide
a
new
configuration
so
that
you
can
just
use
that.
Then
you
just
set
a
one
target,
QPS
right
and
that
target
QPS
is
set
for
all
your
services
so
that
all
operations
will
be
sampled
at
that
QPS
right
and
that's
all
you
have
to
provide
in
the
future.
So
it
won't
be
a
static
anymore.
It'll
be
dynamic,
but
you
don't
have
to
provide
per
service
samples,
I.
D
Stream
ingestion
most
of
the
time
took
today.
We
talked
about
a
lot
of
the
in
gesture
for
the
streaming
like.
What's
the
purpose
for
that
streaming,
because
that's
a
new
architecture
compared
to
the
old
one.
So
what's
that
streaming
can
be
used
like?
Can
we
do
like
back
and
forth
to
search
different
streams
for
something.
B
I
could
take
that
so
the
streaming
essentially
reads
from
the
Kafka
topic,
and
so
we
add,
the
data
Jaeger
output
expands
so
full,
like
it
converts
bands
into
traces.
A
trace
is
a
collection
of
spans,
but
is
only
forgiven
requests.
Now
that
is
difficult
to
do
in
a
non
streaming
architecture,
because
we
do
not
know
when
the
request
completes
so
like
we
do
not.
B
Data
might
still
be
arriving,
so
we
use
the
streaming
job,
firstly,
to
convert
this
pans
into
traces,
and
once
we
do
that
we
can
run
any
sort
of
algorithms
on
that
trace
itself.
So,
as
of
now,
what
we
do
is
like
two
dependencies.
So
what
that
means
is
we
compute
service,
a
called
service,
B
service?
We
call
service
C
and
then
we
aggregated,
so
we
get
a
call
count.
So
we
can
do
things
like
in
in
one
hour
how
many
times
that
service
a
call
service
B
and
we
can
run
any
sort
of
heuristic
jobs.
B
D
B
F
F
B
Long
as
you
write
it
in
the
same
format,
so
the
format
meaning
the
data
model
that
one
showed
earlier.
So
as
long
as
you
follow
that
data
model
and
you
supply
deserialize
err,
we
should
be
able
to
do
it,
so
the
serialization
and
deserialization
is
pluggable.
So
it's
an
interface,
you
can
add
whatever
you
want
and
if
you
are
able
to
write
a
DC
realizer
and
then
you
can
just
put
it
in
whatever
format
you
want
and
yeah
the
other
pipeline
should
work.
Thank
you.
B
B
Pretty
much
so
it's
still
very
much
in
the
Alpha
sort
of
like
weird.
We
haven't
yet
started
work
on
it,
but
ideally
we'd
have
users
determine
whatever
they
want,
so
any
sort
of
policy.
So
if
you
want
to
do
things
like
stir
this
pan,
if
there's
an
editor
or
store
this
pan,
if
this
service
was
involved
and
the
latency
was
greater
than
400
milliseconds
or
any
sort
of
very
fine
grained
settings,
you
should
be
able
to
apply
that
policy
because
out
works
is
it
collects
everything
and
then
the
policies
evaluated
before
story?
Thank.