►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro: Jaeger - Prithvi Raj Venkat Raj & Won Jun Jang, Uber
This session is an introduction to Jaeger and distributed tracing. We will do a short demo of the current Jaeger features, talk about the roadmap for the upcoming year, and finish with a Q&A. After this session the attendees should better understand how Jaeger fits in the observability space for cloud native applications. For more information on the project everybody is welcome to attend the Jaeger Deep Dive Session.
To learn more click here: https://sched.co/FuLC
A
Hello
and
welcome
to
the
introduction
to
mega
talk,
I'm
1-
and
this
is
pretty
we're
both
software
engineers
at
uber,
the
company
that
lost
the
DD
and
are
they
translating
my
joke?
Ok,
whatever
and
more
importantly,
we
are
maintenance
of
the
Jaeger
project.
Ok,
so,
as
for
the
agenda,
I'll
do
an
introduction
to
distributed
tracing
to
get
everyone
in
the
room
familiarize
with
the
problem
area
that
Jaeger
attempts
to
solve
pretty
will
then
run
in
a
demo
of
Jaeger
and
show
how
it
can
be
used
to
debug
certain
observability
issues
within
a
micro
services
architecture.
A
Ok
I
count
three
hands:
ok,
so
ok,
so
cool,
so,
okay!
So
for
the
newcomers,
let's
just
go
over
the
key
concepts
and
terminology
of
distributed
tracing
to
get
everyone
in
the
room
on
the
same
page.
Ok,
so
why
do
we
need
to
distribute
tracing?
In
the
first
place,
this
colorful
visualization
is
the
service
dependency
diagram
of
a
subset
of
micro
services
over.
So
this
means
that
every
node
in
this
graph
is
a
service
and
every
edge
between
two
services
mean
that
those
two
services
depend
on
each
other
uber.
We
have
over
4000
micro
services.
A
A
How
do
you
narrow
down
which
one
of
your
dependencies
is
actually
the
root
cause
of
the
problem
so
distributed
tracing,
can
assist
you
with
narrowing
down
your
search
space,
pretty
we'll
be
demoing
this
capability
in
a
short
while,
so
what
else
can
distribute
your
tracing
be
use
be
useful,
so
it
can
be
used
to
monitor
transactions
and
micro
services.
Let's
say
you
have
a
transaction
that
is
just
isolated
to
a
single
service
like
the
diagram
has
on.
A
There
are
already
existing
tools
to
observe
this
transaction
like
this,
and
you
could
use
login,
for
example.
So
if
you
do
logging,
when
a
transaction
first
starts,
you
can
pass
around
the
transaction
ID
and
use
it
whenever
you
log
something
you
can
then
piece
together
the
timeline
for
the
distributed
transaction
via
some
processing
by
searching
for
all
logs
with
that
transaction
ID.
So
this
very
simple
mechanism
will
work
for
most
both
basic
concurrency
and
asynchronous
concurrency
as
long
as
you
add
the
ability
to
pass
around
this
transaction
ID
throughout
your
code.
A
However,
once
we
move
to
a
micro
services
architecture
with
distributed
concurrency
simply
passing
around
the
transaction
ID
within
a
single
service
won't
get
you
the
entire
picture.
We
now
need
a
mechanism
that
can
propagate
this
transaction
ID
across
services
service
boundaries.
So
we
we
can
piece
together.
The
timeline
of
the
distributed
transaction
so
distributed
tracing
solves
this
issue
with
context
propagation.
So
what
is
context
context
is
just
a
metadata
that
is
passed
around
between
services.
This
metadata
includes
a
unique
ID
that
can
be
used
to
identify
the
distributed
transaction
on
the
left.
A
Here
you
can
see
a
call
graph
where
every
node
represents
a
service
and
the
directed
edges
between
them
signify
requesting
so
service
a
receives
a
request
from
some
external
service
and
it
generates
a
unique
ID
and
inserts
it
into
a
context
object.
The
context
is
then
propagated
along
with
along
two
other
requests:
sorry-sorry.
A
Okay,
so
the
context
is
then
propagated,
along
with
request
to
other
services
here
service,
a
called
service
B.
It
will
also
include
the
context
inside
the
payload
so
to
visualize
the
work
done
by
individual
services
within
this
transaction.
We
use
a
trace
view
as
seen
on
the
right,
unlike
the
call
graph
on
the
Left,
where
all
we
can
see
is
the
causality
between
the
services.
The
trace
view
allows
us
to
see
both
causality
and
the
amount
of
work
done
by
each
service
in
the
graph.
A
So
trace
is
composed
of
multiple
spans
and
each
span
has
a
start
time
a
finished
time
logs
tags
and
represents
the
work
done
by
a
service
within
the
distributed
transaction.
Here
we
can
see
that
service
a
took
the
longest
time.
To
finish,
we
can
also
see
that
service
a
is
calling
service.
B,
which
in
turn
is
calling
service
C,
but
then
service,
B,
wait
until
service
C
is
finished
and
only
then
makes
a
request
to
service
D
and
so
on
and
so
forth.
A
So
one
last
thing
I
need
to
mention
before
the
demo
is
that
is
open
tracing,
so
you've
probably
heard
about
open
tracing
at
this
conference,
since
it's
also
a
CNC
F
member
project,
it's
a
vendor-neutral
instrumentation
API
that
allows
context
propagation
and
distributed
tracing,
amongst
other
things.
So
as
long
as
your
services
are
instrumented,
with
this
API,
you
can
plug
in
different
open
tracing
compatible
tracers,
like
Jager
light
step
and
sky
walking
without
having
to
rewrite
all
your
instrumentation
code.
Okay,
okay,
so
I'll
stop
speaking
now
and
let
proceed
do
them.
B
B
B
B
So
oh
he'll
be
launching
the
demo
application
called
hot
rod,
so
I
would
direct
your
attention
to
its
logs.
So
now,
when
you
look
at
these
logs
here,
like
you
see
the
word
service
with
route
service
front,
end
service,
customer
and
service
driver
now,
these
logs
immediately
tell
us
that
this
demo
application
consists
of
multiple,
smaller
services.
B
In
this
case,
I
think
we
are
most
interested
in
the
front
end,
because
the
front
end
is
usually
the
entry
point
for
these
apps.
So
let
me
click
on
this
address,
so
the
front
end
takes
us
to
this
particular
website.
Now
let
me
give
you
an
introduction
about
hot
rod.
Hot
rod
stands
for
rides
on
demand,
and
it
is
this
simulation
of
a
ride-sharing
app
like
say,
uber
or
DB,
that
allows
people
allows
customers
to
request
vehicles
for
a
given
location.
So
we
only
support
these
four
locations
in
this
app.
B
Now,
when
we
click
on
one
of
these,
you
see
it
says
it's
dispatching
an
app
and
then
a
car,
and
then
here
we
see
this
number,
which
is
a
new
york-style
number
plate.
It's
a
car
number
plate
and
it
says
that
it's
arriving
in
some
amount
of
time,
so
this
application
goes
through,
looks
at
the
nearest
cars
and
sends
one
of
these
cars
to
the
customer.
Some
of
the
things
that
are
important
to
note
in
this
we
have
this
web
client
ID.
B
Now
this
is
a
unique
ID
ID,
that's
given
by
the
JavaScript
on
the
front-end.
If
I
refresh
this
browser
window,
you'll
see
a
different
ID
and
we
have
request
IDs,
so
this
is
also
by
the
front-end,
the
latency,
as
measured
on
the
front-end.
Now
that
we've
done
this,
we
go
to
the
Jager
UI.
Now
it's
very
interesting
that
we
haven't
really.
We
haven't
really
seen
what
this
application
is.
So
we
know
that
this
application
consists
of
many
parts,
but
we
don't
really
know
how
these
parts
relate
to
each
other.
B
What
the
architecture
is
now
recall
that
we
already
did
one
requester
here
now.
This
sends
data
spam
data
to
Jager,
so
Jager
captures
that
data
and
it
stores
it
and
allows
you
to
visualize
it.
So
when
you
go
to
the
dependencies
tab
here,
so
we
can
already
see
that
the
front-end
service
called
three
services,
customer
driver
and
route
and
customer
calls
my
sequel
backing
so
driver
calls
Redis.
So
already
we
are
gaining
an
understanding
of
how
these
services
communicate
with
each
other.
So
we've
only
made
one
request.
B
B
It's
also
collected
today
a
few
minutes
ago.
So
all
of
this
is
just
an
information
screen
right.
We
also
have
this
display
here
that
shows
like
how
much
time
this
tick
it's
about
700,
something
millisecond.
Let
us
click
into
this.
So
when
we
click
into
this,
we
immediately
get
a
timeline
view.
So
here
we
see
that
we
can
just
go
through
this.
B
We
see
that
this
is
front-end
service,
that
called
a
customer
service
which
calls
my
sequel
once
we
collapses,
we
see
that
the
front-end
service
calls
driver,
dot,
find
nearest,
and
then
we
see
get
route
being
called
many
times.
So,
if
you
look
at
this,
this
is
just
a
timeline
view
saying
what
we
do
not
know
what
the
business
logic
inside
is,
but
we
know
how
long
each
component
took
and
how
they
interacted
with
each
other.
B
So
this
is
very
contextual
information.
This
is
all
from
one
craze.
It's
collected
across
multiple
services
and
it's
all
presented
to
you
in
one
screen,
so
it
tells
you
a
story
of
what's
happening
in
your
service
architecture.
Now,
if
we
click
into
this,
we
see
these
logs
and
these
logs
are
very
contextualized.
These
are
pure
request
logs.
So,
looking
at
these
logs,
we
can
see
that
there's
a
HTTP
request
received
the
was
getting
a
customer,
and
then
this
is
search
for
the
nearest
drivers
and
monsters
that
search
is
completed.
B
We
do
we,
the
application,
finds
the
most
efficient
route,
and
once
it's
done,
it
finally
sends
the
dispatch
which
is
sending
a
car
to
the
customer.
Now
we
see
all
of
this
in
this
logging
information.
This
is
in
contrast
with
the
logs
over
here
now.
These
are
the
same
logs
from
all
these
services
for
just
one
request,
and
it
is
very
difficult
to
make
out
what's
going
on
from
here,
as
opposed
to
looking
at
the
trace
view
where
everything
is
in
context
now
I'm
going
to
dig
deeper
into
this
timeline
to
explore
more
interesting
things.
B
So
we
have
this
logo
here
and
when
you
go
down,
we
see
front-end
called
customer
which
calls
my
sequel,
and
then
we
see
a
front-end
calling
the
driver
service
and
the
driver
service
calls
Redis.
One
of
the
interesting
things
is,
you
can
annotate
errors,
so
this
exclamation
mark
or
here
means
that
there's
a
error
in
this
path,
and
we
can
dig
in
to
see
what
that
error
is
like.
We
see
that
there's
a
Redis
timeout,
so
it's
very
informational
that
way
and
users
can
choose
to
like
report
errors
at
any
granularity
that
they
want.
B
We
also
have
for
things
like
my
sequel.
You
can
actually
put
in
the
my
sequel
query
right
inside
your
span,
so
that
you
know
exactly
what
query
was
executing
at
that
point
so
going
down
like
we
might
want
to
see
how
the
performance
of
this
app
or
like
we
might
want
to
learn
like
what
is
actually
going
on
when
someone
calls
a
dispatch-
or
here
we
see
that
driver
calls
read
is
many
times,
and
it's
very
easy
to
notice
in
this
example
that
each
of
these
calls
only
happens
after
the
previous
call
completed.
B
Looking
at
this
frampton
service,
it
also
seems
like
three
of
these
calls
happen
in
parallel
and
then
the
next
three
and
then
the
next
three
and
so
on.
So
also
this
gives
you
like.
We
can
see
clearly
see
this
pattern
right
here,
and
this
also
gives
us
this
notion
of
whether
the
resource
constrained
on
some
other
on
some
other
resource
right,
like
maybe
there's
an
executor
pool
with
only
three
threads
and
the
trace
timeline
view-
gives
you
this
nice
visualization.
B
That
allows
you
to
dig
in
very
deep
into
a
single
request
and
find
out
how
that
request
is
going
on.
The
key
point
is
all
of
this
is
empirical
data.
This
is
actual
data
that
the
app
is
sending
to
Yaeger.
So,
let's
go
back
to
our
simulation
now
I'm
curious
about
how
the
simulation
works.
When
the
multiple
requests
right
now,
we
have
only
seen
one
request
so
I'm
going
to
click
on
something
many
times.
So
if
you
see
the
request,
it's
just
increasing.
So
let
me
quick
some
more.
B
So
we
have
a
lot
of
requests
and
if
you
see
that
if
you
notice
the
latency,
our
initial
request
was
only
759
milliseconds
and
our
final
request
is
almost
three
seconds,
so
it
is
increasing
so
clearly
there's
something
funny
going
on
with
the
app
so
to
see.
What's
going
on,
I'm
going
to
select
one
of
these
one
of
these
number
plates
and
going
to
the
Jaeger
UI
I'm,
going
to
search
via
a
tag.
B
So
well
so,
on
the
left
side
there's
a
search
functionality,
you
can
search.
We
are
key
value
pairs,
so
it's
just
key
equal
to
value
and
then,
if
your
instrumentation
tags
it
correctly,
you
could
just
get
that
particular
results.
So
we
see
over
here
that
this
particular
trace
take
one
point:
seven
nine
seconds:
let
us
dig
into
it
now
going
inside
this.
My
sequel
select
is
almost
taking
one
point
three
nine
seconds
and
clearly
something
is
wrong
here
right
and
we
look
at
it
as
just
go
into
this.
B
You
see
that
this
is
a
select
query.
The
Select
looks
normal,
but
let's
look
into
the
logs
now.
The
interesting
thing
here
is
like
we
see
that
this
particular
log
line
says
that
it's
waiting
for
a
lock
behind
full
transactions
right
and
this
happens
and
the
lock
is
acquired
only
after
one
whole.
Second,
now
open
tracing
allows
us
to
create
rich
instrumentation
that
exposes
these
details
on
to
logs,
and
in
this
case
this
could
potentially
be
a
database.
That's
configured
to
only
use
one
serial
connection
or
like
only
one
or
something
behind
the
queue.
B
So
ideally,
the
next
step
you
take
is
look
into
what
is
going
on
here.
The
other
interesting
thing
here
is
that
you
see
this
blockers
and
these
blockers
are
request
IDs
from
this
page,
so
these
request
IDs
are
available
in
the
in
the
UI
and
your
app
knows
about
it.
The
interesting
thing
is
your
storage
layer
knows
about
it,
the
storage
layers.
Looking
at
the
dependencies,
the
storage
layers,
three
levels,
deep,
it's
the
is
the
thing
at
the
bottom.
A
B
B
A
Checking
check
okay!
Thank
you.
Okay,
thanks.
Let's
see
sorry
thanks
pretty
for
the
demo.
So,
as
you
can
probably
tell
mine
now,
Jaeger
is
an
open
source.
Open
tracing
compatible
distributed
tracing
system,
so
Jaeger
was
inspired
by
Google's,
dapper
and
open
Zipkin,
and
we
started
out
over
in
August
2015.
We
open
sourced
the
project
in
April
2017
and
it
was
accepted
as
an
official
CN
CF
member
project
in
September
2017.
A
So
let's
just
take
a
brief
look
at
the
tech
stack.
The
Jaeger
back-end
is
written
in
:
and
we
support
pluggable
storage
engines
such
as
Cassandra
elasticsearch
and
other,
and
an
in-memory
storage
to
persist
traces.
The
front-end
is
written
in
JavaScript
and
it
uses
the
react
framework.
We
support
many
different
languages
and
have
client
support
for
all
the
languages
listed
on
the
bottom
here.
So
one
of
the
awesome
things
to
note
is
that
the
PHP,
the
c-sharp
and
the
Ruby
Jaeger
clients
were
all
community
contributed.
A
So
speaking
of
community,
we
have
full
ten
full-time
engineers
at
both
uber
and
Red
Hat
working
on
Jaeger.
We
have
sixty
contributors
to
the
main
Yaga
repository,
but
if
we
were
to
include
all
the
sub
repositories
like
client,
libraries
and
such
then,
we
would
have
a
total
of
200
plus
contributors.
We
also
have
many
organizations
using
Jaeger
for
day-to-day
debugging
as
listed
here.
If
your
organization
is
already
using
Jaeger
in
production,
let
us
know
and
we'll
gladly
add
you
to
the
list.
A
Additionally,
we
can
use
traces
to
do
analysis,
which
is
integral
for
companies
like
uber,
where
each
request
will
sometimes
fan
out
to
hundreds
of
different
services.
We
can
also
use
traces
to
detect
a
critical
path
and
do
performance
and
latency
optimizations.
So
in
the
demo
we
saw
previously,
we
saw
that
my
sequel
was
taking
in
the
longest
time.
We
could
have
gone
in
there
and
we
could
have
tried
to
optimize,
as
that
was
where
the
bottleneck
was.
A
Furthermore,
with
some
post-processing,
we
can
generate
dependency
diagrams
that
allow
engineers
to
better
understand
the
system
that
they're
working
on
and
finally
distributed
tracing
provides
a
generic
distributed
context,
propagation
facility,
upon
which
everything
else
is
built.
Ok,
so
these
are
some
some
of
the
recent
highlights
for
the
Jaeger
project,
so
I'm
just
gonna
gloss
over
them.
A
Here
we
have
adaptive
sampling,
we
have
asynchronous
ingestion,
trace
comparisons
and
Jaeger
operator,
so
we'll
actually
do
a
deeper
dive
into
each
one
of
these
new
features
tomorrow
during
the
deep
dive
session,
since,
if
I
go
over
all
of
them
today,
none
of
you
guys
would
show
up
tomorrow.
So
ok,
so
let's
take
a
brief
look
in
at
the
project
roadmap
so,
as
in
the
previous
slide,
I'm
only
just
going
to
gloss
over
some
of
the
new
features
here
and
we'll
in
the
deep
dive
tomorrow
we'll
go
through
and
go
through
all
the
features.
A
Ok,
so
one
of
the
features
I'd
like
to
talk
about
is
a
called
post
tree
sampling
and
it's
something
that
we're
very
interested
in
so
currently
Jaeger
samples,
traces
based
on
a
default
probability
of
one
in
1,000.
This
means
that
if
there
are
1,000
requests,
we
expect
that
we
expect
it
to
be
at
least
one
request
that
is
actually
traced
and
sampled
and
saved
in
Jaeger.
This
means
that
we
have
a
high
probability
of
missing
interesting
and
anomalous.
A
Traces
with
post
trace
sampling,
we'll
sample
all
the
requests
and
then
cherry-pick
only
the
interesting
traces
based
on
certain
heuristics.
These
heuristics
can
be
as
simple
as
keeping
a
histogram
of
trace,
durations
and
then
just
cherry-picking,
the
p1
and
the
p99
duration,
traces
or
counting
the
number
of
errors
in
a
single
trace
and
only
saving
those
that
cross
a
certain
threshold.
A
So
we
have
the
data
pipeline.
This
is
based
on
Kafka
and
a
pan
chief
link,
which
is
a
stream
processing.
Your
air
system,
the
data
pipeline,
allows
us
to
support,
feature
extraction,
trace,
aggregations
and
data
mining.
This
is
a
huge
feature
because
it,
although
individual
traces,
are
useful,
as
you
saw
in
the
demo
aggregations
of
traces,
allow
us
to
build
models
that
can
be
used
to
pinpoint
anomalous
traces
and
automatically
surface
them.
To
the
engineer
you
just
can't
do
that
by
looking
at
single
traces
in
isolation.
A
So
another
way
of
phrasing-
this
is
that
single
traces
are
relatively,
are
sorry,
so
single
traces.
Are
we
actively
looked
up
by
engineers
only
after
that
davisc
only
after
they
discover
a
problem?
On
the
other
hand,
once
you
have
aggregations
of
traces,
you
can
automatically
detect
anomalies
and
then
proactively,
let
users
know
of
degradation
of
their
services.
So
a
lot
of
this
is
already
live
at
uber
and
will
be
open
sourcing
all
of
this
shortly.
A
So
then,
just
the
last
feature
I'll
touch
upon
is
incomplete,
span
support,
so
at
the
moment,
spans
are
only
emitted
once
when
the
service
is
finished.
With
its
with
its
work.
However,
some
services
have
workflows
that
are
hours
long,
which
means
that
the
users
have
zero
observability
into
ongoing
requests
until
the
entire
workflow
is
finished.
A
So
this
new
feature
allows
you
to
incrementally
build
spans
and
view
them,
as
the
workflow
is
running,
giving
you
greater
insight
into
these
longer
workflows
so
again,
we'll
go
into
a
deeper
dive
into
all
of
these
topics
tomorrow,
but
I
just
wanted
to
cover
them
here:
okay
and
okay-
and
that's
all
we
had
for
today,
thanks
for
coming
so
are
there
any
questions?
I
can
answer.
We
can
answer.
B
B
A
So,
just
just
one
last
slide,
so
our
deep
dive
session
is
tomorrow.
So
if
you
were
interested
in
any
of
the
topics
that
I
mentioned,
it's
in
the
same
room
here
at
220
and
we
hope
to
see
you
there
and
also,
if
you're
interested
in
other,
how
other
organizations
are
using
open
tracing,
there
is
a
talk
that
started
at
the
same
time
as
this
talk
and
is
ending
right
now.
So
if
you
have
a
time
machine
you
could
you
can
go
back
and
watch
that?
Okay,
thank
you.