►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Jaeger Intro and Deep Dive - Prithvi Raj, Uber & Joe Elliott, Grafana Labs
In this session we will cover both introductory and deep dive material for the Jaeger distributed tracing backend. For intro we will review methods of getting started installing Jaeger and the basics of distributed tracing. For the deep dive we will be discuss the future of Jaeger built on top of the OpenTelemetry collector and what that means for Jaeger architecture, future development and features. Jaeger is the most popular open source distributed tracing backend. Whether your goal is to get acquainted with Jaeger and distributed tracing or to keep abreast with the latest and greatest, please join us!
A
So
what
is
jaeger
you
use
a
open
source
distributed
racing
platform
is
among
the
most
popular
ones,
jacob
graduated
from
cncf,
and
it
was
started
at
uber
around
2015,
or
so
it
was
shortly.
It
was
open-sourced
in
a
couple
years
later,
what
they
about,
who
am?
I
I'm
fritz
viraj,
I'm
a
jaeger,
maintainer
I've
been
working
with
diego
since
2015
or
so,
and
I
continue
to
work
with
jaeger
at
uber.
Sadly,
most
of
the
work
I
do
these
days
is
a
specific
tube
is
unable
to
contribute
to
open
source.
A
A
A
So
when
someone
requests
a
ride
or
someone
is
looking
up
for
receipt
or
something
it
might
involve,
a
large
amount
of
services
being
touched
or
being
requested,
data
off-
and
this
happens
very
frequently,
so
this
happens
billions
of
times
a
day.
So
in
a
complicated
system
like
this
and
tracing,
really
shows
its
value.
A
A
So
on
this
on
the
left,
we
have
the
service
name,
and
then
we
have
the
operation
name
in
grey
and
on
the
right.
We
have
this
rectangle,
which
represents
the
time
taken
for
this
operation.
This
is
called
a
span
in
tracing
terms.
It
is
a
smallest
unit
of
operation
that
you
can
represent
it's
a
small
logical
unit
on
the
top.
Here
we
have
a
gantt
chart
and
this
allows
users
to
like
focus
and
zoom
into
particular
regions
of
the
trace
on
the
top.
A
A
Another
way
of
visualizing
this
trace
or
a
different
race
to
say,
is
by
using
the
single
trace
service
dependencies
view.
In
this
particular
view,
we
see
that
a
particular
service
calls
a
bunch
of
other
services
which
call
a
bunch
of
other
services,
so
on
and
so
forth,
like
this
view
allows
one
to
orient
themselves
within
a
trace,
and
this
is
really
useful
when
the
architecture
is
both
wide
and
deep.
In
this
particular
case,
we
see
that
the
depth
is
16,
so
this
16
services
deep
and
the
32
services
involved
in
total.
A
So
in
this
sort
of
situation
it
may
be
useful
to
find
out
like
what's
going
on
or
how
these
services
interact
with
each
other.
One
of
the
interesting
things
with
the
microservice
architecture
is
the
notion
of
like
things
going
wrong
and
the
notion
of
detecting
how
things
go
wrong
for
anyone.
Who's
worked
with
microservices.
A
You
know
that
like
finding
where
something
is
going
wrong
is
really
really
difficult,
like
things
might
be
broken
in
a
particular
service
only
for
some
requests-
or
we
might
not
even
be
able
to
detect
it,
using
traditional
tools
like
logs
and
metrics.
Some
categories
of
things
that
are
undetectable
by
logs
and
metrics
are
things
like
a
service
waiting
on
its
dependent
service
to
respond
or
service,
making
a
larger
number
of
calls
to
dependent
services
than
is
required.
A
A
We're
going
to
transition
into
a
high
level
of
how
tracing
works
now
tracing
depends
on
context
propagation.
Now
how
context
propagation
works
is
when
a
request
comes
in
through
edge
service,
and
this
might
be
an
api
gateway,
or
some
such
it
is
stamped
with
a
unique
id.
A
trace
id
and
this
trace
id
might
contain
additional
fields,
and
this
is
called
a
context.
A
This
context
is
then
propagated
to
every
service
in
the
call
chain
by
using
headers
or
any
other
similar
mechanism.
The
main
thing
this
context
allows
tracing
systems
to
do
is
to
assign
causality.
So
in
this
particular
case,
the
tracing
system,
jaeger
would
be
able
to
deduce
that
b
was
called
by
a
and
by
nothing
else.
A
On
the
right
side
of
this,
we
have
a
gantt
chart
view.
This
is
representing
the
same
data
that
is
on
the
left
of
the
screen,
but
I
hear
there's
a
time
dimension,
so
this
latency
and
a
start
time,
that's
added
to
these
spans
spans
might
also
contain
any
additional
information,
so
they
might
contain
logs.
They
might
contain
key
value
pairs
known
as
tags
and
you're
just
limited
by
your
imagination.
What
you
can
put
in
here.
A
A
So
what
is
sampling
sampling
is
just
a
notion
of
saying.
Only
some
percentage
or
some
population
of
traces
are
going
to
collect
data
and
will
be
stone.
Now
they
mult.
There
are
several
different
ways
of
doing
sampling,
but
the
most
common
and
easiest
one
to
do
is
head
head-based
sampling,
where
every
request
has
a
sampling
decision
that
is
made
and
the
sampling
decision
is
propagated.
A
A
So,
essentially,
these
client
libraries
generate
spans
and
these
fans
go
to
their
trace
collection
back
end
and
the
visualization
front
end
retrieves
these
fans
from
the
trace
collection
back
in
and
visualizes
them
as
traces.
This
includes
the
gantt
chart
view
the
dependency
view,
etc.
Optionally,
it
jaeger
may
be
deployed
with
a
data
mining
platform
which
can
generate
dependency
graphs
of
services.
A
A
Specifically,
when
we
look
at
this,
we
see
this
remote
sampling,
remote
sampling
allows
jaeger
collector
to
control
sampling
rate,
and
this
sampling
rate
is
read
by
jaeger
clients,
and
this
really
allows
your
clients
to
sorry.
This
really
allows
a
year
collector
to
control
the
amount
of
traffic
that
your
clients
and
these
applications
send
to
it.
A
So,
for
instance,
if
you
have
a
sampling
rate
of
100,
it
means
that
your
client
on
every
service
is
going
to
send
hundred
percent
of
spend
to
your
collector
once
was
like,
if
you
have
something
like
zero
or
something
like
one
in
thousand
like
you,
the
traffic
that
is
being
sent
to
jaeger
collector
is
really
reduced.
So
this
remote
sampling
is
a
really
important
configuration
that
you
should
know
that
allows
you
to
control
the
amount
of
traffic
that
is
being
handled
by
the
remainder
of
your
system.
A
B
Quick
quick
information
about
me:
I'm
a
jaeger
maintainer
who
works
at
your
final
labs,
primarily
on
our
distributed
tracing
infrastructure,
and
I
enjoy
riding
my
bike
quite
a
bit.
That's
a
picture
from
my
local
park.
I
ride
past
that
fountain
multiple
times
a
week
and
my
twitter
handle
is
in
the
bottom
you're
welcome
to
give
me
a
follow.
I
mainly
just
tweet
about
distributed
tracing
stuff,
so
jaeger
itself,
quick
primer.
We
all
know
what
that
is.
B
I
hope
it's
a
distributed,
tracing
back-end
that
allows
us
to
do
this,
which
is
visualize
a
request
as
it
passes
through
our
infrastructure
as
it
passed
through.
You
know
all
of
the
many
services
that
are
used
to
answer
a
request
can
often
be
difficult
to
understand.
You
know
where
spending
time
distribution
kind
of
gives
us.
Those
answers
shows
us
all
the
different
pieces
and
where
the
time
was
spent
as
well
as,
of
course,
we
can
attach
additional
metadata.
B
B
It's
primarily
maintained
by
jp
crolling
who's,
a
red
hat
engineer
and
a
long-term
jaeger
maintainer
and
there's
two
important
links
here.
If
you
get
involved
with
the
operator,
the
github
repo
itself,
as
well
as
the
docs,
which
are
fantastic-
and
I
spent
quite
a
bit
of
time
in
when
I
was
preparing
this
presentation.
B
So
let's
talk
about
what
we're
doing
today
we're
going
to
talk
about.
You
know
how
to
use
the
the
operator
to
make
a
cluster
to
configure
our
storage
as
well
as
some
other
configuration
options,
we're
going
to
look
at
we're
going
to
talk
about
how
to
deploy
the
agent
and
the
strategies
available
in
remote
sampling,
which
is
an
awesome
feature
of
jaeger,
as
well
as
auto
scaling
more
and
more
quite
a
bit
more.
B
In
fact,
this
presentation
is
not
capable
of
covering
all
of
the
you
know,
features
of
the
operator,
but
we'll
try
to
touch
on.
At
least
you
know
what
it
can
do
for
you
kind
of
at
the
end
that
we
we
don't
cover.
What
we're
not
going
to
cover
in
this
presentation
is,
you
know,
kind
of
in
depth
what
jaeger
is
or
what
tracing
is
or
what
operators
are
or
kubernetes
is
so
we
kind
of
had
this
expectation,
hopefully
that
you
know
you're.
B
So
to
get
started
first,
I'd
recommend
going
check
out
the
docs.
The
link
was,
you
know
in
there
before
here's
the
second
link
and
it's
going
to
tell
you
to
do
a
bunch
of
cube,
cuddle
creatives.
So
there's
a
whole
lot
of
yaml
kind
of
candy
ammo
that
creates
you
know,
deployments
and
services
service
accounts,
roles,
role.
B
B
B
A
very
early
configuration
point
that
we
need
to
talk
about
before
we
move
forward.
Is
this
watch
namespace
environment
variable?
So
in
the
bottom
left
you
can
see
what
comes
in
the
candy
ammo
the
by
default?
It's
going
to
use
the
kubernetes
download
api
to
choose
the
namespace
that
you
deploy
the
operator
to,
and
it's
only
going
to
watch
that
namespace
to
build
new
clusters
or
to
deploy
the
agent
sidecars.
B
Other
options
are
empty.
So
if
you
set
watch
namespace
to
empty,
it's
going
to
actually
watch
all
kubernetes,
all
all
of
the
name
spaces
in
your
cluster.
In
fact,
for
you
know
just
getting
started
playing
with
the
operator,
this
might
be
the
best
one
to
choose,
because
it
will
very
quickly
because
it
will
very
quickly
help
you.
B
Just
like
all
of
the
other
pieces
of
kubernetes
you're
used
to
you
can
use
coop
cuddle
to
edit
them
or
create
them.
You
can
view
them
in
all
the
same
ways.
You
can
query
them
through
the
api
in
all
the
same
ways,
it's
really
fantastic
honestly,
but
in
this
case
we're
going
to
make
a
object
of
type
or
kind
jaeger.
B
The
operator
is
going
to
pick
that
up
and
it's
going
to
make
the
actual
resources
necessary
to
it's
going
to
go,
make
the
resources
necessary
to
create
the
actual
cluster.
So
we're
going
to
make
a
just
very
simple
object
like
you
would
make
a
deployment,
and
then
this
operator
piece
is
going
to
go,
make
the
actual
pods
and
all
of
the
different
infrastructure
necessary
to
have
the
the
object
we
specified.
B
In
this
case.
We
have
this
strategy
option.
It's
kind
of
a
first
choice
here
when
you're
making
a
cluster
and
by
default
it
uses
all-in-one,
which
is
a
great
place
to
start
all
in
one
is
a
single
binary
deployment,
but
we
can
also
choose
production
or
streaming
so
if
you're,
getting
started
playing
with
the
operator,
use,
all-in-one
and
kind
of
just
get
a
feel
for
it.
B
All-In-One
deploys
a
single
binary,
the
single
binary
by
default.
I
have
backend
database
kind
of
over
here,
but
by
default
it
actually
stores
all
of
your
traces
in
memory.
So
if
you
roll
the
pod
your
traces
disappear,
but
you
can
deploy
an
optional
back-end
database.
So
your
application,
through
the
client,
is
going
to
talk
to
the
agent
the
agent
pushes
traces
to
the
or
the
spans
to
the
single
binary
which
it
stores
then
in
memory
for
the
production
and
streaming
options.
B
You
have
you
know
this
stretch
or
you
have
a
set
of
surfaces
that
are
deployed
by
the
by
the
operator
that
you
can
kind
of
scale
independently,
which
helps
you
have
more
control
over.
I
guess
the
more
control
over
the
over
the
over
the
yeager
cluster.
So
here
we
have
the
client
talking
to
the
agent
agent
talks
to
collector
and
if
we're
using
the
streaming
option,
we
also
have
kafka
and
a
jaeger
ingester
and
then
finally
to
the
backend
database.
B
So
if
you
choose
production,
this
green
box
is
not
deployed.
If
you
choose
streaming,
the
green
box
is
part
of
the
deployment
kafka
and
the
backend
database
are
kind
of
our
external
infrastructure
pieces
and
we'll
talk
about
those
in
a
second,
oh
and
then
jaeger
query
also
is
deployed,
which
talks
directly
to
the
back-end
database.
B
So
storage
configuration
is
done
through
this
exact
same
object.
We
looked
at.
We
looked
at
the
crd,
we
saw
we
specified
the
type
right
which
was
production
or
streaming
or
all-in-one,
and
here
we're
going
to
specify
the
storage.
So
here's
where
we
would
specify
we
want
to
use
elasticsearch
cassandra
is
also
an
option.
Elasticsearch
is
the
current
recommended
option
for
a
jager
backend,
but
we
specify
elasticsearch,
and
then
we
have
to
specify
our.
B
We
have
to
specify
where
our
elasticsearch
cluster
is
basically
so
that
makes
sense
and
then,
if
we
want
to
use
kafka,
so
we
have
the
streaming
option
the
str
the
streaming
strategy.
Then
we
have
to
specify
you
know
for
our
ingestor
side.
If
you
recall
this,
this
dot
or
you've
recalled
the
diagram.
It's
collector
goes
to
the
kafka
street.
It
goes
to.
Kafka
goes
to
an
adjuster,
goes
to
your
back
end.
B
So
for
the
injector,
we
have
to
say,
you
know:
you're
kafka,
consumer,
here's
where
coffee
exists
for
a
collector,
we
have
to
say
you're.
A
kafka
producer
here
is
where
kafka
exists
so
for
the
configuration
we
basically
have
to
on
both
sides
tell
the
ingester
and
the
collector
where
to
find
our
kafka
queue,
and
this
is
kind
of
a
neat
feature.
So,
if
you
ever
run,
you
know
one
of
the
pieces
if
I
ever
want
to
know
how
to
configure
jager.
B
I
often
do
this
docker
run
and
I
specify
the
pod
in
this
case
the
collector
and
I
say,
help
or
not
the
pod,
but
the
service
I
say
help
and
I
can
immediately
get
like
a
dump
of
every
single
option
that
I
can
provide
to
the
collector,
which
helps
me
understand
what
the
configuration
options
are.
What
my
kind
of
tunables
are
all
of
these
are
available
through
the
through
the
operator
by
using
this
pattern
you
see
below.
B
So
this
is
really
useful
if
you're
moving
from
an
existing
deployment-
and
you
already
have
a
lot
of
these
configured
or
if
you
just
know
you
want
some
of
these.
For
instance,
q
size
is
common
to
change.
If
you
want
a
larger
queue
size
or
you
want
more
workers
working
on
your
queue
or
whatever
you're
going
to
want
to
change
those
options,
and
you
can
do
that
very
cleanly
through
spec.
B
So
this
again
is
our
jaeger,
crd
or
jager
object,
spec,
collector
options,
and
now
everything
below
options
is
going
to
be
mapped
directly
to
a
cli
parameter,
so
collector
cue
size
will
be
mapped
to
collector
dot
queue
size,
and
this
this
configuration
will
set
the
queue
size
to
100..
So
all
of
the
configuration
options
you
can
find
both
in
the
docs
or
through
running
dash
help
are
going
to
be
available
through
the
operator
by
using
this
options
pattern.
B
In
fact,
if
you
pass
in
garbage
options
so
options
blurg,
blog
10,
then
it's
just
going
to
pass
that
exact
parameter.
You
can
see
it's
saying
unknown,
flag,
blurg.blog,
and
so
this
is
just
kind
of
this
future-proof
pattern
that
I
believe
jp
came
up
with
that
lets.
You
specify
any
parameter
you
want.
So
all
of
those
options
are
available
and
open
to
you
to
use
on
your
jager
cluster.
B
The
agent
strategy
is
another
very
important
piece
of
choosing
or
an
important
choice
when
you're
deploying
your
jager
cluster
and
by
default,
we're
going
to
by
default,
the
operator
is
going
to
set
up
what's
called
a
sidecar,
so
by
default
when
and
a
pod
is
deployed,
if
it's
in
a
watch
name
space
as
well
as
we're
looking
as
well
as
we
have
this
metadata
annotation,
the
jaeger
operator
is
going
to
inject
a
agent
sidecar
next
to
your
pod.
So
you
know
a
pod
is
a
collection
of
containers.
B
Our
pod
might
contain
or
will
will
contain
our
application
and
it
might
contain
a
couple
side-
cars
which
maybe
do
things
like
debugging
or
do
some
kind
of
like
metrics
offloading.
In
this
case
it's
going
to
be
a
sidecar
that
is
the
actual
jager
agent
which
is
going
to
consume
at
the
spans
and
then
push
them
onto
our
collector.
B
This
is
the
default
strategy,
it's
a
very
powerful
strategy
and
it
works
fine
up
to
quite
large
deployment
sizes.
The
one
thing
you
need
to
be
aware
is:
either
your
deployment
needs
this
annotation.
This
annotation
tells
the
sidecar
dot
jaeger
tracing
io.
Slash
inject
true
tells
the
operator
that
I
do
want
the
the
agent
sidecar
on
the
pods
from
this
deployment.
You
can
also
put
this
on
the
namespace
and
then
all
pods
deployed
to
that
namespace.
B
We'll
we'll
have
the
sidecar
injected
so
kind
of
both
are
an
option,
a
second,
more
complicated
option
that
I'd
recommend
only
experimenting
with
kind
of
is,
if
you
need
to
would
be
a
daemon
set
option
so
in
the
daemon
set
we
have
one
agent
per
node
and
all
of
the
applications
on
that
node
can
be
configured
to
send
spans
to
the
node.
Now
it's
kind
of
complicated,
because
there's
no
direct
way
to
reference
the
daemon
set
that
is
on
the
damon
set
pod.
B
B
I
would
only
recommend
it
if
you
need
to
I'm
not
really
sure
what
the
threshold
is.
Hundreds
thousands
of
pods
per
node
maybe
make
this
starts
to
make
sense.
I
can't
say
for
sure,
but
in
this
mode
we're
going
to
deploy
one
agent
per
node.
All
of
the
applications
on
that
are
going
to
node
are
going
to
report
to
it,
and
then
we
are
going
to
send
those
onto
the
collector,
and
you
can
see
here
in
the
jaeger
crd
in
that
object.
B
We
keep
talking
about
it's
the
same
object
this
whole
time
we're
just
kind
of
configuring
different
elements
in
this
case
we're
going
to
say
agent
strategy
is
damon
set,
like
I
said
by
default.
It
is
that
sidecar
strategy,
remote
sampling,
is
another
cool
feature
that
I
have
always
loved
in
kubernetes.
I'm
sorry!
In
yeager
now
remote
sampling
allows
you
to
remotely
control
the
sample
the
sampling
rates
per
operation
of
all
of
your
applications.
B
They
have
to
be
configured
to
go,
find
the
remote
sampling
file
to
bring
it
in
and
to
configure
themselves,
but
this
does
give
you
as
an
operator
it
gives
you
central
control
over
the
rates
at
which
you
sample
all
of
your
different
applications,
and
there
are
different
endpoints
which,
if
an
endpoint
were
to
change
and
suddenly
create
you
know
10
times
the
spans,
it
normally
creates.
It's
overwhelming
your
back
end.
This
would
be
your
way
to
slow
it
down.
B
Basically,
normally
you
have
to
kind
of
create
this
json
file
and
send
the
json
file
to
the
collector.
This.
The
operator
simplifies
that
quite
a
bit
by
just
again
in
this
jager
crd
and
this
jager
object
we're
going
to
have
the
sampling
options,
and
we
can
then
immediately
specify
our
remote
sampling
strategy
and
then
again
we
can
have
us
per
service.
We
can
have
per
operation
even,
and
that
really
gives
us
a
lot
of
central
control
to
handle
the
tracing
load
that
we've
brought
into
our
back
end.
B
Auto
scaling
also
important.
We
have
this
kind
of
collectors
and
injectors
layer
right
so
agents
or
either
sidecar
diamond
set
reports
to
a
set
of
collectors
which
then
maybe
go
to
kafka
if
you
use
streaming
or
maybe
not
and
then
a
set
of
ingesters,
which
are
also
this
a
stateless
piece
of
the
jager
pipeline.
B
So
these
collectors
and
gestures
can
scale
if
you're.
If
your
load
goes
up
considerably,
you
might
need
more
collectors
to
effectively
move
them
through
the
q
in
the
collector
and
put
them
in
kafka
same
with
the
injectors.
If
your
load
goes
up
significantly
and
you
have
a
back,
your
spans
are
backing
up
in
kafka.
You
can
use
auto
scaling
to
bring
up
your
adjusters
by
default.
It's
going
to
create
a
horizontal
pod,
auto
scaler,
with
a
max
of
100,
which
is
quite
a
large
installation,
and
you
might
find
it.
B
You
might
find
that
you
want
to
adjust
that.
So
you
can
say
spec
again
in
this
jaeger
object,
collector
max
replicas.
I
can
control
the
max
again.
Max's
default
is
100
or,
if
you
want
to
turn
this
off,
if
you
don't
want
to
bother
with
auto
scaling,
you
can
also
set
auto
scale
to
false
and
then
set.
I
just
want
10
replicas,
always
or
maybe
a
set
number.
B
B
This
was
a
piece
of
the
jaeger
operative.
I
thought
was
quite
cool
and
didn't
under
didn't
know.
All
of
these
features
exist
when
I
first
started
out,
but
there's
a
number
of
different
other
operator
integrations.
So
for
storage
we
were
saying
you
need
to
go,
make
a
kafka
queue
or
elasticsearch
backend,
and
then
you
need
to
configure
your
application.
You
need
to
configure
your
cluster
to
see
them
to
go
point
at
them.
B
So
if
you
have
the
kafka,
elasticsearch
operators
and
the
jager
operator,
then
you
don't
have
to
provision
this
back
end.
The
jaeger
operator
will
do
it
for
you
through
the
other
operators
and
then
finally,
if
you
have
the
prometheus
operator,
it
will
create
the
appropriate
service.
Monitor
objects
to
monitor
your
jager
cluster,
so
these
operator
integrations
were
quite
interesting.
B
The
jaeger,
your
jager
cluster,
your
back
end
and
your
service
monitors
very
quickly,
there's
quite
a
bit
more.
In
fact,
before
I
started
this
project,
I've
never
used
an
operator
to
run
something
in
a
production
environment.
I've
run
a
number
of
operators
locally
to
learn
about
them
to
kind
of
play
with
them,
and
this
jaeger
operator
is
one
of
the
first
that
I
really
enjoyed
and
felt
like.
It
was
giving
me
value
back
by
spending
a
lot
of
time
with
it.
B
Even
if
you
don't
want
to
deploy
with
an
operator,
I
would
recommend
using
at
least
the
yeager
operator
for
a
while,
because
it
will
teach
you
how
to
operate.
Yeager
it'll
teach
you
all
the
pieces.
When
you
look
at
all
the
configuration
options,
you
will
walk
away
with
a
much
better
understanding
of
jaeger,
so
other
things
it
can
do
are
like
cassandra
schema
creation.
It
can
work
with
elasticsearch
to
handle
your
indexes.
B
Finally,
some
resources,
some
online
resources,
jager
tracing
the
docs,
the
docs-
are
always
great.
Please
go
check
them
out
if
you're
getting
into
jaeger.
At
the
cncf
slack
check
out
the
yeager
channel.
A
lot
of
us
inhabit
the
acre
channel
and
do
our
best
to
help
people
who
are
having
issues
and
then
on
medium
eager
tracing.
You
can
find
a
lot
of
blog
posts
which
are
going
to
give
you
a
lot
of
you
know:
in-depth
information,
other
detailed
information
about
yeager,
please
reach
out
and
connect.