►
From YouTube: Jaeger Deep Dive - Yuri Shkuro, Facebook, Pavol Loffay, Red Hat & Annanay Agarwal, Grafana Labs
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Jaeger Deep Dive - Yuri Shkuro, Facebook, Pavol Loffay, Red Hat & Annanay Agarwal, Grafana Labs
This session is dedicated to an in-depth understanding of the Jaeger project. We will give a short demo of the recently added features, talk about various topics including the architecture, deployment models, configuration, different types of sampling, multi-tenancy, and best practices. We will discuss integration with OpenTelemetry, and review the roadmap. After this session the attendees should better understand the Jaeger architecture, how to deploy it and get the best benefits, and to make contributions to the project.
https://sched.co/ekGR
A
Hi
welcome
to
jaeger
project
deep
dive.
We
have
three
speakers
in
this
talk
and
we'll
go
over
very
many
aspects
of
jager
projects,
but
not
all
of
them
because
they
we
have
a
short
session
and
we
need
to
pack
a
lot
of
content.
My
name
is
yurish
crowe,
I'm
an
engineer
at
facebook,
I'm
a
maintainer
of
jaeger
and
as
well
as
open,
tracing
and
open
telemetry
projects.
A
First,
and
I
will
give
an
introduction
and
then
we
will
talk
about
jaeger
features
in
jaeger
architecture.
I
will
talk
about
sampling,
because
this
is
one
area
where
people
have
often
questions
and
pavel.
At
the
end,
will
talk
about
jaeger
integration,
with
open,
telemetry
and
deploying
jaegers
on
kubernetes,
and
so
first
anae
take
away
with
the
intro
to
tracing.
B
B
B
Each
of
the
green
nodes
present
in
this
graph
represents
a
microservice
and
the
gray
lines
represent
the
communications
between
these
micro
services
when
we
interact
with
the
uber
app,
a
single
request
to
the
uber
infrastructure
may
look
something
like
this,
and
this
typically
happens
billions
of
times
a
day.
So
what
are
some
of
the
monitoring
tools
that
we
use
to
monitor
such
a
complex
architecture?
B
Typically,
we
use
a
combination
of
metrics
and
logs
metrics
are
great
because
they're
aggregatable
they
can
be
used
to
alert
upon
and
they're
a
great
way
to
get
an
overall
picture
under
the
performance
of
the
system.
This
is
a
sample
metric
that
I'm
exposing
from
my
application.
It's
a
standard,
prometheus
metric,
which
gives
me
the
duration
or
the
latency
of
a
request
to
the
system,
and
here
I
can
see
that
this
gives
me
a
very
high
level
picture,
saying
that
my
app
of
the
ice
cream
shop
took
10
seconds
of
latency
for
a
given
request.
B
However,
if
I
want
more
fine-grained
information
about
the
system,
then
I
can
add
more
and
more
tags
to
it,
but
very
quickly.
I
run
into
this
problem
of
cardinality
explosion.
Cardinality
refers
to
the
number
of
items
present
in
a
set,
and
so
in
the
metrics
world.
This
means
the
total
number
of
values
that
a
given
label
set
can
take,
as
I
add,
more
and
more
labels
in
order
to
get
more
information
about
my
application,
I
increase
cardinality
and
this
can
lead
to
degraded
performance
and
is
also
not
cost
effective.
B
On
the
left,
we
can
see
that
we
have
a
very
simple
micro
service
architecture
where
an
edge
service
a
creates,
a
unique
id
for
every
inbound
request
to
it
and
every
time
it
makes
enough,
it
makes
a
downstream
request.
It
passes
along
this
unique
id
as
part
of
the
context
as
these
services
do
some
quantum
of
work
and
generate
spans.
They
attach
this
unique
id
to
the
span
once
they're
emitted
and
stitched
together.
In
the
back
end,
we
can
see
the
trace
as
shown
on
the
right.
B
This
is
formed
with
the
help
of
the
unique
id,
and
so
now,
let's
look
at
some
traces,
so
these
traces
are
generated
from
the
sample
hot
rod,
application
that
shipped
as
part
of
the
jaeger
repository
when
we
click
on
the
system,
dependencies
system,
architecture,
diagram
or
the
dependencies
diagram.
We
see
that
as
a
developer.
This
already
gives
me
a
very
intuitive
picture
of
the
architecture
of
the
system
and
the
request
flowing
through
it.
So
this
shows
me
the
different
microservices
that
are
involved
and
also
shows
me
how
many
requests
were
made
between
these
microservices.
B
Next,
when
we
select
a
trace,
this
is
the
typical
gantt
chart
view
of
the
trace
that
we
see
on
the
left.
We
see
that
the
services
are
arranged
in
a
hierarchical
manner,
which
shows
the
transit
of
nature
of
requests
as
generated
at
the
top.
We
have
a
minimap
which
is
really
useful
if
the
trace
has
like
a
couple
of
thousand
spans,
because
we
can
highlight
a
given
section
and
it
will
only
show
the
spans
for
that
duration.
B
Next,
we
see
just
with
a
simple
look
of
this.
We
can
see
that
some
of
these
operations
were
blocking.
For
instance,
the
front-end
server
service,
which
called
the
customer
is
called
the
mysql
service,
probably
was
trying
to
retrieve
customer
information,
and
all
other
operations
were
blocked
on
this
service.
On
this
request.
B
B
B
Next,
when
I
click
on
a
given
span,
it
generates
some
extra
information
in
the
form
of
tags
and
logs
logs
can
be
optionally
indexed.
We
at
grafana
labs
do
not
index
the
logs
that
are
ingested
into
jaeger,
but
here
you
can
see
that
as
part
of
the
tags,
I
can
add
higher
cardinality
data,
for
instance,
like
the
sql
query
itself,
it
would
be
very
difficult
to
view
the
query
if
it
was
part
of
the
service
and
operation
itself,
but
here
I
can
add
higher
cardinality
data.
B
This
is
a
new
view.
That's
arrived
in
jager,
which
is
the
trace
diff.
We
can
see
that
the
two
traces
being
differed
are
give
shown
at
the
top.
One
of
them
took
2.7
seconds,
while
the
other
took
1.4
seconds
so
clearly,
there
must
have
been
something
very
different
about
this,
because,
even
though
they're
hitting
the
same
end
point
which
is
shown
at
the
top
and
the
difference
is
all
of
these
nodes,
so
we
can
see
that
they
had
a
common
parent,
which
is
the
common
gateway
endpoint
that
they
all
hit.
B
B
What
is
absent
in
trace
b
compared
to
trace
a
and
the
green
node
show
what
is
present
in
race
b,
compare
to
trace
a
so.
The
operations
right
at
the
top
over
here
were
more
or
less
common,
and
we
can
see
that
there's
like
a
substantial
diff,
and
this
is
probably
because
one
of
these
was
a
success
and
the
other
was
a
failure.
B
B
B
B
B
B
The
jaeger
client
sends
spans
locally
to
the
jager
agent,
sends
spans
to
the
jaeger
agent,
which
may
be
running
locally,
either
on
the
same
machine
or
as
a
kubernetes
sidecar,
and
from
here
the
jaeger
agent
sends
pants
to
a
jaeger
collector.
The
jaeger
collector
is
more
of
a
central
component,
whereas
a
jager
agent
may
be
deployed
in
multiple
clusters.
B
This
is
done
for
several
reasons,
because
the
link
between
the
jaeger
agent
and
yeager
collectors
cross
dc
and
might
break
so.
The
jaeger
agent
can
do
stuff,
like
buffering
and
so
on.
The
jaeger
collector
is
more
of
a
central
component
that
sort
of
receives
the
spans
denormalizes
them
can
perform
some
additional
cleansing
of
the
data,
for
example,
removal
of
sensitive
information
and
so
on,
and
then
ingest
this
into
a
database
from
here.
B
Another
important
point
to
note
are
the
red
lines
in
this
graph,
which
shows
the
flow
of
sampling
information.
The
jaeger
collector
is
a
central
store
for
all
sampling
configuration
and
this
can
be
used
to
define
for
service
sampling
per
operation,
sampling
and
so
on.
The
jaeger
client
can
poll
the
jager
agent,
which
in
turn,
can
pull
the
jager
collector
and
receive
sampling
information
without
ever
having
to
rotate
config
maps,
or
something
like
that.
So
this
is
really
useful.
B
B
B
B
Speaking
of
technology
stack,
jaeger
is
written
in,
go
it's
a
go
back
in
for
tracing
data.
It
has
a
plugable
storage
with
support
for
cassandra,
elasticsearch
and
badger
databases.
It
also
has
an
in-memory
store
and
it
has
a
plugable
storage
and
uses
hashicorp
go
plugin.
So
if
there
are
experts
in
other
databases,
you
can
write
a
plug-in
which
can
be
plugged
into
the
jaeger
collector
jaeger
uses
a
react.js
front-end,
which
is
really
feature-rich
and
beautiful
jaeger
has
open
tracing
instrumentation
libraries,
it's
compatible
with
all
open
tracing,
instrumentation
libraries.
B
A
Okay,
let's
talk
about
sampling
in
the
distributed
tracing.
We
use
the
term
sampling
in
the
classical
statistical
sense,
meaning
that
we
try
to
select
a
subset
of
all
individuals
or
traces
from
a
population
of
all
possible
traces
in
order
to
estimate
certain
characteristics
of
that
population
or,
more
specifically,
to
reason
about
application
performance
based
on
those
samples
that
we've
selected.
A
A
Here's
some
napkin
math
assume
that
we
have
a
tracing
span
of
about
two
kilobytes
on
average
and
we
have
a
server
doing
10
000
query
per
seconds
right,
so
that
means
we're
already
generating
20
megabytes
per
second
of
data,
now
assume
that
we
have
100
instances
of
that
service,
so
that's
two
gigabytes
per
second
or
170
petabytes
per
day,
and
that's
just
for
one
service.
If
your
architecture
is
complex,
it
may
have
hundreds
or
even
thousands
of
services.
You
can
imagine
how
much
data
we
could
generate
if
we
actually
were
sampling
every
single
request.
A
They
have
the
same
shape
roughly
similar,
latency
measurements,
and
so
storing
all
of
them
is
kind
of
useless.
We
don't
get
any
more
insights
if
or
store
all
of
them,
and
so
that's
why
we
sample,
however,
in
distributed
tracing
specifically
sampling,
has
a
slightly
interesting
aspect.
What
is
called
what
we
need
to
do
so-called
consistent
sampling
and
what
we
mean
by
that
is
that
if
we
collect
spans
for
trace,
then
we
should
either
collect
all
of
them
across
the
whole
architecture.
A
We
have
the
sum
of
the
notes
that
came
without
a
parent,
and
so
it's
hard
to
reason
about
these
traces
and
so
they're,
not
as
useful
as
if
we
sample
consistently
and
we
get
the
whole
trace
every
time
or
we
don't
get
the
trace
at
all,
which
is
also
acceptable
and
as
far
as
specific
sampling
techniques,
there
are
two
primary
consistent
sampling
techniques
that
are
used
in
the
industry
head
based
is
like
most
most
popular.
It
traditionally
has
been
used
from
the
days
of
google's
dapper
paper.
A
All
the
modern
tracing
systems
support
that
and
recently,
in
the
last
few
years,
tail-based
sampling
started
appearing
as
another
popular
techniques
and
I'll
talk
about
both
of
them.
So,
let's
talk
about
head-based
sampling
or
also
called
upfront
sampling
the
approach.
There
is
very
simple
when
we
start
a
new
trace.
Let's
say
when
we
generate
a
brand
new
trace
id,
because
the
incoming
request
didn't
have
any
trace
id,
then
we
make
a
sampling
decision
at
that
same
time
and
we
capture
that
sampling
decision
in
the
trace
context
which
is
propagated
throughout
the
request.
Right.
A
It's
also
fairly
easy
to
implement,
because
if
you
think
about
it,
the
the
code
is
really
you
just
make
a
probabilistic
decision
or
some
other
algorithm
to
decide
when
to
sample,
and
then
you
just
pass
it
around
and
all
the
other
downstream
sdks.
They
just
respect
that
decision.
However,
that
approach
also
has
a
couple
of
drawbacks.
One
is
it's
it's
not
as
good
as
at
capturing
various
anomalies.
For
example,
let's
say
you're
looking
at
your
metrics
and
you
see
your
p99
latency
spiking
suddenly,
so
you
want
to
see.
Okay.
A
Can
I
find
some
traces
that
represent
that
spike?
Well,
p99
already
means
100
and
if
we're
sampling
with
a
rate
of
one
percent,
that
means
that
our
total
probability
of
actually
catching
your
p99
latency
trades
is
one
in
ten
thousand
right
now.
If
your
traffic
is
very
high
to
service
that
you
have
sufficient
number
of
traces
captured.
Even
with
that
probability,
then
you
probably
can
get
some
example.
A
What
about
sampling,
head-based
sampling
in
jaeger?
So
jager
is
the
case
out
of
the
box,
support
that
and
they
come
with
a
assortment
of
different
samplers,
such
as,
like
always
on
always
off
or
probabilistic
the
common,
mostly
commonly
used
or
rate
limiting,
meaning,
let's
sample
like
10
traces
per
second,
no
more
things
like
that
right
and
the
benefit
of
that
is
those
samplers
are
very
easy
to
implement.
A
A
But
the
point
is
that
you
centralize
the
configuration
rather
than
having
it
all
done
at
the
edges
of
your
traffic.
Here's
an
example
of
a
configuration
for
for
that
type
of
sampling.
So,
on
the
top
left,
we
have
a
default
sampling
strategy
which
would
apply
to
any
service
unless
otherwise
configured
with
something
else
right.
So
here
we
see
it
says,
like
everyone
should
use
probabilistic
sampling
with
fifty
percent
probability
right
except
you
can
also
provide
some
overrides
for
very
specific
operations.
A
A
So
we
can
be
very
advanced
with
that
decision,
and
only
and
basically
either
we
instead
of
capturing
samples
uniformly,
we
can
say,
let's
steer
them
towards
anomalies
like
if
we
have
like
a
long
latency
or
if
we
have
an
error,
let's
always
capture
that
example
right.
So
that
gives
us
control
over
what
kind
of
data
we
can
get
into
the
back
end.
A
That
means
that
we
can
also
catch
anomalies
much
easier
than
with
the
upfront
or
head-based
sampling,
and
another
benefit
is
that,
because
we
are
getting
all
this
data
into
collectors
before
we
make
a
sampling
decision,
we're
actually
dealing
with
a
lot
more
data
that
on
which
we
can
run
various
aggregations.
Let's
say
we
want
to
compute
some
statistical
aggregations
of
like
what
latency
we're
observing
what
histograms
we're
seeing
in
the
services
right.
A
A
You
can
actually
scale
that
system
fairly
well
to
very
large
traffic
of
inbound
traces,
and
the
other
downside
of
tail-based
sampling
is
that
if,
if
the
goal
is
actually
to
collect
all
the
data
up
front
and
then
sample
only
so
much
so
that
our
storage
can
support,
then
all
this
collection
up
front
introduces
an
additional
overhead
on
the
application
itself.
Right,
as
I
mentioned
in
a
napkin
mask
before
you
can
have
up
to
five
to
like
eight
percent
overhead
on
very
high
qps
services.
A
If
you
essentially
collect
data
on
every
single
request
and
export
it
into
collector
tier
before
you
make
the
sampling
decision.
So
it's
a
bit
expensive
in
terms
of
jager
support,
so
jaeger,
as
we'll
talk
later,
is
moving
towards
building
most
of
the
jaegerbacons
on
top
of
open,
telemetry,
collector
and
open
telemetry
collector
does
have
a
logic
for
tailway
sampling.
All
right,
you
can
configure
various
sampling
rules
already
based
on
like
latency
or
certain
tags,
like
the
error
flags.
A
Unfortunately,
at
this
time
it's
only
in
a
single
node
mode,
so
sort
of
you,
you
can
run
a
single
service
and
send
all
the
traces
there.
Then
it
will
work.
But
if,
if
your
traffic
is
such
so
large
that
that
service
cannot
scale
to
it
and
you
need
to
run
multiple
collectors,
normally
they're
status.
So
it's
not
a
problem,
but
in
case
of
a
tail-based
sampling
they
are
become
stateful
and
there
needs
to
be
some
sort
of
a
sharding
solution
which
is
right
now
not
available,
but
it
will
be
available
in
the
future.
A
C
Hello,
everybody,
my
name
is
paulo
phi,
I'm
software
engineer
at
traceable,
ai
and
I'm
core
maintainer
of
jager
project
and
contributor
to
open,
telemetry
and
open
tracing
projects.
In
this
section
I
will
talk
about
eager
and
open
telemetry
integration
before
we
deep
dive
into
open
telemetry.
Let
me
quickly
talk
about
open
tracing,
so
we
better
understand
open
telemetry
as
the
next
evolution
of
open
tracing
and
data
collection,
libraries
in
general.
C
B
C
However,
there
is
one
downside
that
we
want
to
do
that.
We
still
have
to
recompile
and
redeploy
our
application,
and
this
might
be
a
problem.
If
we
have,
you
know
dozens,
maybe
hundreds
or
even
thousands
of
microservices
it
can
be
very
costly
thing
to
do.
The
problem
is
open,
tracing
doesn't
define
any
data
format,
so
all
the
tracing
implementations
they
use
different
data
formats.
C
C
C
C
B
C
B
C
An
ecosystem
where
you
are
using
jaeger
clients,
then
there
is
open
tracing
shin,
which
is
which
is
basically
an
open
tracing
implementation
that
uses
open,
telemetry
sdks,
and
this
allows
you
to
use
all
existing
open
tracing,
instrumentation
libraries
with
open
trace
with
open,
telemetry,
sdk
and,
last
but
not
least,
jaeger
clients.
They
support
trace
context,
which
is
the
default
propagation
format
in
open
telemetry
as
the
case.
So
you
will
be
able
to
use
jaeger
clients
in
a
new
ecosystem
with
where
open
telemetry
sdks
are
used.
C
C
A
Thank
you
paul.
This
is
the
end
of
our
talk.
These
are
the
different
ways
you
can
get
in
touch
with
the
jaeger,
maintainers
and
community.
We
have
a
biblically
meetings
where
you
can
dial
in
and
participate
in
discussions
and
make
sure
to
start
the
projects
on
github
developers
like
those
stars.
Thank
you
very
much
for
joining.