►
From YouTube: Deep Dive: Jaeger – Steve Flanders, Omnition
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Jaeger – Steve Flanders, Omnition
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
This session will provide a deep-dive into the Jaeger project and the power it provides for distributed tracing data. Some quick background information will lay the foundation to discuss advanced topics including adaptive sampling, multi-tenancy, persistency tradeoffs, analytics and more. Demos of recently added features, integrations with solutions including OpenCensus and solving real-world problems with distributed tracing data will be shown. After this session, attendees will better understand the value Jaeger provides and be ready to make contributions to the project.
https://sched.co/OGle
A
No
one
look!
You
can
sorry
there
we
go
copy
audience
awesome
great,
so
I
want
to
talk
about
the
project,
its
current
state
and
where
it's
going
from
a
roadmap
perspective
as
well
as
some
of
the
new
features
I
want
to
actually
show
it
live
you're,
going
to
see
how
you
can
actually
leverage
it
as
well,
and
I'll
try
to
leave
some
room
at
the
end
for
some
Q&A.
First,
a
quick
introduction
on
myself.
My
name
is
Steve
Flanders
I
am
head
of
product
and
experience
at
a
stealth
startup
in
the
bay
area
of
California.
A
In
the
United
States
called
on
mission.
We
are
in
the
observability
space.
We
actually
contribute
to
open
source
software,
including
Jaeger,
open
Sun
system
opens
limit
tree
if
you're
interested
in
learning
more
about
observability
in
general,
I
have
a
feeling
it's
available,
all
right.
So
what
is
tracing
and
why
is
it
important?
Well,
if
you
look
at
very
large
companies,
the
Netflix
Facebook,
the
Amazon's
of
the
world,
what
you
see
is
that
they
have
very
complex
architectures.
A
They
have
moved
to
a
micro
services
based
architecture
where
they
have
a
lot
of
different
service
to
service
communication,
and
this
can
lead
to
complexity
over
who
actually
invented
Jaeger.
They
have
a
very
complex
architecture
as
well
and
so
calls
actually
go
through
multiple
of
these
different
services
in
order
to
call
and
uber
to
get
transportation,
and
so
every
request.
Within
your
system,
that's
known
as
a
distributed
trait,
and
when
you
try
to
troubleshoot
distributed
tracing
systems
problems,
it
can
be
very
very
hard.
A
So
when
you
have
tens
of
services,
maybe
it's
not
that
complex
and
you
have
hundreds
and
services
that
gets
very
complex
very
quickly.
One
of
the
biggest
problems
as
you
move
from
a
monolith
to
a
micro
services
based
architecture
is
problem.
Isolation.
Where
is
the
problem?
Where
is
the
root
cause
and
a
monolith
is
in
the
monolith
with
a
distributed
architecture
like
this
one
micro
service
can
actually
cause
errors
upstream
and
downstream.
A
rippling
of
that.
A
So
while
this
read
directionality
of
arrow
arrows
shows
a
single
call,
there
are
actually
billions
of
them
within
Ebers
infrastructure
every
day.
The
same
applies
to
Facebook,
Microsoft,
Google
and
Netflix.
Any
large
company
that
has
a
lot
of
micro
services
has
a
lot
of
calls
in
the
giving
day.
Now
what
you
want
to
know
is:
what's
going
on
and
who's
to
blame
for
issues
when
a
problem
arises,
if
you're
a
developer
and
you're
on
call
if
you're
an
SRE
and
you're
on
call.
A
You're,
an
f3
or
a
DevOps
engineer
and
you're
on
call.
You
want
to
be
able
to
isolate
the
problem
and
remediate
it
as
quickly
as
possible.
So
how
can
you
do
that
traditionally,
back
before
micro
services,
you
use
metrics
and
logs
to
do
that
primarily
metrics
were
used
to
measure
like
CPU
utilization
memory,
utilization
logs
were
used
more
as
a
root
cause
who
made
what
changed?
Why?
A
This
can
cause
a
lot
of
problems
like
alert
storms
if
one
micro
service
starts
generating
high
CPU
utilization
or
a
large
number
of
errors,
those
errors
or
that
resource
consumption
consumption
can
actually
propagate
upstream
and
downstream
from
a
given
service.
If
different
services
are
owned
by
different
service
owners.
Well,
then,
you
end
up
waking
up
a
bunch
of
people
at
the
same
time.
In
order
to
troubleshoot
an
issue
that
actually
started
in
the
end
service,
so
how
do
you
get
context
and
correlation
distributed
tracing?
A
It's
the
way
prior
to
the
cloud
native
error,
this
was
known
as
application
performance
monitoring.
The
focus
is
really
on
performance,
but
distributed
tracing
can
actually
do
more
than
performance.
It
can
do
availability
as
well.
It
can
help
you
with
on-call
root,
cause
any
troubleshooting
within
your
environment.
What
makes
distributed
trace
and
powerful
is
that
it
can
actually
follow
a
request
throughout
your
entire
system.
That
means
that
knows
who
called
to
that
knows
who
was
originated
in
error
and
who
propagated
in
error
this
graph
here
on
the
on
the
screen.
A
This
represents
a
visualization
of
a
distributed
trace.
This
was
constructed
from
distributed
tracing
data.
You
can
actually
see
different
services.
Red
indicates
errors
that
occurred
in
those
service,
and
you
can
also
see
via
the
arrows
which
service
calls
which
service
completely
constructed
from
distributed
tracing
data.
On
top
of
that,
we
can
actually
enhance
metrics
and
logs
with
distributed
tracing
data
so
that
metrics
and
logs
have
context
and
correlation
as
well.
A
So,
with
that
background,
let
me
introduce
Jaeger,
so
Jaeger
is
a
distributed,
tracing
backend.
It
doesn't
focus
on
metrics
and
logs.
It
focuses
on
distributed
tracing.
It
was
actually
invented
at
uber
if
you've
heard
the
name
URI
before
he's
one
of
the
main
architects
and
contributors
to
Jaeger,
it
was
open
source
and
made
available
in
2017.
It
is
a
CNC
F
project
right
now
in
the
incubating
phase,
but
it
is
up
for
graduation
from
Stan's
bf
and
hopefully
that
will
be
occurring
soon,
which
speak
maturity
of
the
solution.
A
Now
it's
a
very
active
community.
It's
completely
open-source
and
I
definitely
encourage
you
to
get
involved.
If
you're
interested,
you
can
see
some
of
the
stats
here,
it's
primarily
driven
by
maintainer
x'
from
uber
and
Red
Hat,
but
there's
a
lot
of
open
source
contributions
as
well.
There
are
actually
issues
that
are
tagged
with
like
good
first
issue
or
Help
Wanted.
Those
would
be
a
great
way
to
get
involved
with
this
community.
If
you
are
interested
now,
what
is
Jaeger
I
set
it
to
distribute
racing
platform,
it's
actually
made
up
of
four
different
components.
A
You
basically
have
the
in
the
bottom
right,
instrumentation,
libraries.
This
is
how
you
extract
distributed
tracing
data
from
your
application
so
that
you
can
consume
it.
Then
you
have
the
data
mining
platform.
You
can
think
of
this
as
collection
such
as
an
agent
and
a
collector,
as
well
as
the
ability
to
run
analytics
on
top
of
that
data,
so
that
you
can
query
over
it.
A
Okay,
so
there
are
two
different
architectures
for
Jaeger.
This
is
v1
v1
and
v2
are
very
similar.
This
is
actually
in
the
Jaeger
tracing
dot,
IO
documentation,
so
very
easy
to
reference.
Most
of
the
blue
items
or
light
blue
items
represent
components
of
Jaeger.
So
basically
you
will
have
an
application,
as
shown
in
yellow
here.
That's
the
software
that
you
have
written
Jaeger
provides
a
client
or
client
library,
as
it's
known
as
which
can
be
used
to
extract
distributed
tracing
data
out
of
your
application.
A
That
then
offers
an
agent
that
can
run
with
your
application
or
the
host
or
container
that
the
application
is
running
on
the
agent
then
sends
that
data
to
a
collector.
The
collector
can
then
send
that
to
a
database
of
your
choice.
There's
a
few
selected
I'll
talk
about
them
in
a
moment.
There's
this
notion
of
spark
jobs
in
v1.
This
is
what
actually
performs
analytics.
A
It's
used
to
build
dependency
graphs,
so
we
can
actually
visualize
the
distributed
trace
like
I
showed
earlier,
as
well
as
to
compute
SLI
or
service
level
indicators
metrics
from
the
distributed
tracing
data,
and
then
you
have
the
query
interface
in
UI,
so
that
you
can
get
the
data
out
of
the
database
and
actually
be
able
to
see
it
now.
The
only
difference
between
v1
and
v2
is
the
analytics
engine,
so
in
v1
it's
done
via
spark
jobs.
After
the
fact
in
v2,
it's
done
via
a
Kafka
message,
bus
and
flink
streaming
for
analytics.
A
So
it's
done
more
real-time.
Now
it's
not
like
v1
is
deprecated
and
v2
is
what
you
should
use.
There
are
valid
use
cases
for
both
v1
and
v2,
and
both
are
fully
supported
depending
on
your
business
requirements.
The
advantages
are
v2.
This
is
more
of
a
production,
Enterprise
ready,
highly
scalable
architecture.
The
Kafka
message
bus
makes
it
so
that
you
can
handle
bursts
of
traffic
bursts
of
traces
in
your
environments,
the
flink
streaming
jobs.
Allow
you
to
get
Analytics
data,
as
the
data
is
being
ingested
instead
of
after
the
fact.
A
So
it's
a
little
bit
faster
and
a
little
bit
more
scalable.
Both
are
completely
both
architectures
are
encouraged.
Now,
while
the
blue
boxes
represent
Jaeger
components,
you
can
mix
and
mingle
different
components
as
you
see
fit.
For
example,
let's
say
that
instead
of
Jaeger,
you
want
to
use
a
Zipkin
client
library
that
is
supported
as
well.
In
fact,
Jaeger
fully
support
Zipkin.
So
if
you're
using
Zipkin
or
coming
from
a
Zipkin
environment,
you
can
actually
leverage
Jaeger
as
well.
A
In
addition,
if
you
wanted
to
leverage-
let's
say
the
open
census
service,
which
is
made
up
of
an
agent
and
a
collector
instead
of
the
Jaeger
agent
and
collector,
you
can
do
that
as
well.
This
architecture
is
still
fully
supported.
You
could
still
get
the
data
into
a
Jaeger
backend
and
you
could
still
use
the
Jaeger
UI
to
query
the
data
out,
so
you
can
mix
and
mingle.
However,
you
see
fit
the
advantages
of
this
are
clearly
flexibility,
but
from
a
collection
standpoint,
the
Asia
and
the
collector
I
could
now
send
to
Jaeger
and
Zipkin.
A
At
the
same
time,
I
could
send
to
Jaeger
and
omniscient,
let's
say,
for
example,
at
the
same
time,
so
there's
flexibility
if
you
leverage
different
components,
but
that's
not
a
requirement,
you
could
also
skip
the
analytics
engine
if
you
really
wanted
to,
if
that,
if
you
didn't
need
the
service
level
indicators
or
the
dependency
graph,
if
you
just
wanted
raw
traces
and
the
ability
to
query
over
that,
that's
possible
as
well.
So
again,
the
architecture
provides
flexibility.
A
All
of
the
components
are
available
to
use
to
use
it's
a
modular
architecture,
so
every
box
basically
represents
a
separate
docker
container
that
you
can
deploy,
and
so
you
can
mix
and
mingle
to
fit
your
needs.
Now.
I
should
point
out
that
Jaeger
loves
open
standards,
I'm
actually
doing
a
talk
immediately
after
this
one
in
this
room
on
open
telemetry,
which
is
another
CN
CF
project,
the
merging
of
open
census
and
open
tracing
the
Jaeger
client
libraries
that
I
mentioned
they
are
actually
open.
A
Tracing
compatible
today,
open
census
fully
supports
Jaeger
today,
and
these
two
projects
are
merging
to
form
open
telemetry.
In
fact,
yuri
wrote
a
post
on
this.
I
have
a
link
at
the
bottom.
It's
the
medium
post
and
yuri
talks
about
that
in
the
future.
Jaeger
is
considering
removing
its
client,
libraries
and
relying
on
open
to
limit
Ruiz,
client
libraries
and
also
removing
its
agent
and
collector
and
moving
to
the
open
to
limit
reagent
and
collector.
Though
Jager
is
open-source.
Software
Jager
loves,
open
standards.
Open
telemetry
is
providing
some
of
those
standards.
A
That's
definitely
something
to
take
a
look
at
in
the
future.
All
right
going
back
to
Jager
the
technology
stack,
so
Jager
is
written
and
go.
It's
clearly
go
language.
Today
it
has
a
very
pluggable
storage
architecture.
This
is
actually
critical.
It's
one
of
the
newer
features
that
came
out
in
the
one,
nine
one
ten
timeframe
and
one
twelve
is
the
latest
version.
So
basically
the
way
Jager
handles
this
is
they
support
three
core
storage
backends
in
the
Jager
repository
itself.
A
Those
backends
are
cassandra
elastic
and
badger,
and
then
it
has
a
pluggable
architecture,
so
the
community
can
contribute
other
backends
if
they
want
those
as
well
and
there's
a
separate
repository
for
that.
It's
maintained
out
of
Jager
core
today.
That
includes
influx
DB
and
Couchbase,
there's
also
a
github
issue
open
for
other
types
of
backends,
such
as
dynamo
DB.
So
if
you're
interested
in
contributing
to
that
PRS
are
definitely
welcome,
the
front-end
is
a
modern
react.
Javascript
based
front-end,
it's
very
responsive,
you'll,
see
it
here
in
a
minute.
A
A
Okay,
so
integrations
I
just
want
to
cover
this
real,
quick,
depending
on
where
you
are
in
your
cloud
native
journey,
there's
a
bunch
of
ways
to
get
going
with
Jaeger.
So,
for
example,
if
you're
on
kubernetes
there's
a
native
operator
that
you
can
deploy
today
if
you're
on
SEO
there's
integrations
for
that.
If
you
leverage
envoy
as
a
sidecar
for
your
data
plane
or
as
part
of
a
service
mesh,
that's
already
instrumented
and
fully
supports
Jaeger.
So
there's
a
bunch
of
ways
to
get
started
with
little
work
on
your
ends
to
actually
consume
Jaeger.
A
I
definitely
encourage
you
to
take
a
look
at
these
reef.
I
don't
have
time
to
cover
all
of
them.
Alright
new
features,
so
the
latest
version
is
1.12.
I
want
to
talk
about
some
of
the
changes
in
the
1.10
to
1.12
timeframe.
I
think
this
is
important
to
speak
to
the
maturity
of
the
product,
as
I
mentioned,
Jager
is
up
for
promotion
and
CN
CF
to
become
a
graduated
project.
So
a
lot
of
the
focus
that
you
see
today
is
around
hardening
the
existing
feature
set
to
make
sure
that
it
is
production
and
enterprise
ready.
A
So,
for
example,
elasticsearch
being
one
of
the
primary
backends,
there's
been
a
lot
of
improvement
to
the
security
side
of
the
house
to
ensure
that
it's
hardened
to
best
practices
to
ensure
that
your
data
is
secure
in
that
database,
GG
RPC
support
is
natively
being
added
to
it.
They
see
that
as
the
future.
In
fact,
the
front
end
inquiry
interface
was
recently
migrated
over
to
your
PC
in
order
to
get
data
out
of
the
back
end.
Better
Zipkin
compatibility
as
I
mentioned
Jaeger
is
fully
Zipkin
compatible.
A
So,
as
changes
are
made
to
Zipkin,
Jaeger
is
also
enhanced
so
that
it
can
support
that
new
functionality.
You
can
mix
and
mingle
Zipkin
and
Jaeger
as
you
see
fit.
It's
not
uncommon.
I
didn't
talk
too
much
about
instrumentation,
but
there's
something
known
as
context
propagation.
That's
how
you
actually
pass
trace
and
span
data
between
your
applications.
A
There's
different
header
formats
for
that
Jaeger
has
one
Zipkin
has
one
Zipkin
format:
it's
probably
the
most
common.
Today
it's
called
b3.
So
if
you're
using
b3
headers,
you
can
still
leverage
the
Jaeger
back-end
and
Yeager
collection
mechanism.
It's
worth
calling
out.
I
talked
about
open
standards.
There
is
a
new
context,
propagation
format,
standard,
that's
coming
out
from
w3c,
it's
called
w3c
trait
context.
All
of
the
instrumentation
libraries
will
be
supporting
it
going
forward.
Open
census
is
natively,
adding
it
right
now
and
open
telemetry
will
have
it
as
part
of
its
first
version.
A
That
is
the
context,
propagation
format
that
you
should
probably
be
considering
going
forward,
as
that
will
become
the
standard
and
then
from
a
UI
perspective.
There
are
some
enhanced
capabilities.
I'll
actually
demonstrate
some
of
these
and
I
provided
links.
How
do
you
can
actually
get
the
changelog
for
the
changes
that
are
coming
in
these
releases?
You
can
actually
follow
along.
They
do
tag
the
milestones
and
github.
So
it's
very
easy
to
see
the
new
features
that
are
coming
out
and
then
definite
ake.
A
look
at
the
website.
I
mentioned
Jager
tracing
that
I.
A
Oh,
you
can
actually
contribute
to
this
as
well.
It's
part
of
the
github
repository
a
lot
of
rich
resources
here
very
easy
to
get
started
with
that.
I
want
to
switch
to
a
quick
demo
and
then
I
can
show
you
a
few
more
slides.
So
what
I've
done
here
is
I
mentioned
everything's
kind
of
a
docker
container.
You
don't
need
to
memorize.
These
I'll
show
you
the
slides
it's
actually
uploaded
to
this
session.
A
So
if
you
open
up
the
schedule
builder
I
have
these
links
directly
in
there,
but
with
one
command,
I
can
actually
go
ahead
and
fire
up
and
all
in
one
Jager
instance.
It
has
the
collector.
It
has
the
database.
It
has
the
query,
interface
and
the
UI
all
built
in
and
then
in
addition
to
that,
there's
also
an
example
app.
It's
called
the
hot
rod
app.
This
is
on
the
Jager
tracing
site
and
I
will
show
you
the
links
for
that
in
just
a
minute.
A
So,
with
that
in
place,
I
can
actually
locally
run
this
and,
let's
make
sure
it's
easy
to
see.
So
here
is
the
hot
rod
app.
It's
basically
rides
on-demand,
think
like
uber
or
DD
here,
and
you
can
go
ahead
and
request
cars.
I
can
click
it
a
few
times
and
it
will
dispatch
them,
and
this
is
basically
calling
a
micro
services
backends
that
has
been
instrumented
and
that
data
is
going
to
a
Yaeger
instance
and
I
can
choose
different
cars.
I
can
click
it
multiple
times
and
it
makes
a
whole
bunch
of
different
requests.
A
Now,
on
the
flip
side,
this
is
the
Yaeger
UI.
So
the
first
time
you
log
in
this
is
what
Yaeger
would
look
like.
It
basically
provides
a
query
interface
over
your
distributed
traces.
So
you
can
see
here:
I
have
a
bunch
of
different
services.
This
is
from
my
hotrod
app
I
can
go
ahead
and
select
one
in
this
case.
I
can
just
run
a
query
over
that
data
and
I
can
very
quickly
see
the
distributed
tracing
data
in
my
environment.
What
this
will
actually
highlight
for
me
are
important
things
like
the
number
of
errors.
A
So
if
there's
errors
in
the
trace,
I
might
want
to
go
investigate
that
or
the
latency
of
the
request.
So,
for
example,
this
call
took
1.6
seconds.
Each
of
these
boxes
represent
a
trace.
You
can
see
them
summarized
here,
but
you
can
click
into
them
to
get
rich
insights
from
that
data
as
well.
So,
for
example,
now
I
have
clicked
into
a
single
trace.
A
You
can
see
calls
in
to
Redis
and
read
here
tells
you
that
that
an
error
has
occurred
in
this
case
that
looks
like
there
were
some
Redis
timeouts
in
this
particular
requests
to
the
hot
rod,
application
and
there's
rich
metadata
associated
with
this
as
well.
So
when
you
instrument
your
application,
you
can
actually
tag
it
with
important
pieces
of
metadata.
Things
that
you
might
include
could
be,
let's
say,
local
things
to
the
app
I'm
running
on
like
the
host
name.
A
Now
from
a
distributed
trace,
I
mentioned
that
you
can
actually
build
a
topology.
So,
for
example,
here
is
what
the
hot
rod
app
looks
like
visually,
and
each
of
these
represent
a
different
service,
node
and
hovering
over.
It
tells
you
the
call
paths
of
them.
In
addition,
it
provides
a
dag
of
that
same
exact
infrastructure.
So
again
you
can
see
here
you
have
a
front-end
which
calls
the
customer
driver
and
route
micro
services.
The
customer
one
is
backed
by
a
MySQL
database.
A
The
driver
one
has
a
reddish
cash
in
front
of
this
so
again,
very
quickly.
I
can
visualize
my
infrastructure
and
there's
new
calls
are
made.
This
service
graph
can
be
updated.
This
is
typically
handled
by
the
spark
or
flink
jobs
that
you're
running
from
an
analytics
perspective.
To
build
these
graphs
and
then
what's
really
cool
is
that
I
can
do
comparisons
of
different
things.
So,
for
example,
maybe
I
want
to
look
at
the
longest
running,
trace.
I'll.
Take
this
one
and
I
want
to
compare
it
to
the
shortest
running
trace.
A
Let's
say
this:
one
I
can
say,
compare
traces,
and
so
what
this
just
did
is
it
took
two
traces:
it
overlaid
the
data
over
it
and
it's
representing
what
is
similar
and
what
is
different
between
those
two
traces,
so
gray
means
they're
the
same.
That
means
both
traces
contained
that
call
path.
Red
means
that
one
contained
it
while
the
other
one
did
not.
A
So
in
this
case
you
can
see
the
call
that
took
longer
actually
had
the
Redis
call,
and
that's
the
call
that
timed
out
so
in
this
trace
I
can
see
that
Redis
was
causing
the
problem
that
resulted
in
this
1.8
seconds,
whereas
a
typical
call
path
is
typically
under
a
second
about
700
milliseconds,
so
very
easy
to
compare.
What
a
trace
looks
like
against
another
trace.
Looking
for
differences,
this
makes
it
easy
to
do
problem
isolation
and
determine
where
you
should
be
focusing
your
efforts
going
forward.
A
So
that's
a
very
quick
demo
of
the
Jager
UI
I
do
want
to
show
you
some
future
things
as
well
that
are
planned
and,
as
I
mentioned,
I
do
have
these
commands
in
the
slide
deck.
So
it'd
be
easy
for
you
to
get
after
the
session
so
graph
visualizations.
What
I
showed
you
with
the
hot
rod
out
is
really
a
basic
app.
They
only
had
five
or
six
micro
services
and
if
you
have
a
small
microservices
based
architecture,
that
will
work
just
fine
for
you.
A
A
So
if
you
have
small
traces,
we
can
do
a
comparison.
I
showed
you
two
different
hotrod
calls
that
was
pretty
easy
to
visually
compare,
but
if
you
have
hundreds
or
thousands
that
may
be
a
lot
harder
to
see.
In
addition,
while
comparison
of
how
the
trace
looks
is
valuable,
what
could
be
even
more
valuable
is
differences
in
the
latency,
the
time
it
took
between
those
calls.
So
let's
actually
see
what
that
looks.
Like
here's
a
much
more
complex
example.
A
A
At
the
top
of
the
charts,
so
the
divergence
here
at
the
bottom
probably
accounts
for
y1
span
or
one
trace.
One
call
one
request
took
longer
than
the
other.
Now.
What
we
might
want
to
do
instead
is
what,
if,
instead
of
comparing
the
structure,
we
compare
the
time
of
the
spans
and
see
if
there
are
differences
there,
just
because
it's
red
doesn't
mean
it's
faster
or
slower.
In
the
previous
example,
here,
I
want
to
show
a
future
road
map
item
for
Jaeger.
A
The
idea
is
again
I'm,
comparing
two
spans
where
one
took
2.7
4
seconds
and
the
other
one
took
50%
longer
and
I
want
to
answer
the
question
of
why?
Why
did
it
take
longer?
So
I
can
see
that
there
are
some
new
spans.
I
might
be
curious.
Are
those
to
blame?
Is
that
why
it
took
50%
longer
I
can
also
ask
the
question:
is
it
just
because
there's
lag,
there's
increased
throughput
for
this
request,
so
that's
why
it
took
longer,
and
so
the
idea
here
is
instead
of
comparing
spans
visually
from
a
structural
perspective.
A
I,
compare
them
from
a
duration
perspective,
and
here,
if
I
look
at
duration,
I
can
see
very
quickly
that
the
durations
are
very
similar
even
between
the
different
nodes.
I
can
see
that
nodes
that
are
not
shared.
They
don't
really
have
a
difference
here,
but
I
do
see
a
few
black
and
red
hard
to
see
in
the
chart
here,
but
those
are
different
from
a
duration
perspective
and
the
idea
here
is
I'll
be
able
to
hover
over
any
of
these
spans
Bini
of
these
calls
and
quickly
see
hey.
A
This
call
took
two
seconds
for
1.7
seconds.
Why
did
it
take
that
long?
This
is
problem.
Isolation,
I
can
now
go
to
the
service
owner
for
this
particular
service
and
say
hey.
Why
was
your
service
slow?
How
can
we
speed
it
out
because
it's
impacting
the
rest
of
the
calls
downstream?
It's
impacting
customer
experience,
I
made
there
may
be
multiple
services
that
are
slow,
and
so
you
can
see
that
very
visually.
Now
this
is
not
available.
A
Today,
you
can
actually
compare
to
span
structures
to
one
another,
but
you
cannot
compare
to
span
to
trace
durations
to
one
another
if
you're
interested
in
contributing
to
that
there
are
PRS
and
github
issues
open
to
get
that
added
to
Jager.
Of
course,
if
you're
interested
in
learning
more
the
website
and
the
blog
are
really
great
resources
for
that,
in
addition,
there's
a
bunch
of
different
ways
to
get
involved.
The
github
repos
listed
here
all
communication
happens
in
Gitter.
So
if
you're
not
familiar
with
that,
you
can
take
a
look
at
it.
A
There
is
a
mailing
list
where
you
can
ask
questions
as
well.
Twitter
is
also
very
active
and
there
are
bi-weekly
community
meetings
that
anyone
is
welcome
to
join.
You
can
ask
questions
or
get
involved,
to
become
a
contributor
to
the
project
as
well
with
that
I
would
open
it
up
to
some
QA
and
I've
covered
a
few
of
the
most
common
questions
that
I
hear.
While
giving
this
talk,
so
I
figured
I'll
skim
through
them
very
quickly
since
I
have
a
couple
minutes.
One
is
where
can
I
get
more
resources?
A
Uri
just
recently
published
a
book
on
mastering
distributed
tracing
I,
definitely
encourage
you
to
take
a
look
at
that.
It
should
be
available
to
download
or
purchase
overhead
of
tracing
one
of
the
most
common
questions
that
gets
asked
is:
is
this
going
to
slow
down
my
application?
The
answer
is
typically.
No.
The
client
instrumentation
is
very
efficient
with
that
said,
if
you
have
a
very
latency
sensitive
application,
you
should
of
course,
test
this
beforehand
and
make
sure
it's
efficient
for
your
environments.
Gabi
actually
recently
did
an
internship,
basically
with
the
Linux
Foundation.
A
She
posted
a
medium
post,
comparing
that
climb
instrumentation
and
the
overhead
that
it
introduces.
She
has
some
really
cool
graphs
that
actually
show
how
much
resource
utilization
was
consumed.
It's
a
great
resource
if
you're
looking
for
some
benchmarking
information
sampling
is
a
common
question
as
well.
I
didn't
touch
on
this,
because
I
only
have
30
minutes
to
talk
about
Jaeger,
but
in
distributed
tracing,
it's
very
common
to
sample
the
data.
Instead
of
consuming
a
hundred
percent
of
it,
you
only
take
a
smaller
percentage
of
that
data.
A
This
is
not
as
common
in
logs
and
metrics,
but
for
better
or
for
worse,
it
is
in
distributed
tracing
now.
Jaeger
the
agent
and
the
collector
has
a
cool
feature
called
a
damp
adaptive
sampling.
That
means
it
can
say
hey
the
back
end
is
overly
overloaded.
I
want
to
dynamically,
tell
an
application
to
slow
down
and
send
me
less
traces.
It's
a
really
cool
feature
that
is
in
iaeger
today.
The
goal
is
to
add
that
to
open
telemetry
going
for
for
going
forward,
but
it's
not
available
in
open
tracing
or
open
census.
A
Today,
what
open
census
and
open
telemetry
do
offer,
though,
that
Jaeger
does
not
is
tale
based
sampling,
so
there's
two
different
types
of
sampling,
Xin
Jaeger
head
base,
which
means
it
happens
at
the
application
level
when
the
trace
is
to
be
initiated
and
tael
base,
which
means
after
the
entire
trace
has
been
collected.
The
advantage
of
tail
based
is
I
can
make
a
sampling
decision
with
all
the
data.
So,
for
example,
I
can
say
only
sample
traces
that
contain
errors.
A
I
can't
do
that
with
head
based
sampling,
because
unless
only
the
first
span
would
be
known.
If
that
one
contained
an
error,
I
could
collect
it,
but
if
a
downstream
span
had
an
error
with
head
based
sampling,
I,
don't
know
that
I've
already
made
a
sampling
decisions,
so
I
lose
that
data
and
then
finally,
message
bust,
a
syncs,
long
traces.
There
are
always
corner
cases
when
it
comes
to
observer
ability
with
distributed
traces.
These
are
three
of
the
biggest
ones.
The
answer
is,
it
varies
between
them.
Open
tracing.
A
Has
this
notion
of
follows
from
so
from
a
message,
bus
or
from
an
async
perspective?
I
can
use
the
follows
from
construct
in
order
to
basically
link
traces
together,
open
census
and
open
telemetry
have
a
similar
notion
of
linking
this
data,
but
the
visualization
is
still
hard
to
be
fair.
Yeager's
primary
focus
has
been
for
more
real-time
RPC
calls
not
for
some
of
the
longer
running
hours
or
day
distributed
traces.
That
would
be
possible
to
add
if
there
was
enough
interest.
Of
course,
PRS
are
welcome.
With
that
we
have
five
minutes.
A
Question
was
around
I:
don't
want
to
send
the
client
library
distribute
traces
to
in
local
agent,
I
want
to
send
it
directly
to
the
collector
or
directly
to
the
Jaeger
back
end.
Is
that
possible?
So
the
answer
is
yes.
That
is
possible.
Actually,
you
can
send
the
client
library
directly
to
the
collector
itself.
The
collector
can
run
with
the
database.
Node
I
wouldn't
recommend
that
in
production,
but
you
can
do
it.
For
example,
Jaeger
publishes
an
all-in-one,
so
that
is
totally
possible.
With
that
said,
the
best
practice
is
to
run
an
agent.
A
Let
me
explain
why
it's
kind
of
important
there's
a
couple
reasons.
One
is
client
instrumentation
requires
you
to
edit
your
code,
which
means
you
have
to
commit
it
to
your
repository
and
push
it
through
your
CI
CD
pipeline.
That's
overhead
for
software
developers.
They
don't
like
to
touch
it
multiple
times.
If
I
run
an
agent
I
can
tell
the
client
instrumentation
to
point
to
local
host
local
host.
A
Never
changes
always
local
hosts,
no
matter
where
I
deploy
it
anywhere
in
my
environment,
the
client
instrumentation
just
points
to
local
host
and
the
agent
will
work
if
I
point
to
a
collector
and
it's
in
a
different
data.
Center
and
I
have
latency
problems
where
I
run
a
rename
it
or
re
IP.
It
I
now
have
to
go
touch
my
code
and
maybe
I'm
in
a
polyglot
architecture
and
I
have
five
different
languages.
I
now
have
to
go
touch
it
in
at
least
five
different.
A
My
code
bases
that's
overhead,
so
in
general
the
recommendation
would
be
used.
The
agents
because
having
the
ability
to
send
the
local
host
is
very
powerful.
The
other
reason
is
because,
let's
say
I'm
running
in
kubernetes,
I
can
actually
run
it
as
a
daemon
set,
a
local
agent
of
the
host.
Well,
now
I
can
do
aggregation
to
the
host.
The
agent
provides
buffering
and
retry
capabilities.
If
I
don't
have
an
agent,
then
my
client
instrumentation
has
to
have
buffer
and
retry
capabilities
or
I
might
lose
that
data.
Well,
that's
more
complexity.
A
In
the
client
library
I
mentioned
that,
like
latency,
sensitive
apps,
don't
want
overhead.
That's
more
overhead
in
my
application,
but
I
don't
really
want
to
manage.
I'd
have
to
do
that
for
every
language
go
Java,
C++,
node,
Python
whatever.
So,
ideally
you
want
the
client
library
to
be
as
simple
as
possible
and
not
have
to
have
that
logic
and
to
be
able
to
point
locally
to
itself,
so
it
doesn't
have
to
be
reconfigured.
That's
some
of
the
advantages
of
the
agent
why
you
should
probably
use
it,
but
it
is
not
a
requirement
so.
C
A
C
A
A
Let's
think
after
this
I'd
love
to
hear
more
about
that,
and
let's
see
if
we
can
figure
that
out
because
it
should
be
supported,
our
goal
is
to
make
it
flexible
for
any
platform,
whether
it's
kubernetes
or
Cloud,
Foundry,
or
something
else
it
should
work.
Absolutely.
Thank
you.
Thank
you.
Other
questions.
I
have
one
more
minute.
Yeah.
B
We
send
out
a
request
under
then
we
get
to
the
response
we
suspend,
but
after
we
introduce
things
so
we
smash,
for
example,
is
to
we
sink
most
horrible
things
dedicated
by
the
smash
they
recorded
the
requests,
the
record
of
the
response,
so
we
find
the
most
of
things
could
handle
the
bias
to
now.
So
do
you
think
it's
a
good
idea?
We
dedicate
our
scenes
to
use
to
we're
removing
our
record.
Oh
you
think
some
things
should
be
handled
by
a
CEO
under
some
as
sure
things
is
better
to
handle
by
application
data.
So.
A
The
questions
around
service,
smash
and
sto
is
one
example
of
that
linker
D
is
another
one.
What's
nice
about
service
mesh
is
that
it
basically
puts
a
sidecar
into
all
of
my
application
containers
and
that
sidecar
is
already
instrumented.
So
in
the
case
of
sto,
it's
typically
envoy
on
is
instrumented
with
Zipkin
and
Jaeger
compatibility,
so
I
can
get
distributed
tracing
data
for
free.
A
Community
track
performance
or
availability
issues
in
the
function
calls
themselves.
The
second
is,
even
if
you
don't
instrument
your
app,
you
still
need
context
propagation,
because,
while
sto
generates
the
span
and
trace
information,
it
has
to
propagate
that
information
through
your
app.
So
in
the
case
of
like
rest-based
are
pcs,
you
still
need
context
propagation
in
your
application
at
a
minimum,
so
you
can
pass
baggage
and
other
trace
information.
A
B
Another
question
about
the
message:
bus.
We
also
have
some
communication
with
a
message
bus.
Then
the
scenes
must
be
transferred
from
the
message
between
the
message:
bus
I'm,
not
sure
you
mentioned
the
you
decorate
aways,
the
cop
car.
There
was
maybe
the
transparent
about
this
span
party.
We
using
another
message
pass
than
what
you
see
how
many
extra
scenes
showed
us
to
fall,
chasing
Guetta
completely
change.
Yes,.
A
So
from
a
message,
bus
perspective,
Kafka
works,
but
other
ones
you'd
have
to
add
the
the
primary
form
of
doing
this
is
through
open
standards.
When
the
screen
comes
back,
open
tracing
is
one
of
them
opens
limitary
and
open
census.
Also,
have
this
notion
of
follows
from,
or
sometimes
it's
called
links
you
would
have
to
make
sure
to
pass
that
as
part
of
your
trace
data,
and
that
way
it
could
literally
drop
it
off
when
the
message
is
received
into
the
message,
bus
and
then
you
have
to
propagate
that
through
your
message,
bus.
A
So
when
it's
picked
back
up
at
tags
that
follows
from
or
linked
information
when
it
sends
it
back
out,
and
that
way
you
can
make
the
entire
trace
okay.
So
thank
you
I'm
available
for
questions
after
we
are
at
time,
but
if
you
want
to
meet
me
over
here,
I
definitely
be
willing
to
answer
them
for
you.
Thank
you.