►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Jaeger Deep Dive - Yuri Shkuro, Uber Technologies & Pavol Loffay, Red Hat
This session is dedicated to an in-depth understanding of the Jaeger project. We will give a short demo of the recently added features, talk about various topics including the architecture, adaptive sampling, multi-tenancy, and configuration, and review the roadmap. After this session the attendees should better understand the Jaeger architecture, how to deploy it and get the best benefits, and to make contributions to the project.
https://sched.co/Uags
A
Okay,
we
were
going
to
start
everyone
for
coming.
This
is
a
Jaeger
deep
dive.
We'll
live
demo
yesterday.
If
you
want
to
see
that
go
back
in
time
and
see
that
one
here
is
more
about
architecture
of
Jaeger
and
I
always
deployed
and
all
found
like
more
technical
details
that
you
may
wanna
know.
So
we
will
talk
about
the
project
briefly,
as
open
source
project.
We
will
talk
about
how
tracing
really
works.
A
Yesterday
I
gave
a
sort
of
a
hand,
wavy
story
about
how
that's
implemented,
but
today
we
want
a
bit
like
dive,
a
bit
deeper
on
that
and
we'll
talk
about
like
features
of
Jaeger
and
enrolled.
My
opinion
all
other
things
and
we'll
try
to
leave
a
lot
more
time
to
Q&A
than
yesterday
so
again
to
introduce
ourselves.
I'm,
URIs,
crow,
I
work
at
uber
on
observability
systems,
mostly
on
tracing
and
I,
maintain
Jaeger
and
open
tracing
and
open
telemetry
and
I
published
a
book
on
tracing.
A
B
B
Swagger
is
not
only
a
distributed
tracing
tool,
it's
also
like
the
whole
platform,
because
we
provide
different
bits
as
pieces
which
can
be
used
separately
so
starting.
We
have
a
lot
of
instrumentation
libraries
written
in
different
languages.
This
libraries
implement
open
tracing
API
and
you
can
use
them
in
your
applications
to
report
data
to
our
trace,
collection,
back-end
and
then
the
stress
collection,
mecca
and
writes
data
to
the
database,
which
is
later
represented
in
the
UI.
Then
there
is
also
data
mining
platform,
which
can
you
know,
get
the
data
from
the
database
analyze.
B
It
do
some
aggregation
and
stores
it
back,
which
is
then
used
in
the
UI
for
different
visualizations,
so
Jaeger
itself.
The
architecture
is
inspired
by
Google's
diaper
system
and
open
Zipkin.
It
was
initially
it
started
at
uber
in
2015,
open
source
in
in
2017
when
it
also
joined
CN,
CF
s
incubating
project,
and
last
month
we
graduated
as
a
top-level
CN
CF
project.
I
would
like
to
thank
all
of
the
contributors
who
made
it
possible.
B
On
the
technology
stack
the
system
itself,
all
the
backend
components
are
written
in,
go
Lang.
We
super
different
storage
layers,
so
you
can
write
data
to
Cassandra,
elasticsearch,
better
and
in
memory.
The
UI
is
pretty.
Nice
is
written
in
react,
as
I
mentioned
before
a
lot
of
we
have
a
lot
of
clients
in
different
languages.
So
pretty
much.
We
support
every
mainstream
language
like
:,
Java,
Python,
nodejs,
C++,
C
sharp,
and
there
are
two
community
based
PHP
LMB.
B
Then
there
is
integration
with
Kafka.
We
will
talk
about
it
a
little
bit
later
and,
of
course,
Apache
fling
and
Apache
spark
for
data
aggregation
and
data
analytics
about
the
community.
So
the
main
repository
has
about
nine
thousand
stars
in
the
moment,
a
lot
of
contributors
and
at
the
moment,
fifteen
maintainer
occurs
all
the
projects
in
the
main
organization.
We
are
very
happy
to
accept
more.
B
A
It's
your
birthday,
so
they
promised.
Let's,
let's
talk
about
how
tradition
works
in
practice.
This
might
look
a
bit
look
through
it.
So
what
we
see
here
in
green
are
two
to
services
right
and
we
want
to
trace
request
and
at
the
top
like
left,
left
to
right
and
right
to
left
is
the
request
response
chain
of
like
execution.
A
So
what
we
see
here
is
that
when
the
request
comes
to
the
first
micro
service,
it
first
hits
some
server
end
point,
and
there
is
an
instrumentation
in
that
point,
which
sometimes
is
called
a
trace
point
right
that
trace
point
before
certain
things.
Specifically
in
this
case
it
takes
the
trace
context,
which
is
typically
passed
on
the
wire
between
those
micro
services
and
converts.
A
If
you
have
multiple
trace
points
in
the
service,
they
all
have
to
have
access
to
the
trace
context
in
order
to
be
able
to
identify
the
event
correctly,
and
so
what
I'm,
showing
here
is
like
kind
of
rectangular
traced,
points
and
circulatory
points.
The
circular
ones
are,
you
can
think
of
them
with
something
like
inside
the
application,
but
maybe
it's
making
a
database
call
right.
A
A
Maybe
it's
some
in
some
framework
that
that
your
your
application
is
using
and
and
that
piece
of
code.
This
gathers
the
data
about
the
operation
that
you
about
you
to
do
like
I
said
that
could
be
a
scale
query,
or
maybe
some
name
of
the
temperature,
and
it's
it's
writes
that
that
data
into
what
I
call
in
this
die
a
tracing,
API
right
it
just
you
can
think
of
it
like
similar
to
logon.
When
you
say
a
log
dot
info,
you
provide
the
string.
You
provide
some
parameters
right.
A
What
happens
at
that
case,
you
you
actually
instrument
in
your
code
with
the
log
statement
and
that
information
gets
written
to
some
underlying
login
implementation
that
runs
in
your
application,
which
could
later
on
write
it
to
disk,
maybe
ship
it
somewhere
else.
Your
application
doesn't
really
care
at
that
point.
You
all
you
did
just
like.
You
call
the
cloak
with
infrared.
So
it's
the
same
thing
with
trace
you
just
have
you
call
a
slightly
different
API?
Maybe
you
start
a
span
and
you
finish
a
span.
A
You
attach
attack
to
expand
your
given
name
to
span,
but
ultimately,
conceptually
it's
very
similar
to
just
doing
a
more
structured
logging
and
then
tracing
API.
It's
an
abstract
thing
like
similar
to
login.
Api
is
just
no
like
it
doesn't
actually
do
by
itself
anything
you
just
call
it
it's
a
callback
and
then
usually
that
API
is
implemented
by
some
tracing
library.
In
this
case
it
could
be
Jaeger
client,
it
could
be
any
other
tracing
library.
Actually,
you
can
have
a
Zipkin
library
or
liquid
open
telemetry.
A
If
you
go
to
those
talks,
so
they
also
sometimes
cause
the
SDKs
the
doors,
libraries,
so
those
Vipers,
they
simply
implement
the
tracing
API.
And
then
then
they
do
something
with
that
data.
In
case
of,
if
you
take
Jaeger
client
as
an
SDK,
what
it
will
do
it
will
kind
of
buffer
that
data
for
a
while
and
then
package.
It
convert
it
into
thrift
format
and
send
it
out
of
process
to
either
a
guru
to
some
of
the
tracing
back
and
collection
mechanisms
which
we'll
talk
about
there,
several
of
them
and
Jager
and
then.
A
Finally,
as
the
request
goes
further
and
then
your
micro
service,
one
makes
a
call
to
micro
service
to
the
last
trace
point.
Does
something
special?
It
takes
that
in
memory
context,
they
request
idea,
whatever
that
we
kept
passing
around
and
encoded
on
the
wire,
so
that
it
has
a
separate
representation
on
the
wire
and
so
that
the
next
micro
service
can
also
like
starts
engine
date,
because
your
next
micro
service,
maybe
Ryan
and
the
different
processes
on
different
data
center.
A
But
it
will
be
able
to
correlate
its
events
or
its
trace,
trace
events
with
the
micro
series
one
and
then,
as
they
all
call
this
and
do
the
tracing
back
and
eventually
we'll
be
able
to
reconstruct
that
whole
request
execution.
And
so
what
collection
here
is
like
the
blue
box
collection
and
normalization,
because
you
can
have
events
captured
with
different
instrumentation
like
instrumentation,
the
walls
over
time,
sometimes
formats
change,
etc.
You
may
have
one
application
instrumented
with
zipped
in
another
instrumental
to
the
aggressor,
instrumental
open
telemetry.
A
It's
actually
not
that
hard
to
make
them
all
understand
each
other
and
they
may
all
sound
different
data
models
out
of
the
box.
But
then
collection,
back-end
will
sort
of
normalize
those
into
one
single
model
that
jäger
back
and
understands,
and
it
then
it
will
get
stored
to
the
database
and
from
the
database
from
the
storage.
A
So,
as
I
was
talking
through
that
previous
picture,
I
mentioned,
there's
a
trace
in
API
and
the
tracing
SDK,
tracing
implementation
right,
so
open
tracing
is
that
tracing
API?
If
you,
if
you
do
something
in
your
application
and
you
make
calls
to
trade,
to
open
tracing
API,
nothing
is
going
to
happen
right.
It's
an
oval
basically,
unless
you
actually
plug
in
some
SDK
with
it
and
Jaeger,
is
that
SDK
or
you
can
plug
in
any
other
SDK
vendor
SDK,
even
custom,
blender
and
and
so
at
the
top.
A
There
Malik
Powell
told
me
that
it's
a
bit
confusion,
but
what
this
really
shows
is
that
it's
conceptually
your
application
can
be
instrumented
in
three
different
ways
on
the
Left
we
see
is
what
what's
called
manually.
Instrumented
application
was
basically,
let's
say,
you're,
not
using
any
framework.
You
probably
write
in
and
see
you
know,
and,
and
so
like
all
the
network
calls
all
the
inter
process
communication
that
you
do
and
you
need
to
manually
go
and
instrument,
and
you
add
the
open
tracing
code
to
that
right.
A
Send
you
start
the
span
manually
before
call
finishes
planned
afterwards.
Typically,
no
one
does
that
because
it's
just
like
everyone
uses
some
frameworks
it
and
so
with
frameworks.
It
gets
a
lot
easier
because
then
the
instrumentation
really
needs
to
exist
in
the
frameworks
rather
than
in
the
application.
A
The
third
option
is:
if
you
can
pull
that
off,
it's
like
the
easiest
way
to
adopt
in
a
large
organization
where
people
sometimes
use
auto
instrumentation,
where
with
the
middle
box.
So
even
though
the
framework
itself
is
like
instrumented,
but
the
way
the
instrumentation
is
typically
done
for
those
frameworks
is,
is
you
still
need
to
need
to
do
some
minimal
configuration
right?
So
if
you
read
any
open
tracing
tutorials,
they
will
start
to
a
cable
first
instantiate.
A
The
tracer
then
use
this
HTTP
framework
create
the
middleware
for
it
pass
the
tracer
to
the
middleware
a
middle
word
and
will
attach
the
HTTP
framework
and
start
doing
its
thing
right.
So
the
minimal
step
that
your
application
still
needs
to
do
is
create
a
tracer
and
pass
it
to
some
other
frameworks.
Right
would
be
nice
if
you
didn't
have
to
do
that,
because
you
can
just
and
deploy
a
tracing
without
touching
any
of
the
application
code
and
that's
what
what
instrumentation
does
historically
like
big
long-standing
vendors
like
New
Relic
AppDynamics,
they
provided
those.
A
What
instrumentation
is
the
case
where
you
just
maybe,
if
you
enjoy
world,
you
attach
a
jar
to
your
runtime
and
suddenly
it
goes
and
rewrites
byte
code
in
your
application
and
and
provides
this
kind
of
instrumentation.
We
have
a
tuple
we're
using
one
library
in
Python
that
does
that
and
kind
of
all
my
experiences
of
actually
distributing
instrumentation
to
services.
That
was
the
easiest
one,
because
we
made
a
change
once
in
one
place,
and
every
service
in
Python
got
the
tracing
automatically,
whereas
with
other
languages,
especially
with
go
some
someone
needs
to
do
something
manual.
A
So
that
those
are
very
early
in
my
mic,
ultimately
all
three
types
of
instrumentation:
they
just
talk
to
trace
in
API,
and
then
that
case
it
could
be
open,
tracing,
API,
right
and
then
the
rest
is
like.
You
have
an
iteration
implementation
which
implements
the
tapir
and
then
collects
data
and
sends
it
to
the
backend.
A
Finally,
on
on
the
right
here,
you
can
see
like
open,
trace,
an
icon
and
then
two
boxes
open
trace
in
country,
but
the
api.
So
the
api
is
the
actual
product
of
open
tracing
the
libraries
that
you
can
use,
and
then
there
is
a
separate
organization
called
country
which
was
created
specifically
to
to
create
those
shared
middleware
instrumentations
that
you
can
use
with
popular
frameworks
like
if
you
use
in
spring
or
a
jungle
in
python.
You
don't
have
to
keep
rewriting
everyone
the
same
instrumentation,
because
it's
identical
everywhere.
A
We
can
just
write
it
once
as
an
open-source
project
and
then
because
the
only
thing
that
project
is
exposed
is
open,
tracing
api,
you
can
unplug
any
vendor.
Any
tracing
is
decay
afterwards
and
it
will
work.
So
that
was
the
story
last
year,
and
now
we
have
open
telemetry
that
you've
probably
heard
a
lot
and
open
till
poetry.
You
can
think
of
it
as
the
next
iteration
of
open
tracing.
A
It
also
is
a
merger
with
another
project,
open
census,
which
was
kind
of
competing
with
open
trace
in
which
was
very
unfortunate,
because
no
one
actually
wanted
that
competition.
We
want
the
standard
way
for
everyone
to
instrument,
rather
than
everyone
have
to
decide
which
way
to
go,
and
so
open
telemetry
in
that
sense
is
very
similar
to
open
tracing.
It
has
a
part
which
is
an
API
only
right,
and
you
can
implement
that
API
1200.
A
Okay,
thank
you.
Oh
oh,
please
go
better.
So
one
difference
between
telemetry
and
open
traces
that
open
slam.
It
does
count
with
an
SDK
right,
so
you
don't
need
the
Jaeger
client
anymore.
You
can
just
use
standard
SDK
across
or
board,
and
all
the
others
are
sort
of
like
join
forces,
to
do
one
single
thing
and
it
will
work
everyone.
So
everything
else,
aside
from
that
is
this
kind
of
the
same
you-you-you
instrument,
the
code
or
you
use
auto
instrumentation,
which
will
talk
to
that
API
and
then
it
will
send
data
through
the
SDK.
B
So
the
sampling
is
it's
a
way
how
you
can
control
how
much
data
or
how
much
traces
are
stored
in
your
thoracic
system,
because
if
you
end
up,
you
know
tracing
all.
The
request
is
just
too
much
data
to
store.
So
we
have
to
have
a
way
how
to
control
that
so
Jager
by
default
does
head
base
and
blink
or
upfront
tracing,
which
is
the
way
when
we
start
the
trace,
we
decide
whether
the
trace
should
be
stored
or
not.
B
We
there
are
multiple
ways
how
you
can
decide
it
if
the
trace
should
be
traced
or
not,
there
are
like,
for
example,
there
is
a
probabilistic
sample
which
says:
I
want
to
store
only
1%
of
all
traces.
Then
there
is
rate
limiting
will
you
say,
I
want
to
store,
only
I
know
10
traces
pair
1.
Second,
this
is
all
per
service,
so
you
can
configure
it
in
the
service,
but
sometimes
the
service
has
multiple
endpoints
and
these
endpoints
have
a
different
traffic
patterns.
B
So
endpoint
a
may
receive
I,
know,
1,000
requests
and
even
B
receives
two
requests.
What
happens
if
you
use
probabilistic
sampler
is
that
endpoint?
You
will
get
only
you
know
very
small
traces
stored
for
the
ten
point,
so
we
have
a
way
how
you
can
overcome
this
problem
by
using
endpoint
level
sampling
within
the
service.
So
you
say
for
exam
and
pee
which
receive
is
a
lot
of
traffic,
will
sample
a
10%
and
and
B,
which
is
receives
only
you
know.
Fraction
of
that
will
receive
I,
don't
know.
B
50%
of
50%
of
the
traffic
will
be
sampled.
This
can
be
also
controlled
on
the
Jaeger
server.
So
you
don't
have
to
go
to
the
clients
and
change
the
configuration
you
just
change
it
in
the
in
the
deployment
of
the
separate
server
on
the
road
map.
We
have
also
two
new
samplers
coming
in.
There
will
be
a
tag
matching
sampler,
which
basically
means
that
when
you
are.
B
A
If
because
really,
if,
if
you
think
what
you
use
tracing
for
you're,
not
using
it
for
monitoring
every
single
request,
what
you
typically
use,
it
is
for
troubleshooting
problems,
it's
an
observability
tools,
not
a
monitoring
tool
per
se,
and
when
you
troubleshoot,
you
actually
want
to
capture
requests
which
have
problems,
but
hopefully
in
your
production,
99%.
The
requests
don't
have
problems
right,
and
so,
if
you
use
probabilistic
sampling,
then
you're
gonna
capture,
99%
of
all
your
requests
that
you
capture
will
be
okay,
they're,
they're,
fine,
they're,
all
the
same
they're.
A
All
successful,
there's
no
reason
actually
to
capture
them
that
menu
right.
You
might
capture
like
ten
of
them.
You
know
that
you
just
establish
the
overall
flow,
but
and
so
that
that's
per
head
by
sampling.
Really
he
struggles
with
that
because
it
makes
the
decision
very
at
the
beginning
of
the
request.
A
But
there
is
lots
of
infrastructure
work
involved
to
make
this
work
so
I
we
actually
working
on
rolling
this
out
at
uber,
so
I'm,
hoping
that
it
will
show
up
next
year
in
jäger,
open
source
and
the
tag
matching.
That
Pavel
was
starting
to
say:
that's
still
an
upfront
sampling
approach,
but
it's
it
allows
us
to
control
to
allow
like
users
to
change
sampling
based
on
specific
patterns
that
they
see.
A
So,
as
the
name
implies
you
you
essentially
you
you
push
a
configuration
to
the
client
saying
if
you,
if,
if
the
span
contains
the
stack
with
this
value,
then
sample
that's
right
or
sample.
That
was
like
a
different
strategy
very
high
probability,
and
so
that's
it's
very
experimental
right
now.
We
we've
implemented
that
over.
We
haven't
rolled
it
out
in
production,
but
there
is
one
team:
that's
really
interested
in
using
us,
so
I'm
curious
how
that
will
turn
out.
A
B
B
B
B
You
don't
have
to
understand
how
to
do
how
to
deploy
a
Jaeger
to
use
the
operator
you
just
define
I
want
to
use
this
storage
I
want
to
use
this
configuration
and
operator
will
handle
all
of
the
deployment
for
you,
but
also
the
upgrades
if
there
is
a
new
version
of
Jaeger,
so
by
default,
Jaeger
exposes
matrix
in
primitives
format.
So,
for
example,
eager
operator,
you
know,
watches
this
matrix.
If
there
is
something
wrong
it
can,
you
know,
change
the
deployment
accordingly,
the
open
telemetry
already
ships
with
Jaeger
exporters
and
Jaeger
receiver
in
the
collector.
B
A
So
I
guess
we
didn't
mention
Kafka,
but
Jaeger
is
overall
implemented
with
a
push
architecture,
unlike
premises
which
you
just
pull
for,
for
the
metrics
I
mean
there.
The
problem
comes
to
either
one
of
them
so
like
for
us
push.
Architecture
was
easier
because,
especially
the
applications
need
to
know
very
little
about
how
to
locate
the
Jaeger
components
in
order
to
talk
to
them
right
and
the
classic
architecture.
A
A
It
was
easy
for
us
to
just
run
at
one
single
agent
per
host
because
we
don't
have
like
channels
requirements
and
then
the
agent
then
takes
care
of
location
where
the
rest
of
the
jäger
back
and
this
communication
to
it
and-
and
you
know,
and
passing
the
data
through
and
also
the
control
flow,
that
we
mentioned
the
sampling
strategies.
You
can
configure
them
on
the
Jager
back-end
and
they
make
they
become
available
to
the
Jager
clients
at
runtime.
So
that's
done
with
the
thread
feedback
loop.
It
I
think
that
only
works.
A
If
you
use
the
agent
you
cannot
use.
If
you
talk
directly
to
collector
and
then
on
the
right
side,
collectors
were
just
like
they
were
completely
stateless
ant
and
they
were
essentially
passed
through.
They
would
get
tracing
date
and
write
it
to
the
database
as
well
and
I
think
we
have
a
next
slide
for
it,
so
yeah.
A
So
the
challenges
with
that
synchronous
or
asynchronous
push
approach
was
that
if
you
have
a
traffic
spike
in
your
application,
or
maybe
some
application
rolled
out
with
the
configuration
of
jäger
client
with
like
hundred
percent
sampling
and
suddenly
starts
flooding,
your
back-end
there's
very
little
that
Jaeger
back-end
could
actually
do
to
protect
itself.
With
that.
A
First,
what
we
introduced
as
the
Kafka
in
the
middle,
it
supported
an
open-source
version
of
Jaeger,
and
so,
in
which
case
the
push
model
still
remains,
but
once
the
data
hits
collector,
it
just
writes
data
to
Kafka
and
that
provided
a
couple
of
benefits.
One
is
cough
Queens,
usually
a
lot
more
elastic
than
storage
like
Cassandra
or
elasticsearch,
because
those
actually
like
I
mean
they
have
a
schema.
They
do
the
indexing.
A
Of
course,
you
can
have
a
spike
to
size
degree
that
your
Kafka
blows
up,
but
the
usually
wouldn't
reach
those
points
and
and
Kafka
tuber
is
like
super
large,
so
we're
a
tiny
drop
in
that
and
and
then
we
have
another
component
called
Jaeger
in
gesture
that
reads
from
Kafka
and
stores
data
database.
It
can
be
deployed
in
two
modes.
A
One
is
just
as
a
single
thing
or
you
can
split
it
and
twin
gesture
in
indexer,
for
example,
the
two
where
we
actually
split
those
database
into
two
clusters,
so
that
one
cluster
stores
just
the
traces
that
you
can
retrieve
by
trace,
ID
and
another
cluster
stores
indexes
for
those
traces
that
and
that
indexing
can
lag
behind.
If
we
want
you,
oh,
if
it
happens
to
be,
we
don't
really
care
that
much
and
the
second
bit
is
that
we
we
started
building
the
data
pipeline
on
top
of
that,
specifically
with
streaming
fleeing
jobs.
A
A
As
specially
when
you
have
a
lot
of
data
than
to
read
the
whole
thing
in
this
park
is
pretty
expensive,
so
it
can
timeout
and
you
have
all
kinds
of
performance
issues
that,
whereas
freeing
streaming
is
just
does
it
all
the
time
and
flashes
data
periodically
and
plus
I
mean
that's
more
specific
to
do
over,
but
we
also
have
a
standard
framework
for
anything.
That's
in
Kafka
can
be
saved
to
HDFS
so
that
we
can
query
it
with
a
MapReduce
job.
A
B
B
Yaeger
can
can
receive
data
in
format,
so
you
can
just
you
know,
redirect
all
the
data
from
the
king
clients
to
Jager
collector.
So
we
could,
you
can
receive
the
trip
to
brought
above
and
the
JSON
in
words
into
format,
but
then
also
you
can
configure
eager
clients
to
understand,
zipping
context,
propagation
format,
so
once
this
can
be
instrumented
with
open
tracing
and
Jager
and
other
with
zip
in
and
they
can
still
continue
the
same
trace.
There
is
also
integration
with
Kafka,
which
pretty
much
just
uses
the
uses.
B
Okay,
so
new
features
since
basically
last
cube
con
in
Seattle.
There
is
new
Jager
operator,
we
pretty
much
it.
It
exposes
all
the
configuration
what
we
have
in
the
Jager
itself,
so
it's
you
can
do
basically
everything
with
it.
There
is
new
local
storage
Basin
badger,
so
this
is
something
like
Prometheus,
where
you
have
no
Oriole
data
and
Jager
in
a
single
pot,
and
we
also
publish
the
storage
plugin
based
on
G
RPC.
At
the
moment
there
are
two
available
implementation,
one
for
couch,
B,
Couchbase
and
honorable
influx
DB.
B
This
is
delayed
sampling.
We
talk
about
the
same
plane.
It
is
basically
that
tech
based
sampling-
it's
not
fully
open
source
right,
but
it's
something
we
are
working
on.
It
will
be
soon.
We
made
a
lot
of
improvements
on
the
security
side
of
the
things,
so
you
can
use
TLS
with
GRP,
see
between
Jaeger
agent
and
the
collector,
but
also
even
configure
tails
TLS
between
all
the
storages.
B
A
Back
so
this
is
something
that
we
didn't
mention.
Yet
this
is
still
work
in
progress.
There's
actually
a
bug
in
the
last
release.
So
that's
why
I
didn't
want
to
show
it
really
a
live
version,
but
it
will
get
fixed.
It's
an
easy
bug.
So
what
we've
been
working
on
is
service
graphs.
We
talked
about
series
graph
yesterday,
but
traditionally
those
service
graphs
are
based
on
just
a
single
hop
relationship
between
services,
right,
which
means
that
you,
you
can
reason
about
what
your
immediate
dependences
are
for
service,
but
not
beyond
that.
A
You
cannot,
for
example,
see
like
if,
if
your
dependencies
depend
on
something
else
and
then
depend
on
something
else,
can
you
tell
whether
you
depend
on
something
also,
not
especially
if
you
trying
to
figure
out
your
SLI,
so
things
like
that,
you
can't
really
from
the
single
hub
diagram.
So
all
you
know
is
that
you
depend
on
that
guy.
Maybe
it
just
replies
with
the
characters
out
all
the
time
right.
A
So
we
we
started
working
on
what
we
call
like
deep
dependency
diagram,
so
transitive
diagrams,
where
the
the
diagram
is
built
on
the
full
path
that
we
observed
in
the
traces.
And
then,
when
you
look
at
it
in
the
integral
form,
you
can
actually
use
like
mouse,
overs
and
and
seed
all
the
paths
going
through
their
services
and
in
this
example
like
it
to
to
build
that
diagram
for
the
full
arc.
Each
actually
requires
some
data
mining
jobs
which
is
not
available
open
source.
A
A
You
can
see
on
screen
for
some
reason
anyway.
There's
supposed
to
be
a
button
right
here
in
the
top
right
corner,
say
and
like
show
show,
is
graph
and
when
you
do
that
it
looks
like
this.
So
again
it's
it's.
It
looks
very
much
like
a
regular
service
dependency
diagram,
but
the
the
connections
are
life
and
they're
sort
of
sensitive
to
where
the
weather
you
are,
and
you
can
focus
this
diagram
to
on
any
service.
So
it's
actually
oriented
to
specific
service
right.
A
So
it
only
shows
all
the
path
that
comes
with
a
service
shown
in
red.
Not
the
rest
of
the
work
is
actually
so
that
that
mode
is
like
very
useful
when
you're
trying
to
investigate
accurate
dependencies,
or
we
had
sometimes
scenarios
at
uber
where
people
were
trying
to
move
services
between
data
centers
and
where
they
couldn't
figure
out
what
the
real
dependencies
are,
because
no
one
really
knows
in
production.
A
A
B
B
On
the
roadmap,
we
are
trying
to
make
it
easy
for
data
scientist
or
anybody
who
wants
to
play
with
traces
to
to
basically
get
them
analyze
them
and
try
to
you
know,
find
interesting
information
about
what
happened.
So
there
are
two
parts
we
want
to
create
a
DSL
trace
DSL,
which
makes
it
very
easy
to
extract
features
from
trace
data
and
also
provide
Jupiter
notebooks.
So
you
can
very
easily
deploy
it
and
go
play
with
it.
B
A
That
might
be
useful
in
some
cases,
but
they're
not
utilizing
like
the
full
power
of
the
tracing
data.
That's
it's
a
new
database
and
also
querying
for
the
database
from
the
database
directly
is
is
actually
difficult
because
there
are
some
questions
that
you
may
want
to
ask.
That
cannot
be
answered
by
regular,
whether
it's
a
column
or
regular,
relational
database.
A
It's
you
can
just
express
those
those
questions,
for
example,
if
if
your
question
is
I
know
I
want
you
to
know
something
about
like
a
trace
which
contains
series
a
colon
series
because
ER
and
C
right,
it's
it's
a
graph
question
essentially,
but
if
typical
Trey
simply
trace
storage
is
implemented
either
as
a
key
value
store
or
the
relational
database.
So
with
relational
when
you
store
one
span
at
a
time.
A
That's
like
you
asking
a
question
about
three
records,
essentially
in
their
relationship,
so
you
potentially
can
express
it
with
like
nested
joins,
but
it
gets
really
really
ugly
and
that's
why
we
wanted
to
to
shift
towards
more
like
a
graph
ways:
DSL
I
Kremlin,
where
you
get
the
whole
trace
in
memory,
and
you
can
do
anything
you
want
with
it.
You
can
ask
any
kind
of
questions
extract
any
metric
from
it
metric
me
and,
like
you,
calculate
something
some
feature
about
this.
For
example,
how
much
this
particular
service
call?
A
What
percentage
of
the
services
on
a
critical
path
right
and
and
then
the
reason
we
want
to
do
tribute.
A
dog
book
is
because
developing
those
kind
of
programs
that
extract
features,
that's
pretty
advanced
use,
usage
of
tracing,
but
without
having
that
framework
or
that
capability
in
place.
It's
pretty
difficult
to
do
otherwise,
like
I
said
SQL
join
like
joins
us
very
hard
to
write
for
this
data.
A
Big
Map,
Reduce
George's
like
difficult
so
Facebook
had
this
implementation.
Actually
in
their
internal
system,
which
was
very
successful
where
you
can
start
with
a
notebook,
you
can
potentially
write
your
query
or
your
feature
extraction
code
over
small
data
set,
which
you
can
iterate
and
run
quickly,
and
then
you
got
a
very
good
signal
from
it.
A
Hopefully,
and
then
you
just
take
that
same
thing
and
you
run
it
against
a
large
volume
of
data,
or
maybe
you
put
it
in
production
and
versus
streaming
data
right,
and
so
this
is
kind
of
where
we
want
to
go,
because
it's
of
course
it's
a
power
user
tool.
I
don't
expect
like
regular
engineers
necessarily
go
in
there.
But
if
you
have
a
dedicated
performance
engineering
tool,
sorry
performance,
engineering
team,
whose
responsibilities
like
okay?
Well,
let's
optimize
efficiency
across
the
fleet
or
saw
some
performance
issues
in
the
service.
A
B
We
would
like
to
also
build
a
community
around
this,
so
people
would
run
this
aggregation
jobs
internally
and
comment
whether
they
find
any
value
in
it
and
once
we
won,
we
proved
that
you
know
this
specific
job
is
useful.
We
can
be
relevant
that
and
created
very
simple
deployment
without
using
Jupiter
on
books
and
so
on.
A
This
is
basically
the
same
thing
as
what
we've
said
about
the
tag.
Matching
sampling
is
kind
of
ad
hoc,
something
where
you
can
allow
users
to
express
their
sampling
policy
in
more
details
than
just
probabilistic,
or
something
very
simple
that
we
have
today,
it's
more
about,
like
oh
I,
want
to
look
at
these
parameters
of
the
trace
and
and
make
a
sampling
decision.
So
that's
part
of
that
work.
I
guess
tell
by
sampling,
is
what
we've
discussed
before
that.
B
Open
telemetry,
we
have
a
lot
of
clients
in
different
languages
which
takes
a
lot
of
time
to
maintain
them,
and
once
the
open
telemetry
is
ready,
its
production
ready.
We
would
like
to
basically
deprecated
our
clients,
so
we
can
focus
more
on
these
data,
analytics
jobs
and
provide
more
features.
You
know,
I,
think
that
the
data
collection
should
be
somehow
open,
sourced
and
used
by
maybe
different
systems,
and
then
the
jäger
can
focus
really
on
the
providing
value
from
this
data
to
the
end
users,
rather
than
just
collecting
that.