►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Jaeger Project Deep Dive - Juraci Kröhling, Red Hat (Intermediate Skill Level)
About Juraci
Juraci is a software engineer at Red Hat working with the Distributed Tracing team on the Kiali project. He is a core contributor to Jaeger and is also active in the OpenTracing community.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Thank
you
everyone
for
joining
today.
This
is
the
Jaeger
project
deep
dive.
My
name
is
julissa
Shawn
Crowley
I'm,
a
software
engineer
at
Red,
Hat
and
I
work
at
the
chaotic
project,
more
specifically
under
disability
tracing
team,
meaning
that
I
work
mostly
on
Jaeger
and
open
tracing.
Now,
how
many
of
you
were
here
yesterday
for
the
project?
Intro,
yeah,
good,
perfect,
so
just
like
yesterday,
I'm
starting
with
some
assumptions
about
your
knowledge
right.
So,
let's
we
have
only
35
minutes
again
and
I'll,
then
just
move
relatively
quickly
and
I'm,
starting
from
where
I
left
yesterday.
A
Just
for
those
who
were
not
here
yesterday,
our
context
here
for
you
is
micro-services,
so
we
are
solving
a
problem
that
is
very
common
in
micro-services,
the
problem,
the
observability
problem,
or
specifically,
where
one
of
the
pillars
of
observability
daughter
to
be
another
2min,
matrix
and
logging
right.
We
talked
yesterday
about
what
is
the
tracing.
Why
is
it
needed?
Why
is
it
relevant
for
micro
services?
We
talked
at
why
open
tracing
so
why
open
tracing
is
required
or
what
it
were
are
the
advantages
of
open
tracing?
What
is
the
context?
A
There
is
one
that
runs
alongside
with
your
application
and
that's
the
component,
that
is,
that
is
responsible
for
capturing
the
data
and
sending
somewhere
right,
and
this
is
the
the
Jaeger
client
and
the
client
when,
when
we
think
about
Java
or
goal,
it's
a
library
that
you
ship
with
your
application,
either
part
of
the
binary
in
goal
or
as
a
jar
in
your
wor
packaging
or
a
jar
packaging
for
Java.
Now
that
that
that
client
sends
data
to
an
agent
or
to
a
collector,
the
collector
writes
data
to
the
storage
and
the
query.
A
The
query
service
reads
from
storage
right,
so
Mars
going
tip
dive
into
the
client.
The
client
is
the
party
that
is
responsible
for
actually
building
these
pens
right.
So,
if
you
remember
yesterday,
span
is
a
data
structure
that
records
the
time
information.
So
how
long
when,
when
a
span
started
and
how
long
it
took
to
finish,
it
also
stores
a
log
information.
A
If
you
tell
it
to
tags
and-
and
it
may
also
carry
data
along
to
you-
know
each
other-
the
child's
events
like
baggage
items,
the
Jaeger
client
packs
data
into
this
pen
and
dispatches
to
an
agent
in
the
default
case.
The
agent
is
something
that
is
running
locally
like
localhost
and
sends
data
via
UDP,
meaning
the
client
itself
can
be
very
thin
and
it
should
never
fail
right.
So
if
there
is
a
problem
in
sending
data
to
the
agent,
it
will
just
you
know
not
care
about,
so
it's
it
logs
metrics.
A
So
the
client
has
a
metric
standpoint
or
you
can
extract
metrics
and
consume
that
on
Prometheus.
So
you
know
that
X
spends
or
X
traces
were
expected
to
be
seen
on
the
server.
But
if
you're
seeing
less
number
of
traces
and
then
you
have
a
networking
problem
somewhere,
the
client
is
extensible,
so
you
can
implement
your
own
senders
like
HTTP
center
and
UDP.
Centers
are
two
that
we
have.
So
if
you
need
a
different
one,
you
can
implement
those
and
different
reporters
as
well
yeah.
A
So,
as
I
mentioned,
it
emits
internal
metrics
for
the
Java
client,
you
can
plug
a
micrometer
integration
and
with
that
you
can
plug
any
micrometer
back-end
that
you
want
all
right
so
who,
if
you're
familiar
with
micrometer,
okay,
yeah
cool,
so
yeah
you
can
plug.
That
means
one
micrometer
back
end
is
for
me
it's
right,
so
you
can
get
this
information
for
me
to
use
quite
easily
or
stats,
D
or
jmx
or
any
other
backend.
That
is
supported
by
a
micrometer.
A
So
it
is
a
client,
so
the
agent
again
it's
a
component
that
sits
very
close
to
the
client,
usually
localhost.
Now
what
localhost
means
today
is
quite
cloudy
right,
so
yeah,
so
we
don't
know
what
localhost
is.
Is
it
the
the
bare
metal
that
we're
running
on
or
is
it
in
Cuba?
This
context?
This
is
the
node
or
is
it
the
pod
right
so
yeah?
So
for
our
case
here
we
just
say
it
sits
close
to
the
client,
the
agent
acts
as
a
buffer.
A
It
receives
data
via
UDP
and
sends
data
to
a
collector
somewhere
science
via
TCP
by
a
usual
thrift
or
regular
HTTP
call
now
the
collector.
The
collectors
sole
purpose
is
to
write
data.
Today,
storage,
not
sounds
quite
simple,
but
when
we
are
talking
about
storage,
nothing
is
really
simple.
Is
it
so
we
write
two
elasticsearch
and
cassandra.
A
We
also
have
an
in-memory
datastore,
so
ephemeral,
but
we
also
accept
the
agriculture
also
accepts
data
in
different
formats.
So
if
you
have
a
client
instrument
using
Zipkin
libraries,
bigger
collector
can
accept
that
and
also
your
client
may
talk
the
jäger
protocols
directly
from
your
client
to
your
collector,
so
bypassing
the
agent.
A
If
your
environment
does
not
support
UDP,
for
instance,
and
then
we
have
the
Jaeger
query,
the
eager
query
is
usually
bundled
with
the
UI.
So
technically
the
UI
and
the
query.
There
are
different
components.
The
query
is
only
a
rest
service
that
accepts
queries
and
gets
data
from
the
vacuum,
storage
and
returns
JSON.
A
A
We
actually
thought
about
splitting
the
Y
with
the
query,
but
we
see
no
real
purpose
on
earth,
because
the
query
is
the
least
of
your
problems
when
you're
trying
to
scale
alright.
So
it's
only
a
limited
amount
of
people
looking
at
the
traces,
but
you
have
a
very
high
number
of
people
or
machines
writing
data.
So
the
query
is
the
we
have
no
benefits
in
splitting
the
Y
right,
so
this
is
the
main
overview
for
the
components
themselves.
A
Now
it's
a
deep
dive,
meaning
I,
have
to
show
you
how
to
configure
it
or
tell
you
how
to
configure
jäger.
So
there
are
the
main
idea
for
the
components
for
the
back
back-end
components
for
jäger.
If
you
follow
this
idea
from
the
12
factor,
apps
that
your
application
should
be
configurable
through
a
different
sources
of
your
mate
of
you
know
from
from
different
sources,
so
configuration
file,
environment
variables
and
comment
line,
interface,
switches
or
options.
A
So
the
first
part
is
how
we
can
configure
using
a
cuban
ideas,
config
map,
this
copy
and
paste
from
one
of
our
examples.
So
you
specify
config
map
and
within
the
data
for
that.
For
that
key,
you
specify
spam
storage
type
on
Cassandra.
If
you
want
to
specify
Cassandra
data
register,
you
can
override
that
by
passing
a
a
command-line
interface
option,
like
means
being
storage,
spender
type,
is
memory
equals
memories.
A
So
this
is
the
pattern
I'm
not
going
to
show
you
all
the
options.
The
easy
way
of
finding
those
options
is,
you
know,
docker,
run
and
then
ministry
needs
help.
It
prints
everything
or
if
you
have
the
binary,
just
run
the
binary
ministry
yourself.
We
try
to
document
every
option
so
so,
and
so
if
you,
if
you
are
looking
for
tuning
yogur,
we
provide
some
options.
Some
hooks
one
of
them,
or
perhaps
the
tube
more
important
ones,
are
queue,
size
and
workers.
So
you
can.
A
If
your
machine
is
really
beefy,
you
can
increase
the
number
of
workers,
and
if
you
have
a
lot
of
memory,
you
can
increase
the
queue
size
and
you
can
also,
if
you,
if
you
have
a
lot
of
requests
coming
in.
If
you
have
a
lot
of
data
to
digest,
you
may
want
to
play
with
a
simply
right.
So,
instead
of
sampling,
every
single
request
that
comes
in
you
may
want
a
sample.
A
A
Another
another
aspect
of
the
backend
components
that
we
have
is
a
support
for
exposing
telemetry
data.
So
it's
both
exposing
metrics
right.
So
by
the
fall
we
expose
data
in
Prometheus
firmage,
so
we
provide
an
endpoint
with
a
bunch
of
metrics
like
how
many
traces
were
started,
how
many
of
those
were
finished,
so
you
should
have
a
one
to
one
relationship
most
of
the
time,
so
it's
okay
to
have
some
is
not
finished.
A
So
there
are
two
backends
that
we
support
right
now:
X
part
expires
or
XP
bar,
which
is
I,
think
the
default
Fargo
and
Prometheus,
which
is
a
default
for
everything
right.
So
everybody's
talking,
Prometheus,
oh
yeah,
so
that
was
it
for
the
configuration
monitoring
and
you
know
that
the
simple
parts
of
the
back
end,
and
now
we
talk
more
about
how
to
run
it.
So
the
simplest
way
of
running
Jaeger
is
the
all-in-one
docker
image.
It's
also
known
as
the
standalone.
A
If
you
download
the
binary
from
the
github
release
page-
and
this
is
as
the
name
implies-
it's
a
package
that
contains
everything
right,
so
you
just
run
the
binary
and
it
starts
the
collector.
The
query,
the
agent,
the
why
everything
on
the
same
process
opens
a
bunch
of
ports
and
you
can
start
interacting
with
it.
A
This
is
quite
ok
and
it's
you
know
it's
it's
quite
easy
to
use
and
if
you
have
only
one
servers,
one
one
server
with
one
service
running
on
the
same
server,
that's
what
you
should
be
using,
but
that's
not
always
the
case
you're
not
always
using.
You
know
one
server
with
one
service
everything
there.
Then
you
probably
want
to
use
a
deployment
protection.
Let
me
call
it
so
it's
not
that
the
other
one
is
not
suitable
for
production.
A
It's
just
that
this
use
case
is
probably
what
you
want
to
use
when
you
go
to
production,
because
each
component
is
a
binary,
so
the
collector
is
a
binary.
The
agent
is
a
binary.
The
query
UI
is
another
binary
on
kubernetes.
You
could
apply
them
on
their
own
pods
and
scale
them
up
and
down.
Accordingly,
all
right.
So
to
help
you
with
that,
we
provide
some
openshift
in
kubernetes
templates.
It's
as
easy
as
cooker.
All
munis
f
apply
min
as
a
fan
and
the
template
file.
A
The
tricky
part
here
is
storage,
so
for
those
of
you
who
have
played
with
the
storage
on
kubernetes
or
on
any
other,
a
scalable
environment,
you
know
that
storage
is
a
tricky
part.
Alright.
So
the
word
of
caution
here
is:
do
not
use.
Our
storage
templates
use,
bring
your
own
storage
template,
so
we
provide
some
for
guidance,
elastic
search
and
Cassandra,
but
you
should
really
look
into.
A
You
know
apply
that
to
your
infra,
so
if
you
have
a
ready,
Cassandra
use
the
same,
the
same
patterns
that
you
are
using
for
other
applications
and
we
also
have
a
community
contributed
helm
chart.
It's
been
a
while
since
I
personally
use
it,
but
I
do
recommend
you
take
another
look
and
see.
If
that
helps
you
I
know
it.
It
just
helped
a
lot
of
people.
So
so
that's
the
main
idea
for
a
deployment.
You
know
the
main
two
options
is
the
all-in-one
or
production.
A
The
next
one
is
talking
about
the
agent,
so
I
kind
of
hinted
it
earlier
that
we
don't
know
what
local
host
trees
right.
So
when
we
are,
you
know
some
years
ago,
it
was
quite
easy.
Is
local
host
is
local
host
and
that's
it
once
you
sell
one,
two:
seven:
zero,
zero
one:
that's
it
bare
metal
host
process,
a
host
level
process.
That's
it
now!
A
A
So
as
a
daemon
set,
you
probably
you
probably
want
to
use
a
demo
set
when
you
have
all
the
pots
on
that
node
running
belonging
to
the
same
tenant
or
having
the
same
sort
of
security
concerns
right,
so
they
belong
to
the
same
owner.
Basically,
if
you
are
a
deploying
year
or
the
agent
in
a
in
a
machine
that
is
that
belongs
to
one
company
alone,
if
you're
not
a
like
a
cloud
provider,
then
it's
probably
okay
to
have
as
a
diamond
setter
now
the
tricky
part
is.
A
The
agent
has
a
long-lived
connection
to
a
collector.
So
if
you
have
ten
collectors
but
one
node,
it
means
you
have
one
agent
talking
to
one
collector
alone,
all
right,
so
you
have.
If
you
have
a
lot
of
data,
you
probably
do
not
want
to
use
a
demo
set.
You
probably
want
to
use
a
sidecar
so
that
you
spread
your
load
throughout
during
from
and
the
sidecar
approach
is
I.
Think
I
don't
need
to
tell
what
sidecar
is
so.
A
The
second
container
in
the
pod
would
be
the
agent
and
your
application,
the
main
container
for
the
pod
sends
data
to
the
agent
by
a
local
host
and
that
agent
that
makes
a
long-lived
connection
to
the
collector.
So
if
you
have
10
collectors
and
you
have
10
instances
of
your
service,
you
have
10
agents
each
one
have
any
long-lived
connection.
A
So
if
you,
then
that
means
you
know
he's
in
theory,
if
you
use
all
your
collectors,
so
this
about
the
point,
the
agent
now
one
question
that
came
a
couple
of
times
already
was
how
about
multi-tenancy.
How
about
if
I
want
to
run
Jaeger
in
a
multi-tenancy
setup,
and
then
we
ask
you
what
is
multi-tenancy
then
for
eager
alright.
So
there
are
several
aspects
of
multi-tenancy
that
are
not
well
defined
for
us.
So
what
is
Tennessee?
Is
it
Tennessee
at
this
pain
level?
A
So
if
you
have
two
or
three
tenants
each
one
of
those
in
a
concubine
areas,
namespace
or
in
a
OpenShift
project,
you
can
just
then
just
install
a
Jaeger
in
one
in
each
of
those
namespaces
right.
I.
Think
the
overhead
for
that.
For
the
agent
it's
like
the
20
megabytes,
so
the
binary
is
only
20
megabytes
and
the
whole
infra
the
whole
binary
they
stand
alone
is
less
than
30
megabytes.
A
You
can
is
to
extend
the
client
to
implement
your
own
tenancy
model,
alright,
so
in
that
case
we
would
be
having
the
client.
So
this
is
an
image
of
the
the
traditional
model.
I
would
call
it
where
one
agent
talks
to
one
collector
belonging
to
a
tenant,
and
if
there
is
another
application
we
have
the
same
but
talking
to
a
different.
So
we
just
duplicating
from
right.
A
A
B
A
Very
similar
approach
we
took
with
regards
to
security
all
right,
so
what
is
security
for
us?
So
what
should
we
protect?
Should
we
protect
the
agent?
So
should
we
trust
that
the
agent
is
receiving
data
from
untrusted
sources?
Like
do
we
have
micro
services
in
our
infraorbital
areas,
cluster
which
are
not
trusted,
or
should
we
just
assume
that
everything
that
is
running
there
is
running
there
for
a
reason,
and
we
trust
people
who
deploy
those?
A
So
there
are
scenarios
where,
like
in
startups,
which
was
every
code
that
is
running
there
in
theory
right
but
for
banks
or
for
healthcare.
We
probably
want
to
do
some
more
security
tightening
there
and
just
like
with
multi-tenancy.
It
looks
like
everyone
has
a
different
requirement
and
again
because
the
components
are
lightweight
we
can
afford
to
also
have
you
know,
to
duplicate
or
to
have
a
auditor
components
order
to
delegate
those
things
to
other
parts
of
the
infra,
like
security
proxies
all
right.
A
A
Alright,
so
one
feature
that
is
missing
is
telling
the
agent
to
send
how
to
get
a
token,
a
bearer
token
or
JWT
how
to
get
it
and
how
to
send
it,
but
a
client
a
they
are
able
to
do
that
already
right,
so
they
you
can
specify
when
configuring
the
client
you
can
either
exposing
environment
variable
or
you
can
just
configure
the
client
programmatically
saying
this
is
the
username
and
password
or
the
bearer
token
to
be
used.
The
collector
itself
is
not
aware
of
the
security
part
because
the
proxy
runs
before
it.
A
So
if
it
got
a
request
and
it's
then
it's
acceptable
right.
There
is
a
an
example
of
how
to
do
that.
Using
key
clock
click.
Clock
is
a
single
sign-on
solution.
Open-Source
project
as
well-
and
you
know
it
just
works
right.
So
why
should
we
do
this
kind
of
thing
on
eager
if
we
can
leverage
other
parts
of
the
in
front
now
on
east
geo,
we
could
use
as
well
multiple
Telus
connections
so
that
we
trust
that
we
are
talking
to
services
which
are
trusted
in
our
infra.
A
It
would
be
a
nice
blog
post
by
the
way
so
yeah
and
again
the
same
with
multi-tenancy.
If
your
security
requirements
they
are
not,
they
cannot
be
satisfied
with
the
scenario.
You
can
extreme
point
your
own
reporter
right,
so
it's
extensible
yeah,
so
the
road
map,
so
we
we
have
in
the
roadmap
for
for
jäger.
Those
three
features.
A
The
first
one
is
data
pipeline
right,
so
imagine
that
you
that
we
are
receiving
spans
and
instead
of
just
letting
data
flow
to
Cassandra
or
elasticsearch,
we
want
also
to
have
this
data
on
spark.
We
want
to
have
some
knowledge
about
this
data.
We
want
to
do
some
extract
insights
from
it.
We
want
to
do
what
you
know.
Wanna
understand.
A
What
is
the
service
path
for
a
given
request
pattern
so
that
we
can
say
well
whenever
my
order,
my
check-out
endpoint,
is
called,
then
those
are
the
services
which
are
involved
so
that
when
I'm
doing
an
upgrade
or
you
know,
I
can
see
what
are
the
critical
paths
in
my
application.
So
when
I
do
end
up
I
know
well,
those
are
the
things
that
I
have
to
take
care
of
before
doing
the
upgrade.
A
The
second
thing
is
adaptive
sampling
that
I
mentioned
before
so
adaptive
sampling
is
on
the
fly,
letting
the
same
plane
change
based
on
our
current
requirements.
Right
so
right
now
we
can
kinda
achieve
adaptive
sampling
by
playing
with
cuber
NIDA
spots,
so
we
could
have
a
one
service,
backed
by
two
sets
of
deployments
on
kubernetes
and
then
or
sto
as
well.
We
can
just
switch
like
blue,
green
and
a
B
and
so
on
and
play
with
sampling
rates,
but
we
can
do
actual.
A
You
know
more
than
that,
so
adaptive
sampling
is
to
provide
more
features
on
a
smarter
decision
about
the
same
thing.
So
should
I
sample
this
request
or
should
I
not
sample
is
required?
That's
the
basic
decision,
the
third
one
is
discharge
plugging
and
this
one
is
quite
hot
right
now,
so
we
are
it.
We
are
actually
working
on
it
on
this
one
at
this
moment
and
we
are
trying
to
figure
out
what
is
the
best
way
of
doing
storage
plug-ins
for
our
back
end.
So
the
back
end
is
a
goal.
A
Application
is
a
goal
and
application.
So
there
are
quite
a
few
different
ways
of
dream.
Plugins
looks
like
the
preferred.
One
was
cheese-like
goal:
plugins
there
is
something
called
the
goal
plugins.
The
problem
is:
if
the
plug-in
code
has
a
dependency
on
a
on
a
version
of
package,
a
package,
a
version
one
and
the
main
code
has,
you
know,
packaged
a
version
two.
Then
you
have
both
in
the
binary
in
the
end
and
that
leads
to
funny
situations
yeah.
So
that
was
pretty
much
it
that
I
had
I
talked
actually
too
much.
We
are
almost.
A
We
have
only
a
few
minutes
left
for
questions.
One
message
that
I
want
to
share
with
you
or
what
I
want
to
ask
you
is
to
join
our
community.
So
if
you
guys
were
at
the
keynote
today,
there
was
a
slide
where
people,
just
you
know,
raise
hands
and
asked
asked
the
community
for
something
or
ask
the
demo
chance
for
something,
and
so
remember
that
that
slide
when
you
know,
do
raise
your
hand
and
ask
us
when
you
and
you
need
something
and
yeah.
B
A
Our
community,
you
know,
if
you
have
something
interesting
to
share
feedback
on
how
on
you
did
something
with
Jaeger,
so
you
you
deploy
it
in
some
way
share
with
us.
We
are
also
happy
to
take
gas
blog
post
on
our
medium
yeah,
so
we
meet
every
two
weeks
on
zoom,
it's
open
for
everyone.
The
meeting
details
are
on
the
main
Jager
core
repository
github
repository,
we're
also
active
on
Twitter,
and
we
also
have
a
mailing
list.
So
it's
quite
easy
to
reach
out
to
us
yeah
and
that's
pretty
much.
A
The
idea
is
that
this
would
be
a
sort
of
question
is
if
there
is
a
way
of
limiting
how
many
pens
a
client
can
report
right
yeah.
So
this
is
a
good
idea
actually,
so
this
is
something
that
I
can
see
implemented
as
a
as
a
sampling
strategy.
Yeah,
sorry,
so
you
can.
You
can
talk
to
him
and
exchange
later.
If
you
want.
A
So,
but
so
also
part
of
the
adaptive
sampling
is
blacklisting
services,
so
you
could
probably
blacklist
that
service
after
sometime
after
spamming,
the
the
agent
or
the
collector,
but
it's
not
something
that
we
have
right
now.
So
this
is
a
great
idea,
so
opening
an
issue
on
the
get
github
repo
with
this
idea,
I
think
if
it
is
not
on
people's
mind
right
now,
it
should
be.
A
A
Right
so
that
yeah,
that's
exactly
what
I
think
the
guy
might
even
be
here.
So
if
you
are
just
yeah
so
there
so
talk
to
that
guy,
so
the
idea
is
to
if
you
have
an
io
weight
and
you
don't
know
why
you
would
use
trace
information
to
understand
what
went
wrong
or
what's
going
on
there
right
so
I
think
he
asked
him
to
start
a
conversation
with
the
community
about
that.
So,
if
you're
also
interested
in
this
part
so
just
join
the
conversation.
It's
easy.
A
A
So
open
tracing
is
an
instrumentation
API.
Open
census
is
a
so
do
you
know
micrometer
for
Java.
No,
so
open
census
is
a
a
complete
set
of
components
to
have
not
only
tracing
what
also
monitoring
and
logging
for
applications.
So
there
is
a
Jager
exporter
for
open
census.
It
works
it's
quite
different
than
open
tracing.
My
personal
opinion
is
that
as
a
so
for
monitoring
job
applications,
I
would
use
micrometer,
because
that's
what
I
see
as
a
standard
right
now
for
tracing
I
would
use
the
open
tracing
API.
A
Because
that's
vendor-neutral,
basically
so
I
can
ship
my
application
with
a
no
op,
tracer
and
and
I'm
not
using
any
vendor
I'm,
not
using
any
tracing,
but
the
tracing
hooks
are
there.
So,
but
if
you,
if
you
want
a
complete
solution,
if
you
want
one
API
to
do
everything
for
you,
then
it
should
certainly
go
with
open
census.
A
Spence
dropped
so
at
the
agent
level
I
Spencer
opted.
So
if
you
have
a
mismatch
between
the
spans
reported
by
the
client
and
the
the
spends
that
you
see
on
the
server,
then
there
are
spends
been
dropped
somewhere
either.
The
queue
is
whole.
You
know
at
the
agent
side
or
the
collector,
maybe
I
think
the
collector
is
not
dropping
anything,
but
the
agent
certainly
is
if
the
queue
is
full,
for
instance
right.
So
that's
the
first
early
indication
of
something
going
wrong.
A
A
A
So
yeah
one
cool
idea
that
we
are
starting
to
think
about
is
like
the
table
based
simply
meaning
the
decision
to
trace
or
not
trace
is
done
after
the
fact
right.
So
we
store
data
somewhere
and
we
either
drop
it
or
we
record
it
and
we
send
it
somewhere
because
you
may
have
an
error
happening
like
three
micro-services
down
there,
but
the
first
microservice
is
not
aware
of
that
or
you
know
at
that
time
that
it
started.
It
did
not
know
that
it
would
fail
right,
so
it
it
will
place
data
somewhere
or
store
data.
A
A
So
alerting
so
the
question
is
whether
there
is
alerting
for
jäger
right.
Basically,
there
is
not,
but
there
is
an
open
tracing
contra
plug
that
converts
your
spend
data
into
metrics
and
expose
that
to
promise
use
and
so
on,
and
then
you
can
use
that
data
to
create
an
alert
right
so
either
with
alert
manager
from
from
from
premises
or
plug
any
other
alerting
system.
So
here
itself
or
you
know
so,
the
trace
is
good
for
debugging
an
issue
and
not
for
really
monitoring
outside
all
right.