►
From YouTube: Jaeger: The Future with OpenTelemetry and Metrics- Jonah Kowall, Logz.io & Joe Elliott, Grafana Labs
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Jaeger: The Future with OpenTelemetry and Metrics - Jonah Kowall, Logz.io & Joe Elliott, Grafana Labs
In this session, we will start with an introduction to the Jaeger distributed tracing project and the basics of distributed tracing. Jaeger recently deprecated its native clients in favor of the clients maintained by the upcoming OpenTelemetry project. We will explain what this means to you as users and why we are changing the path forward. To help facilitate this transition, we will cover auto instrumentation of an application using an OpenTelemetry client and we will cover multiple best practices to build a scalable trace pipeline to deliver this data to a Jaeger backend. Moving Jaeger from a tracing system to a monitoring system has been the big push for the project in the last year. Made possible by OpenTelemetry and the processor layer which allows for the creation of metrics derived from traces in the pipeline. Operational monitoring is now possible using the new monitoring tab, which adds metrics capabilities to Jaeger UI via another graduated project, Prometheus. We are always seeking new collaborators, contributors, and users. We need your help, please join us!
A
Hey
everyone
welcome.
Welcome,
welcome
to
the
Jaeger
maintainer
talk
we're
going
to
be
talking
about
a
lot
of
cool
stuff,
I'm
Jonah,
Cowell
and
I'm
also
joined
on
stage
by
Joe
Elliott
nice
to
meet
you
and
we're
going
to
be
covering
a
lot
of
things
about
Jaeger,
open
Telemetry,
where
the
project
is
going
kind
of
where
it
is,
as
you
know,
Jaeger's
a
graduated
project,
one
of
a
handful
very
mature
and
well
adopted
in
the
ecosystem.
Joe
tell
us
a
little
bit
about
yourself.
B
A
Since
and
I'm
Jonah
Cowell
I'm,
the
CTO
at
logs
IO
also
work
on
Jaeger
among
other
projects.
I
do
a
lot
of
diving
because
I
live
in
a
nice
place
for
that.
So
that's
what
I
tend
to
do
when
I'm
not
working
some
photos
I've
shot
recently
underwater
and
yeah.
So
in
the
agenda
today,
we're
going
to
talk
about
distributed,
tracing
and
Jaeger
a
little
bit
deeper
into
open,
Telemetry
and
then
kind
of
as
we
go
through
we'll
get
a
little
bit
deeper
into
certain
features
that
we
built
out.
A
We're
going
to
do
a
really
quick
live
demo
of
some
of
the
stuff
that
we're
doing
hopefully
I'm
going
to
do
it
over
Wi-Fi
too,
which
will
be
really
fun
and
then
I
doubt
we're
gonna
have
time
for
questions
but
come
up
after.
We
also
have
a
slack
Channel
which
will
hit
on
right
at
the
end.
So
with
that
I'm
going
to
dive
into
a
really
quick
intro
for
those
of
you
that
are
just
getting
started
or
or
maybe
don't
know,
a
lot
about
distributed.
A
A
Each
of
those
hops
or
steps
that
are
going
through
is
a
span
and
that's
kind
of
a
little
teeny
piece
of
the
end-to-end
transaction
inside
the
span.
You
can
have
all
kinds
of
metadata.
You
can
have
tags
you
can
attach
logs.
You
can
attach
all
kinds
of
interesting
data,
that's
going
to
help
you
either
do
analytics
or
troubleshooting
and
there's
just
a
lot
of
different
use
cases
and
interesting
stuff.
You
can
do
with
tracing,
which
is
kind
of
the
most
advanced
signal
in
Telemetry
today
in
general
and
observability.
A
So
with
that
kind
of
talk
a
little
bit
about
the
relationships,
it's
really
important
in
a
trace
to
understand
how
those
spans
all
come
together
and
roll
into
into
what's
actually
happening
with
the
end
user
or
the
API
that's
being
used
or
exercised,
and
so
here
you
can
see
some
visualizations
and
you'll
see
more
of
them
inside
the
Jaeger
UI
that
explain
how
the
different
components
of
a
trace
come
together
and
roll
into
that.
So
obviously,
inside
the
data
itself
there
are
Pointers.
A
And
now
what
you
can
do
is
you
can
also
calculate
the
data
off
of
that
to
understand
errors
and
latency
and
do
all
kinds
of
cool
stuff
that
we're
going
to
talk
about
a
little
bit
later
and
here's
the
visualization
you
get
when
you
use
Jaeger.
So
if
you
do
install
Jaeger,
we
make
it
really
easy
to
get
started.
We'll
talk
about
that
and
you
can
get
these
cool
visualizations
of
the
tracing
data
that
you're
looking
at
and
with
that
Joe's
going
to
talk
talk
a
little
bit
about
Auto,
instrumentation,
cool.
B
So
we're
going
to
talk
about
the
open,
Telemetry
clients,
those
of
you
who
are
already
using
Jaeger
may
have
in
the
past
year
or
so
seen.
Maybe
a
message
or
a
warning
like
this
in
the
docs
or
on
your
Jaeger
client.
So
about
a
year
ago,
Jaeger
deprecated,
all
of
its
clients
for
all
of
the
major
languages.
In
favor
of
the
open,
Telemetry
clients,
the
community
decided
to
kind
of
converge
on
the
open,
Telemetry
clients,
as
they
kind
of
implemented.
B
The
good
news
here
is
that
there
is
very,
very
wide
language
support
for
jaeger.
In
fact,
I'm
pretty
sure
it's
a
super
I'm,
sorry
for
the
open,
Telemetry
clients,
which
I'm
pretty
sure
is
a
super
set
of
the
originally
supported
languages
in
Jaeger.
I
was
surprised
to
see,
for
instance,
Swift
is
supported.
So
for
all
of
you,
you
know,
writing
your
back
end
applications
in
Swift.
B
So
if
you're
using
Jaeger
Jaeger
implements
the
open
tracing
shim
as
well,
and
it's
quite
possible,
you
can
very
easily
swap
out
the
Jaeger
back
end
I
mean
your
client
for
the
open,
Telemetry
implementation
of
open
tracing
with
maybe
little
or
no
code
changes,
so
open
tracing
is
a
older
standard
that
was
an
API
standard.
Jaeger
implemented
open
Telemetry
also
supports
that
standard
and
I
have
a
good
Link
in
the
bottom
of
this
slide.
B
I'll
also
say
over
my
part
of
this
presentation:
there's
a
lot
of
languages
and
a
lot
of
details
and
it's
impossible
to
get
into
all
of
it.
So
I've
tried
to
include
links
in
all
the
slides.
The
PDF
we've
attached
to
the
schedule
should
have
this.
All
of
these
links.
I
encourage
you
to
go,
get
that
and
use
this
presentation
as
a
jumping
off
point
to
you
know,
get
into
the
details
of
your
own
systems.
B
Another
piece
of
good
news
is
that
the
configuration
of
open,
Telemetry
clients
is
very
similar
to
the
configuration
of
your
Jaeger
clients
all
through
environment
variables
and
in
a
lot
of
cases,
or
even
kind
of
like
a
one-to-one
mapping.
So
like
service
name,
there's
service
names
in
open,
Telemetry,
the
you
know
the
data
model
is
very
similar,
so
a
lot
of
this
configuration
is
just
going
to
be
able
to
translate
right
over,
and
this
is
of
course
not
exhaustive.
I
just
wanted
to
show
a
few
examples
of
of
environment
variable
rules.
B
You
were
likely
using
to
configure
your
Jaeger
clients,
and
you
know
showing
you
that
all
these
same
options
are
going
to
exist
over
here
for
your
open
Telemetry
clients.
So
we
have
like
sampling.
Options,
of
course,
is
very
important
in
tracing,
and
you
can
see
open.
Telemetry
is,
of
course,
going
to
support
that
as
well.
B
You
know
your
Jager
agent
host,
where
you're
pointing
your
agent,
the
propagation
which
is
very
important,
we're
going
to
talk
about
in
a
second.
All
of
these
kind
of
configuration
points
should
be
available
in
open,
Telemetry
or
something
similar
and
again
that
link
at
the
bottom
has
every
environment
variable
in
every
silly
little
combination
of
parameters.
You
can
pass
to
talk
about
the
autoness
of
this
instrumentation.
Everybody
gets
excited
about
tracing
Auto
instrumentation,
because
sometimes
tracing
can
feel
a
little
manual.
B
We
wanted
to
talk
about
open,
Telemetry
is
capable
of
in
this
case,
and
I
have
used,
maybe
three
or
four
of
the
different
open
Telemetry
clients
at
different
times
and
I.
Think
the
the
good
news
here
is
that
open,
Telemetry
generally
does
a
good
job
in
its
clients
of
meeting
the
expectations
of
the
community
and
the
Frameworks
that
they're
serving
so
in
Java.
B
There's
often
an
expectation
of
heavy
Auto
instrumentation
spring
in
particular,
has
a
lot
of
hooks
where
you
can
do
what
you
want
after
events
like
hdb
calls
and
other
events
in
your
in
your
framework.net
also
kind
of
has
a
history
of
some
of
this.
A
lot
of
these
Powers,
whereas
maybe
go,
is
a
language
where
people
expect
a
little
bit
more
manual
work
a
little
bit
more
manual
coding.
So,
in
my
experience,
diving
into
these
languages,
I'd
say
your
expectations
as
a
developer
is
Hope
are
hopefully
met
by
the
community.
B
If
you
just
jump
into
the
the
link
there
at
the
bottom,
you'll
find
your
own
language,
hopefully
and
they'll,
be
like
the
you
know,
hello,
World
style,
getting
started
guide
and
you
hopefully
in
an
hour
or
two
can
have
very
basic
instrumentation
set
up
and
you
can
be
writing
to
Jaeger,
and
you
can
start
seeing
that
you
know
open,
Telemetry
and
Jager
is
going
to
work.
Just
fine
and
you
can
migrate.
You
find
a
migration
path
for
your
existing
applications.
A
There's
also
I
just
wanted
to
add,
there's
actually
a
new
Go
Auto
instrumentation
based
on
ebpf.
That's
pretty
interesting.
It's
super
new,
but
I
think
we're
going
to
see
more
and
more
innovation
of
using
things
like
ebpf
to
do
more.
Automated
data
collection
so
definitely
keep
watching
the
profiling
and
ebpf
with
open
Telemetry,
because
it's
really
interesting
stuff
and
it'll
make
it
even
easier
and
deeper
data
collection.
Sorry
to
chime
in
no.
B
Absolutely
not
yeah,
Trace
propagation
is
also
going
to
be
very
important,
so
you
have
like
a
fleet
of
applications
and
they're
all
using
the
Jaeger
clients
and
they're,
using
what
are
called
the
Jaeger
headers,
probably
so,
Jaeger
propagates
Trace
information
with
the
headers
and
if
you
don't
Deep
dive
into
tracing,
you
might
not
know
this,
but
you're
going
to
need
to
know
this
real
soon.
If
you
start
rolling
out
open
Telemetry
clients,
because
open
Telemetry
by
default
communicates
with
w3c
headers
Jaeger
by
default,
communicates
with
jaeger
headers,
and
there
is
some
of
overlap
there.
B
Most
open,
Telemetry
clients
also
support
Jager
headers,
most
Jaeger
clients
can
talk
with
w3c
headers,
but,
as
you
start
rolling
out
some
of
the
first
apps
with
open
Telemetry,
you
might
find
some
traces
are
broken.
You
might
need
to
go.
Look
at
how
you're
propagating
header
values,
maybe
double
checking
standardize
on
one
of
the
two.
B
Whatever
is
more,
whatever
is
easier
for
your
particular
environment
and
then
you
can
get
your
full
traces
as
they
pass
through
both
you
know
your
your
instrumented
applications
and
your
open
Telemetry
instrument,
applications
and
finally,
I
wanted
to
show
an
example
of
Auto
instrumentation.
So
these
are
some
screenshots
from
a
Java
Auto
instrumented
app
that
I
threw
together
I'm,
not
a
Java
developer.
All
I
did
was
take
a
restful
application
that
you
know
has
git
and
post
endpoints
and
then
hits
a
relational
database.
B
We've
all
seen
this
kind
of
application
a
million
times
and
then
I
threw
the
Java
agent
jar
I,
just
linked
it.
Using
the
using
the
instructions
on
the
getting
started
page
in
open,
Telemetry
and
I
was
immediately
able
to
get
some
basic
tracing
going.
I
had
very
nice
attributes
from
both
my
hdp
side,
as
well
as
my
database
site,
I'm,
not
sure
how
visible
that
is.
B
Hopefully
it's
okay,
but
you
can
see
that
there
are
the
HTTP
method,
the
HTTP
path,
the
URL,
the
HTTP
version
that
was
in
use
are
all
automatically
being
added
as
a
tributes
to
my
spans,
my
trace
and
then
on
the
database
side.
I
was
kind
of
a
surprise
to
also
see
a
lot
of
deep
information
about
the
database
call
automatically
added
with
no
code
changes.
B
I'm,
seeing
like
the
database
query
string
the
name
of
the
database,
the
name
of
the
table
that
was
accessed
the
kind
of
operation
that
was
performed,
and
it
was
a
really
a
rich
set
of
information
for
almost
no
effort.
I
really
made
no
code
changes
just
linked
that
jar
file
and
it
was
ready
to
go
so
if
you
are
in
the
Java
Spring
World,
you
might
have
some
nice
surprises
on
your
the
ease
of
moving
to
open,
Telemetry
and
maybe
also
some
of
the
other
more
Auto,
auto
instrumented
kind
of
languages.
B
B
So
the
good
news
here
is
that
the
open,
Telemetry,
collector
and
the
clients
speak
every
Jaeger
protocol,
so
the
Jaeger
agents
expect
one
of
two
protocols
which
you
might
be
aware
of:
thrift,
compact
or
Thrift
binary.
There's
the
port
numbers
up
there.
The
Collector
supports
one
of
two
protocols:
normally
it's
grpc
from
the
agents
and
there's
also
a
lot
of
us
might
have
a
Kafka
queue
in
our
Jaeger
Pipeline
and
our
Trace
pipeline.
B
The
open,
celebrity
collector
can
read
and
write
all
of
these
protocols
and
it
can
write
to
the
Kafka
queue
in
the
formats
that
Jaeger
expects.
So
that
means
you
can
really
mix
and
match
open
Telemetry
components
along
with
your
Jaeger
components.
In
almost
any
combination
you
want
I
also
put
a
little
link
up
there.
It's
something
important
about
the
collector
itself
is
there's
two
there's
two
distributions
of
The
Collector
there's
one.
The
core
distribution
does
not
have
Jaeger
support,
but
there's
one
called
contrib
and
I've
linked
the
the
docker
image
there.
B
The
reason
we
might
consider
using
that's
also
probably
impossible,
to
read
the
reason
we
might
consider
using
the
open,
Telemetry
collector,
along
with
our
Jaeger
pipeline,
is
the
open.
Geometry
collector
has
a
lot
of
really
nice
processors
and
ability
to
mutate
our
spans
as
they
move
through
it.
You
might
find
that
some
of
these
nice
abilities,
for
instance,
I've,
highlighted
a
processor,
called
the
redaction
processor,
and
you
can
use
it
to
look
for
maybe
pii
or
other
patterns,
other
kind
of
information.
B
You
don't
want
to
pass
into
your
back
end
and
it
will
redact
that,
but
there's
other
processors
that
will
add
or
delete
attributes,
there's
processors
that
will
decorate
spans
with
additional
metadata.
The
open
television
collector
is
very
rich
in
its
ability
to
modify
and
work
with
Trace
data,
so
I
would
encourage
you
to
explore
the
collector,
as
you
are
kind
of
getting
into
the
open
Telemetry
ecosystem,
because
the
collector
supports,
like
we
said
all
of
the
same
protocols,
that
the
that
the
the
Java
I'm
sorry
the
Jaeger
pipeline
supports.
B
B
Now,
with
the
open,
Telemetry
client
I
can
use
the
Jaeger
agent
like
I,
always
have,
because
the
open
Trail
make
open,
Telemetry
client
can
write
directly
to
it
or
I
could
swap
that
out
with
the
open,
Telemetry
collector
there's
you
know
these
applications
can
do
both
can
do
the
same
thing.
They
can
both
read
the
same
protocols.
They
can
both
write
to
The
Collector
successfully.
So
you
have
a
lot
of
flexibility
in
your
pipeline.
B
A
lot
of
people
use
queuing
in
a
in
a
in
Jaeger
and,
like
we
said
before,
because
the
open
Telemetry
collector
can
put
Trace
data
in
the
Kaka
queue
in
the
format
Jaeger
expects
we
can
use
open
television
collectors
in
this
situation
as
well.
So
my
client,
maybe
I'm,
writing
to
a
mix
of
the
Jaeger
Pipeline
and
the
open,
Telemetry,
collector,
Pipeline
and
then
I
can
still
use.
B
My
Kafka
queue
I'm
pulling
with
my
Jaeger
and
Jester,
and
now
I'm
writing
into
the
database
like
I
always
have
so
there's
a
lot
of
flexibility
here.
There's
a
lot
of
power.
A
lot
of
a
lot
of
configuration
and
a
lot
of
documents
to
read
I
would
recommend
digging
in
and
seeing
what
you
can
start
doing
with
your
Trace
Pipeline
and.
A
So
what
the
reason
why
you
may
want
to
move
from
simple
to
Advanced
is
either
scale
reasons,
resiliency
reasons
or
other
kinds
of
business
requirements
that
you
may
have,
but
that's
typically,
why
people
ask
Kafka,
add
Kafka
one
of
the
common
questions
we
get
all
the
time
is
my
collectors
are
overloaded
or
my
back
end
is
overloaded.
How
do
I
help?
How
do
we
help
and
in
the
Jaeger
Channel
we're
going
to
tell
you
you
have
to
look
at
where
the
pressure
is
and
probably
add
some
kind
of
cue,
some
kind
of
back
pressure
mechanism.
B
Final
slide
for
me
just
kind
of
trying
to
be
silly
here,
the
point
it
being
because
these
applique
or
these
these
two
processors
are
so
easily
interchangeable.
You
know
you
can
do
almost
anything
you
want.
This
particular
pipeline
is
highly
unrecommended,
but
if
you
want
to
do
repeatedly
write
from
collectors
to
agents
to
Kafka
and
back
you
can
do
it,
and
maybe
you
have
some
bizarre
business
reason
to
go
about
building
a
pipeline
as
complicated
as
this,
but
yeah
with
that.
I
think
we're
heading
back
to
Jonah
sure
cool.
A
Yeah,
so
we're
going
to
talk
a
little
bit
about
Joe
mentioned
the
processors
there's
so
many
powerful
ones
and
new
ones
being
built
constantly,
like
you
just
started:
building
some
of
the
cool
things
that
show
architectures
service
Maps
exactly
these
are
all
processors.
So
one
of
the
cool
things
that
we
built
over
the
new
the
last
year
was
actually
a
new
processor
and
I'm
going
to
talk
about
that
and
how
it
comes
forward
in
in
Jaeger.
So
Jaeger
is
a
distributed
tracing
system,
as
you
saw
in
the
screenshots.
A
It
can
help
you
debug
and
understand
where
your
transactions
are
going,
but
a
lot
of
people
today
need
APM
I
want
performance
monitoring
performance
data.
So
how
do
we
move
from
distributed
tracing
to
understanding
performance?
It's
about
introducing
new
signals
aside
from
traces.
We
also
need
metrics,
and
we
need
to
understand
that
so
that
we
can
look
at
my
services
slowing
down
I'm,
seeing
more
errors,
trending
data,
this
type
of
thing
we
need
metrics
on,
so
we
decided
to
build
some
additional
components.
A
You
can
see
how
this
fits
into
a
pipeline
where
you're
calling
the
trace,
metrics,
processor
and
sorry,
it's
so
small
within
the
pipeline
right
here,
and
the
result
of
this
is
that
we
then
generate
metrics
from
these
traces
and
we're
able
to
do
cool
things
like
show
you,
the
metric
data
from
your
applications,
without
any
additional
instrumentation
and
I'm,
going
to
show
you
this
in
a
live
demo,
but
we
can
show
these
Prometheus
metrics.
You
can
visualize
them
in
grafana.
You
can
alert
on
them
in
alert
manager.
A
You
can
use
any
Prometheus
compatible
back
end
with
this
feature,
and
so
that
includes
obviously
Prometheus
cortex
Thanos
memir
m3db,
you
name
it
anything
that
does
prom
ql
will
work
with
this
and
has
the
right
exporter
in
open,
telemetry
and
I'm
going
to
show
you
a
really
quick
demo
of
how
this
looks
in
grafana
just
so
that
you
can
see,
hopefully,
I'm
not
logged
out.
So
we've
built
a
couple
of
dashboards
here.
One
is
service,
performance,
monitoring,
dashboard
and
really
quickly.
A
Here
we
can
look
at
the
total
number
of
spans
we're
looking
at
six
hours
latency
by
service.
We
can
actually
look
at
the
breakdown
of
all
the
metrics
and
then
each
particular
service.
You
can
actually
see
my
catalog
service,
the
latency,
the
spans
the
error
rates
and
then
the
second
really
cool
dashboard.
Is
the
service
level
view?
A
So
if
your
team
has
a
particular
service
like
I
own
the
front
end
of
my
application,
you
can
actually
look
at
all
of
the
particular
metrics
for
your
individual
service,
and
these
are
just
coming
right
off
Prometheus,
broken
down
by
by
the
operation.
So
it's
pretty
powerful
to
have
this
data
available
to
the
team,
because
these
metrics
are
very
tied
to
your
business
in
particular,
and
the
second
thing
that
we
did
aside
from
creating
the
metrics
is
we
made
them
usable
inside
the
Jaeger
UI.
A
So
when
you
run
Jaeger,
you
can
actually
see
in
this
new
monitoring
tab
how
it's
visualized
and
I'm
going
to
show
you
a
live
demo
of
this
as
well,
so
in
Jaeger,
if
you
haven't
updated
recently,
you'll
notice
that
there
is
a
new
tab
right
here
called
monitor
when
you
click
on
monitor
what
it's
going
to
do.
Is
you
pick
the
service
that
you
want
and
it's
going
to
show
you
the
latency
and
you
see
here:
we've
got
percentile
buckets.
A
Hopefully
it's
not
too
small,
the
error
rate,
the
request
rate
and
then
I
could
actually
go
down
and
see
the
different
operations
and,
let's
say
I'm,
starting
to
see.
Oh
look
here.
This
is
showing
11
errors
on
my
order.
Service
I
can
actually
drill
in
directly
by
clicking
on
that
link
and
it's
going
to
pull
up
the
specific
traces
related
to
that
service.
So
just
give
it
a
second.
So
these
are
the
the
traces
specifically
related
to
the
service.
So
the
idea
here
is:
it
gives
you
a
little
bit
of
a
monitoring
workflow.
A
Obviously
we
don't
do
alerting
in
Jaeger
yet
I
mean
you
know.
That
is
something
that
if
anyone
wants
to
contribute
or
work
on,
we
definitely
would
love
to
have
alert
manager,
integration
and
there's
all
kinds
of
cool
things
that
that
we
can
do
with
this
data.
So
that's
kind
of
some
of
the
new
stuff
that
we've
done
so
I
did
want
to
also
hit
on
a
couple
of
new
features
and
for
those
of
you
that
were
at
the
keynote
this
morning,
I
summarized
these
in
a
45
second
video.
So
it
might
be
a
little
redundant.
A
So
so,
right
now
Jaeger
takes
open
Telemetry
line
protocol.
That's
the
raw
protocol
for
open
Telemetry,
so
you
can
send
Trace
data
directly
from
anything
speaking
open
Telemetry
into
the
collector,
which
then
gets
stored
onto
disk.
The
other
piece
which
Joe
mentioned
was
adaptive
sampling.
We
added
that
specifically
into
open
Telemetry,
and
do
you
want
to
talk
a
little
bit
about
that
one
because
you
you
did
more
on.
B
That
that's
the
one
I,
actually
it
yeah
so
adaptive
to
handling,
is
a
a
new
feature,
sampling
feature
in
Jaeger
that
allows
you
to
adjust
your
sample
rates
in
response
to
your
volume
or
your
Trace
volume
dynamically.
So,
for
a
long
time,
Jaeger
has
been
able
to
remote
sample,
which
means
like
pull
a
document
remotely
from
a
remote
source
and
then
use
that
to
adjust
sample
parameters,
but
it
was
always
a
static
document.
What
this
new
feature
adaptive
sampling
does
is.
B
It
will
watch
your
Trace
data
and,
if
maybe
one
particular
service
or
one
particular
endpoint
starts
overwhelming
your
back
end.
You
can
dynamically
or
the
system
will
dynamically
reduce
the
sample
rate
for
that
particular
endpoint
or
service
and
allow
your
back
end
knot
to
fall
over.
Also,
if
the
maybe
the
throughput
of
a
particular
endpoint
goes
down,
you
could
increase
your
sample
rate,
so
you
have
a
larger
percentage
of
traces
from
that
particular
service.
So
it's
just
like
a
dynamic.
The
ability
to
dynamically
adjust
your
sample
rates
based
on
your
current
traffic
yeah.
A
And
there's
a
lot
of
getting
back
to
those
processors
and
open
telemetry.
There's
a
lot
of
really
interesting
sampling
strategies
you
can
Implement
and
sampling
and
tracing
is
a
really
important
topic
and
Big
Talk
unto
itself.
The
next
feature
that
was
added
is
you
saw
the
kind
of
tabular
views
in
Jaeger.
We
actually
had
a
community
contribution
for
flame
graphs,
which
is
kind
of
another
interesting
way
to
look
at
the
trace
data.
A
A
Yeah
exactly
we
like
cleaning
things
up,
so
that's
that's
important
too.
So
some
of
the
things
that
we
want
to
do
today
in
Jaeger
those
of
you
that
are
users,
there's
a
few
different
dependency
graphs.
It
would
be
nice
to
have
one
and
potentially
one
of
the
other
things
that
we've
discussed
is
using
those
new
service
views
that
you've
been
working
on
to
generate
that,
because
today
you
have
to
run
spark
to
generate
the
dependency
views
in
Jaeger
or
you
can
run
like
we
do
at
my
company.
A
B
General
is
hinting
at
there
is
there.
We
talked
about
those
processors
in
The
Collector.
There
is
a
new
processor
called
the
service
map,
processor,
I,
think
which
generates
service
graph
metrics
and
then
graphonic
can
currently
visualize
those
I
don't
know
if
there's
any
other
place.
I
can
visualize
them,
but
they're
open,
Telemetry,
metrics
and
they're
an
open
standard.
At
this
point,
so
you
know
you
can
use
those
metrics
yourself
to
generate
visualizations.
Grafana
can
also
do
it.
A
Yeah
so
the
more
context
we
can
add
on
to
those
graphs,
the
visual
views,
the
the
more
useful
the
tool
becomes,
and
obviously
that's
super
important
in
when
you're
doing,
troubleshooting
or
monitoring
the
other
piece
is
eventually.
We've
had
stops
and
starts
on
the
second
major
bullet
here,
which
is
how
do
we
move
away
from
the
Jaeger
collector,
which
today
writes
the
data
to
the
back
end
and
implements
something
like
a
paired
down
open,
Telemetry
collector?
Instead,
it's
just
less
code
for
us
to
manage
in
the
project
and
it's
somewhat
redundant.
A
A
Yeah
they
just
added
that
into
the
project
about
two
weeks
ago.
You'll
want
to
search
for
key
Val
k
e
y
v.
A
l
is
the
little
startup
that
built
that
and
just
do
key
Val
go
and
you'll
see
it.
We've
tested
it
out
and
it
definitely
works
well
and
does
context
propagation
with
ebpf,
which
is
really
cool,
that
they're
able
to
do
that.
That.
A
D
E
Differently,
a
great
talk,
Joe
and
Jonah,
so
I
definitely
helped
to
understand
different
things
in
open,
Telemetry
and
eager,
but
what's
the
future
of
eager.
So
today
we
definitely
love
the
eager
UI
and
the
storage.
Integrations
are
really
good
for
Cassandra
elasticsearch
and
recently
the
grbc
integration
has
been
added.
But
what's
the
future
for
eager
and
or
how
do
we
separate
out
the
boundaries
between
open,
Telemetry
and
eager.
A
This
one's
always
a
debate,
I
mean
we
would
love
to
have
one
database
right
ideally
to
do
this,
but
everyone
kind
of
wants
something
different.
So
we
have
people
that
love,
Cassandra
and
use
that
and
people
that
want
to
use
elastic
or
open
search
and
then
obviously
there's
other
data
stores
like
like
what
is
it
prom
scale.
D
A
B
Don't
know
I've
considered
it,
but
no
I've
not
done
any
work
on
it,
but
yeah.
You
mentioned
the
grpc
plugin,
which
is
a
great
thing
to
mention.
So
there
is
a
grpc
back
end,
so
you
could
write
a
plug-in
for
any
back
end.
You
wanted
at
this
point
and
I
think
there
might
be
a
Proto
well,
whatever
I'm,
not
sure.
Definitely
a
jrpc
back
end,
which
allows
for
extension,.
A
F
Talk
I
have
two
questions
for
you.
First
one
is:
do
you
have
plans
to
introduce
other
streaming
Solutions
other
than
Kafka,
for
example
nuts,
just
stream,
and
the
second
question
is:
do
your
metrics
also
provide
exemplars,
for
example
the
trace
ID,
so
that,
let's
say,
if
I
have
a
graphang
dashboard
I
can
see
a
spike
I?
Can
click
on
that
Spike
and
go
to
the
particular
Trace
graph
sure.
A
A
It
I
believe
that
the
The
Collector
does,
but
in
this
case
the
the
individual
traces
don't
really
matter
on
the
metric
itself,
because
you
already
know
the
context,
because
the
the
dimensions
that
you
define
in
the
processor,
for
example
the
path
or
whatever
tag
you
decide
to
use,
is
already
going
to
show
you
the
examples
for
the
metrics.
So
it's
not
necessarily
relevant,
but
I.
Guess
we
could.
B
I
guess
I,
don't
know
off
the
top
of
my
head.
What
the
open,
Telemetry
support
for
exemplars
right
now
is
I'm
getting
a
thumbs
up
from
gerasi.
So,
yes,
open,
teleporary
supports
exemplars,
so
I
would
presume
that
right
we
could
do
if
it
doesn't
already
generate
exemplars
out
of
this
or
out
of
the
span
metrics
processor,
which
would
make
for
a
compelling
use
case
like
it's
really
nice
to
see
metrics
generated
from
your
traces
and
very
quickly
be
able
to
jump
over
so
I
would
review
the
docs.
B
A
B
There
are
no
plans
at
this
moment
to
support
other
ones.
The
current
implications
kind
of
like
a
very
bespoke
Kafka
implementation.
There's
no
I,
don't
think
there's
a
general
shim
there,
so
a
new
implementation
will
require
some
effort
from
someone
else.
If
you
are
interested
in
contributing
something
like
that,
I
would
start
an
issue
first
and
discuss
with
the
maintainers
like
how
you
want
to
implement
it,
and
we
could
talk
about
it.
I
think
the
collector,
the
open
Telemetry
collector
also
only
supports
Kafka
I'm,
not
sure
on
that
yeah.
A
C
C
A
That's
that's
correct,
so
there's
a
few
ways
to
to
instrument
in.net
and
you
can
either
install
the
instrumentation
or
it
can
be
included
as
a
package
inside
your
build
if
you're
going
somewhere,
that's
serverless
in.net.
Basically,
so
it
depends
on
how
you
want
to
deploy,
but
the
auto
instrumentation.
Is
there
there's
no
code
changes
needed
and
there's
a
great
team
in
that
Community
that
are
that's
working
on
it
and
we're
happy
to
take
questions
after
also
happy
for
you
to
jump
on
our
slack,
which
is
up
on
the
screen.