►
From YouTube: Jaeger: Intro and Deep Dive - Jonah Kowall, Logz.io
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Jaeger: Intro and Deep Dive - Jonah Kowall, Logz.io
In this session we will cover both introductory and deep dive material for the Jaeger distributed tracing backend. For intro we will review methods of getting started installing Jaeger and the basics of distributed tracing. For the deep dive we will be discuss the future of Jaeger built on top of the OpenTelemetry collector and what that means for Jaeger architecture, future development and features. Jaeger is the most popular open source distributed tracing backend. Whether your goal is to get acquainted with Jaeger and distributed tracing or to keep abreast with the latest and greatest, please join us!
A
Thank
you,
everyone
for
coming
in
person,
and
this
is
cool
to
see
everyone
together.
A
lot
of
the
talks
and
the
observability
track
have
been
virtual.
Unfortunately,
I
hope
it
changes,
but
I'm
glad
to
get
everyone
together
and
and
showing
up
and
meeting
and
talking
today,
I'm
actually
the
title's
a
little
misleading,
but
this
talk
is
going
to
be
about
jager
and
we're
going
to
dig
into
the
jaeger
operator,
which
is
a
really
cool
piece
of
technology.
That's
part
of
the
project
and
then
we're
going
to
talk
about
some
of
the
metrics
work.
A
The
first
part
of
the
talk
about
the
operator,
joe
elliott,
actually
put
the
slides
together
and
he
was
unable
to
make
it
last
minute
he's
one
of
the
maintainers
as
well
he's
over
at
grafana
labs
and
he's
the
person
responsible
for
tempo
for
those
of
you
that
are
maybe
using
that
he's.
A
great
engineer,
sorry
he's
not
with
us
today.
A
My
name
is
jonah
cowell,
I'm
the
cto
at
logs.
I
o.
I
do
a
lot
of
work
on
jaeger,
open,
telemetry
and
open
search,
which
is
the
new
open
source
fork
of
elasticsearch,
and
I'm
happy
to
be
here
when
I'm
not
working
at
the
computer
or
maybe
playing
a
little
video
games.
I
spend
a
lot
of
time
underwater.
A
It's
been
nice
and
covered
because
you're
very
isolated.
That
way,
so
I
do
a
lot
of
diving,
that's
my
passion
for
the
last
several
years
and
explore
a
lot
of
cool
places
where
I
live
in
south
florida
and
all
over
the
world.
So
if
you
want
to
talk
diving,
that's
definitely
another
thing.
Besides
observability
that
I
like
to
talk
about
and
my
company
is
a
sas
observability
company,
there
are
many
out
there.
I
can
see
many
of
you
representing
other
ones.
A
We
focus
on
an
open
source
based
platform
and
jager
is
part
of
our
platform.
So
it's
the
same
ui,
the
same
usability
with
a
bunch
of
other
stuff
around
it
and
jager
is
a
is
a
great
cncf
project.
It's
been
graduated
for
quite
some
time
for
those
of
you
that
are
not
familiar
with
jaeger,
it's
and
you're,
maybe
using
open,
telemetry
or
looking
at
it.
A
The
collector
component
of
jager
was
actually
forked
to
create
the
open,
telemetry
collector
and
we're
starting
to
bring
some
of
that
back
into
jaeger
and
we're
essentially
going
to
just
be
consuming
upstream
collector
and
that's
the
plan.
So
jaeger
is
basically
a
storage
format
and
a
ui
and
a
bunch
of
other
pieces
around
it
in
order
to
scale
it
out,
but
in
general
we
build
things
in
a
very
different
way:
they're
very
componentized.
A
It's
designed
to
run
on
kubernetes,
and
this
is
something
that
uber
created
many
years
ago
and
open
sourced
donated
to
cncf
yuri.
The
gentleman
who
created
jaeger
at
uber
has
a
great
book
on
distributed
tracing.
Many
of
you
have
probably
read
it.
It's
very
helpful,
even
though
the
examples
are
open,
tracing
they're,
still
very
similar
to
open
telemetry
and
those
that
are
not
familiar
I'll.
Just
give
you
a
really
quick
rundown
of
what
you're
looking
at
here.
This
is
a
trace
and
in
the
trace,
there's
a
bunch
of
different
transactions
that
are
occurring
here.
A
So
we
can
look
at
the
different
spans
which
are
each
of
these
little
segments
and
how
much
time
they're
taking
and
then
we
can
look
at
the
end
to
end
trace
how
much
time
that's
taking
are
there
errors?
Are
there
other
things
that
are
going
on
and
you
can
drill
down
into
these?
There
are
tags
that
are
added
that
show
you
different
information
in
the
example
here
that
mysql
call
you
can
see
the
database
query
how
long
it
took
whether
there
were
issues.
A
A
He
was
previously
an
engineer
red
hat.
He
recently
joined
grafana
labs
and
he
did
a
lot
of
work
on
the
operator.
Does
some
really
cool
things
and
I'm
gonna
show
you
how
you
can
use
this
on
your
laptop
to
do,
set
set
up,
jager
and
start
using
it,
and
even
how
you
can
use
the
operator
to
scale
out
and
operate
kubernetes
in
a
large
distributed
environment?
A
The
operator
is
designed
to
do
all
of
those
things.
Here's
a
link
to
it
to
the
docs,
which
are
also
really
good
and
comprehensive
and
explain
everything
in
depth.
So
a
lot
of
work
went
into
this
and
thanks
to
the
folks
at
red
hat,
who
are
big
into
the
operator
model
that
basically
want
everything
to
be
deployed.
A
A
How
do
I
basically
do
deployments
configuration
scale
out
and
even
talk
about
agents
and
sampling
and
auto
scaling,
so
keep
in
mind
that
some
of
the
things
that
were
going
over
here
are
jager
specific,
like
the
instrumentation
libraries,
open
telemetry
is
going
to
be
the
standard
jaeger
had
its
own
libraries
before,
but
open
telemetry
is
basically
going
to
replace.
All
of
that
something
that's
currently
being
discussed.
A
A
lot
of
people
do
use
the
jaeger
libraries
in
their
applications
and
they're
still
supported
we're
just
using
hotel
in
the
future
similar
to
people
that
maybe
used
open
tracing
before
open
tracing
is
being
deprecated,
there's
nothing
new
going
into
it
all.
The
work
is
in
hotel,
but
that
doesn't
mean
that
the
hotel
collector
will
not
accept
open
tracing
data.
It
does,
and
it
will
continue
to
do
so,
and
this
is
the
challenge
with
instrumentation.
A
A
So
a
couple
things
to
keep
in
mind.
The
logs
are
very
useful
when
you're
getting
this
up
and
running
and
also
it
will
watch
specific
name
spaces
to
deploy
new
new
instances.
Essentially,
if
you
leave
it
blank,
it
will
deploy
on
every
namespace
or
you
can
specify
certain
namespaces
as
well.
So
it's
up
to
you,
depending
on
how
you're
deploying
this,
if
you're
testing
it
in
your
test
environment
or
on
your
laptop
just
leave
it
blank
and
it'll,
deploy
everywhere.
Essentially.
A
So
so
the
really
basic
way
of
deploying
jaeger
on
your
desktop
laptop
whatnot
is
basically
a
single
binary.
So
this
includes
the
the
application.
Well,
so
the
single
binary
is
the
centerpiece,
the
jager
single
binary,
the
other
pieces
around
it.
It
needs
a
data,
store,
a
place
to
store
information,
and
then
data
comes
from
your
application,
instrumented
with
the
client
and
goes
to
a
jager
agent.
A
The
second
and
you'll
use
this
on
your
laptop.
If
you
want
to
test
it
out
start
playing
around
the
second
one
is
production
which
includes
everything,
except
for
the
green
box,
which
says
streaming
and
I'll
explain
streaming
after,
and
so
what
this
means
is
now
we're
starting
to
deploy
a
collector,
and
the
collector
allows
us
to
scale
out
the
way
the
data
ingestion
comes
in.
A
So
you
can
think
of
this
as
a
way
to
essentially
support
more
jaeger
agents,
more
infrastructure
that
is
sending
trace
data
and
we're
going
to
talk
more
about
the
back
end
database
and
about
kafka
and
how
those
work
in
the
production
mode,
because
there
are
options
for
those
jager
query-
is
also
a
service
that
helps
scale
out
the
ui.
So
when
you're
running
in
production,
it's
a
microservice,
that's
also
deployed
and
managed.
A
A
So
today,
when
you
decide
what
you're
going
to
run
jager
on,
there
are
two
options
today
in
the
operator
one
is
elasticsearch
and
the
other
is
cassandra.
The
operator
will
handle
deploying
those
and
being
able
to
scale
them
out,
so
it's
actually
quite
comprehensive
what
the
options
are.
You
can
also
point
this
at
another
cluster.
If,
let's
say
in
your
organization,
you've
got
a
big
elastic
search
cluster.
A
A
So
we
do
support
open
search,
which
is
the
apache
2
version
of
elasticsearch-
it's
not
in
the
operator.
Yet
it
would
be
trivial
to
add
it
we'll
see
right
now,
red
hat
hasn't
decided
whether
they
want
to
switch
to
open
search
just
because
they
they
do
keep
things
apache
too
in
their
code
base.
So
we'll
see
what
happens
with
that.
A
A
So
anyone
that's
running
big
elastic
search
clusters,
you're
already
running
kafka,
because
there's
essentially
no
other
way
to
scale
it
well,
and
so
in
the
streaming
strategy
or
configuration
it
will
deploy
kafka.
You
can
use
various
topics
as
well.
If
you
want
to
separate
and
manage
the
data
effectively,
and
this
allows
you
to
deal
with
bursts
of
data
coming
in
that,
maybe
your
back
end
can't
deal
with
kafka
will
cue
them.
A
The
other
thing
that
kafka
can
be
used
for
is
to
do
some
stream
analysis
and
there's
been
some
nice
little
experimental
projects
that
are
in
one
of
our
repos
in
the
jager
org
of
people
that
have
built
kafka
streams
and
spark
streaming
type
of
analytics
on
the
trace
data.
None
of
it's
really
well
baked,
but
some
of
it
came
from
uber
and
other
companies
that
do
that
type
of
analysis.
A
We
built
something
similar
in
kafka
streams
that
does
some
of
our
streaming
analysis.
We
haven't
open
sourced
all
of
it,
but
we're
we're
going
to
do
that
soon.
It's
definitely
in
the
plan,
so
kafka
is
something
that
you're
going
to
want
to
deal
with
if
you're
running
a
bigger
cluster-
or
you
have
other
use
cases
there,
but
the
operator
will
do
that.
So
to
give
you
kind
of
an
idea
of
how
this
works.
A
The
nice
thing
about
jager
is
that
each
of
the
components,
because
it's
all
in
go
it's
just
command
line
arguments,
so
the
operator
is
basically
passing
in
the
right
command
line.
Arguments
for
what
you're
trying
to
deploy
so
some
examples
here
and
we
can.
We
can
specify
a
lot
of
this
on
the
command
line.
This
basically,
is
an
example
of
us
deploying
you
know
it.
A
It
essentially
does
all
the
configuration
and
passes
the
command
line
arguments
in
for
you
by
running
the
operator,
so
you
don't
have
to
deal
with
this
all
yourself,
which
is
what
happens
when
you
run
it
on
your
own.
So
it's
definitely
gives
you
a
lot
of
options
and
makes
it
easily
scalable
in
a
kubernetes
environment.
A
And
the
other
thing
to
note
is
that
if
you
decide
to
pass
in
other
command
line
parameters,
so
in
this
case
the
example
is,
there
are
there's
blurrg
and
blarg
that
are
basically
being
specified
in
the
crd
here
and
you'll
notice
down
at
the
bottom,
when
the
command
line
is
run
unknown,
flag
blur
dot
blarg,
it
actually
just
passes
all
of
those
parameters
in.
So,
if
you
feel
like
passing
a
different
parameter
in
to
one
of
the
jaeger
components,
the
operator
will
just
essentially
run
that
in
the
command
line.
A
A
So
the
other
thing
to
think
about
is
the
agents
and
in
jaeger
the
agent
is
essentially
something
that
the
instrumentation
talks
to
so
in
the
open
telemetry
world.
It's
it's
essentially
a
collector
in
jaeger.
We
actually
have
two
different
things:
we
have
an
agent
and
a
collector
in
open
telemetry.
The
collector
can
be
an
agent
or
it
can
be
a
collector
that
was
kind
of
part
of
the
design
of
open
telemetry
was
to
have
that
same
binary,
be
flexible
for
both
use
cases.
A
So
when
you
decide
to
deploy
your
agent
as
a
sidecar
in
the
operator,
what
this
means
is
that
for
every
node,
that's
running
an
application,
an
agent
is
going
to
go
there,
there's
going
to
be
centralized
collector
that
the
agents
all
talk
to
this
is
one
option
that
people
use
and
I'll
show
you
the
other
options
for
for
agent
deployment
that
the
operator
supports.
So
this
is
one
way
this
way
is
kind
of
better,
because
it's
more
segmented
with
each
of
the
applications
that
you're
deploying
in
in
kubernetes.
A
A
Now
the
challenge
here
is
that
you
can
overload
the
agent
and
you
may
have
different
teams,
deploying
applications
that
would
talk
to
a
single
agent
and
the
config
and
and
just
managing
that
may
be
difficult
depending
on
your
configuration
and
how
your
teams
work
and
then
those
get
sent
on
to
a
collector.
So
these
are
basically
the
options
for
agent
strategy
deployment
that
are
in
the
operator.
A
A
So
today,
most
of
the
sampling
happens
at
the
collector,
so
the
traces
come
in
and
the
collector
decides
whether
it's
going
to
send
those
on
to
your
back
end.
The
challenge
is
that
all
of
that
data
is
still
going
over
your
network
and
if
your
collector
isn't
really
close
to
your
infrastructure
that
could
cause
other
contention
or
cost.
A
If
you're
running
in
the
cloud
with
remote
sampling,
the
instrumentation
library
actually
is
able
to
read
a
file
or
get
a
signal
to
change
its
sampling
strategy
in
real
time,
and
so
because
that's
happening
in
the
instrumentation,
you
can
eliminate
the
fact
that
the
instrumentation
is
even
going
to
send
trace
data
or
not
and
there's
a
lot
of
really
great
use
cases
for
this
I'll.
Give
you
a
good
hypothetical,
let's
say
in
general.
A
A
So
the
other
thing
that
the
operator
supports
is
auto,
scaling
and
scaling
the
collectors
and
ingestors
which
is
really
useful
as
well.
How
many
of
the
collectors
do
I
want
to
potentially
scale
to,
and
you
can
also
extend
this
on
your
own
and
maybe
use
a
metric
for
auto
scaling
and
I'll
talk
about
the
metrics
that
jager
supports,
because
it
exposes
a
lot
of
data
and
prometheus
that
you
can
scrape,
and
then
you
can
use
that
as
part
of
your
auto
scaling
as
well.
A
A
A
So
a
lot
of
folks
are
are
doing
that
and
it
helps
operationalize
yeager
for
sure.
So,
there's
also
a
bunch
of
other
pieces
that
the
operator
can
do
that.
I'm
not
going
to
dig
into
because
they're
even
more
advanced
use
cases,
but
here's
a
few
example
examples
of
them
in
cassandra.
It
can
create
schemas.
A
There's
a
lot
of
index
management
and
elasticsearch,
where
jaeger
will
actually
do
the
whole
life
cycle,
management
of
indices
for
you
and
then
there's
all
kinds
of
other
things
that
you
know
that
this
can
do
and,
of
course,
open
shift
support.
That's
the
main
reason
that
red
hat
contributed
all
the
work
into
the
operator,
because
they're
big
into
that
and
there's
lots
of
things
that
you
can
do
with
this
operator
as
well.
So
it's
it's
worth
checking
out
it's
worth
using,
even
if
you're
just
starting
to
play
with
jaeger
on
your
machine.
A
A
A
So
most
of
you
that
have
used
an
apm
tool
or
or
other
products
that
call
themselves
that
it's
because
they
use
traces
and
they
also
use
metrics
together-
and
this
allows
you
to
do
things
like
monitoring,
alerting
capacity,
planning,
other
types
of
planning
when
I
deploy
something
new
how's
it
going
to
trend.
What's
going
to
happen
in
my
environment,
this
is
really
the
difference
between
distributed
tracing
and
monitoring
tools
like
apm
tools,
so
we
built
recently
over
the
last
several
months,
a
new
capability.
A
We
call
atm
or
aggregated
trace
metrics,
and
what
this
is
is
there's
a
couple
of
pieces
to
it.
One
is
an
open
telemetry,
there's
a
a
span,
metrics
processor
and
in
this
processor,
when
traces
come
in
the
little
diagram
over
to
the
right
traces
come
in.
We
then
are
able
to
look
at
those
traces
and
derive
metrics
from
the
traces
that
then
use
you
send
to
a
any
type
of
metrics
back
end,
so
whether
you're
using
prometheus
or
you're,
using
a
commercial
tool.
A
Anything
in
open
telemetry
can
take
those
metrics
from
that
trace
processor
and
then
the
traces
continue
on
to
your
tracing
back
end.
So
the
most
common
use
case
is,
if
I
want
to
use
all
open
source,
is
I'm
going
to
send
my
metrics
to
prometheus
and
I'm
going
to
send
my
traces
to
jaeger
as
an
example,
and-
and
so
this
allows
us
to
create
all
kinds
of
metrics
off
of
the
trace
data
and
I'll,
give
you
some
examples
and
and
show
you
what
that,
what
that
does
so
in
the
configuration
for
open
telemetry?
A
This
is
an
example
config.
We
basically
specify
the
histogram
buckets
that
we
want
to
create
inside
of
prometheus
and,
and
we
can
define
definitions,
there's
a
bunch
of
examples
in
the
code
base.
The
challenge
is
that
it
can
create
a
lot
of
metrics,
depending
on
cardinality
that
you
define
most
commonly.
A
We
do
status
code
based
so,
okay
versus
errors
versus
you
know,
redirects
and
other
things
like
that,
and
then
you
have
all
the
histogram
buckets
and
prometheus
to
do
that
down
at
the
bottom
is
an
example
of
how
you
would
use
that
together
so
using
jaeger
and
prometheus
together.
This
is
basically
a
simple
config
of
the
pipeline,
so
how
that's
all
set
up
in
open,
telemetry
and
the
code
in
github
is
down
at
the
bottom
as
well.
It's
part
of
the
collector.
A
So
once
I
have
this
data,
I
can
do
all
kinds
of
cool
things:
here's
a
screenshot
from
grafana
showing
the
histogram
buckets
the
performance
of
all
my
traces
and,
and
we
basically
are
calculating
the
require
the
requests
per
second
or
the
request
time,
the
latency
the
errors.
So
it
allows
you
to
do
a
lot
of
different
monitoring
use
cases
and
gives
you
real
visibility
into
it.
A
We
also
added
a
view
into
yeager
to
query
these
metrics
and
I'll.
Show
you
that
in
a
second
and
that
supports
anything,
that's
prom
ql
compatible.
So
in
jager,
whatever
supports
prom
ql,
you
can
point
it
there
and
it
will
query
those
metrics
and
bring
them
into
the
ui
and
I'll
show
you
how
that
looks.
The
community
can
add
other
things,
but
we
use
m3db
in
my
organization
and
prometheus
of
course,
but
it'll
work,
fine
with
cortex
or
cortex-based
systems
and,
of
course,
thanos
time
scale
db.
Whatever
else
is
prom
ql
compatible.
A
If
you
listen
to
the
talk
yesterday
about
compatibility,
it
does
matter
whether
things
support
prom
ql
the
right
way,
so
naturally
it
does
have
to
support
the
right
type
of
query
language
for
it
to
work.
But
if
someone
wants
to
contribute
a
you
know
the
ability
for
this
to
work
with
another
pr
another
metric
system.
That's
fine,
we'll
definitely
take
any
prs
in
the
project,
and
so
we
have
a
new.
A
This
is
actually
appending
pr
right
now
in
jager
ui
as
there's
a
new
monitor
tab
in
the
ui,
and
this
allows
you
to
visualize
this
data.
So
you
know
latency
response
time,
error
rates,
the
trending
of
those
and
then
some
views.
This
is
kind
of
the
first
step.
I
think
for
jaeger,
obviously,
would
be
nice
if
we
built
alert
manager,
visibility
into
jaeger
and
really
created
more
monitoring
capabilities,
really
to
move
jaeger
from
a
just
a
distributed
tracing
system
into
more
of
a
an
operational
monitoring
tool
and
debugging
tool.
A
So
that's
kind
of
what
we
put
together
definitely
check
out
the
docs
come
visit
us
in
slack
and
we're
always
posting
blogs,
and
if
you
have
blogs
or
cool
ideas
that
you
want
to
post
on
our
medium.
Just
let
us
know
we're
always
looking
for
new
content
and
ideas,
and
I
think
that
is
about
it.
I'm
open
for
questions
here,
I'll
repeat
them
for
those
of
you
that
are
listening
online
and
I
will
also
bring
up
the
slack
and
the
online
thing
so
that
we
can
make
sure
to
include
everyone.
A
So
the
question
was,
I
talked
about
the
scalability
of
jaeger.
Is
there
any
questions
about
the
scalability
of
the
atm
component?
That's
inside
open
telemetry,
so
one
of
the
challenges
with
with
anything
that's
doing,
sampling
or
calculation,
is
that
you're
going
to
get
it's
it's
tricky
because
you're
going
to
be
somewhat
limited,
based
on
where
the
traces
are
going.
So
let's
say
you
have
10
different
collectors
that
are
calculating
metrics
you're,
going
to
have
10
different
sets
of
metrics
in
prometheus.
A
But
if
you
roll
those
up
so
let's
say
I
put
the
collector
name
in
the
metric
path.
I
can
then
use
it
without
the
collector
name
and
get
the
aggregate
of
that
data.
So
the
idea
and
that's
what's
nice
about
prometheus-
is
that
you
can
aggregate
data
up
a
level
from
where
you
are,
and
if
you
decide
that,
maybe
I
don't
want
a
service
I
mean.
Maybe
I
don't
want
a
micro
service
level
view
and
I
want
to
move
a
level
up
and
look
at
a
let's
say,
a
pod
level
view
or
a
kubernetes.
A
You
know
construct.
You
can
use
all
different
kinds
of
things
in
the
metric
path.
So
it
gives
you
a
lot
of
flexibility
in
terms
of
scalability.
We
haven't
really
posted
numbers
on
it,
but
we
put
a
lot
of
metrics
through
it
like
to
the
point
where
it
got
prohibitively
expensive
and
we
actually
tuned
it
down.
A
A
So
yeah,
definitely
I
mean
jager
is
a
microservices
architecture.
It's
scalable,
there's
no
real
issues
around
that.
If
you
have
different
storage,
back-ends
jager
doesn't
work
across
separate
clusters
of
elasticsearch
or
cassandra,
for
example.
So
you
should
scale
those
out
which
presents
its
own
set
of
challenges
because
they're
not
the
easiest
things
to
scale
out,
but
it
does
work
well,
but
it
only
supports
basically
one
back
end
because
of
the
way
that
the
that
the
product
really
works
today
or
the
project
really
works
today
over
there.
A
A
So
you
can
build
sampling
and
sorry,
I'm
sorry
for
people
online,
I'm
trying
to
to
rephrase
it,
but
you
can.
You
can
combine
different
things
in
the
sampling
strategy,
so
you
can
use
tags.
You
can
use
any
type
of
part
of
the
transaction
itself
to
create
a
sampling
strategy
and
do
sampling
and
filtering
the
challenge
is
with
remote
sampling
is
that
we
haven't
implemented
all
the
way
in
hotel,
but
it
will
go
that
way
because
there
are
customers
or
sorry
they're,
not
customers.
A
There
are
users
using
the
jager
libraries
that
support
that,
and
they
want
that
feature,
including
companies
like
uber
that
created
it.
They
they
need
it
and
they
want
to
go
to
hotel.
So
the
only
path
they
have
for
it
is
is
for
that
to
happen.
So
it
will
be
supported.
You
will
be
able
to
combine
them
together,
but
it
might
give
you
weird
results.
Depending
on
what
your
goals
are:
combining
remote
sampling
and
centralized
sampling.
A
A
A
A
If
you
want
meaningful
traces
like
the
example
that
I
that
I
showed
you
and
I
we've
I've
had
end
users
at
my
company
that
have
done
that,
and
they
say
why
is
the
trace
data
so
boring?
It's
like
the
same
as
a
log
message
and
I
said
well:
you've
got
to
instrument
the
code
or
we're
just
going
to
see
what
the
service
mesh
is
doing.
A
A
What's
the
expected
memory
or
cpu
footprint
of
running
agents's
sidecars,
so
the
cpu
is
going
to
depend
on
the
throughput
of
what
you're
sending
through
the
agent
similar
to
the
collector.
But
the
footprint
on
the
memory
side
should
be
very
small.
It's
a
go
binary.
It
doesn't
do
much.
It's
actually
not
a
super
complex
piece
of
technology
at
all.
Similarly,
if
you
decide
to
use
open
telemetry
in
that
way,
you
can
run
a
strip
down
collector
that
doesn't
have
all
the
exporters
and
other
things
in
there.
A
There's
a
custom
builder
for
open
telemetry
that
lets
you
build
your
own
distribution.
Essentially,
it's
really
simple
to
use,
and
then
you
can
create
a
really
small
collector
and
it
shouldn't
be
using
much
in
terms
of
resources
beyond
what
you're
sending
through
it,
which
is
obviously
the
the
processing
power.
That's
that's
really
required
there.
D
A
Sure
so
the
question
was
that
if
you're
running
it
across
multiple
regions-
and
you
have
multiple
elastic
search
clusters-
the
thing
I
said
it
doesn't
support-
we
don't
really
query
across
them,
so
you
would
have
a
single
jaeger,
ui
instance
for
each
region
and
you
can't
really
search
across
them.
So,
unfortunately,
the
answer
today
is:
we
don't
support
it.
It
could
definitely
be
built,
but
it
would
be
pretty
hard
to
federate
those
queries
and
make
the
ui
work
the
right
way
there
are.
A
There
is
work
going
on
in
open
search
to
to
to
do
multi-cluster
query.
It's
actually
was
just
released
recently
in
open
search
and
so
you'll
have
some
capabilities
there.
But
it's
it's
not
something.
That's
supported
in
the
jaeger
ui.
Someone
could
contribute
it,
but
I
would
guess
that's
a
pretty
big
feature
to
build.
A
And
I
don't
really
have
a
good
recommendation
around
it,
my
company,
we
do
something
differently
where
we
allow
you
to
create
different
sub
accounts,
that
point
to
different
clusters
and
indexes
and
we
can
search
across
those,
but
we
built
a
whole
bunch
of
abstraction
to
make
that
happen.
It's
part
of
our
platform,
but
it
would
be
really
hard
to
contribute
that,
because
it's
it's,
it
changes
the
way
that
all
the
back
end
works.
Basically,
but
if
you
want
to
contribute
something
like
that,
then
you
know
by
all
means.
A
I
think
it
would
be
a
cool
feature.
There
may
already
be
an
open
issue
on
it,
but
feel
free
to
suggest
it
in
the
channel
and
if
it's
something
that
you're
looking
for
so
I
mean
I'll,
keep
going
until
someone
cuts
me
off
I'm
over
though
well,
I
think
bartek's
talk
is,
is
starting
actually
there's
a
little
bit
of
time,
I'll
I'll
answer
one
more
before
I
call
it.
So
here's
a
good
one,
because
I
like
app
decks.
Kenya,
sorry,
okay!
This
is
the
last
one.
A
Can
jager
plus
atm
generate
app
deck
scores,
so
aptx
is
an
interesting
way
of
trying
to
summarize
user
frustration
or
how
well
your
application
is
performing.
It's
not
something
that's
supported,
but
it
could
easily
be
added
to
the
to
the
to
the
metrics
processor.
That's
there
or
you
could
actually
make
a
different
a
different
one
to
do
that.
So
it's
a
good
idea.
I
think,
but
not
something
that's
supported
today,
and
so
I'm
going
to
wrap
it
up.