►
From YouTube: Getting Started with Jaeger - Joe Elliott, Grafana Labs
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Getting Started with Jaeger - Joe Elliott, Grafana Labs
Jaeger is the most popular open source distributed tracing system in the world and, as such, often comes with people asking how to bootstrap their first cluster. If you are interested in getting started with Jaeger join us in this lightning talk where we will cover the basics of Jaeger architecture, installation and configuration in a Kubernetes cluster. We will also review the major components of Jaeger and their responsibilities, backend options, and various considerations for building your first production cluster. Additionally, key Jaeger specific metrics will be covered and other operational concerns.
https://sched.co/fGW8
A
Hey
everyone:
my
name
is
joe
elliott
and
we're
talking
about
getting
started
with
jaeger.
Today,
this
presentation
is
mainly
aimed
at
people
who
are,
you
know,
getting
started,
maybe
building
their
first
production
cluster.
Maybe
you've
already
set
up
a
proof
of
concept
and
you're
wondering
kind
of
what
it
takes
to
get
this
into
production,
or
maybe
you're
just
interested
in
tracing
in
general,
see
if
it's
something
that's
worth
doing
for
you
all
and
you
want
to
know
what
jaeger
looks
like
and
kind
of
how
jager
is
built.
A
A
A
and
before
we
get
into
the
actual
architecture.
I
want
to
talk
a
little
bit
about
what
a
trace
is,
because
you
know
kind
of
what
a
trace
is
and
how
traces
are
generated,
kind
of
inform
how
the
actual
back
end
works,
so
understanding
what
traces
helps
us
understand.
Why
the
back
end?
Why?
The
the
jager
injection
pipeline
is
the
way
it
is
so
this
is
a
trace
from
loki
and
this
particular
trace
represents
a
single
request.
A
So
this
is
one
request
to
get
35
milliseconds
to
get
through
the
loki
or
to
be
to
be
answered
by
loki,
I
mean
it's
passed
through
a
number
of
different
applications
and
services,
so
you
can
see,
there's
like
these
green
bars,
loki
gateway
at
the
top
and
then
query
front
and
orange,
and
then
loki
queers,
these
blue
ones,
all
over
the
place.
These
spans
kind
of
represent,
maybe
like
a
single,
functional
or
logical
unit
inside
of
our
application
and
the
application
each
process
and
there's
lots
of
these.
A
A
This
is
the
architecture
so
giving
that
in
mind.
We
have
all
these
applications
right
and
all
these
applications
are
going
to
then
be
pushing
all
their
spans
into
the
agent
from
the
client
into
the
agent,
the
client
itself
to
start
there.
It's
on
this
side,
this
side,
the
client
itself,
is
a
a
part
of
your
applications,
like
a
library
or
a
framework
that
you
bring
into
your
application
to
create
these
spans
and
offload
them
to
jaeger.
There's
lots
of
different
applications
or
frameworks,
and
language
supported,
like
popular
ones
like
go
and
java
node
ruby.
A
So
all
all
these
different
popular
enterprise,
languages
have
support
for
jaeger
or
jager
has
support
for
them.
Perhaps-
and
you
create
these
bands
in
your
applications
using
client,
which
is
then
offloaded
to
the
agent
where
the
agent
is
positioned,
is
important.
We'll
talk
about
that
a
little
bit
in
a
second
kind
of
there's,
a
lot
of
options
about
where
you
put
the
asian
agent
in
relationship
to
your
application.
A
The
agent
then
moves
them
over
to
the
collector
piece.
The
collector
then
has
to
make
a
choice
about
or
sorry
it
doesn't
make
a
choice.
You
make
a
choice
about
how
to
set
up
your
pipeline.
You
can
either
at
this
point
point
it
directly
to
a
backend
database
or
you
can
use
this
kafka
ingester
optional
kind
of
piece
and
we'll
talk
about
some
of
the
choice
there.
Why?
A
You
would
choose
to
use
this
so
agent
or
sorry
client,
in
my
application
to
the
agent
layer
to
the
collector
optionally
kafka
and
then
an
adjuster
and
then
our
backend
database
and
then
the
jaeger
query
piece
which
we
won't
really
talk
about
much.
The
query
piece
just
speaks
directly
to
your
backend
database.
A
So
how
am
I
going
to
deploy
this
I'd
say?
There's
three
probably
popular
options
here,
helm
a
lot
of
people
like
helm.
It's
great!
It's
great
application.
It's
a
great
way
to
deploy
things
to
kubernetes.
The
operator,
mainly,
I
believe
maintained
by
jp,
is
an
excellent
choice
for
deploying
graphi
or
for
deploying
jager
and
then
there's
the
custom
option
as
well.
Just,
however,
you
all
want
to
do
it.
So
maybe
you
just
have
custom
yaml,
maybe
they're
just
sitting
in
get
somewhere.
We
use
json.
A
actually
at
grafana
to
deploy,
to
deploy
yeager
I'll
say
that
jager
itself
is
not
very
difficult
to
operate.
Your
operational
complexity
is
going
to
come
from
your
back
end.
The
database
you
choose
to
put
your
to
put
your
to
put
your
spans
in
so
jaeger
itself
is
mainly
stateless.
It
has
cues,
it
keeps
things
in
memory,
but
you
know
it's
easy
to
operate,
it's
easy
to
monitor
and
to
manage
it's
it.
It's
not
very
difficult.
A
The
operator
is
a
cool
choice,
because
the
operator
is
a
cool
choice
because
it
supports
some
different
agent
modes,
like
sidecar
injection
or
as
daemon
sets
sitting
on
your
hosts
and
we'll
talk
about
why
you
would
choose
one
of
the
other
in
a
second
and
then,
of
course,
helm
help's,
a
great
option.
We
used
helmet
my
previous
position
quite
a
bit
and
helms
is
a
great
way
to
deploy
applications
as
well.
A
A
Cassandra
is
not
bad,
in
fact,
we
use
cilla,
which
is
kind
of
a
cassandra
alternative
at
griffon,
and
it
works
fine,
there's
like
maybe
slightly
less
flexibility
in
searching,
but
we
use
it.
We
have
no
problems.
So
if
you
have
a
ton
of
operational
experience
around
cassandra
and
you
don't
want
to
kind
of
take
the
risk
on
of
building
or
understanding
a
new
back
end,
then
cassandra
is
a
hundred
percent
worth
a
100,
a
good
option.
A
The
other
option
you
have
to
pick
is
kafka
or
not.
So
we
talked
about
this
like
optional
kafka
piece,
you
have
collector
or
sorry
client
agent
collector,
and
then
this
kafka
injector
optional
part.
The
kafka
piece
is
useful
in
some
ways
you
kind
of
have
this
like
durable
cue
for
a
lot
of
people.
In
fact,
kafka
is
very
easy
to
run,
especially
maybe
compared
to
alaska
cassandra,
at
least
for
us.
A
It
is,
and
it
provides
some
more
flexibility
kind
of
in
your
pipeline
if
you're
back
in
stutters
for
a
little
bit
cough
it
very
nicely
and
easily
queues
these
pieces
up
and
then
once
your
back
ends
back
online,
it
can
continue
pushing
spans
in
there
there's.
Also,
some
kind
of
neat
integrations
with
kafka,
with
kafka
streaming
to
do
analytics
and
to
do
some
other
do
some
other
kind
of
like
data
processing
on
the
spans
they
come
through
and
also
coffee
gives
you,
your
your
org
or
your
group
or
your
team.
A
It
gives
you
the
flexibility,
maybe
of
writing
your
own
kind
of
like
applications
that
watch
the
kafka
stream,
maybe
generate
some
metrics
or
you
know,
provide
some
kind
of
information
about
your
infrastructure
that
is
difficult
to
find.
Otherwise,
if
you
do
make
something
cool
like
that,
please
share
it.
Please
share
it
with
community
yeager's,
open
source,
we
we
love
contributions
and
involvement,
and
if
you
build
something
very
neat,
we'd
love
to
hear
and
see
it.
A
A
final
choice
here
is
kind
of
the
positioning
of
the
agent,
which
we've
kind
of
alluded
to
a
few
times
previously.
In
this
discussion,
the
agent,
the
common
wisdom
here-
is
to
put
the
agent
as
close
as
you
can
to
the
application.
So
the
operator
does
this
kind
of
nicely
for
you
it'll.
Do
this
sidecar
injection,
where
you
have
your
application
and
then,
as
a
sidecar
you'll,
be
running
the
agent.
A
So
it's
sitting
there
in
the
exact
same
network
name
space
and
every
application
is
going
to
have
its
own
agent
right
side
by
side.
Excuse
me,
and
this
would
be
the
absolute
fastest.
You
know
way
to
offload
spans
from
your
application.
The
kind
of
game
that's
being
played
here
is
your
spam.
Your
applications
generally
expands
extremely
quickly,
you're
filling
up
a
queue
in
the
client,
a
configurable
queue
in
the
client.
A
You
there's
workers
that
are
draining
that
queue
as
fast
as
possible
by
sending
spans
over
udp
to
the
agent.
So
the
closer
your
agent
is
to
your
client,
the
fewer
drop
spans
you're
going
to
get
so
a
sidecar
for
the
most
for
the
most
high
volume.
Applications
of
sidecar
is
as
good
as
you
can
get
putting
your
application,
a
single
agent
as
close
as
possible
to
your
application
to
get
those
spans
off
as
quick
as
possible
to
do.
You
have
as
few
drop
spans
as
possible.
A
Other
popular
options
are
to
run
it
as
maybe
a
daemon
set
or
if
you're,
not
in
kubernetes,
just
on
every
host
and
then
have
all
the
different
applications
on
that
host
speak
to
your
local
agent
and
then
at
grafana.
We
just
run
the
agents
as
a
deployment
with
a
service
in
front
of
it
like
any
normal
kubernetes
application.
A
A
So
sampling
is
very
important
choice.
Understanding,
sampling
sampling
allows
you
to
only
record
a
certain
amount
of
your
total
traces
into
your
back
end.
Most
of
us
cannot
afford
to
sample
to
record
100
percent
of
every
request
in
our
infrastructure
or
back
end.
They
can
kind
of
quickly
get
out
of
control.
A
Development
teams
are
going
to
start
adding
spans.
Hopefully
your
volume
is
going
up.
Somebody
throws
a
new
service
in
which
is
going
to
add
more
spans.
Of
course,
all
these
different
kind
of
changes
always
are
growing.
It's
always
making
your
span
count,
grow
and
you're
going
to
find
that
it's
very
difficult
to
do.
100
sampling,
or
at
least,
maybe
not
difficult,
but
costly.
Your
back
end,
cassandra
elasticsearch,
is
going
to
be
very
large.
A
Once
you're,
hitting
hundreds
of
thousands
or
you
know,
a
large
number
of
spans
per
second
hundred
thousand
is
enormous
numbers
spans
per
second,
so
our
options
for
down
for
down
sampling
from
100
are
like
probabilistic
or
rate
limiting.
Those
are
common
options.
Probabilistic
is
where
you're
going
to
say
10
of
every
trace
initiated
by
the
service
I
want
to
actually
save
rate
limit
is
going
to
say
five
requests
per
second
five
traces
per
second
initiated
by
the
service.
A
A
The
next
part,
I
cannot
emphasize
enough
there
are
these
different
sampling
options
like
probabilistic
and
rate
limiting,
which
are
fun
to
work
with.
I,
I
think,
for
a
production
instance
shared
by
multiple
teams.
There
is
no
substitute
for
remote
sampling,
I'm
gonna,
just
tell
you
to
set
up
remote
sampling.
A
Remote
sampling
allows
you
to
centrally
control
all
of
the
different
services
and
their
sampling
rates,
so
you,
as
the
jager
operator
sitting
here
one
day
and
all
of
a
sudden,
your
spans
per
second
go
from
50
or
from
5000
to
7000,
and
you
don't
know
why
and
your
back
end's
struggling
and
you
can't
figure
out
what's
going
on
and
you
don't
have
control
over
these
applications
that
are
creating
spans.
Remote
sampling
gives
you
that
control.
So
it
allows
you
to
kind
of
maintain
this
document,
and
you
kind
of
see
it
over
here.
A
It
allows
you
to
maintain
this
document
that
says
for
each
service
what
what
percentage
of
span
or
what
percentage
of
traces
do
we
want
to
do?
We
want
to
actually
save
from
that
service.
You
can
even
do
it
per
operation.
So
maybe
you
have
like
end
points
that
you
don't
care
about.
You
don't
ever
want
to
sample
those.
A
As
an
operator,
some
development
team
somewhere
is
going
to
start
slamming
your
back
end
with
a
ton
of
traces
or
a
ton
of
spans
or
both
traces
and
spans,
and
unless
you
can
control
that
as
the
operator
you're
just
going
to
get
overwhelmed
and
you're
going
to
have
to
go
like
scramble
around
talk
to
these
development
teams
and
you're
kind
of
at
their
mercy,
remote
sampling
gives
you
the
control.
You
need
absolutely
100,
recommend
remote
sampling
monitoring,
so
monitoring
jaeger
is
actually
pretty
straightforward.
A
It
has
really
good
logging
standard
out
logging,
like
everything
else,
json
logging.
If
that's
your
thing,
it
also
has
prometheus
metrics
so
or
really
the
openmetrics
format,
so
there's
lots
of
different
applications,
of
course
prometheus
itself.
That
will
scrape
those
store,
those
and
allow
those
for
querying,
and
the
metrics
are
very
descriptive,
they're
very
easy
to
read
in
fact
a
personal
thing
I
do
when
I
start
any
application,
is
I
curl
the
metrics
endpoint?
I
just
start
reading
it
because
the
metrics
end
point
on
a
well-designed
application.
A
Will
give
you
good
information
about
the
internals
of
that
application
and
jaeger
absolutely
falls
into
that
category.
So
just
reading
the
metrics
is
going
to
tell
you
a
lot
about
what's
going
on
inside
and,
finally,
everything
is
a
cue,
so
the
client
in
your
application
has
a
cue.
The
agent
has
a
queue.
A
The
collector
has
a
cue
kafka
is
a
cue
if
you
run
that
the
adjuster
is
the
only
piece
that
doesn't
really
have
a
queue
and
all
of
these
cues
can
get
filled
up
and
start
dropping
spans,
and
that
is
the
number
one
question
your
developer
is
going
to
come
to
you
with.
Where
are
my
spans
like?
Why
is
my
trace
broken?
Where
did
all
my
spans
go?
It's
going
to
happen
over
and
over
again,
so
understanding
how
to
diagnose
this
ingestion
pipeline.
A
There's
a
medium
article
that
I
recommend
called:
where
did
all
my
spans
go
and
it
walks
you
through
every
single
piece
and
all
the
metrics
to
watch
and
tells
you
exactly
how
to
kind
of
track
down
where
your
drop
spans
are
also
there's
a
couple
other
links
there.
The
monitoring
and
troubleshooting
links
on
the
docks
are
also
great
for
figuring
this
out.
A
Finally,
a
couple
resource
links:
there's
the
docs,
the
docs
are
really
well
done.
Please
go
read
those
if
you're
interested
in
jaeger.
Please
submit
prs
to
improve
them.
If
you,
if
you
have
some
problems
or
if
there's
something
you
want
to
clarify
the
getter,
the
getter
channel
is
haunted
by
myself,
as
well
as
some
of
the
other
maintainers
and
is
a
great
place
to
get
answers
to
getting
started.
Questions
a
great
place
to
just
kind
of
move
past
some
initial
hurdles
to
understand.
A
What's
going
on
or
how
to
how
to
get
your
basics
going,
or
also
just
a
chat
generally
about
jager,
of
course,
and
then
the
medium
yigger
tracing
blog
has
a
lot
of
a
lot
of
great
articles
from
all
of
the
maintainers
that
talks
about
all
kinds
of
different
ways
to
use
jager
and
new
and
clever
ways
to
work
with
it
and
how
to
diagnose
issues
and
absolutely
a
great
place
to
go.
If
you
find
yourself
working
with
jaeger
a
lot
all
right.
Thank
you
all.