►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deploy, Scale and Extend Jaeger - Louis-Etienne Dorval, Ticketmaster
Distributed Tracing has been a hot topic in the last few years. While a lot has been said about the benefits of having tracing as well as how to instrument your (micro?) services, this talk will instead focus on how we deployed, scaled and extended Jaeger at Ticketmaster. We will look into how we drove adoption internally as well as the strategies that were put in place to scale it as it gained traction. We’ll cover the first few pull requests needed to get the ball rolling and the lessons learned while helping different development teams get started. We will share how the architecture of Jaeger enabled us to deploy it inside Kubernetes as a multi-region cluster used by application all across our hybrid cloud and explore the various extension points that are available.
https://sched.co/MPbm
A
A
So
here's
the
agenda
for
today
as
you'll
see
it
follows
the
title
of
the
talk
pretty
closely
so
I
hope,
you're,
not
too
surprised.
This
talk
will
indeed
focus
on
Jaeger
and
what
we've
learned
along
the
way
using
it
a
ticket
master.
Our
mission
is
to
power
unforgettable
moments
of
joy
and
to
realize
this
we
need
to
make
sure
our
systems
are
behaving
and
functioning
correctly
when
you
interact
with
them.
A
Those
of
you
who
already
use
get
mastered
before
might
recognize
the
screen.
It's
our
website.
More
specifically,
our
seating
chart.
That's
what
you
used
buy
tickets
sounds
some
simple
right.
You
click
you
pay
for
it
and
you
wait
for
the
show,
there's
actually
some
quite
unique
challenge
in
that's
getting
industry,
and
that
includes
the
fact
that
every
single
product
we
sell
are
unique
and
carefully
picked
by
our
customer.
So
it's
obvious
that
we
don't
want
to
sell
the
same
seat
twice
and
have
you
shared
your
seat
with
someone
else
all
that
we
have
to?
A
But
there's
a
lease
one
other
time
you
interact
with
Ticketmaster
that
you
might
not
know
or
realize
as
much
as
buying
the
tickets
online
and
that's
when
you
enter
the
venue
during
the
day
of
the
concert
of
your
favorite
artists
or
sports
team,
when
the
validation
is
done,
it's
done
by
our
system
and
no
matter
how
you
got
your
tickets
from
our
mobile
application
to
a
real
Paper
Tickets.
That
validation
goes
to
our
system
and
to
ensure
that
everything
works
fine.
A
Even
at
that
scale
we
have
some
additional
constraint
being
a
40
years
old
company
and
we've
been
innovating
for
all
these
years.
So
internally
we
have
not
one
but
27
to
kitten
system
that
are
backed
by
more
than
300
products.
Our
internal
cloud
included
more
than
20
20,000
VM
in
seven
data
center
and
because
we're
at
Q
Khan
I
felt
that
the
last
ATS
was
mandatory.
So
we're
at
15
Rene's
cluster
right
now.
A
All
those
serve
all
those
compute
power
and
those
infrastructure
are
used
power
services
from
all
the
language,
as
shown
here,
there's
probably
much
more
I
probably
can't
fit
all
that,
but
those
are
the
biggest
one,
but
it
also
served
to
power
one
of
our
ticketing
system.
That's
a
VAX
program,
and
if
you
don't
have
a
computer
like
that
on
your
desk,
you
can
also
run
it
on
mini
cube
nowadays.
A
So
listening
to
me
talking
about
all
that
should
already
tell
you
the
answer
to
the
why
tracing
question
and
I
won't
really
go
into
all
those
questions.
So
what
is
tracing,
how
to
do
tracing
and
even
why
do
tracing
there's
already
some
great
talks
about
that.
Some
very
great
present
presenter
did
that
in
various
conference,
but
I've
linked
for
talk
that
resonate
with
me
in
the
last
two
two
years
in
the
last
two
years,
so
I
recommend,
if
you
haven't,
watched
those
to
listen
to
them.
A
They
all
been
recorded,
and
there
now
well
after
the
talk,
there's
still
one
reason
that
wasn't
mentioned
in
the
all
those
talk,
and
that
was
my
one
of
my
reason
to
do
that,
and
it
was
to
prevent
people
from
searching
log
with
20
long
line
20
with
a
query
that
long
in
any
search
engine.
It's
not
meant
to
do
that.
It
can
be
forced
to
do
something
like
that
and
we've
done
it.
A
A
A
A
So
how
did
we
started
this
effort?
How
did
you
growth,
adoption
internally,
I
tell
them,
and
the
main
advice
I
could
give
you
is
to
start
with
the
development
teams
before
switching
to
the
upside
I
was
a
Java
developer
for
working
on
various
API
and
I
can
really
tell
you
that
tracing
does
fill
a
gap
between
logging
and
matrix
working
with
them
is
the
best
way
to
get
started
and
you'll
see
that
they
care
about
it
and
that
it
will
it's
something
they
will
need
to
do
won't
do
usually.
A
We
start
with
demo
with
the
team
followed
up
by
a
Q&A
session
and
generally.
The
first
question
is:
how
can
I
get
started
so
try
to
understand
what
what
language,
what
framework
you
use
internally,
I've,
there's
multiple
question
on
that
slide:
try
to
answer
these!
It's
going
to
be
what
you'll
be
working
with
from
now
on.
A
A
It's
not
the
funniest
part,
it's
not
always
enjoyable,
but
it's
gonna
save
you
quite
some
time
down
the
road,
as
you
scale
from
one
team,
to
much
more
try
to
answer
these
question
in
your
documentation.
That's
what
we
did
and
people
were
finding
that
helpful,
but
also,
if
you
see
a
question
being
asked
over
and
over,
that's
probably
a
good
indication
that
your
documentation
does
not
mention
anything
about
that.
A
A
Try
working
with
this
with
your
first
team
instrument,
the
bare
minimum
framework-
and
that
might
be
that
might
looks
harder
than
it
really
is
because
there's
already
a
good
coverage
on
the
open
tracing
that
IO
registry
and
that's
open
source
instrumentation
for
various
project
from
java
python
to
scala
and
a
lot
of
these
I
say
document
I
talk
about
documenting
how
to
use
them.
Usually
the
open
source
project
already
mentioned
that,
but
we
found
that
some
little
piece
of
our
service
template
required
additional
documentation.
A
You'll
have
eventually
to
start
testing
that
that
instrumentation,
you
might
find
a
team,
that's
willing
to
do
that
with
you,
but
we've
usually
started
by
sending
PRS
directly
to
the
team
and
the
process
of
doing
the
first
PR
you'll
usually
find
the
most
obvious
problem
so
fix
these
after
that.
Have
them
deploy
it
somewhere,
because
a
non
merge
PR
doesn't
get
you
anywhere
closer
to
where
you
want
to
be
having
it.
It
does
validate
some
things,
including
a
network
ACL
or
deployment
script.
A
A
They
wanted
to
know
more,
but
most
importantly,
people
that
on
boarded
tracing,
wanted
to
know
more
about
those
blah
books
and
those
long
span
calling
and
other
services,
and
they
started
asking
other
teams
and
pushing
them
toward
having
tracing
as
well.
So
that's
really
a
snowball
effect.
We
not
notice
your
documentation
will
start
to
be
Andy.
At
this
point,
we've
started
with
one
or
two
teams
and
scaling
255,
the
last
30
or
at
a
very
rapid
rate,
so
that
helped
and
always
continue
to
be
involved
with
the
teams.
A
So,
let's
move
on
how
we
deployed
Jaeger.
This
is
the
architecture
diagram
of
Hager
I'll,
walk
you
through
it
quickly,
if
you've
never
seen
that
before
it's
part
of
the
official
documentation
on
the
left.
There's
a
gray
box
which
kind
of
look
pink
here
from
my
angle:
it's
our
it's
your
house
or
your
VM,
your
docker
container,
and
that
gray
box.
You
see
the
yellow
box
being
your
already
deployed
application,
something
that
you
already
have
everything
below
that
the
tree
block
below
that
will
be
the
new
code.
A
If
you
look
at
the
bottom
of
the
gray
pink
ish
box,
there's
the
Jaeger
agent
that
runs
alongside
of
your
application,
it's
a
separate
process
and
it
and
those
taking
those
pans
from
your
application
and
sending
that
to
the
collector
the
collector
is
in
the
middle.
It.
A
job
is
to
save
those
batch
of
Spanish
received
from
all
your
clients
to
your
data,
store
the
data
store
being
the
green
green
box
at
the
bottom.
A
A
If
you
over,
if
you
already
looked
at
the
documentation,
this
might
be
new,
it
wasn't
well,
it's
still
not
there.
Yet
it's
an
alternative
way
to
deploy
Jaeger
the
main
difference
being
that
the
Jaeger
collector
will
write
to
a
stream
instead
of
directly
to
the
data
store.
The
stream
currently
can
only
be
Kafka,
and
that
could
be
extended
at
some
point.
A
Once
the
information
lands
in
Kafka
there's
a
new
component
called
the
Jaeger
and
gesture
that
reads
all
the
data
from
Kafka
and
sends
that
the
data
store
Kafka
being
able
to
handle
much
easily
spikes
of
traffic
than
something
like
elasticsearch.
We
use
that
to
have
a
constant
flow
of
data
going
to
elasticsearch
and
not,
however
whelmed
it.
The
rest
state
stays
the
same.
The
query
and
the
UI
are
deployed
together
the
query.
The
data
store
and
your
agent
still
on
little
Twitter
to
your
collector.
A
We
have
three
different
environment
and
I
deployed
in
scout
master.
We
have
the
non
production,
the
pre
production,
which
could
be
named
staging
in
the
production
one.
It's
important
to
note
that
every
application
that
talks
to
one
that
talks,
one
to
the
other,
needs
to
share
the
same
Jaeger
cluster.
A
A
Yes,
as
I
said
earlier,
we
don't
run
it
inside
kubernetes
because
we
wanted
to
save
time
and
focus
on
Jaeger
instead
of
elasticsearch,
but
everything
else
that's
mark
with
kubernetes
logo
is
deploy
in
our
queries
cluster.
So
that's
the
Jaeger
that
the
Jaeger
collector
the
Jaeger
and
gesture
and
the
query
as
well.
A
The
architecture
of
Jaeger
makes
it
super
simple
to
deploy
that
in
deploy
that
in
super
Nerys,
because
all
those
three
components
are
stateless,
so
they
don't
have
a
state
on
disk
internally,
we
have
a
custom
chart,
but
it's
just
a
deployment
object
with
some
templating
on
top
of
it
for
all
three
component.
We
also
use
a
cron
job
object
for
the
es,
elasticsearch
and
X
cleaner,
which
is
self-explanatory.
A
A
Regarding
load,
balancing
the
Jager
collector
received
both
HTTP
and
TCP
for
use
TCP
protocol,
so
we
use
a
service
with
type
load
balancer
to
expose
it
externally
from
the
cluster
there's.
Also,
an
internal
endpoint
save
some
and
network
round-trip
in
AWS,
which
is
where
we
deploy
currently
translate
to
an
ELB.
As
for
the
Jager
query,
we
use
an
ingress
resource
because
it's
all
HTTP
in
our
case
it's
an
Al
being
Gris.
A
All
right,
so,
the
last
component
of
the
year
infrastructure
is
the
eager
agent
there's
two
different
way
to
deploy
it.
There's
the
deme
daemon
set
way
and
the
sidecar
way
the
first,
while
the
sidecar
one
is
deployed
alongside
your
application.
So
for
every
application
you
deploy,
there's
a
Jaeger
agent
and
the
second
one
being
a
demon
on
your
instance.
So
multiple
applications
share
a
single
Jaeger
agent.
A
A
This
means
that
if
you
have
a
bad
client,
that's
sending
too
much
traces
for
some
reason
it.
It
cannot
affect
another
client,
the
second
one
being
security.
You
don't
have
to
punch
a
hole
in
your
network
to
to
send
UDP
packets
across
to
the
same
agent.
Those
are
fully
isolated
in
the
dock,
one
to
the
other
that
comes
with
additional
resources
to
zation
you'll
have
to
run
one
and
you'll
have
to
load
and
run
one
yagura
jumper
application
and
there's
some
additional
complexity
that
I'll
toss
it
out.
A
A
Which
is
in
the
Pro
section
for
some
reason
it's
not,
and
the
isolation
is
slightly,
is
not
as
best
as
the
sidecar,
because
there's
multiple
client
that
can
affect
each
other.
The
added
complexity
I
was
referring
to
is
that
by
default
the
Jaeger
client
only
repelled
by
default.
It
reports
to
local
laws,
and
it's
not
always
the
case.
If
you're
using
a
darker
run
with
link,
it's
going
to
be
a
different
hostname
than
local
O's,
and
the
same
goes
for
for
the
daemon
set
approach.
A
A
There's
there
is
now
a
Jaeger
operator
for
kubernetes
that
abstract
all
that
this
sport
for
us
deployment
strategy
from
all-in-one
being
a
kind
of
developer
mode,
trying
it
out
easily
the
production,
one
being
the
first
diagram
I
showed
earlier,
so
there's
no
character
involved.
It
could
run
elasticsearch
or
cassandra
with
with
that
and
with
the
streaming
approach,
you
have
basically
the
same
deployment
as
we
do
internally,
but
with
much
easier
config.
A
It
also
sparked
about
Jaeger
agent
strategy,
so
you're
not
stuck
with
using
either
either
one
it
can
even
inject
the
sidecar
for
you
in
your
deployment.
So
that's
a
very
good,
very
nice
thing
to
have
there's
newly
version,
the
new
version
sport
version
upgrade
for
Jagr.
So
that's
one
of
the
thing
we
were
missing
for
before
using
that.
A
A
A
You
write
that
to
a
different
queue
and
use
a
replicator
in
the
east
region,
or
whichever
is
your
primary
to
sync-
that
data
from
west
to
east
and
merge
that
in
a
topic
that
contains
both
region
with
that
configured,
you
can
use
the
Jaeger
and
jester
to
consume
all
the
data
from
that
topic
which
contain
everything
and
writes
at
your
data
store
the
replicator
in
question.
Is
we
use
a
conference
message?
A
Replicator
there's
various
options,
even
one
from
you
were,
so
you
can
look
at
the
various
trade
off
on
you
know,
there's
so
a
problem
with
the
previous
approach
is
that
if
we
were
to
lose
East,
we
would
lose
access
to
our
trace,
even
in
West,
which
is
not
ideal.
If
you're
trying
to
survive
original
outage
to
achieve
milk,
see
multi-region
that
can
survive.
That
you'll
have
to
use
well,
in
this
case,
to
the
last
or
a
few
more
replication
and
that's
pretty
much
it.
A
A
So
let's
talk
about
scaling
now
I
there
wasn't
much
documentation
on
the
scaling.
When
I
submitted
the
talk
now,
there's
a
little
bit
more,
so
my
main
advice
was
to
start
small
there's
the
remote
control
sampler
that's
available
and
it
lets
you
control
how
much
data
ingest
from
your
various
clients,
which
lets
you
control
when
to
scale
and
not
having
it
forced
by
your
client
joint
in
your
cluster.
A
This
is
the
blog
post.
I
was
referring
to
from
one
of
the
manners.
I've
added
in
this
slide,
all
the
various
cues,
the
bounded
queues
that
are
in
the
architecture.
There
is
one
from
the
jäger
agent
to
the
collector,
even
one
from
the
jäger
client,
the
agent
pretty
much
everywhere,
there's
a
queue.
Those
queues
are
designed
to
prevent
hagar
from
crashing.
If
there's
too
much
load,
it
will
start
dropping
data
instead
of
a
and
they
can
all
be
configured
with
the
Vargas
flag.
A
A
A
And
we've
skipped
pretty
much
everything
that
was
well.
We
skip
a
lot
of
thing
that
were
that
were
hard:
how
to
scale
elasticsearch
how
to
scale
Cassandra,
Kefka
or
even
kubernetes.
There's
good
talk
about
that
on
the
web
and
even
good
talks
right
here
for
community
so
and
you
close
the
scaling
section
I
wanted
to
show
what
is
our
current
infrastructure
we
I've
shown
elasticsearch
on
the
left
and
the
Jager
collector
on
the
right.
We
run
on
leaf
for
pods
right
now.
A
A
And
now,
let's
talk
about
extending
Jaeger
because
it
might
not
fit
your
needs
right
now,
but
there's
still
alot
to
do
the
biggest
feature.
I've
seen
in
github
about
extension,
point
and
Jaeger
was
to
add
a
storage
plug-in.
It
has
been
large
since
1.12,
which
was
released
this
week,
if
I
recall
correctly,
so
it's
now
possible
to
add
DynamoDB
BigTable
in
flux,
DB
or
any
other
database
instead
of
using
elastic,
search
or
Cassandra.
A
If
you
want
to
protect
your
eager
UI,
it's
been
requested
a
few
time
right
now,
there's
no
such
feature,
but
you
can
front
it
with
Kiko.
This
blog
post
will
explain
you
and
walk
you
through
exactly
how
to
do
that,
but
that
basically
had
an
SSO
in
front
of
it.
What
we
use
extensively
at
get
master
was
UI
customization.
You
can
add
custom
menus
and
jäger,
which
includes
link
story,
internal
documentation
or
slack.
You
can
add
tags
on
your.
A
You
can
add
links
on
your
tags,
so
we
use
that
to
link
with
our
inventory
tool
to
find
a
slack
channel
of
the
team
to
find
the
on
call
information
of
the
team,
that's
having
a
problem
or
even
the
source
code,
but
you
can
link
anything
any
links.
You
could
link
directly
to
the
AWS
console
with
this
this
templating
or
to
your
spline,
clogs
or
ALK
log
with
those
right
here.
A
The
possibilities
are
endless
with
that
and
it's
been
a
feature.
That's
been
added
from
the
community,
so
it's
per
great
to
see
people
having
great
ideas
like
that,
the
last
one,
not
the
last
one
but
another
one-
is
a
concept
of
processor
inside
Q&A
inside
Jaeger.
It's
not
exposed
right
now,
but
there's
an
open
feature
for
it.
There's
four
type
of
processor,
the
first
one
is
the
preprocessor
it
in
those
batch
of
message.
You
could
normalize
the
data
extract,
metrics
on
it
with
it,
the
second
one
being
the
filter
span.
A
It
will
allow
you
to
build
a
whitelist
or
blacklist
with
of
services.
You
could
enforce
mandatory
tag,
re
or
yeah.
Enforcing
remote
sampler
would
be
a
good
one,
but
it's
not
possible
right
now.
Third,
one
is
being
the
spam
sanitizer,
that's
what
you
will
use
to
normalize
tag
or
even
remove
sensitive
data
before
replicating
that
across.
A
A
The
last
one
is
to
do
analytics
on
the
trace.
So
if
you
have
all
the
trace
flowing
through
Kafka,
you
can
holler
eyes
them
to
build
SL
a
chart.
Our
dependency
map
that
are
much
more
flex
you
could
just
like
you.
Borders
will
trace
quality
to
our
latency
Instagram,
so
that
was
the
engine
thanks.
Everyone
for
listening.