►
From YouTube: How We Used Jaeger and Prometheus to Deliver Lightning-Fast User Queries - Bryan Boreham
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
How We Used Jaeger and Prometheus to Deliver Lightning-Fast User Queries - Bryan Boreham, Weaveworks (Intermediate Skill Level)
This talk comes from practical experience of running a cloud-based SAAS under Kubernetes for the last two years. Prometheus is good for the big picture view of how things are running, while Jaeger acts like a microscope on the internal workings of your system. You will learn about:* Setting up Jaeger and Prometheus in your Kubernetes cluster* Inspecting Jaeger and Prometheus output for tell-tale signs of trouble * Instrumenting your code (with examples in Go) for more detail* Applying the information gained to optimise your code and tune your system
About Bryan
Bryan is Director of Engineering at Weaveworks, delivering deployment, observability and monitoring for containers and microservices. Previously, Bryan has worked on many kinds of systems including financial trading, factory control and bakery. Outside of work he is a father, a keen cook and a helicopter pilot. Bryan is a maintainer on the CNI project, and also works on Weave Cortex, a scaleable multi-tenant back-end for Prometheus. Previous speaking includes: at FOSDEM 2017, GoLang UK 2016, Cloud Native London meetup, Silicon Valley DevOps meetup.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
What
am
I,
who?
What
am
I
Who-
am
I?
Yes,
let's
start
with
that,
so
I
work
at
a
company
called
weave
works.
I
am
a
software
engineer.
We
run
a
system,
a
SAS
system,
we've
been
running
it
on
kubernetes
in
production
for
two
years
more
than
two
years,
but
my
personal
thing
is
is
making
things
go
faster,
optimisation?
It's
like
a
video
game
to
me
because,
because
you
can
do
it
for
hours
and
hours
and
hours
and
there's
a
score,
and
in
that
sense,
so
that's
what
I'm
going
to
talk
about
I.
A
A
A
The
back
end
of
that
thing
is:
is
an
open
source
project
called
weave
cortex.
It
is
a
multi-tenant
Prometheus,
because
we
we
run
all
the
storage
and
back-end
servers
and
everything
if
you
want
to
use
that
you
can
sign
up,
and
so
this
is
just
to
indicate
it's
it's.
You
know
it's
it's
kind
of
a
bit
more
complicated
than
that
in
real
life,
but
there's
there's
multiple
parts
to
it.
It's
a
distributed
system,
so
it's
quite
a
complicated
Beast
to
optimize.
A
What
are
the
three
most
important
things
in
software,
optimization
I
hear
you
ask
I
like
to
I
like
to
put
this
in
I.
Think
it's
really
important
to
get
some
kind
of
ground
rules
so
number
one
measure
you
must
measure
before
you
act,
don't
just
start
changing
things,
because
you
think
it's
gonna
make
it
go
faster
measure.
First,
the
next
most
important
thing
is
measure
measure
the
real
system.
You
know
don't
measure
some
tiny
benchmark
or
a
blog
post,
or
you
know,
measure
the
real
system
really
doing
what
it's
doing
and
once
you
have
measured.
A
A
Talk
about
measuring
a
distributed
system.
This
is
where
we're
kind
of
getting
back
to
the
projects
I'm
going
to
talk
about,
so
any
any
system
which
has
multiple
parts.
Basically,
what
we're
going
to
need
to
do
is
is
get
the
information
out
of
all
of
them
or
as
many
as
are
involved
in
whatever
requests
I'm
interested
in
and
we're
going
to
want
to
collect
that
information
in
some
central
place.
A
So
that's
the
basic
pattern
prometheus
most
of
you
put
your
hands
up
so
I'll.
Just
briefly
explain:
it
is
a
time-series
metrics
system,
so
time
series
meaning
a
quantity
that
varies
over
time.
Now
you
might
be
measuring
CPU
utilization
or
memory
usage,
or
things
like
that.
Prometheus
project
ships,
a
bunch
of
exporters
which
are
things
that
gather
the
raw
data
and
send
them
out
as
metrics
and
but
you
should
definitely
also
have
metrics
from
inside
your
services
and
we
we
rec.
A
We
talked
about
the
the
read
methodology,
request,
rate,
error
rate
and
delay
or
we
usually
call
it
latency,
but
it
makes
a
crap
mnemonic.
So
so
we
went
with
delay
so
in
the
it
just
go
back
to
that
picture.
In
the
Prometheus
case,
the
Prometheus
server
sits
there
and
it
requests
the
metrics
from
all
those
exporters
and
from
all
the
parts
of
your
system
and
it
collects
them
on
central
place
and
you
can
query
it.
A
So
so
Jaeger
is
very
much.
What
is
my
system
really
doing?
What
is
it
doing
in
much
more
detail?
What
is
what
is
it
doing
as
the
flow
of
control
moves
from
one
part
of
the
code
to
another,
as
it
moves
from
one
component,
one
micro
service
to
another?
That's
what
what
Jaeger
is
all
about
and
again
we're
going
to
want
to
to
collect
all
of
that
information
from
all
the
different
parts
and
bring
it
together.
So
we
can
look
at
it.
A
A
A
A
Would
be
slower
than
this
time
so
take
the
89th
percentile,
the
the
median,
the
50th,
percentile
and,
and
the
mean
those
are.
Those
are
three
measures
that
you
can
compute
well,
the
first
two
anyway,
with
a
with
a
prometheus
feature,
called
a
histogram
which
is
very
powerful
and
that's
so
that's
a
very
typical
kind
of
sre.
In
today's
terminology.
A
A
In
the
Prometheus
syntax,
you
tell
it
what
you
want
to
summarize
by
so
the
next
thing
I
did
was
I
I
dug
in
well
I
mean.
Let
me
just
draw
your
attention
to
the
fact
that
some
of
this
is
this
is
a
showing
a
chart
over
a
couple
of
days
and
some
of
the
time
it's
running
at
4
seconds
and
some
of
the
time
it's
running
it
like
half
a
second.
So
so
you
might
want
to
think
you
know
what's
happening.
A
What's
the
difference
between
those
times
who
knows
when
I
drilled
in
I
added
this
extra
parameter
to
the
query
route
I
can
say
route
because
I'm
in
Europe
in
America,
I
have
to
say
route,
and
it
gets
me
confused,
because
I
can't
translate
all
that
quickly
anyway,
drilling
so
route
is
like.
Where
are
we
sending
the
request
internally
and-
and
it
becomes
a
very
different
picture
it?
Actually,
it
says
that
one
request
in
particular
is
actually
pretty
steady
at
five
seconds
and
and
the
rest
of
them
are
kind
of
flapping
about.
A
So
this
is
an
illustration
of
how
averages
can
confuse
or
misdirect,
because
we
were
taking
the
average
of
everything
one
in
a
hundred
was
sometimes
at
a
certain
level,
and
sometimes
at
another
level,
and
in
fact
those
differences
were
driven
by
how
many
other
requests
were
in
the
mix
because
we're
averaging
them
so
so
definitely
a
tip
there.
Beware
of
averages
because
they
may
be
hiding
something
important,
but
Prometheus,
fantastically
powerful.
Just
slight
edit.
To
that
query,
we
can
drill
in.
We
can
find
out
what
that
is,
and
it
turned
out.
A
So
I
didn't
have
detailed
information
on
how
we
were
hitting
the
database,
so
I
needed
to
get
some
detail
on
that
I
decided
to
use
Yaeger.
How
does
that
look
I'm,
not
going
to
go
through
that
in
detail,
but
but
basically
it's
it's
like
three
four
lines
to
kick
this
thing
off
at
the
beginning.
This
goes
in
your
sort
of
main
program
and
and
then
one
more
line
to
bring
in
a
library
to
instrument.
A
My
sequel
calls
sorry,
there's
a
few
libraries
out
there
I
happen
to
pick
this
one
called
expensive
worlds:
don't
write
it
down
until
I,
get
to
the
end
of
the
story.
No
ice
skates
good,
it's
good,
so
yeah!
So
now,
I've
got
Tracy.
So
what
does
that
look?
Like
I
mean?
Let's,
let's
tempt
the
demo
gods.
There
was
a
dashboard
earlier.
This
is
the
the
eager
dude.
A
So
you
put
in
some
kind
of
this
is
the
Jager
screen
you
put
in
some
kind
of
thing
that
you
want
to
look
at
okay
and
and
Jager
goes
and
finds
all
the
information
it's
got
on.
What's
been
running
and
it's
a
little
kind
of
histogram,
some
things
are
slower
than
other
things
and
you
click
on
one
of
those
you
drill
Ian.
So
let's
go
back
to
my
slides
and
it's
not
attempt
to
demo
gods
anymore.
A
They
just
walk
through
this
for
those
of
you,
but
two-thirds
of
you
did
not
raise
your
hands,
but
when
I
asked,
if
you'd
used
yogur
so
I'm
going
to
tell
you
what
it
all
looks
like
on,
don't
here
on
the
left,
we've
got
the
kind
of
the
call
stack.
This
thing
calls
this
thing
calls
this
thing,
and
you
know
that
this
is
the
details
of
our
user
authorization
service.
A
So
you
don't
need
to
know
anything
about
that.
A
point
is
because
I
added
that
that
sequel
tracing
I
can
actually
now
see
the
the
sequel
query,
that's
being
executed,
whatever
arguments
anything
like
that
and
each
point
I
get
the
what's
called
the
span.
The
time
span
the
beginning
and
the
end
of
each
operation,
so
I
can
see
that
in,
however
much
detail
life
instrumented
my
code
and
that
works
across
component,
so
we
just
we
just
trace
everything
and
we
get.
A
So
it
turns
out
that
wasn't
enough
information
to
find
the
problem,
sometimes
life
isn't
like
that.
It
turns
out
the
actual
problem
which
I
just
sort
of
I
just
figured
out
well
thinking
about
what
could
possibly
be
going
wrong
in
this
particular
case.
The
problem
was
that
we
had
accidentally
disabled
connection
pooling
to
the
database
and
and
that
the
goal
sequel
library
doesn't
have
a
hook
to
find
out
when
it's
opening
a
new
connection
in
the
connection
pool.
A
A
The
this
big
average,
the
the
99th
percentile,
it's
sort
of
flattened
out
it
got
less
spiky,
but
it's
not
that
impressive,
so
I'm
showing
you
the
median
instead,
which
came
down
from
I,
don't
know
22
milliseconds
to
four
or
something
like
that.
So
so
that's
that's
that
story.
You
know
it
doesn't
entirely
have
have
that
neat
narrative
that
I
could
just
point
at
the
answer,
but
it
it
was
very
good
at
eliminating
a
lot
of
other
possible
concerns.
A
A
Okay,
let's
go
back!
Remember
this!
This
was
at
the
beginning
of
my
talk.
So
I
had
this
on
the
screen
and
I
to
show
me
a
dashboard
and
I
had
all
the
tracing,
so
this
is
actually
a
this.
Is
the
screen
grab
I
pasted
into
the
github
issue,
so
this
is.
This
is
the
actual
evidence
that
I
had
on
hand
at
the
time
for
those
slow
queries
and
again,
the
the
pattern
is
that
it
kind
of
it
goes
down
through
the
hierarchy
of
calls
and
it's
doing
a
bunch
of
different
things.
A
So
zoom
them
in
first
thing
is
that
that
that
little
exclamation
mark
there
that
says
that
this
operation
errored,
so
my
cache
call,
was
erroring,
and
this
was
a
very
large
part
of
why
it
was
too
slow.
So
that
was
immediately.
You
know,
that's
the
big
picture,
it's
immediately
clear,
just
one
of
those
operations
errored,
and
that
was
really
important
and
the
other
thing
I
just
zoomed
in
that
bitmap.
The
other
thing
is
this
long
diagonal
line.
A
Long
diagonal
lines
to
someone
like
me:
that's
been
optimizing
and
using
these
kind
of
tools
for
I,
don't
know
a
decade
that
says
serialization
that
that
says
those
things
are
being
executed
one
after
the
other,
and
it
says
to
me
as
an
engineer:
why
don't
you
parallel
eyes?
It
can
these
things
be
executed
in
parallel?
A
Well,
yes,
they
can.
So
this
is
the
trace
from
a
couple
of
weeks
ago,
the
all
those
operations
start
in
parallel.
The
whole
thing
is
horizontally
scalable,
they
all
execute
in
parallel.
They
all
finish
enormously
faster,
so
so
that
there
you
go.
There
were
two
two
very
important
things
in
that
one
screengrab
of
Jager.
That
was
showing
me.
A
So
that's
my
war
stories.
I
just
want
to
kind
of
distill
that
down
into
what
I
call
the
the
patterns
to
look
for
and
made
a
list
of
them.
Hopefully
some
help
to
you.
If
you're
thinking
about
using
this
technology,
so
I
made
a
list
and
then
I
thought
well
actually,
I've
got
a
few
I've
got
one
at
the
very
beginning.
A
That's
that's
where
you're
going
to
get
the
biggest
bang
for
the
buck.
If
you
look
for
the
longest
one
and
then
I,
actually
it's
it's
on
this.
It's
on
this
picture
when
I
was
doing
the
slides,
I
I
thought.
Oh
actually,
there's
another
there's
another
kind
of
lesson
hiding
in
there,
which
is
this
this
whole
region,
that
what
happens
is
that
you
chose
you
this
bit
kind
of
zoomed
in
and
this
bit
zoomed
out
this
whole
region
over
here
doesn't
have
any
detail
in
it,
and
so
that's
my
my
second
pattern.
Zero.
A
If
you
don't
have
any
detail,
you
need
to
go
back
to
the
code
and
add
some
detail,
because
you
have
no
idea
what
it's
doing
in
that
gap,
and
that
is
it's
an
iterative
process
right.
You
speed
up
one
bit
and
then
that
makes
the
another
bit
the
most
important
thing,
and
then
you
go
around
the
loop,
so
so
yeah.
A
We
need
some
detail
in
in
that
gap
next,
one
that
long
diagonal
line
that
I
spoke
about
now,
sometimes
you're
running
an
algorithm
where
one
step
depends
on
the
step
before,
in
which
case
you
can't
just
paralyze
it,
but
in
most
cases
you
can
parallel
ice.
It
there's
a
whole
scientific
computing
world.
If
people
have
been
doing
this
for
many
decades
and
rearranging
the
work
and
so
forth,
so
that
it
can
be
done
in
parallel
and
summarized
at
the
end,
so
yeah
very
much
trigger
for
me
to
go.
A
Another
interesting
one
if
I
see
a
bunch
of
things
exactly
the
same
length,
I
think
why
why
are
they
exactly
the
same
length
like
everything,
everything
in
a
in
a
concurrent,
dynamic
system,
everything's
a
little
bit
random?
Everything
is
running
at
once,
everything's
contending
for
resource
things
kick
off
at
different
times.
Why
am
I?
Seeing
these
things
take
exactly
the
same
time,
so
I
would
the
one
that
actually
triggered
this.
In
my
mind,
was
a
was
a
time
out.
A
In
fact,
the
work
was
just
being
curtailed
at
a
particular
time
and
everything
was
being
thrown
away,
but
it
could
be
some
other
reasons.
I
mean
that's
kind
of
suspicious,
like
a
bunch
of
spans,
all
exactly
the
same
length.
Think
about
that
and
then
a
kind
of
a
variant
of
that
one
is
when
they're
they're
all
different
lengths
random.
That's
what
we'd
expect
they're
all
starting
at
different
times.
That's
random!
A
That's
what
we'd
expect,
but
for
some
reason
they
all
finish
at
the
same
time
and
again
as
someone
who's
been
looking
into
these
problems
for
a
long
time.
This
is
a
this
is
a
trigger
to
to
think
about
some
kind
of
interlock.
You
know
if
this
is
hitting
a
traditional
database,
I
think
there's
a
transaction
holding
up
the
long
run
or
some
kind
of
multi
thread
lock
or
some
kind
of
resource
that
they're
all
contending
on,
or
something
like
that.
So
so
what
happened
at
exactly
that
time?
A
So,
okay,
that's!
My
tracing
patterns,
summarize
measure
measure
measure.
That's
my
big
lesson
from
atheists
and
yoga
a
great
tool,
so
I
gave
a
kind
of
Whistlestop
tour.
I!
Guess
we'll
have
time
for
questions
in
a
minute.
So
if
you
want
to
know
any
more
detail
about
how
we
use
that
look
for
those
patterns
which
which
give
you
a
real
insight
into
what's
going
on
all
right,
you
know
just
pay.
Someone
like
me
to
run
a
hosted
service,
because
that's
that's
what
we
do
for
a
living.
So
thank
you.