►
Description
Warp-Speed Debugging with Prometheus Exemplars - Ian Billett, Red Hat
Effectively debugging distributed systems almost always requires inspecting more than just your Prometheus metrics data - logs, traces and profiles all provide essential information that help you quickly and efficiently pinpoint the root cause of your bugs. However, navigating between different systems with disjointed data sources interrupts your debugging flow state and ultimately increases the time taken to identify and resolve your bugs. Wouldn't it be nice if Prometheus had a native capability to help you hop between data sources? Enter exemplars! In this beginner-focused talk, Ian Billett will walk you through what exemplars are, how they work and provide practical examples of how you can leverage them in your applications today to super charge your debugging experience.
A
I
guess
just
the
first
thing
to
say
is
you
know,
I
assume
that
most
of
you
like
work
remotely
most
of
the
time
and
it
definitely
gets
kind
of
quite
lonely
so
coming
to
events
like
this
being
in
person
with
you
all,
with,
like
my
lovely
team,
from
red
hat
and
old
friends
from
previous
events-
and
you
know
the
new
people
you're
going
to
meet
today.
It's
just
yeah,
it's
really
nice
to
be
back
in
person.
A
So
I
guess
let's
try
to
take
this
off
now
yeah.
My
name
is
ian
billet,
I'm
an
engineer
at
red
hat,
where
I'm
the
team
lead
for
red,
hat's,
internal
observability
service
and
today
we're
going
to
talk
about
prometheus
exam
plus
and
specifically
how
you
can
use
them
to
accelerate
your
debugging
to
warp
speeds
and
I'm
being
serious
when
I
say
warp
speed,
I'm
not
using
this
term
lightly.
A
Exemplars
are
one
of
these
features
that,
if
you're,
not
if
you're
kind
of
earlier
on
in
your
prometheus
story
or
you're,
not
kind
of
paying
super
close
attention
to
the
project,
it
could
be
a
feature
that
kind
of
slips
under
the
radar
but
kind
of
when
you
see
it
in
action.
You
know
it
kind
of
it
looks
like
magic,
and
I
want
to
kind
of
show
you
the
magician's
tricks
today.
A
Firstly,
I'm
going
to
convince
you
why
exemplars
are
such
a
powerful
addition
into
your
observability
stack
and
kind
of,
as
with
any
change
that
you
make
to
production
systems,
it
has
absolutely
has
to
be
worth
it.
It
has
to
be
worth
the
investment
in
time
and
effort
and
energy,
and
I'm
going
to
construct
that
argument
with
you.
A
Secondly,
we're
then
going
to
talk
about
the
actual
mechanics
of
exemplars.
You
know
how
they
work,
how
they're
ingested,
how
they're
stored
you
know
the
real
nuts
and
bolts
of
how
exemplars
work
and
then
finally
we're
going
to
see
some
exemplars
in
action,
and
I
I
sacrificed
the
pano
chocolate,
the
demo
gods,
so
fingers
crossed
for
that
one
all
right.
So
I
want
to
kick
off
just
by
kind
of
painting,
you
all
a
picture
which
will
kind
of
give
us
all
the
shared
context
with
which
we
can
understand
the
value
of
prometheus
exemplars.
A
And
then
you
receive
an
alert
or
you
observe
some
symptoms
of
the
system,
not
behaving
so
as
the
responsible
human.
It
is,
then
your
job
to
figure
out
what
is
going
on
and
to
return
the
system
to
a
normal
working
order,
and
this
is
exactly
why
we
have
observability
tools
right.
They
are
tools
that
collect
and
emit
and
store
streams
of
data.
Your
your
metrics,
your
logs,
your
traces
and
more
recently
your
profiles
and
then,
with
these
streams
of
data.
A
It
is
then
the
human's
job
to
ask
questions
of
the
system
using
these
streams
of
data,
and
it's
the
and
it's
these
these
streams
of
data
that
then
provide
the
answers
to
the
questions
that
you're
trying
to
answer
and
every
question
you
ask
and
every
answer
you
get
narrows
down
the
space
of
where
the
problem
that
is
afflicting
your
system
lies.
So
this
is
kind
of
a
very
roundabout
way
of
just
describing
your
normal
debugging
workflows,
but
in
context
of
your
observability
tools
and
your
observability
data.
A
And
yes,
that
is
a
picture
of
mr
zuckerberg,
giving
his
congressional
hearing
that
was
photoshopped
to
look
like
data.
It
was
too
funny
not
to
include
the
internet's
amazing.
Isn't
it
all
right,
and
I
would
say
that
it
is
the
speed
with
which
you
can
repeatedly
go
through
this
cycle
of
asking
questions
and
getting
answers.
That
really
is
the
making
or
breaking
of
your
observability
tooling,
and
when
I
say
speed
here,
I
don't
just
mean
like
mashing
through,
like
the
questions
and
answers
quickly
and
quickly.
A
As
you
can,
I
mean
I'm
I'm
more
talking
about
the
speed
with
which
you
can
hone
in
on
relevant,
contextual
information
and
kind
of
you
know,
then
we
can
then
use
the
mean
time
to
detection
and
mean
time
to
recovery
metrics
to
kind
of
capture,
just
how
effective
we're
being
here-
and
these
are
metrics
that
are
kind
of
most
commonly
used
in
incident
response
like
crisis
response
like
situations,
but
I
would
argue
that
they
are
absolutely
applicable
anytime.
Your
software
is
not
behaving
in
a
way
that
you're
expecting.
A
A
I
guess
the
point
I'm
making
here
is
that
the
opportunity
cost
of
that
debugging
is
always
quite
high.
There's
almost
always
something
better.
You
could
be
doing
with
your
time
something
more
productive,
but
instead
you're
debugging.
This
problem-
and
the
final
point
I
want
to
make
here-
is
that
your
time
is
incredibly
precious.
A
You
know
like
in
engineering
organizations
the
I'm
going
to
assume.
This
is
true
of
most
of
you
that
the
human
cost
is
the
biggest
cost
that
those
organizations
pay
out.
Thank
you
very
much
and
I
read
a
kind
of
I've
seen
it.
I've
seen
a
an
article
that
said
that
the
on
average,
the
average
engineer
only
spends
only
has
30
percent
of
your
time
in
your
kind
of
working
week
is
focused
engineering
time
a
disclaimer.
I
didn't
know
if
that
was
backed
up
by
any
data,
but
it
kind
of
feels
intuitively
correct.
A
So
the
overarching
point
I'm
kind
of
making
here
is
that
finding
and
eliminating
inefficiencies
in
your
debugging
process
is
not
only
good
for
you,
it's
good
for
your
organization,
it's
good
for
everyone.
Okay.
So
next
question
is:
how
do
prometheus
exemplars
help
us
achieve
a
better
efficiency
in
our
debugging
process?
And
you
know
what
is
the
magnitude
of
this
efficiency
gain
that
we
can
get
by
using
them?
A
So
when
you're
in
this
debugging
loop
of
asking
questions
and
and
getting
answers
very
often
the
piece
of
information
you
require
to
answer,
your
question
will
lie
in
a
different
stream
of
data
than
the
one
you
are
currently
focused
on.
For
example,
you
know
you
see
that
a
metric
that
shows
increased
request
latencies,
so
you
go
and
you
need
to
find
some
tracing
data
which
kind
of
breaks
down
these
latencies
and
it
helps
you
answer
the
question
of
like:
where
is
this
latency
increase
going
from,
and
I've
been
there
like
many
times?
A
A
And
it
is
this
process,
this
switching
of
context,
that
I
think
is
is
is,
is
a
small
period
of
time,
but
is
actually
incredibly
costly.
You
know
the
human
brain
can
only
hold
four
chunks
of
information
in
working
memory
at
any
one
time.
So,
if
you
have
to
go
through
this
costly
context,
switching
process,
you
have
to
take
your
current
debugging
context,
kind
of
unload
it
into
your
longer
term.
A
A
The
documentation
says
that
exemplars
are
references
to
data
outside
of
the
metric
sets.
I
would
kind
of
go
further
and
kind
of
almost
use
more
vivid
language
and
say
that
exemplars
give
you
the
necessary
pieces
of
data
so
that
you
can
hop
directly
from
your
metrics
to
the
relevant
contextual
locations
in
different
streams
of
your
observability
data,
and
that
is
exactly
why
this
talk
is
called
warp.
A
Final
point
I
want
to
make
here
is
like
okay,
that
sounds
cool.
That
sounds
helpful,
but
just
how
helpful
is
this?
So
this
is
going
to
sound
silly,
but
I
think
of
this
almost
in
terms
of
swimming
pools
bear
with
me
here
right.
You
take
the
average
time
it
takes
you
to
perform.
One
context
switch
then
you
add
say
20
to
kind
of
load
and
reload
that
context
in
your
head.
A
A
All
right
first
question
is:
how
do
you
create
an
exemplar
okay,
and
this
is
done
in
when
you
are
instrumenting
your
application
code
so
effectively?
It's
like
the
client
libraries
make
it
very,
very,
very
simple.
When
you
are
making
your
observation
in
the
instrumentation
code
in
your
application,
the
the
functions
that
are
provided,
it
just
observe
with
examples.
A
A
Okay,
so
that's
how
you
create
an
exemplar
in
your
application
code
yeah
there.
It
is
so
once
we've
defined
an
an
exemplar
in
our
application
code.
How
do
we
get
that
into
prometheus?
Well,
if
you
think
back
to
your
prometheus
101,
you
remember
that
prometheus
operates
on
a
pull-based
model
of
data
ingestion.
A
This
means
that
prometheus
kind
of
goes
out
to
each
of
its
targets
and
the
things
that
it's
configured
to
scrape
and
then
request
the
metrics
page,
which
is
a
text
page,
which
kind
of
looks
like
this.
In
the
normal
case,
which
includes
you
know,
the
metric
label
set
the
set
of
data
that
that
uniquely
identifies
that
metric
and
then
the
value,
so
that's
standard
prometheus
without
exemplars.
A
So
that's
what
an
exemplar
looks
like
in
terms
of
the
exposition
format,
but
this
model
has
a
couple
of
nuances
that
I
just
want
to
call
out
here.
The
first
one
is:
let's
assume
that
your
server
is
serving
say,
10
requests
per
second
and
every
request
is,
is
being
observed
with
a
set
of
exemplars,
but
your
prometheus
server
is
only
coming
around
to
scrape
that
metrics
page
every
30
seconds.
So
so,
what's
going
to
happen
here
and
what
you
find
out
that
happens
is
that
only
the
most
recent
exemplars
are
captured.
A
A
The
second
nuance
you
should
be
aware
of
is
that
exemplars
are
not
supported
in
the
standard
bog
standard,
prometheus
exposition
format,
but
they
are
supported
in
the
open,
metrics
format
and,
like
these
formats
are
very,
very
similar.
This
isn't
something
to
be
like
worried
of
it's
just
something
to
be
aware
of,
and
there
is
some
more
peculiarity
in
their
differences
that
I
do
not
feel
qualified
to
talk
about.
A
A
So,
within
the
tsdb,
our
exemplar
data
is
stored
in
a
fixed
size
in
memory
circular
buffer
is
the
terminology
using
the
documentation
and
the
size
of
this
buffer
is
controlled
by
the
max
exemplars
configuration
option.
What
this
means
is
that
you
know
prometheus
will
just
hold
on
to
a
set
number
of
examples
at
any
one
given
time
and
that
you
know
when
a
new
one
is
added.
A
You'll
lose
one
from
the
back
end,
so
you
just
have
the
most
recent
ones
in
memory
at
any
given
time,
and
things
being
in
memory
then
also
mean
that
you
know
maybe
troubles.
There
may
be
difficulty
when
your
prometheus
restarts,
so
exemplars
are
not
persisted
in
a
long-term
fashion
like
prometheus
blocks.
A
Are
you
do
get
the
persistence
of
the
wow,
which
is
the
right
ahead
log,
which
then
what
this
effectively
means
is
that
the
most
recent
two
to
three
ish
hours
of
data
is
like
there
on
disk,
so
that
if
your
prometheus
server
restarts
and
starts
back
up,
you'll
read
all
that
data
back
into
memory.
So
if
your
prometheus
restarts,
you
will
get
them
like
the
two
or
three
ish
most
recent
hours,
but
not
the
longest
ones,
because
they
were
just
in
memory
and
they're
kind
of
lost.
A
All
right,
don't
forget:
exemplars
are
not
enabled
by
default
in
the
prometheus
server.
You
know,
they've
been
in
the
master,
the
main
branch
story
since
version
2.26,
which
was
about
march
last
year,
but
there's
still
a
feature
that
is
marked
as
experimental.
So
you
need
to
explicitly
enable
it
that
will
definitely
catch
you
out.
If
you
come
to
use
exemplars.
A
Okay,
so
once
your
exemplar
data
is
in
prometheus,
the
next
thing
we'll
want
to
do
is
actually
get
hold
of
that
data,
so
how
are
exemplars
queried
so
very,
very
simply:
prometheus
exposes
on
get
and
post
methods.
The
query
exemplars
endpoint
super
super
simple,
and
to
this
end
point
you
provide
a
query.
You
provide
three
things.
You
provide
just
a
standard,
prometheus
query,
any
query
doesn't
matter
the
start
timestamp
and
the
end
timestamp,
so
what's
kind
of
going
on
behind
the
scenes
here
is
that
prometheus
is
taking
this
query.
A
It's
very
simple,
but
you
know
the
final
question
that
I
then
want
to
address
when
we're
talking
about
the
specific
mechanics
of
prometheus
exemplars
is
how
can
you
walk
between
the
data
stream
because
it's
it's
all
very
well
having
a
http
endpoint
that
gives
you
jason
back,
but
you
know
in
reality.
That's
not
going
to
give
you
a
real
first-class
debugging
experience,
and
so
the
answer
to
this
question
like
at
this
point
in
time,
is
simply
grafana.
A
A
A
It
just
picks
a
random
number
and
calculates
that
fibonacci
number
and
then
kind
of
it
exposes
metrics
and
traces
and
logs,
but
we're
not
going
to
kind
of
go
into
the
logs
and
then
these
are
then
ingested
into
prometheus,
I'm
using
for
metrics,
obviously
I'm
using
grafina
tempo
to
ingest
the
metrics
and
then
obviously
using
grafana
to
visualize
this
all,
so
I'm
kind
of
looking
at
it
from
here.
So
these
are.
This-
is
my
prometheus
data
source
in
grafana
and
I've
done
this
in
configuration.
But
we'll
show
up
here.
A
B
A
95Th
percentile
of
my
demometric
that
I'm
exposing
and
you'll
see
and
you'll
see
here
what
this
data
looks
like.
So
all
of
like
this
is
obviously
this
is
just
the
normal
like
line.
That
kind
of
you
draw,
but
what's
really
cool
is
that
after
I've
enabled
exemplars
and
I've
ingested
them
and
I've
configured
all
the
things
to
the
right
level.
These
little
green
triangles
are
that
exemplar
data.
A
So
what's
really
neat,
is
you
write
this
like
quite
high
level
aggregate
level
metric,
but
then
it
then
gives
you
the
possibility
of
taking
individual
specific
points
that
relate
to
relevant
contextual
pieces
of
relevant
contextual
locations
in
other
streams
of
data,
so
just
take,
for
example,
this
data
point
and
it
kind
of
gives
you
the
option
to
like
first
query
with
tempo.
So
let's
just
see
what
looks
like
so
and
you'll
see
straight
away
that
that
trace
you
can
see
here.
A
This
is
us
calculating
fibonacci
numbers
that
is
almost
as
pointless
as
bitcoin
mining,
but
perhaps
not
so
much.
That
was
actually
probably
the
most
controversial
thing.
I'd
say
in
this
talk
all
right
and
then
you
know
looking
at
these
traces
we
were
then
calculating
and
like
the
the
steps
I
would
have
had
to
have
gone
through
to
get
to
this
point
would
have
been
such
a
pain
in
my
neck.
A
But
you
know
it's
just
a
one
click,
and
that
is
that
one
click
hop
that
one
click
warp
is
the
prime
debugging
experience
here,
but
the
other
cool
thing.
So
that
was
an
internal
link
to
tempo,
but
the
other
cool
thing
that
I
will
show
you
is
just
you
know
by
templating
I
can
you
can
template
into
this
external
link,
which
then
just
links
me
to
a
totally
different
data
source.
That
then
lets
me
dig
into
where's
my
where's
where's,
the
icicle
graph.
A
That's
the
real
there
we
go
and
then
like
it's
a
contextual
warp
to
another
relevant
place
in
a
different
stream
of
data,
but
for
those
of
you
that
are
hawkeye,
you'll
notice
you'll
realize
that
this
is
not
actually
the
thing
that
was
running,
because
this
is
just
on
the
pastor's
demo,
but
the
templating
works
and
you
can
link
to
an
external
source.
So
that
is
how
you
would
do
it.
It
looked
a
lot
like
that.
A
C
Thank
you
ian.
Are
there
any
questions
in
the
room?
I
saw
someone
typing
on
slack.
Oh
yeah,
there
question
from
slack
for
you
all
right.
What
magic
was
that
I
was
about
to
see
the
demo
and
it
went
to
the
big.
Oh,
that
was
a
streaming
question.
Sorry
about
that
all.
If
anyone
has
any
trouble
viewing
the
stream
or
wants
to
watch
the
sessions
again,
everything
is
recorded.
Everything
will
be
up
in
a
short
period
of
time.
A
D
A
A
really
good
question,
I
think
the
short
answer
is:
that's
the
only
way
that
it
works,
and
I
think
the
kind
of
the
clue
is
in
the
name
right
exemplar.
It
took
me
a
while
to
figure
out
why
why
that
was
chosen,
but
it's
it's
an
example
of
something:
it's
not
everything,
it's
an
exemplar
of
the
data,
so
I
guess
just
to
kind
of
follow
that
logic
through
then.
A
Is
that,
like
the
exemplar
that
you
observe
that
prometheus
scrapes,
if
you
have
a
system
like
tracing
that
you
can't
hold
everything
you
know,
there's
no
guarantee
that
that
exemplar
you've
observed
is
going
to
link
to
anything
that
your
tracing
system
can
serve
so
like
yeah,
there's,
there's
nuance
and
to
this,
but
about
the
most
interesting
thing
I
don't
think
you
would
know
a
priori
like
before.
Prometheus
has
come
to
scrape
it.
A
B
Thank
you
for
the
talk
you
mentioned
that
only
the
latest
exemplar
is
stored.
Right,
so
is
that
the
latest
per
set
of
label
set
like
for
a
particular
label
set.
The
latest
value
will
be
stored.
The
latest
example-
and,
if
that's
true,
let's
say
a
particular
metric-
was
raised
with
the
exemplar
and
for
the
next
five
minutes.
There
was
no
new
example,
so
will
that
example
be
sent
out
to
in
each
scrape
to
prometheus?
B
A
A
A
I
would
think
probably
that
the
the
overhead
required
in
terms
of
storing
and
serving
those
would
be
too
too
high
in
a
similar
way,
that
the
kind
of
the
tracing
kind
of
can
get
very,
very
big,
very,
very
quickly.
I
would
imagine
that
there'd
be
a
similar
constraint
felt
on
the
prometheus
side,
and
you
know
what
what
is
the
use
case
of
this?
Is
it
like?