►
From YouTube: Prometheus Sparse High-Resolution Histograms in Action - Ganesh Vernekar, Grafana Labs
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Prometheus Sparse High-Resolution Histograms in Action - Ganesh Vernekar, Grafana Labs
Sparse high-resolution histograms are going to totally revamp how Prometheus works with histograms. Maybe you have heard about the ongoing development efforts in previous talks. Now, for the first time, you will witness a complete working setup, from instrumentation over ingestion, storage, and querying all the way to graphical representation. Ganesh will demonstrate the breathtaking possibilities of these histograms, which include precise quantile estimations and high-resolution heatmaps, both aggregated and partitioned at will, even if, over time or between different targets, histograms of different resolutions are involved. Accompanied by benchmark results from real world load.
A
A
So
before
we
talk
about
the
shiny
new
histograms,
let's
see
what
is
a
histogram,
a
histogram
lets.
You
distribute
your
observations
into
multiple
buckets.
Let's
take
this
example.
So
in
all
the
examples
that
I
talk
it's
I
am
going
to
observe
the
latency
of
a
request.
So
on
the
y-axis,
it's
the
number
of
requests
on
the
x-axis.
It's
the
request,
duration.
A
So,
in
this
particular
histogram
we
can.
What
we
can
get
out
of
this
is.
There
are
15
requests
that
are
less
than
0.1.
Second
latency.
There
are
25
requests
from
0.1
to
one
second
and
one
to
two
seconds
and
the
last
bucket
is
special.
It
encapsulates
all
the
requests
that
were
greater
than
two
seconds
into
a
single
bucket.
A
So
how
do
we
store
this
in
prometheus?
So
in
prometheus
we
give
one
time
series
for
each
bucket,
so
we
have
a
label
for
a
series
called
l
e,
which
means
less
than
or
equal
to
and
prometheus
recognizes
this
special
label
as
the
bucket
value
for
a
time
series.
So
so
for
this
particular
histogram.
We
have
four
bucket
time
series
each
mentioning
here
the
bucket
boundaries
and
if
you
notice
it
is
less
than
or
equal
to
so.
A
The
first
time
series
which
is
0.1
includes
all
the
count
of
all
the
requests
that
had
latency
less
than
0.1
seconds,
which
is
15
and
the
next
one
is
less
than
or
equal
to
1
seconds,
which
is
all
the
requests
that
were
before
one
second,
so
it
includes
the
first
bar
and
the
second
bar.
Similarly,
the
third
time
series
is
the
sum
of
first
second
and
third
bar,
and
there
is
a
plus
infinity
bucket,
which
is
everything
before
infinity.
Basically,
the
total
count,
and
we
have
two
additional
time
series
for
the
entire
count
and
some.
A
So
this
is
a
problem.
The
first
problem
is,
we
have
to
pre-define
these
bucket
boundaries,
even
before
instrumenting
like
when
you
write
the
instrumentation
code.
You
have
to
mention
these
bucket
boundaries
and
it
can
get
tricky
and
it
can
take
some
experimentation
to
get
these
bucket
boundaries
right
and
the
buckets
are
cumulative.
If
you
see
this
particular
example,
we
have
four
buckets
that
are
filled
and
I
have
changed
the
bucket
boundaries
a
little
bit,
but
we
have
defined
a
whole
lot
of
time
bucket
boundaries
for
this
particular
histogram.
A
So
in
this
example,
you
may
be
able
to
compare
the
bucket
1.0
and
bucket
2.0,
but
all
other
variations
are
incomparable,
so
you
will
have
to
wait
for
some
time
so
that
you
have
all
the
new
buckets
ready,
yeah
and
for
every
histogram
that
you
define
the
number
of
total
memory
series,
which
is
the
time
series
that
promises
consumes,
is
number
of
buckets
plus
three.
So
why
why
this
is
a
problem?
Take
an
example:
you
are
instrumenting
the
requests
and
you
have
sharded
the
histograms
like
one
histogram
per
status
code
per
route.
A
So
this
is
a
simple
example.
Let's
say
you
have
1000
parts
which
are
instrumenting
these
histograms
and
a
single
bucket
takes
about
few
thousand
series
across
all
your
deployment.
So,
even
if
you
add,
let's
say
five
additional
buckets,
it's
going
to
take
exponential
number
of
not
exponential,
but
a
huge
number
of
time
series.
A
So
here
comes
the
new
histograms
that
we
are
working
right
now,
it's
in
poc
stage.
We
have
a
huge
design
and
we'll
build
it
step
by
step
in
a
simple
way.
I'm
leaving
out
a
lot
of
details
out
of
this
so
that
it's
easier
to
understand.
So
what
I'm
going
to
talk
in
the
next
five
to
ten
minutes
explaining
this
is
a
multi-year
study
and
research
by
beyond.
Who
is
here
with
us
right
now.
So
beyond
myself
and
dieter,
my
colleague
from
graphene
labs,
we
worked
on
the
code
and
all
this
poc.
A
A
You
don't
have
to
pre-define
your
buckets.
The
buckets
are
already
pre-defined
for
you,
but
you
can
set
the
precision
like
the
resolution
factor
of
the
new
histograms.
For
example,
let's
take
the
factor
of
2
power
of
1.
What
does
it
mean?
It
means
the
you
multiply
a
bucket
boundary
with
2
to
get
the
next
bucket
boundary
and
we
always
start
with
the
number
1..
A
So
if
the
bucket
boundary
is
1,
the
next
bucket
boundary
will
be
2,
4,
6,
8
and
so
on.
Now
this
is
one
resolution
of
histograms
and
we
always
have
the
factor
as
some
power
of
two.
If
you
want
to
lower
the
resolution,
like
the
gap
between
two
consecutive
bucket
boundaries
is
huge.
You
just
take
the
factories,
2
2
and
you
get
the
bucket
boundaries
as
1,
2,
4,
1,
sorry,
1,
4,
16,
64
and
so
on,
and
you
can
take
the
factor
of
2
power,
4
or
2
power,
8
and
so
on.
A
A
If
you
want
to
go
in
the
other
direction,
the
factors
look
like
2
and
then
2
power
1
by
2,
which
is
square
root
of
2.
Then
x
will
be
2
power
1
by
4,
which
is
square
root
of
square
root
of
2
and
so
on.
So
you
multiplied
square
root
of
2
with
1
you
get
the
next
bucket
boundary
and
then
you
multiply
it
again.
You
get
the
2..
You
don't
have
to
worry
about
this
math.
A
A
For
example,
let's
take
the
factor
of
two
power
one
by
two
and
two
power,
one
by
four,
so
we
have
three
bucket
boundaries
in
above
and
when
you
increase
the
resolution,
one
step,
a
new
bucket
boundary
comes
between
all
the
bucket
boundaries
that
were
there
before
and
if
you
take
the
third
and
fourth
example,
all
the
boundaries
from
above
remain
the
same.
You
just
get
new
boundaries
in
between.
A
A
So
if
we
just
glance
at
this
for
a
moment,
we
see
that
between
resolutions
there
are
some
common
buckets,
so
that
helps
us
move
from
a
higher
resolution
histogram
to
a
lower
his
resolution.
Histogram.
In
this
example,
the
first
histogram
uses
a
factor
of
2,
so
the
boundaries
are
1
to
2,
2,
4,
4,
to
8
and
so
on.
A
So
if
you
wanted
to
decrease
the
resolution
of
this
histogram,
you
choose
the
the
next
factor,
which
is
2
power
of
2,
and
you
add
the
buckets
which
fall
into
the
new
buckets
and
you
get
the
new
histogram
and
you
increase
the
like
lower
the
resolution
again
and
you
get
the
new
histogram.
So
why
do
we
want
this?
A
We
saw
earlier
that
if
you
change
the
bucket
bucket
layout,
you
cannot
match
the
buckets
between
two
histograms
again
because
they
can
be
incompatible.
But
with
this
the
first
histogram
above
is
of
resolution
to
power
of
one.
The
second
is
of
two
power
of
two.
So
if
you
let's
say,
if
you
wanted
to
add
these
two
histograms,
you
just
convert
the
histogram
of
a
higher
resolution
to
a
lower
resolution,
and
now
you
can
do
any
kind
of
arithmetic
that
you
want.
A
So
that's
a
part
of
predefining
buckets
to
some
power
of
something
so
that
histograms
of
different
resolution
can
be
compared
together.
Yeah.
So
I
have
skipped
the
step
of
converting
the
higher
resolution
to
a
lower
resolution,
but
you
can
match
the
colors.
The
yellow
bucket
matches
here
and
the
blue
buckets
are
added
together
to
get
the
blue
bucket
yeah
I'll
move
on
to
the
next
slide
and
because
we
have
predefined
the
bucket
boundaries.
Now
you
have
to
just
only
specify
the
the
factor
that
you
want
to
multiply
for
every
new
bucket
boundary.
A
So
now
that
the
bucket
boundaries
are
fixed,
you
don't
need
to
store
the
bucket
boundaries
itself.
In
the
storage,
because
encoding
float,
numbers
is
expensive
and
not
very
efficient,
so
we
can
use
integer
numbers
which
starts
from
0
goes
up
the
number
and
down
the
number
line,
and
this
is
very
efficient
to
encode
and
takes
less
space
and
less
cpu
to
encode
and
decode.
A
So
we
give
the
id
0
to
the
bucket,
which
has
the
upper
boundary
as
one.
So
in
these
are
three
different
solution.
One
is
two
power
of
one.
The
second
histogram
is
two
power,
one
by
four,
which
is
a
higher
resolution
bucket.
The
next
is
two
power
of
two,
so
we
start
at
zeroth
bucket
and
every
new
bucket
gets
the
id
one,
two
three
four:
when
the
bucket
boundaries
are
increasing
and
when
it's
decreasing,
we
go
in
the
negative
direction.
A
So
so
we
have
built
up
all
the
information
that
we
need
about
the
histograms,
and
this
is
how
we
encode
it.
We
take
it.
We
take
an
example
of
this
histogram,
so
the
first
part
in
the
histogram
is
the
metadata
which
encodes
the
resolution.
The
total
sum
resolution
is
just
the
factor
that
we
talked
about
2
power
of
2
power,
of
something
and
the
sum
and
count.
This
is
enough
to
decode
the
rest
of
the
histogram,
so
the
next
part
we
call
it
a
span
which
tells
you
what's
the
bucket
layout
in
this
particular
histogram.
A
So
if
you
observe
the
histogram,
the
first
four
buckets
are
consecutive,
they
are
one
after
another
and
then
there
is
a
gap
of
two
buckets
and
then
there
is
a
bucket
again.
We
are
using
the
factory
two
power
one
and
let
me
use
that
to
decode
what's
written
in
the
span,
so
we
have
zero,
comma
four,
then
two
comma
one.
It
means
the
first
bucket
starts
at
index
zero,
because
the
first
bucket
has
upper
boundary
of
one
and
at
zero
index
there
are
four
buckets,
hence
zero,
comma
four
and
the
next
two
comma.
A
One
says
that
after
the
previous
set
of
buckets,
you
have
a
gap
of
two
buckets,
so
you
have
to
skip
two
buckets
and
the
next
stream
of
buckets
is
of
length
one.
So
you
have
one
bucket,
so
this
compressed
format
tells
you
what's
the
bucket
layout
and
now
you
know
which
buckets
are
filled
and
which
buckets
are
unfilled
and
we
just
store
the
count
in
each
bucket
consecutive.
There
are
there
on.
A
We
map
one
time
series
to
all
the
systems
currently
in
prometheus,
a
sample
which
is
a
sample
has
timestamp,
as
in
64
and
a
value
as
float
64..
We
just
replace
the
float64
with
the
new
encoding
that
we
just
described
right
now
and
yeah,
and
you
just
have
one
time
series
like.
If
you
increase
the
number
of
buckets
or
decrease,
it
doesn't
change
the
number
of
series
and
it's
efficient
like
there
was
a
talk
in
prom
con
last
year,
and
we
saw
that
this
new
encoding
without
having
one
series
per
bucket
gives
about.
B
A
Than
90
percent
of
index
size
savings,
if
you
have
too
many
buckets
previously
and
roughly
around
50
of
disk
size
savings
yeah.
So
how
do
you
instrument
this?
It
is
as
simple
as
this,
so
everything
remains
same
as
previous
instrumentation
and
you
don't
define
your
buckets.
You
just
define
what
factor
you
want
to
use
so
here.
I
have
example
of
two
and
four
four
is
to
power
of
two
and
you
don't
need
to
get
the
precision
very
right.
A
The
instrumentation
library
automatically
chooses
the
closest
precision
from
what
you
define
and
as
and
when
you
observe
any
values.
The
buckets
are
filled
automatically
you
don't
have
to
like
yeah.
It's
just
filled
automatically.
The
new
buckets
are
created.
If
it
has
value,
if
it
doesn't
have
any
value,
the
bucket
does
not
exist.
Hence
the
name
sparse
high
resolution.
Histogram
sparse,
comes
from
the
fact
that
we
don't
care
about
empty
buckets.
A
A
The
scraping
looks
like
this.
Currently,
we
have
a
http
request
which
asks
for
a
text
format
matrix
which
gives
the
time
series
in
the
format
that
we
saw
earlier,
and
we
do
another
http
request
to
get
the
new,
sparse
histograms.
We
are
encoding
this
new
space
histogram
in
protocol
buffer
format,
because
it's
more
efficient
and
can
be
easily
described
like
the
new
histograms,
can
be
easily
described
in
protobuf
compared
to
the
text
format.
So
prometheus
does
two
requests
to
a
target
to
get
all
the
data
that
it
needs
now.
A
A
A
So
I
have
like
the
instrumentation
example
that
I
showed
is
the
same
thing
that
we
are
seeing
here
right
now,
so
I
have
two
histograms
one
is
called
midi,
a
lower
resolution
whose
factor
is
2
power
of
2,
which
is
4,
and
the
medium
resolution
is
2
power
of
1,
which
is
2.
Don't
worry
about
that
number?
I
am
going
to
use
different
timestamp
to
show
you
different
values
and
arithmetic,
like
we
have
implemented
a
bunch
of
from
ql
functions
which
work
on
these
new
histograms.
B
A
A
B
A
A
B
A
A
A
Okay,
now
this
is
actually
scraping
a
live
cluster.
We
have
a
mimir
which
is
like
memory
cluster
running
in
our
dev
environment.
We
have
instrumented
it
with
this
new
histograms
with
a
very
high
resolution,
and
we
have
a
prometheus
running
in
dev,
which
is
scraping
these
histograms
and
I'm
port
forwarding
that
pod
and
connecting
into
a
local
local,
prometheus
yeah.
These
are
just
requests,
and
here
you
can
clearly
identify
that
there
is
a
band
of
latency.
There
are
different
bands
of
latencies.
A
B
A
How
can
you
use
this,
so
everything
that
we
saw
here
is
open
source.
The
instrumentation
is
available
in
the
client,
golang
library,
a
client
blue
line
repo,
but
it's
in
a
branch
called
space
histogram.
Similarly,
the
prometheus
server
that
we
ran
is
open
source
and
it's
in
the
sparse
instagram
branch
and
the
grafana.
A
A
C
Hi
yeah
thanks
for
the
talk,
was
very
interesting,
so
I
get
the
mathematical
properties
of
this.
I
think
it's
really
pretty
nice,
but
let
me
know
if
I'm
looking
at
this
correctly
so
let's
say
we're
measuring
latency
and
we,
our
latency
for
a
specific
endpoint,
is,
has
a
normal
distribution
centered
at
around,
like
200
milliseconds
right.
C
So
if
we
don't
do
anything
most
of
our
and
so
we
have
a
standard
deviation
that
makes
it
go
from
like
150
to
250
right.
So
if
we,
depending
on
the
factor
that
we
use
we're,
losing
a
lot
of
precision
there
right
like
most
of
it,
is
going
to
the
to
go
to
the
same
kinds
of
packets,
is
there
a
like?
Are
you
thinking
about
this,
like
we
thought
to
play
with
offsets
and
scale
so
that
you
can
get
the
best
resolution
for
your
metric.
A
Okay,
so
when
you
set
a
precision,
the
bucket
boundaries
are
fixed,
so
you
cannot
ask
to
focus
on
a
particular
range
of
values,
but
what
could
be
done
is
like
there
was
another
option
in
the
client
library
which
I
did
not
mention.
Is
you
limit
number
of
buckets
that
you
want,
so
you
can
set
a
very
high
precision
and
a
limit
on
buckets
like
150
or
200,
which
is
still
very
practical
with
these
new
histograms
and
once
it
hits
the
for
example,
200
bucket
limit
it
will
automatically
go
into
the
next
lower
precision.
A
A
There
is
something
called
a
compressed
unit
of
histograms
called
chunks
which
contains
120
histograms
at
a
time.
So
once
those
120
histograms
are
done,
you
start
okay.
This
is
wrong.
Okay,
sorry,
so
it's
possible
that
you
can
reset
a
histogram,
for
example,
after
every
10
minutes
or
half
an
hour
one
hour,
you
can
set
it
to
reset
the
histogram
so
that
you
again
start
with
the
highest
precision
and
low
number
of
buckets
filled.
So
that's
another
way
to
move
back
to
high
resolution
again.
Okay,.
D
Hi
thanks
for
this,
it's
seems
very
elegant.
It
might
be
a
misunderstanding
of
mine,
but
I
feel
like
the
it
seems
like
the
the
sort
of
distribution
of
the
buckets
across
the
number
line
is
assuming
the
data
is
going
to
be
followed
like
a
long
tail
distribution
like
there's
going
to
be
more
sort
of
like
samples
at
the
lower
end
of
the
number
line
and
they'll
get
more
and
more
sort
of
spread
out
as
you
go
up.
Is
that.
A
A
A
Yeah
and
that's
correct
like
if
you
set
higher
precision,
the
size
of
buckets
is
going
to
be
small
up
to
some
extent
and
it
just
exponentially
grows.
So
the
idea
with
this
was
the
gap
between
two
consecutive
buckets,
like
the
percentage
difference
between
two
consecutive
buckets
is
fixed
so
and
whenever
you
do
a
quantile
estimation,
the
error
of
your
estimation
is
limited
to
what's
the
difference
between
the
buckets
like
the
percentage
difference.
A
So
the
idea
was
to
reduce
the
percentage
error
of
the
histogram
quantile
estimations
and
if
you
take
the
plus
infinity
bucket
currently
in
prometheus,
if
there
is
a
value
which
falls
in
that
bucket,
you
cannot
accurately
predict
within
a
certain
percentage.
If
the
quantile
estimation
is
correct,
if
it
falls
there,
so
that's
right,
like
the
spread
will
be
big.
But
if
you
take
the
percentage
error
of
your
estimations,
that's
going
to
be
limited
wherever
you
go.
A
B
E
A
E
D
Is
that
okay,
okay,
you
can
ask
the
question
okay,
I
was
just
gonna
ask
like
if
you
were
to
to
go
ahead
with
this.
It
sort
of
sets
the
precedent
for
non-uh
floating
point
series
values.
Do
you
think
there's
other
use
cases
for
that,
like
with
you
know,
once
once
that
precedence
has
been
set?
Are
there
other.
A
D
A
So
doing
changes
to.
If
I
understand
your
question
right,
you
mean
we
replace
float64
with
a
different
data
structure.
Yeah
is
this
scope
to
add
different,
more
data
structures
yeah?
So
there
is
it's
possible,
but
the
big
problem
that
comes
with
it
is
making
any
change
to
the
data
type
requires
changing
the
tsdb,
the
from
kill
engine
and
everywhere
you
access
the
sample.
D
A
Okay,
so
your
question
was:
how
can
we
access
individual
buckets,
so
we
have
a
prom,
a
huge
design
dock,
explaining
all
the
proper
things
that
we
want
to
do
with
this,
and
this
is
included
there.
So
we
will
have
the
ability
to
ask
for
a
value
at
a
bucket
just
that
it's
not
implemented
at
the
moment.
A
G
B
G
A
So
when
you
say
the
center
of
mass,
it
is
just
the
calculation
which
starts
at
one
like
the
like
the
calculation
of
bucket
boundaries,
which
starts
at
one,
but
the
observations
that
you
do
can
be
centered
anywhere,
but
the
problem
with
centering
the
calculation
somewhere
else
is
it
creates
a
problem
again.
You
cannot
mix
and
match
other
histograms.