►
From YouTube: Streaming Recording Rules for Prometheus, Thanos, and Cortex Using... Gibbs Cullen & Rob Skillington
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Streaming Recording Rules for Prometheus, Thanos, and Cortex Using the M3 Coordinator - Gibbs Cullen & Rob Skillington, Chronosphere
As the Prometheus ecosystem has matured, the compatibility between projects has accelerated and it is possible to use features from one project with another. In this talk, Gibbs and Rob will show how to use various parts of the M3 ecosystem to perform in-memory streaming metrics aggregation, without using M3DB itself, thanks to developments in core Prometheus and M3. Using two deployment options, they will perform a demo using a concrete example of creating a highly efficient cAdvisor overview dashboard that shows fast aggregate views of CPU, memory, network and disk activity for pod groups from tens of thousands of active containers. They will also demo how to use streaming recording rules to store per-metric 10m or 1h downsampled metrics with a global metric suffix in Prometheus, Cortex, and other Prometheus remote storage solutions, like M3 and Thanos, that don’t yet have the ability to downsample metrics.
A
All
right
so
just
a
little
bit
about
us
before
getting
started,
so
my
name
is
gibbs
cullen.
I'm
a
developer
advocate
at
chronosphere
during
my
time
at
chronosphere.
I've
helped
work
on
the
m3
project
as
a
contributor,
also
a
member
of
the
cncf
observability
tag
and
and
then
I
think
I
have
my
twitter
up
there
as
well.
A
All
right
so
just
gonna
go
through
the
agenda.
Real
quick,
so
gonna
start
out
going
through
the
problem
statement
and
then
gonna
talk
about
aggregation
using
prometheus
recording
rules,
and
then
we
will
go
into
streaming
aggregation
using
prometheus
in
the
m3
coordinator,
which
rob
will
then
give
a
quick
demo
on
and
then
time
for
q,
a
all
right
so
querying
high
cardinality
metrics.
A
A
So,
as
you
can
see
here
and
this
and
this
query
here
so
we're
looking
at
the
cpu
usage
metric
and
you
can
see
that
it
it's
going
to
take
quite
a
long
time
for
you
to
render
all
of
your
query
results
and
that's
because
your
query
is
pulling
again
across
pulling
all
the
labels
across
all
of
your
pods
and
that's
going
to
result
in
about
51
000
time
series
metrics.
So
with
that
sort
of
load,
you're
going
to
have
it's
going
to
take
about
20
seconds
for
your
query
results
to
render.
A
A
So
in
this
example,
we
are
looking
at
just
two
labels,
so
we're
aggregating
on
two
labels,
container
name
and
namespace,
and
by
by
doing
this
aggregation
on
these
two
labels
kind
of
before
you
do
the
query
you're
able
to
cut
down
by
quite
a
bit
the
time
it
takes
for
your
results
to
render.
So
in
this
example,
now
it's
only
going
to
take
less
than
a
second
for
your
query,
results
to
render
and
then-
and
that's
mostly
because
you've
cut
it
down
to
about
230
time
series.
A
You
have
all
of
your
prom
ql
functions
available
to
you
and
in
this
slide
we're
just
going
to
look
at
an
example
of
a
recording
rule.
So
I
believe
the
the
code
here
we
have
the
cpu
kind
of
usage
metric
that
we
are
summing
at
a
regular
interval.
So
every
minute
in
this
recording
rule
and
then
it's
going
to
be
grouped
by
the
two
labels
and
the
config
so
container,
name
and
namespace,
but
basically
inside
this
inside
this
diagram,
you
have
your
single
prometheus
node.
A
You
have
your
rule
manager,
it's
pinging
the
querier
at
these
regular
intervals.
So
these
are
stimming
samples
inside
the
diagram
here
and
at
that
point
it's
going
to
create
a
brand
new,
aggregated
time
series
which
will
then
be
sent
to
your
time
series
database-
and
this
is
all
just
just
to
know.
This
is
going
to
be
all
done
in
memory
and
as
like
a
single
process.
A
But
let's
say
you
wanted
to
do
aggregation
outside
of
your
prometheus
instances
or
and
kind
of
use,
a
remote
storage
solution
for
larger
scale
use
cases
and
to
also
aggregate
at
almost
any
level
of
time
series
without
increasing
the
load
of
your
heavy
recording
rules.
Then
m3,
then
kind
of
you
could
do
this
using
m3
and
m3
coordinator,
which
allows
for
the
streaming
aggregation.
A
Okay,
so
kind
of
just
to
go
into
a
little
bit
more
about
how
m3
does
the
streaming
aggregation?
First
of
all,
just
a
quick
review
of
a
recap
of
what
m3
is
so
it's
a
remote
storage,
prometheus
remote
storage
solution.
It's
created
back
at
uber
in
2016
to
help
manage
their
metrics
monitoring,
use
cases
internally
and
now
is
used
by
many
other
big
companies,
including
including
chronosphere,
and
then
with
m3.
So
the
aggregation
is
going
to
be
moved
from
kind
of
your
recording
rules
to
streaming
aggregation
and
we're
going
to
do
that.
A
Using
what
we
call
roll
up
rules
and
rollup
rules
essentially
solve
the
same
problem
as
prometheus.
Recording
rules,
except
we
just
take
a
slightly
different
approach
and
what
we
do
is
we
through
the
aggregation
using
the
m3
coordinator
and
in
some
use
cases
the
aggregator
tiers
like
in
the
diagram
here.
We
use
them
to
reconstitute
metrics,
so
counters
gauges,
histogram
metrics,
as
if
they
had
been
exposed
as
an
aggregate
from
a
single
instance
of
an
application
versus
from
several
instances
of
an
application.
A
And
then
we
also
do
provide
another
way
to
aggregate
metrics
through
mapping
rules
which
essentially
just
does
down
sampling
of
metrics,
but
regardless
both
ways
of
aggregation.
Once
once
the
aggregations
are
completed,
these
new
metrics
will
be
then
sent
over
to
your
timeshares
database
or
your
remote
storage
solution.
A
And
then
yeah
just
zooming
in
a
little
bit
and
kind
of
recapping
on
those
two
ways
of
aggregating
metrics.
So
with
these
visuals
here,
which
I
think
are
helpful,
so
we
have
kind
of
roll
up
rules
which
essentially
just
roll
up
across
or
aggregate
across
multiple
time
series-
and
this
is
essentially
just
reduces
your
query
time
cardinality
and
then
we
have
mapping
rules
which
do
the
down
sampling
by
doing
aggregations
inside
a
single
time
series
and
then
making
kind
of
these
longer
term
or
larger
scale.
Queries
more
efficient
as
they're
have
fewer
samples.
A
A
So
we
have
the
three
main
steps
we'll
go
through
here,
but
you
can
see
there's
the
on
the
configuration
we
have
kind
of
the
the
roll-up
rule
name
and
then
there's
the
filter
that
we
use
as
well
and
then
and
then
in
terms
of
the
steps
we
have
our
first
step,
which
is
to
take
the
increases
or
the
deltas
between
each
of
our
monotonic
counters.
So
you
can
see
in
the
bar
chart.
We
have
we're
kind
of
measuring
the
little
the
increments
in
between
each
of
the
counters
measuring
the
increases
or
the
deltas.
A
So
once
we've
done
the
summing
of
those
increases,
then
we
can
now
do
the
final
transform
and
the
final
step
here,
which
is
to
create
turn
this
metric
into
a
monotonic
cumulative
counter,
which
will
then
be
sent
over
and
persisted
to
your
time,
series
database
or
remote
storage
and
kind
of
one.
That's
one
that
has
been
done.
You'll
use
the
storage
policies
set
in
this
configuration
here
to
kind
of
dictate
the
resolution
and
retention
periods.
B
Cheers
thanks
gibbs,
so
yeah.
I
was
going
to
just
quickly
step
through
a
few
of
the
the
differences
at
a
high
level
and
then
give
a
quick
demo
if
the
demo
gods
are
kind
today
which
they
may
not
be.
But
we'll
see-
and
you
know
I
think
what's
first
worth
talking
about
is
this
is
actually
a
similar
example
that
I
talked
just
in
the
short
earlier
keynote.
B
But
when
we
look
at
like
specific
use
cases
that
can
be
very
high,
cardinality
and
again,
histograms
tend
to
be
a
obvious
choice
here,
because
the
the
amount
of
buckets
which
you
define,
which
gives
you
the
level
of
granularity
you
want
in
terms
of
how
precise
your
histogram
measurements
can
be,
can
multiply
your
cardinality
quite
extensively.
And
so
you
know
in
this
example
we're
looking
at
yeah
single
microservice
with
200
pods,
20,
endpoints,
http
or
grpc.
B
B
B
B
But
again,
the
majority
of
the
problem
here
is
not
in
any
other
part
of
the
expression
except
a
few
distinct
high
cardinality
labels
that
you
can
aggregate
away
and
reconstitute
a
new
counter
as
if
it
was
aggregated
from
one
place.
So
it
stays
on
that
expensive
step
that
that
runs
over
the
network.
It
also
avoids
this
reverse
index:
query
pressure
and
just
release:
query
bandwidth
on
prometheus
or
your
remote
storage
in
general.
B
B
I
just
want
anything
that
matches
a
specific
filter
to
exclude
a
certain
high
cardinality
label
that
I
know
exists
on
all
of
my
metrics,
like
the
pod
or
instance,
and
so
we're
gonna
have
a
look
at
what
this
looks
like
in
reality
now
bear
with
me,
because
I
am
gonna.
Try
to
just
remove
the
screen
mirroring.
C
D
B
B
Then
you
go
to
integrations
and
instead
of
going
to
the
default
prometheus,
remote
storage,
aggregation
inquiry
with
m3,
you
go
to
the
aggregation
for
prometheus,
thanos
or
other
remote
right
storage,
and
this
will
highlight
the
dock
and
compose
stack
that
I'm
about
to
run
through.
So
it
has
a
deep
link
to
the
development
stack
that
I'm
about
to
run
locally.
B
What's
interesting
here
in
the
development
stack
is
basically
we
have.
If
we
go
look
at
the
document
compose
file,
we
have
a
coordinator
which
is
receiving
traffic
from
a
prometheus
scraper,
so
this
scrape
this
prometheus
instance
just
acts
purely
as
a
scraper
sends
all
of
its
data
to
m3
coordinator
and
one
through
remove
right
yeah.
That
is
a
good
call.
Thank
you
I'll.
Do
that
on
the
terminal
as
well.
B
So
this
this
prometheus
scraper,
basically
scrapes
a
whole
bunch
of
c
advisor
metrics
in
this
demo
and
sends
them
to
m3
coordinator
via
prometheus
from
our
right
and
then
either.
You
can
use
m3
coordinator
by
itself
to
aggregate
in
in
in
memory.
You
know
with
a
single
replica
over
running
as
a
sidecar,
and
then
it
can
basically
aggregate
across
those
metrics
and
give
you
rolled
up
metrics
and
and
then
it
sends
basically
data
either
to
two
different
prometheus
instances.
B
One's
called
raw,
which
is
just
storing
back
the
samples
that
the
scraper
was
sending
it
and
then
the
other
one
is
aggregate
which
is
receiving
both
time
series
that
were
aggregated
using
rules
that
you
defined
as
well
as
down
sampled
to
like
either
one
minute
or
five
minute
of
resolution,
so
that
it
basically
means
you
can
get
different
levels
of
resolution
similar
to
how
thanos
does
batch
down
sampling.
But
this
is
all
happening
in
real
time.
So
it's
available
like
every
five
minutes
as
a
data
point,
is
ready.
B
B
Take
the
increase
of
it
group
by
these
dimensions,
which
is
the
container
the
namespace
and
the
cpu
index.
If
you
want
to
look
at
stats
on
a
per
cpu
basis,
but
not
across
all
the
instances
and
then
we're
building
the
the
monotonic
counter
again
here
and
we're
storing
it
in
this
storage
policies
tells
us
which
prometheus
those
those
two
prometheuses
that
we
had,
which
one
to
store
it
in
and
those
are
defined
up.
Here
we
have
the
raw
prometheus.
B
B
B
You
need
to
view
your
data
in
your
graphs
and
so
obviously
over
large
periods
of
times
with
you
know,
lots
of
pods
rather
than
a
small
demo
environment.
I'm
marrying
this
really
adds
up
because
it's
you
know
a
factor
of
a
thousand
divided
by
23.
So
you
know
43
43,
x,
larger
in
terms
of
cardinality
and
yeah.
That's
that's
pretty
much
what
I
wanted
to
step
you
through.
I
think
we
have
a
brief
amount
of
time
to
do.
Q.
A
perhaps.
B
E
F
B
So,
instead
of
writing
directly
to
the
faintest
receiver
via
remote
right,
you
write
to
the
coordinative
I
remote
right
and
then
have
the
coordinator
point
to
the
thanos
receiver
and,
and
then
the
optional
part
of
the
of
this
whole
thing
as
well
is,
if
you
wanted
to
not
just
use.
A
single
process
for
the
m3
coordinator
is
is
having
a
cluster
of
entry
aggregators,
which
the
coordinator
can
use
for
for
distributed.
Stateful
aggregation,
awesome.
C
C
B
B
That
that's
really
interesting
question
so
and
just
to
repeat
it,
I
guess
yeah,
you
know,
cpu
spikes
latency
deviation,
yeah.
I
think
everyone
struggles
with
this
one.
You
know
it's.
Everyone
has
their
own
strategies,
obviously
using
a
sustained
high
period,
for
you
know
for
like
latency
spikes
can
help
in
terms
of
like.
B
What's
known,
good
standard
deviation
does
tend
to
work
well,
but
it
requires
a
lot
of
care
right,
like
looking
at
week
over
week,
data
like
my
using
an
offset
by
seven
days
and
comparing
that
to
the
current,
that's
that's
all
pretty
yeah.
It
requires
a
lot
of
human
curation.
I
I
I
would
say,
there's
probably
better
sources
than
me
that
that
have
deeply
written
about
it.
B
D
Of
course
hi
there.
Thank
you
for
your
talk.
One
of
the
challenges
with
prometheus
is
the
high
carbon
high
cardinality,
with
the
metrics
that
you
have
available
and
something
that
can
help
with
that
challenge
is
the
recording
rules
which
you
talked
about
going
a
step.
Further
m3
appears
to
be
able
to
help
with
the
roll-up
and
mapping
rules
again
to
help
with
the
high
cardinality
of
the
recording
rules
that
you
might
be
creating
on
a
label
by
label
or
aggregation
by
aggregation
basis
on
what
are
some
best
practices.
D
D
Yeah,
for
example,
you
might
be
taking
a
look
at
multiple
types
of
aggregations
on
top
of
a
single
metric,
such
as
the
sum
or
the
p99,
or
things
like
that.
Why
not
all
of
them?
How
does
that
help
shift
the
problem
of
the
high
cardinality
of
the
labels
that
exist
into
the
high
cardinality
of
rules
that
you
might
be
aggregating
over
yeah.
B
That's
where
the
metric
template?
Sorry,
the
aggregation
template
really,
you
know
shines
in
that.
You
define,
like
anything
that
matches
underscore
total.
If
you
assume
that
to
be
a
counter,
then
you
can
exclude
by
you
can
apply
a
rollup
of
increases
and
deltas
and
then
and
then
build
a
monotonic
counter
on
those
and
use
the
metric
name
as
the
the
resulting
input
as
a
variable
for
the
output
metric
name.
B
E
B
Yeah
I
mean,
I
think,
gibbs
touched
on
it
a
little
bit.
It's
been
part
of
m3,
since
m3
was
built
as
a
remote
storage
at
uber
and
the
main,
the
main
reason
why
you
know
that
that
was
really
done
was
that
it
it.
We
had
all
these
high
cardinality
use
cases,
and
it
was
just
very
difficult
for
all
the
developers
to
to
go
and
curate
and
specifically
protect
against
cardinality
explosion
and
every
single
one
of
their
different
things
that
they
were
monitoring.
B
And
so
you
know
the
aggregator
was
built
to
be
able
to
like
give
the
power
back
to
the
end
users,
so
they
don't
have
to
go
and
reinstrument
their
application.
They
can
just
derive
different
pivots
of
the
same
very
high
cardinality
data
that
they're
pumping
in
so
at
some
point.
You
know
you
can
just
throw
money
in
the
problem
to
collect
the
metrics,
but
then
the
dashboard
spinner
or
the
timeouts
you
know
of
of
your
vowels
start
to
hit.