►
Description
Integrating Long-Term Storage with Prometheus [A] - Julius Volz, Prometheus
While Prometheus's built-in storage does not support long-term data retention and durability, Prometheus provides nascent interfaces for integration with external long-term storage systems. This talk explains how these interfaces work, and how you can integrate with them to create custom long-term storage solutions. It will also showcase some real-world integration examples and highlight the key challenges in implementing them.
About Julius Volz
Julius co-founded Prometheus and lead the project to success at SoundCloud and beyond. He now focuses on growing the Prometheus community and is the main organizer of PromCon, the first conference around Prometheus. In a previous life, Julius was a Site Reliability Engineer at Google.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
I'm
Julius
I'm,
one
of
the
cofounders
of
Prometheus
nowadays
I'm
freelancing
for
multiple
companies
around
Prometheus
doing
various
stuffs.
Some
touches
the
aspect
of
long-term
storage
integration
in
Prometheus,
so
I'm
going
to
basically
tell
you
a
bit
today
where
we
are
currently
and
where
we're
going.
This
is
just
a
start,
so
historically,
Prometheus
has
only
had
a
local
storage
Prometheus
stores,
all
its
data
on
local
disk.
That's
complicated
enough.
We
did
it
like
this
very
specifically
for
a
reason.
First
of
all,
it's
simpler
to
build,
but
it's
also
it
can
be
simpler
to
run.
A
You
know
if
you
want
to
wipe
your
storage,
you
just
wipe
your
local
storage
directory.
It
avoids
any
complicated
clustering,
especially
for
monitoring
system
that
is
supposed
to
be
up
all
the
time
having
it
break
down
in
the
face
of
a
network
partition
when
you
really
need
it
is
a
bad
idea,
and
so
you
can
build
really
highly
available
primitive
setups
out
of
just
having
two
promises
that
are
identically.
Configured
scrape
the
same
data
do
both
alert
evaluation
in
parallel
totally
independently
from
each
other.
A
A
We
have
always
said
that
we
were
at
least
hesitant
to
put
more
advanced
features
into
the
local
storage
advanced
features
such
as
down
sampling
data,
because
we
said
well,
it's
not
really
supposed
to
be
a
long-term
storage.
Anyway,
it's
not
flexible
in
two
ways.
On
one
hand,
it's
only
storage,
but
you
might
want
to
do
something
completely
different
with
the
incoming
samples
like
put
a
stream
processor
in
front
of
it
or
on
the
other
hand,
if
you
run
Prometheus
on
a
cloud
native
system
like
kubernetes,
you
might
have
noticed.
A
Ok,
like
every
time,
I
reschedule
my
kubernetes
pod.
That
has
parameters
in
it.
My
storage
history
is
completely
gone
unless
I
use
some
strategy
to
mount
in
persistent
storage.
So
obviously,
like
we
had
this
idea
very
early
on
that
it
would
be
nice
to
have
a
local
storage.
This
is
one
of
the
first
primitive
issues
tiled
ever
I've
naively
filed
it
in
2012
within
the
first
month
of
the
project,
and
we
say:
ok
well,
we
want
to
have
something
like
this
eventually
I
think
this
issue
has
by
now,
under
over
150
comments.
A
A
So
let's
post
some
design
questions
on
the
right
path.
We
had
this
kind
of
argument
of.
Do
we
send
entire
chunks
over
so
Prometheus
local
storage,
already
chunks
samples
in
every
time,
series
and
compresses
these
chunks
very
nicely,
or
do
we
just
send
life
as
we
get
the
samples?
Do
we
send
them
life
over
to
the
remote
system?
Also,
how
much
buffering
and
retries
do
we
want
to
do
if
the
remote
end,
for
example,
is
just
not
reachable
at
a
moment
for
the
read
path?
The
biggest
question
was:
do
we
do?
A
A
That
would
actually
be
way
more
efficient
network
wise
to
have
the
local
storage
really
efficiently
compress
up
many
samples
for
the
same
series
and
then
set
that
whole
compressed
bunch
batch
of
samples
over
with
only
the
labels
and
the
metric
name
exactly
once,
but
the
problem
is
then
like
what
chunk
size
does
the
remote
system
actually
need?
Is
it
the
same
chunk
size
as
parameters
has
a
totally
different
one?
A
A
Okay,
let's
keep
it
simple
for
now
we're
just
sending
samples
individual
samples
over
and
yeah
that
basically
enables
simple
use
cases
for
now
on
the
buffering
and
retries
front,
we
wanted
to
be
a
bit
careful
on
one
hand:
it's
we
actually
charge
the
writer
because
it
tends
to
be
actually
quite
high,
throughput
business
if
you're
writing
a
lot
of
data
to
a
remote
end
tom
who's
way.
Back
there,
he
wrote
like
a
little
pit
controller
that
dynamically
increases
or
decreases
the
number
of
parallel
charts
with
which
we
send.
A
We
don't
want
to
do
too
much
queuing
in
this
part
that
writes
out
samples
to
the
remote
storage
one,
because
the
representation
of
the
samples
in
there
is
quite
memory
intensive
and
B,
because
if
we
just
stop
for
up
too
much
and
retry
too
much,
it
might
actually
use
so
much
memory
that
it
takes
down
the
rest
of
Prometheus,
which
is
really
not
what
we
want.
We
still
want
perimeters
to
be
like
this
decoupled,
very
reliable
system
that
just
happens
to
also
talk
to
a
remote
storage,
the
protocol
and
encoding.
A
We
actually
started
out
using
G
RPC
and
then
hit
a
snack
because
some
components
on
the
internet
and
in
you
know,
on
the
path
between
the
primitives,
that
was
collecting
data
and
the
remote
storage
did
not
support
HTTP
2.
Yet
there
were
some
G
RPC
problems
with
that.
So
for
now
we
are
doing
plain
HTTP
and
protocol
buffers,
but
in
the
future
probably
will
also
enable
G,
RPC
or
even
replace
the
HTTP
and
protobuf.
We
have
to
see
about
that.
A
A
Yeah,
the
way
it
works
is
just
a
sense,
every
individual
sample,
as
it
comes
over
directly
to
the
remote
storage.
There
is
some
batching,
of
course,
as
I
mentioned,
but
not
a
lot
in
the
optimal
case.
You'd
have
some
storage
system
that
really
natively
into
integrates
with
this
Prometheus
remote
right
protocol
and
just
takes
that
directly.
But
more
realistically,
you'll
probably
want
to
write
something
into
an
existing
storage
system
where
you
would
have
a
bridge,
it
gets
the
sample
stream,
and
then
it
manages
that
into
whatever
format
your
actual
system
needs.
A
We
have
an
example
bridge
in
the
documentation
directory
of
Prometheus,
which
supports
in
flux,
DB
grass
eyes
and
open
GS
DB
and
for
the
right
path,
there's
no
repair,
they
have
to
be
careful
and
or
more
generally,
you
could
put
any
arbitrary
processor
behind
that
bridge
to
do
whatever
you
want.
You
know
Kafka
or
whatever.
A
So
what
you
write
out
doesn't
necessarily
have
to
be.
A
storage
system
doesn't
necessarily
have
to
be
a
system
that
you
read
back
from
configuration
rise.
All
you
need
is
a
URL
in
your
config.
You
can
also
have
so
I
grayed
out
a
bunch
of
options
that
optional.
You
know,
authentication
and
TLS
option
options.
What's
interesting
here
is
you
can
have
relabeling
in
this
path
as
well,
so
you
can,
for
example,
choose
which
series
to
send
and
which
to
drop
for
the
remote
storage.
A
So
on
the
read
path,
we
had
the
central
this
question
around
distributed
or
centralized
evaluation
on
one
hand
the
scalability,
if
you
want
to
make
really
big
crews
over
many
time
series
of
a
lot
of
history.
At
some
point,
you
would
need
some
kind
of
distributed
evaluation
down
sampling,
aggregation
of
local
results
and
so
on,
but
that
would
require
some
kind
of
remote
knowledge
of
our
construct,
or
you
know
you
would
need
to
translate
from
to
our
construct
into
influx
to
L
or
whatever
you're
dealing
with
and
what
we
require
more
complexity.
A
On
the
remote
end,
we
think
that
the
local
evaluation,
where
you
do
all
the
evaluations
in
the
primitives
that
requests
the
data,
is
probably
good
enough
for
most
queries
that
are
not
too
big.
It
helps
in
that
case,
if
the
remote
end
supports
some
kind
of
down
sampling.
So
that
you
don't
request
as
much
data
as
you
know,
the
raw
data
that
you
want
pumped
into
it,
because
that
can
be
quite
high
resolution.
A
So
we
do
the
centralized
computer
evaluation
and
the
read
path
looks
like
this:
we
send
over
to
a
bridge
or
even
a
tow
system,
maybe
a
set
of
time-series
matches
that
are
very
similar
to
what
you
would
type
in
prom
QL.
You
know
equals
unequals
and
so
on,
along
with
ranges
of
times
that
we
want
for
this
data,
then
the
bridge
talks
to
whatever
back-end
system
there
is
gets
the
data
and
sends
it
back
as
Ross
samples,
annotated
with
metric
name
and
labels
and
then
all
the
funky
evaluation.
A
A
This
would
enable
a
model
where
you
can
actually
configure
all
these
sub
Prometheus's
as
remote,
read
endpoints
and
then
just
do
it
a
query
time
different
reliability
aspects
and
so
on,
but
at
least
its
possibility.
There's
there's
more
possibilities
around
this.
Where
that
come
from
again,
you
just
need
to
configure
a
URL.
We
might
want
to
have
service
discovery,
support
there
in
the
future,
but
right
now
it's
just
simple
URL
and
some
options.
For
the
connection
note,
we
still
want
primitives
to
be
the
very
reliable
system
for
what
its
original
Eve
meant
to
do.
A
A
The
metadata
fetching
any
kind
of
metadata
is
only
against
the
local
storage
for
now.
So,
if
you
want
to
list
all
the
label
names
that
exist
or
all
the
values
for
a
given
label,
name
and
so
on,
but
we
want
to
also
at
least
have
the
option
of
fetching
that
from
the
remote
site,
we're
going
to
do
some
magic
based
on
the
external
labels
of
a
primitive
server.
A
A
primitive
server
has
a
set
of
external
labels
that
identify
it
against
other
primitive
servers
that
you
might
run,
and
so,
when
we
write
out
samples
to
remote
and
we
attach
external
labels
and
when
they
come
back,
we'll
remove
them
and
when
we
query,
we
include
them
in
the
koreans
on.
So
we
try
to
just
do
the
right
things
there.
That
is
not
yet
in
the
code
base,
but
will
be
by
next
week,
hopefully
and
cool.
So,
let's
get
to
to
actual
real
world
example.
B
A
First,
one
is
chronics,
which
is
a
solar
based
horizontally,
scalable,
long-term
storage
built
by
QA,
we're
in
Munich
and
I
freelanced
a
bit
with
them
last
year,
and
they
wanted
to
get
primitive
data
into
solar
into
chronics.
To
be
able
to
do
deeper
analysis
on
the
like
anomaly,
detection
and
so
on.
A
This
only
supports
the
right
path
for
now,
though,
so
there's
this
in
gesture,
which
basically
takes
the
generic
right
format,
samples
chunks
it
up
and
stores
it
in
a
chronic
specific
chunk
format
in
chronic
there's,
no
read
path
there.
What's
more
interesting,
maybe
is
the
more
recent
development
I've
been
working
with.
We've
works
on
cortex,
and
this
is
a
system
I'm
going
to
talk
a
bit
more
about
cortex,
supports
full,
read
and
write.
It's
a
horizontally
scalable
Prometheus
re-implemented
in
the
cloud,
and
it's
like
API
compatible.
A
You
can
use
it
through
weaves
cloud
service,
but
at
the
back
end
of
it
is
also
open
source.
So
you
can
run
it
yourself
and
you
can
use
it
and
contribute.
There's
the
link
down
there.
It
stores
all
that
data
currently
in
AWS,
DynamoDB
and
s3,
but
there's
another
company
also
being
interested
in
it
at
the
moment
and
planning
to
extend
it
for
a
Google
compute
platform
support.
A
A
It
was
one
of
the
first
real
user
out
there
also
for
the
right
support,
because
the
way
it
works
is
you
still
run
a
Prometheus
locally
in
your
data
center.
That
can
do
the
service
discovery
and
pulling
of
targets,
because
you
know
that's
kind
of
the
tough
bit.
How
do
you
solve
that
when
you
want
to
have
a
hosted
Prometheus
and
then
it
uses
Prometheus's
remote,
write
protocol
to
send
samples
to
to
cortex
cortex
aces,
all
and
just
as
a
kind
of
nice
benefit
I
recently
added
the
remote
read
support
to
cortical
as
well.
A
So
if
you
want
to,
you
can
now
also
read
back
data
that
you
send
to
cortex
through
the
premises
towards
your
Center.
So
let's
do
a
quick
demo
of
that.
I
hope
that
works
so
I'm
on
the
development
environment
here
of
weave
cloud
I'm,
going
to
the
monitor
section
here
and
I
hope
it
I
hope
the
Wi-Fi
is
with
me
today.
The
yeah.
This
might
be
a
bit
difficult
like
sending
actual
samples
over
the
Wi-Fi
to
the
service,
but
just
load
is
a
second
ago.
C
A
So
you
can
take
pictures
and
stuff,
but
I'll
do
leave
it
after
it
and
yeah.
So,
basically,
all
the
straw
meters
does
is
it
scrapes
itself
and
it
scrapes
a
local
node
exporter
which
is
running
in
this
other
tab,
to
gather
some
nodes
hosts
that
not
a
lot
of
samples
and,
as
you
see,
there's
a
push
URL
and
reach
URL
and
yeah.
So
let's
try
that
out.
Let's
start
running
that
Prometheus,
it
will
log
some
errors
because
of
the
network
here
and
some
other
issues
against
the
deaf
environment.
A
But
hopefully
some
samples
will
make
it
there
Doh
doh-doh
yeah.
So
you
see
like
some
recharging
messages
here.
So,
let's
see
at
some
point,
this
should
notice
that
there
are
that
there's
aa
premies
disconnected
instances,
one
jobs
one.
There
should
be
two
jobs
eventually,
but
we
can
already
go
to
the
monitor
section.
A
A
A
Let's
do
one
minute
here
as
well
or
five
minutes,
so
it
will
not
look
exactly
the
same
right
now,
probably
because
of
some
other
texture
and
yeah.
So
now
it's
already
it
queries
its
local
storage,
but
it
also
creates
the
remote
storage,
which
of
course
now
is
super
slow
over
the
Wi-Fi,
because
the
primitive
runs
on
my
laptop.
But
now
the
cool
thing
is
see:
let's
refresh
this
one
more
time
did
it
turn
query
running
yep
cool
so.
A
A
I'm
glad
that
worked
okay,
awesome
and,
of
course,
what
we
also
so
go:
go
forth
a
tryout
with
cloud
users
give
feedback
and
but
also
we
are
hoping
for
people
to
build
their
own
right.
Now,
it's
not
documented.
Yet
this
is
all
kind
of
like
the
remote
reach,
especially
has
just
been
added
to
master.
It
will
probably
be
in
the
next
release
and
there
will
also
be
documentation
on
prometheus
IO,
hopefully
yeah,
together
with
the
next
release.
A
Right
now,
I
can
only
point
you
at
the
protocol
buffer
definition
that
includes
the
actual
data
types
that
are
sent
over
the
wire
there's.
Some
snappy
compression
involved
as
well
that'll,
be
documented
yeah,
so
just
yeah
we're
curious
to
see
what
people
do
with
it
and
if
it's
useful,
it's
also
experimental.
A
A
A
In
in
cortex,
oh
yeah,
we
use
actually
the
same
chunk
compression
that
we
use
in
Prometheus,
but
the
system
around
it
is
completely
different.
It's
like
replicated
scalable
horizontally
and
so
on,
but
yet
actual
like
the
very
lowest
level,
we
have
primitive
style
chunks,
but
they're
kind
of
we
can
swap
them
out
if
we
need
to
so
yeah
Tom
back.
There
is
the
master
mind
the
one
in
the
blue
shirt
he's
a
master,
won't
be
behind
leaf
cortex
so
like.
If
you
have
weave
cortex
specific
questions
around
how
that
works.
E
B
B
Rather,
cool
server
use
what
we
have
to
close
and
make
sure
all
will
reveal
and
conclude
with
results
of
an
excuse
ability,
because
I
apologize,
Madoff,
ten
hundred
staff
importance
for
that
location
as
well
Brittany's
like
charting,
headed
smooth
person
and
would
start
the
insane
look.
These
will
be
bold
and
full,
and
all
these
are
also
be
able
to
evolve.
As
don't
you
see.
A
Yeah
and
in
general
you
ask
the
question:
will:
is
it
worth
at
all?
Having
remote
read
and
yeah
I
mean
at
least
there's
a
lot
of
user
demand
for
it.
So
far,
people
have
said
well,
yeah
great.
We
can
write
data
into
some
remote
system,
but
we
want
to
be
able
to
query
back
through
chrome
QL,
because
otherwise,
now
we
have
to
go
to
a
different
system
to
read
it
back
out
and
we
can't
use
our
normal
graphing,
a
dashboards
and
so
on.