►
Description
Chronix as Long Term Storage for Prometheus - Moritz Kammerer, QAware GmbH
"Prometheus is great when it comes to monitoring and alerting. But the long term storage opportunities are comparatively weak compared to related time series databases (missing data distribution, sharding etc.). At this point Chronix [1] enters the stage. Chronix is an open source time series database. It focuses on an efficient long term storage both in terms of storage volume and access times. Chronix achieves a compression rate of 98% compared to data in CSV files while an average query took 21 milliseconds, determined in a benchmark asking 96 queries for different time ranges and time series. Chronix offers a multi-dimensional generic data model for storing all kinds of time series, functions for anomaly detection used in the frameworks EGADS [2] and SAX [3], and an integration with Apache Spark [4] allows for distributed time series processing.
In this code-intense session we show the integration of Prometheus and Chronix. We also dig into the details of Chronix and explain why Chronix loves Prometheus and vice versa. Furthermore we demonstrate a toolchain: collect data with Prometheus, pipe them to Chronix, visualize both data sources in Grafana [5], and easily analyze tons of data with Spark and Apache Zeppelin [6].
[1] www.chronix.io
[2] https://github.com/yahoo/egads
[3] https://github.com/jMotif/SAX
[4] http://spark.apache.org/
[5] http://grafana.org/
[6] https://zeppelin.apache.org/"
About
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
So
I
assume
you
never
heard
about
chronics,
but
that's
no
problem
because
I'm
going
to
introduce
it
to
you
and
that's
the
current
state
of
the
art
for
monitoring
cloud
native
applications.
You
have
your
running
cloud
native
applications
and
use
Prometheus
to
pull
the
matrix
in
there
and
then
you
can
use
it
for
real-time
monitoring
and
alerting,
and
the
default
retention
day
is
about
14
days
in
prometheus.
After
that
the
data
gets
discarded.
A
A
Those
metrics
and
we'd
use
the
default
retention
day
of
14
days
to
store
the
data,
and
it
also
pushes
the
data
into
your
other
database
and
that's
where
chronics
come
into
play
because
chronics,
it's
a
lossless,
long
term
storage
and
it
can
store
your
data
literally
forever
and
it's
as
long
as
you
give
it
enough
hard
disk
space.
It
can
store
it.
We
have
we.
Let
us
see
the
storage
requirements
for
that.
A
You'll
see
a
graph
of
a
running
a
system
here
and
there
are
some
hiccups
in
there
and
now
imagine
you
don't
want
to
analyze
just
the
last
14
days
but
like
a
year
or
something
like
that
to
find
problems
before
that
and
imagine
you're,
not
monitoring.
Just
one
system
or
some
systems
but
you're
monitoring,
your
fleet
of
IOT
devices
or
something
like
that
which
generates
huge
amounts
of
metric
points.
A
So
that's
40
motivation.
Today,
I'm
gonna
talk
a
little
bit
of
electronics,
how
we
build
it
from
what's
in
there
and
the
performance,
how
we
do
the
integration
with
Prometheus
and
then
I
have
a
showcase
for
you,
which
combines
Prometheus
the
chronic
sin
gesture
which
is
needed
for
the
Prometheus
chronics
and
the
graph
on
a
dashboard
which
we
heard
and
the
talk
before.
A
A
The
ingestion
bridge
is,
if
you
have
a
software
which
generates,
for
example,
kairos
DB
metrics.
You
can
point
that
des
oft
where
to
the
ingestion
bridge
endpoint
and
that
will
make
the
translation
between
Kairos
TV
formats
into
chronics
format.
So
I
can
just
point
it
anywhere.
It's
also
supports
open
tiers
to
be
in
flux,
to
be
Cairo's,
graphite
or
the
native
Prometheus
format.
That's
the
Prometheus
text
format,
not
the
proto
buffers
one.
A
Then
the
chronic
score,
it
consists
of
three
systems.
They
are
not
panel
together,
you
can
use
them
at
their
own,
for
example,
chronic
storage.
It's
based
on
leucine
leucine,
is
the
job
of
framework
for
indexing
text
data,
something
like
that:
the
chronic
storage,
for
example.
If
you
don't
need
a
big
cluster
server
system,
but
want
to
run
it
embed
it,
you
can
use
the
chronic
storage
part.
A
The
chronic
server
part
is
based
on
Apache
Solr,
which
is
a
no
SQL
data
store
which,
which
you
can
scale
on
multiple
clusters,
high
availability,
fellow,
but
something
that
the
chronic
server
Intelli
uses
the
chronic
storage
format,
but
also
utilizes
solar
to
store
the
data.
And
on
top
of
that,
if
you
have
really,
if
you
have
many
data,
you
can
also
use
the
SPARC
integration.
A
Sparc
is
to
massively
multi
analysis
of
of
tunneling
solar,
and
on
top
of
that,
you
can
use
Apache
Zeppelin,
for
example,
with
in
combination
with
spark
or
you
can
use
the
chronic
analytics
suit,
which
is
ongoing
work
or
you
can
use
the
graph
on
a
dashboard
and
today
I'm
going
to
show
you
the
ingestion
bridge
how
to
get
data
from
Prometheus
into
chronics.
We
also
take
a
look
at
the
chronic
server
part
and
we
take
a
look
at
the
graph
owner
in
the
showcase.
A
A
My
second
definition
is:
what
is
a
time
series
and
a
time
series
is
an
arbitrary
list
of
chronological
ordered
samples.
So
the
time
series
consists
of
multiple
of
those
short
definition
is
what's
a
chunk
and
the
chunk
is
just
a
part
of
a
time
series,
so
you
take
that
split
it,
and
then
you
get
multiple
chunks
and
my
last
definition,
Flipboard
is
a
little
bit
in
a
way,
a
time
series
database.
What's
the
time
series
data
base,
it's
a
specialized
database,
which
is
which
is
specialized
for
storing
and
retrieving
time
series
and
efficient
and
optimized?
A
A
The
next
step
is,
it
creates
chunks
from
that
68
billion
points
because
to
store
68
building
points
in
a
NOAA
scale.
Data
base
isn't
such
a
great
idea.
If
you
get
68
billion
documents,
that's
too
much
for
them
to
handle,
so
you
create
chunks.
For
example,
we
created
1
million
chunks,
which
all
consists
of
68,000
points
and
the
chunks
have
attributes,
for
example,
from
which
house
they
came,
which
metrics
taste
or
something
like
that
and
the
third
step.
A
It
applies
basic
compression,
so
we
take
the
chunk
and
we
just
compress
it
and
because
it's
timestamp
value
pairs
it
achieves
a
very
high
compression
rate.
In
that
case,
69
percent
compression.
We
see
some
details
later
and
then
you
take
a
multi-dimensional
storage
and
store
them
in
our
example.
If
you
use
the
chronic
server,
it
uses
solar
to
store
them.
A
How's
the
record
look
like
here:
you
have
a
your
compressed
chunk.
A
chunk
consists
of
a
time
series
which
is
a
timestamp
and
a
value,
and
that
case
in
America
come
later
that
Fidel
or
you
can
also
store
traces,
for
example,
a
time
stamp
and
in
an
exception
or
you
can
store
locks,
which
is
the
time
stamp,
and
your
lock
data
or
any
data
you
want
chronics,
isn't
interested
in
what
your
store
in
it.
A
Then
you
have
some
technical
fields
and
you'll
need
them
for
solo,
for
example,
a
document
ID
and
a
version
for
optimistic
locking,
that's
a
start
and
an
end
because
you
store
chance
in
it.
That's
the
start
point
of
the
first
time
series
point
in
the
chunk
and
that's
the
one
of
the
last
and
then
you
have
optional
attributes.
For
example,
from
which
hosted
data
came
from
rich
process,
the
data
came
to
which
group
it
belongs,
which
metrics
is
stored
and
what's
the
max
value,
it's
the
pre
computed
max
value
of
that
chunk.
A
A
Now
only
storing
is
a
bit
boring,
so
chronics
also
useless
aggregations.
For
example,
you
can
cure
it
for
the
min
value
of
the
metrics
for
the
max
value
of
the
metrics.
We
have
an
average,
you
can
sum
them,
you
can
count
them.
You
can
calculate
percentiles
and
the
deviations
first
of
all
you
the
last
value
or
scientific,
and
then
it
also
supports
transformations
which
is,
for
example,
the
bottom
five
values
or
the
top
five
values
of
dead
metrics.
You
can
calculate
moving
averages
for
about
in
a
five
minute
window.
A
A
It
also
supports
supports
analysis,
which
is
that
trend
analysis
using
a
linear
regression
model
to
determine,
if
your,
for
example,
if
you're,
using
or
if
you're
plotting
your
memory
usage,
if
the
memory
usage
is
is
increasing
over
time.
That
is
your
trend,
analysis
or
you
can
find
an
outlier
analysis,
for
example,
to
find
time
windows
where
the
memory
consumption
is
much
higher
than
other
times.
A
It
will
get
flagged,
so
you
can
see
the
time
windows
where,
where
you
have
more
exceptions
on
your
production
systems,
you
can
you
also
use
fast,
dynamic
time,
warping
warping,
which
is
used
to
compare
time
series
if
you
have
time
series
from
two
different
services,
since
we
want
to
see
if
they
are
in
identical.
If
they
are
behaving
the
same,
you
can
use
that
method
or
you
can
use
two
symbolic
I
could
get
approximation,
which
is
the
if
you
search
for
it
it.
It's
also
called
si
X,
that's
used
for
a
similarity
and
pattern
search.
A
So
I
said
you
can
store
anything
in
chronics.
How
does
that
work?
Here's
the
Java
interface
for
the
time
series
converter,
and
here
you
get
its
it's
a
generic
over
T,
and
here
we
have
a
time
series
chunk,
which
is
the
generic
T
and
it
just
it
has
to
create
a
binary
time
series.
The
binary
time
series
is
that
part
which
is
stored
into
junk
and
that's
the
other
way
around.
A
You
get
the
binary
time
series
you
get
a
query
start
and
a
query
end
that
that's
what
the
user
wants
to
start
an
end,
because
it's
junk
and
then
you
get
your
generic
back,
so
you
can
store
anything
in
chronics
chronics
by
default,
because
that's
the
main
use
case
for
us
has
support
for
numeric
values.
It's
just
here
over
orbital,
so
you
can
put
there
in
a
chunk
of
doubles
or
a
list
of
doubles
and
it
gives
you
the
binary
representation.
But
there
is
more
to
come.
A
Okay,
how
to
run
that
stuff?
Here's
your
computer,
you
need
a
Java
8,
you
don't
load
the
chronics
distribution,
the
chronics
distribution
includes
a
solar
and
the
solar
entirely
uses.
Leucine
and
chronics
is
implemented
a
solar
plugins.
So
you
have
a
crew.
If
you
know
solar,
you
have
a
Curie
Handler.
You
have
the
injection
handler,
which
translates
the
data
from
those
databases
into
chronics.
You
have
a
compaction
handler
which
is
used
to
compact
data
into
chunks
more
on
that
later,
and
you
have
to
retention
handle
off
there.
A
A
A
That's
this
is
the
implementation
of
the
interface
I,
showed
you
two
slides
before
that
converts
the
binary
data
into
a
list
of
Tuttle's,
and
then
you
can
trade,
a
solar
Curie,
which
says
I'm
interested
in
all
metrics,
which
contains
load,
and
then
you
call
the
stream
method
and
you
get
a
Java
eight
stream
and
you
can
work
with
the
data
or
you
can
also
add
some
aggregations
or
functions.
That's
all
that
stuff!
A
Here,
for
example,
we
are
interested
in
the
max
value
of
the
metrics
and
the
min
value
of
the
matrix.
How
many
metrics
there
are
and
signed
difference
here.
It's
an
example:
if
your
first
metric
values
20
and
your
last
metric
value
is
minus
80,
you
will
get
minus
80,
so
it
falls
over
time
and
then
you
also
call
the
stream
method
and
you
get
a
Java
8
stream.
Yes,.
B
A
Chronic
Sisk
created
an
in
cooperation
with
a
German
university
if
you're
interested
in
the
art
fact
that's
from
their
paper
yeah.
If
you're
interested
chunk
just
come
to
me
after
the
talk,
we
are
evaluated,
chronics
against
influx
to
be
open,
TCP
and
Kairos
to
be,
and
that's
important
one.
All
databases
are
configured
as
a
single
null
database.
They
are
not
running
cluster
they're
running
on
one
machine,
yeah.
A
If
you
take
at
the
query
times
and
the
paper,
a
Curie
mix
test,
if
is,
was
defined
about
usual
curious,
which
are
run
against
time,
series
databases
and
in
this
query,
mix
Cronus's
and
the
bottom
tier
70
is
three
percent
to
a
ninety
time.
92
percent
faster
on
data
retrieval,
and
it's
in
the
query
mixed
case
82
97
percent
faster.
If
you
want
to
query
data
out
of
chronics,
if
you
take
the
storage
demand
for
you--even,
108
gigabyte
rossi
use
full
file
and
you
want
to
import
it
in
the
database
and
in
chronics.
A
A
Okay,
now
I've
talked
about
chronics.
What
about
Prometheus
so
to
get
it
work
Prometheus.
It
collects
the
metrics
from
your
service,
it
writes
them
to
the
default
storage
and
it
also
writes
them
in
our
configuration
through
the
standard,
remote
write,
interface
to
electronics
and
Jasta,
the
chronics
and
gesture
which
sits
in
the
middle
between
chronics
and
your
Prometheus,
and
it's
used
to
collect
the
samples
from
the
Prometheus
and
write
them
into
Prometheus
in
batch
mode.
A
A
Now,
some
scenarios-
this
is
running
Prometheus
using
chronics
on
a
single
host,
use
the
Prometheus
to
get
your
metrics
from
your
systems.
It
writes
them
it's
right
to
samples
using
your
remote
right
in
the
face
in
the
chronics
and
gesture
this
some
batching
and
writes
the
rights
the
batches
into
the
chronic
server.
A
A
A
Here
you
can
see
a
lopa
Lansing
example.
You
have
multiple
Prometheus,
you
all
point
them
to
an
engineer.
Slope
will
answer
in
balances
them
over
to
chronics
and
justice.
They
do
the
batching
and
they
write
it
through
the
chronics
server
or
the
full
cloud
mode.
You
use
Prometheus
type
them
into
nginx
load
balancer,
which
shot
which
balance
them
over
multiple
chronic
system.
You
write
the
data
to
a
chronic
smoker
or
a
solar
Master
and
solar
duster
Sharleen
and
distributes
them
over
multiple
chronic
service.
A
A
It
sits
between
your
Prometheus
and
your
chronic
server
I'm
using
a
remote
write
interface
and
you
just
configure
your
host
into
port
and
ingest
or
and
Prometheus
remote
write
off
the
rest.
The
main
feature
is
it
batches
to
samples
in
memory?
So
here
you
get
samples
outcomes
batches
for
the
chronic
server,
because
chronics
needs
needs
large
chunks
to
work
well,
and
you
can
also
configure
the
max
patch
age.
It's
question
restart
with
resilience,
for
example
your
piping
samples
in
there
and
any
crashes,
and
that
hasn't
read
all
the
batch.
A
A
A
So
that's
the
reason
why
Prometheus
and
cryonics
are
really
good
to
work
with
each
other,
because
Prometheus
uses
so-called
labels
to
store
the
management
values
and
if
you
compare
it
to
chronic
that
uses
the
attributes
which
are
also
key
value
pairs
to
store
dimension
values
so
that
the
only
difference
is
is
called
labels
and
it's
called
attributes.
It's
the
same
feature.
The
the
labels
are
edited
dynamically
and
in
chronics
a
debate
is
based
on
solar.
A
You
can
use
a
schema,
but
you
can
also
use
the
schema
less
mode,
which
adds
dynamic
fields,
so
they
are
also
added
dynamically
and
Prometheus
store
sample
pairs
for
a
pair
of
a
timestamp
at
the
world.
You-
and
this
is
also
supported
in
chronics.
It's
puzzles
stores
a
timestamp
NW,
and
that's
how
that's
why
the
chronics
and
gesture
is
so
simple.
It
just
takes
the
data
it
matches,
it
writes
a
dog
it
has,
it
doesn't
have
to
do
some
transformation
or
something
like
that.
A
That's
a
chronic
schema
that
some
solar
internals,
it's
not
very
important
here.
You
see
the
fields,
for
example
the
ID
field,
the
star
field,
the
unfilled,
the
data
field,
which
is
binary,
and
here
you
see
a
dynamic
field
which
is
used
to
store
the
text.
So
if
you
store
the
Prometheus
label,
it
gets
translated
into
a
chronic
level.
For
example,
if
you
use
the
premises
label
host,
it
gets
the
dynamic
fields
host
or
underscore
as
for
a
dynamic
field,
and
then
you
can
query
for
that.
A
A
A
few
words
about
our
performance.
It's
11
days
of
data
in
the
slides
in
my
life
presentation.
It's
it's
more
data,
but
here
they
were
12,
mil
112
million
samples
in
Prometheus,
that's
about
786
megabyte
on
disk
and
in
chronics
it's
265
megabyte
on
disk
and
without
compaction.
I
mentioned
compaction
earlier
compaction
works
by
you
specify
how
many
points
you
want
per
chunk
and
chronics
takes
those
data
points
and
chunks
them,
and
here
you
can
see
how
that
has
an
effect
on
your
disk
usage
size.
That's
the
default.
A
It
just
likes
100
data
points
but
shrunk
and
then
cuts
the
chunk,
and
here
you
can
see
if
you
run
100
points
per
charge.
You
end
up
with
357
megabytes
and
if
you
run
500
thousands
points
and
junk,
you
all
knew
just
get
119
megabytes,
because
the
compression
is
more
efficient.
The
caveat
is,
if
you
store
500,000
data
points
per
chunk.
Your
query
time
gets
higher
because
it
has
to
unpack
500,000
data
points.
If
you
corrode
your
data.
C
B
A
A
C
B
B
C
B
B
A
No
I'm
not
sure
no,
no,
no,
they
are
not
at
the
moment.
There
are
not
options.
They're
hard-coded
in
the
software
in
the
research
paper
I
showed
earlier.
They
are,
they
run
multiple
benchmarks
with
different
compression
algorithms,
different
compression
settings
and
they
end
up
with
gzip
and
the
block
size
of
I.
Don't
know
you
have
to
read
the
paper,
but
they
are
hard-coded
at
the
moment.
A
So
that
the
junking-
and
in
that
case
we
disabled
to
Chungking
or
we
disabled
to
compaction
of
chunking
and
that's
a
screenshot
from
our
software
kaki
tool,
which
I
showed
earlier-
and
here
you
can
see
the
red
lines-
are
the
CPU
usage
for
the
chronic
Sanjay.
Sir.
The
green
lines
are
for
Prometheus
itself,
the
CPU
usage
it's
max
at
400%
because
we
have
for
cost
and
the
blue
one
is
the
chronic
server.
A
A
That's
the
default
solar
Korean
face
and
if
you
run
a
query
against
it,
we
can
see
it
found
1
million
and
something
solar
documents.
And
here
you
can
see
the
fields
I
mentioned
earlier,
for
example
the
ID
the
start
date.
The
end
date
and
here
is,
for
example,
a
Prometheus
tag
called
tender
and
it's
it
is
stored
in
the
handler
underscore
s
a
field
of
that
solar
document,
and
here
is
the
compressed
chunk.
A
A
This
is
our
Prometheus.
It's
running,
showing
the
memory
chunks
of
our
hand,
gesture
and
now
the
interesting
thing
I
have
a
dashboard
which
is
called
pro-mix.
It
combines
the
lower
data
is
coming
from
Prometheus
and
the
upper
data
is
coming
from
chronic
sit.
That's
this
exact
same
matrix
also,
that
is
coming
from
the
Prometheus
data
store,
and
that
is
the
data
stored
in
chronics,
and
here
we
can
see
that's
about
the
14
day.
Mark
Prometheus,
throw
the
data
away
and
in
chronic
state
it's
also
stored
and
also
interesting,
is
where
are
the
spikes?
A
A
C
B
A
And
you
could
configure
that
this
in
gesture
is
also
is
always
writing
in
that
chronic
server,
but
not
based
on
the
metric.
So
all
metrics
from
that
and
gesture
will
get
in
that
chronic
server.
If
you
point
it
directly
to
that
chronic
server,
if
you
point
it
to
the
master
node,
it
shards
on
your
folding
policy,
for
example
the
ID
field
or
the
metrics
field,
or
something
that.
A
D
A
And
that's
one
big
use
case
because
we
are
not
getting
these
metrics
with
Prometheus,
but
with
an
self-written
collector,
which
rights
use
file
files
and
we
import
them.
This
year's
file
files
into
our
chronic.
So
that's
all
data
from
the
past
and
chronics
sort
salt
ordering,
and
then
you
get
it
in
the
right.
Chronological
order.