►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Cortex - Infinitely Scalable Prometheus - Bryan Boreham, Weaveworks
Prometheus is an excellent monitoring solution, however it is designed to store all metrics on one host in one set of files, which limits scalability and availability. This talk provides an overview of Cortex, an Open Source project proposed for donation to the CNCF, which takes metrics from multiple Prometheus instances and stores them in a NoSQL database. Outline: - Microservices architecture of Cortex - NoSQL back-end (DynamoDB, Bigtable or Cassandra) - Fault-tolerance via DHT-based ingestion tier and stateless query tier - Deployment via Kubernetes - Architectural comparison to projects in the same area, such as Thanos and M3 - Lessons learned from operating “Prometheus as a service” for two years
To learn more: https://sched.co/GrXL
A
A
There's
no
way
it
can
be
infinitely
scalable,
so
welcome,
and
then,
when
I
was
thinking
about
when
I
was
thinking
about
the
name
and
and
kind
of
running
it
around
in
my
head
and
and
somehow
it
turned
into
infinity
war.
You
know
which
is
was
a
was
a
film
that
was
out
recently
and
it's
kind
of
a
dumb
film.
They
go.
They
go
chasing
around
after
these
stones,
Infinity
stones,
so
I'm,
not
gonna,
mention
any
of
that
again.
A
So
cortex
is
a
time-series
store
built
on
promises
who
uses
Prometheus
and
I
put
my
own
hand
up,
because
we
also
do
use
Prometheus
it
is.
It
is
built
on
prometheus.
We
we
have.
We
huge.
We
use
the
code
very
much
built
on
it.
Great
benefit
of
the
old
source
community
and
the
key
attributes
of
cortex
are
that
it
is
horizontally.
Scalable
highly
available,
provides
long-term
storage
and
is
a
multi-tenant
system,
so
I'm
gonna
say
a
lot
more
about
all
of
those
things
over
the
course
of
this
talk.
A
Cortex
is
a
CN
CF
project.
We
we
started
at
we've
works.
We
turned
over
the
IP
to
the
CN
CF,
it's
Apache
licensed
you
can
the
source
code
from
that
URL
at
the
bottom
of
the
page.
Court
reject
slash
cortex,
okay.
Why
did
we
build
cortex?
Well,
we
run
we've
works.
We
run
this
software
as
a
service
monitoring
as
reserve
ability
and
continuous
deployment
platform
we've
cloud,
and
we
wanted
to
do
things
like
like
dashboards.
We
draw
these
automatic
magic
dashboards
and
we
wanted
to
do
that
as
a
as
a
sign
up.
A
A
This
is
the
weather,
the
minds
behind
cortex.
This
is
a
screen
dump
from
github,
so
it
started
June
2016
two
and
a
half
years.
Ish
we've
been
running
it
in
production
about
two
years,
probably
the
number
one
committer
Tom
Wilkie
in
the
audience.
We
got
a
few
of
the
maintainer
of
cortex.
Here
we
got
Ken
here
myself
and
there
there
are
people
on
this
list
who
are
also
maintainer
of
Prometheus.
So
again,
I
want
to
stress
a
lot
of
lot
of
sharing
and
collaboration
there.
A
A
Anyway,
so
yeah
the
the
community,
the
minds
behind
cortex,
the
other
side
of
the
community
who
uses
cortex
it
is,
it
has
been
adopted
by
a
number
of
people
running
platforms.
We've
worked
obviously
profound
labs,
Aspen,
mash
and
platform.
9
are
here
at
cube,
Cohen
and
as
our
Maya
data
neaby
s,
they're
part
of
the
people
on
the
right
are
some
of
the
users
of
some
of
the
platforms
on
the
left
and
and
I
put
this.
So
this
is
the
sole
of
the
pro
is
it
is
they
were
liking.
A
A
But
just
to
to
cover
the
basics:
Prometheus
runs
as
one
process
that
that's
one
of
the
key
things
very
simple,
just
run
it,
and-
and
you
pointed
at
things
that
have
metrics,
which
might
be
your
own
apps
or
it
might
be
some
kind
of
exporter
that
gets
data
from
somewhere
else.
We
call
this
great
thing:
it
pulls
the
metrics
in
it,
compresses
them
stores
them
on
disk,
and
it
also
serves
up
queries
which,
which
come
from
some
kind
of
dashboard
viewer.
Something
like
that.
A
What
are
you
going
to
do?
Well,
you
can
just
get
a
bigger
box.
Prometheus
scales.
Really
well
you
can.
You
can
certainly
serve
millions
of
time
series
from
one
large
Prometheus
box,
very
effective,
very
good,
very
well
engineered,
but
but
some
people
then
find
that
that
tops
out
it
still
not
an
you
know,
can't
get
a
box
big
enough
or
don't
don't.
Maybe
you
want
that
much
risk
in
one
place,
so
the
next
place
that
people
go
is
to
split
up
the
problem.
So
this
point
you
have
to
kind
of
pick.
A
You
know
your
manually
sharding
the
data
you,
whether
you
slice
it
by
apps
versus
infrastructure
or
you
slice
it
by
team,
a
versus
team
B
or
you
slice
it
by
data
center.
Or
are
you
slice
it
you
you?
You
are
gonna
have
to
pick,
and
then
that
leads
to
a
problem
on
in
terms
of
viewing
the
data
that
how
do
you
build
one
chart
that
brings
together
data?
A
Instead
of
one
process,
we
run
many
processes.
It
is
it's
what
Tom
calls
on
micro
services,
oriented
architecture
flame
me
about
that
one.
So
this
this
is
the
this
is
the
cortex
architecture.
We
we
actually
use
standard
Prometheus
to
do
the
scraping
the
the
operation
of
pulling
in
the
individual
metrics
from
the
endpoints.
A
A
So
the
the
thing
that
processes
time
series
samples
coming
in
on
ingest,
we
call
an
in
gesture
so
yeah
at
the
point
of
my
infinitely
scalable
headline.
Is
it
sits
horizontally
scalable?
If
you,
if
you
need
more
power,
you
add
more
in
gestures,
we
route
through
a
process
called
the
distributor.
We
route
every
time
series
2-1
in
gesture.
A
A
Actually,
I
want
to
stress
core
Texas
is
built
on,
unlike
decades
of
work
and
research,
that
other
people
did
very,
very
well
born
techniques
for
distributed
databases.
Basically,
so
III,
don't
wanna
I,
don't
want
to
claim
any
of
the
insight
or
ingenuity
in
in
in
taking
those
ideas.
These
are
these
are
pretty
standard
ideas,
but
they
they're
also
pretty
good
ideas.
A
So
the
the
great
thing
about
a
DHT
is
there's
no
bottleneck
in
terms
of
coordinating
you
can
you
can
keep
on
adding
in
jesters
and
they
will
be
inserted
into
the
DHT
and
we
will
route
time
series
2-1
in
jester
and
they
they
don't
have
to.
You
know,
communicate
every
single
time
to
one
central
point
which
is
is
going
to
limit
scalability,
so
distributor
working
through
the
DHT
gives
us
for
this
piece
gives
us
an
infinite
scalability.
A
Let's
look
inside
one
of
those
in
jesters,
so
the
the
samples
come
in
each
wiggly
line
here
is
is
a
time
series
actually
in
reality,
most
time
series
are
zero
continuously,
it's
it's
kind
of
dull.
When
you
look
at
it,
but
I've
imagined
that
all
of
my
time
series
are
much
more
exciting
here,
so
so
the
the
the
wiggly
lines
come
in.
They
build
up
at
different
rates
and
that
they're
compressed
in
memory.
The
the
compression
technique
that
cortex
uses
and
basically
every
time
series
database
I
know
about
uses,
is
called
gorilla.
A
Compression
came
from
Facebook
link
to
the
paper
there,
so
we
compress
it.
We
get
this
massive
compression
of
the
data
and
when
we
reach
a
certain
level
which
is
well
originally,
it
was
1k.
Recently,
we've
brought
that
up
to
sort
of
four
or
five
K,
typically
a
few
hours
of
data.
In
one
chunk,
we
we
stick
the
chunks
in
a
store.
So
this
is
the
power
of
the
Infinity
stone.
The
space
stone
is
compressing
our
data.
A
A
What
do
we
want
from
that?
We
want
scalability,
because
we
were
infinitely
scalable.
That's
why
you
came.
We
want
speed,
because
we
don't
want
to
wait
any
time
at
all
for
our
dashboards
to
draw
the
third
thing.
You
might
not
be
so
immediately
aware
of
durability.
What
do
I
mean
by
this?
So
so
you
can
put
something
on
a
disk
and
if
you're
going
to
store
it
for
years
or
months
or
whatever
you
gonna
store
a
lot
of
it.
We
have
hundreds
of
terabytes
of
this
stuff.
A
Then
your
disk
is
going
to
fail.
So
you
need
to
store
multiple
copies
on
disk,
to
save
yourself
when
one
of
those
copies
fails
and
then
when
one
of
them
inevitably
does
fail,
you
need
to
make
a
new
copy
from
one
of
the
other
copies
that
you
were
hanging
on
to
right.
So
that's
that's
a
kind
of
anti
entropy
function.
They
call
it
in
order
to
have
durability.
A
So
cortex
can
speak
to
a
variety
of
back-end
stores,
Cassandra
very
well
known
one
dynamodb
or
s3
from
Amazon
BigTable,
Google
cloud
store,
so
there's
two
kinds
of
store
on
there:
the
the
table
oriented
ones
like
DynamoDB,
are
used
for
the
index
and
the
the
sort
of
bucket
oriented
ones
generally
used
for
the
the
chunks,
the
the
the
actual
time
series
chunks
that
we've
stored
from
the
in
gesture.
Although
that
that
isn't
an
an
iron
rule,
we
we
do
also
have
the
capability
to
store
chunks
in
DynamoDB
and
in
BigTable
and
in
Cassandra.
A
So
anyway,
there's
there's
actually
more
backends.
Coming
it's
quite
a
nice
kind
of
adapter
pattern,
but
yeah
we
do.
We
do
rely
on
someone
else
to
do
this
work
of
making
the
data
durable
and
also
also
running
index,
lookups
at
enormous
speed
at
enormous
scale.
So
look
a
little
bit
more
into
the
detail
of
the
index.
A
What
are
we
mean
by
this?
We
so
I've
got
an
example
there
of
a
Prometheus
metric.
So
every
metric
has
a
name
HTTP
duration
seconds
in
this
case,
and
it
has
a
set
of
labels
and
each
label
has
a
value,
so
job
equals
shipping
instance
equals
a
path
equals
foo
result
equals
200.
The
average
tends
to
be
about
10-12
labels
parametric
and
the
the
concept
of
in
inverted
index
is
we
we
break
all
of
those
values
out
of
that
role
and
we
we
store
them
separately.
A
We
store
a
list
against
those,
so
so
we
kind
of
have
an
index
for
each
label
against
each
metric,
and
we
have
all
the
possible
values
pulled
out
there
and
against
those
we
have
all
the
time
series.
This
is
basically
the
way
that
Google
indexes
the
web
right.
The
the
you
know
that
the
for
a
word
that
you're
searching
for
cortex
they
have
a
list
of
all
the
web
pages
that
mention
that
word.
A
A
What
I
do
is
I
go
to
that
row
in
the
index.
First
of
all,
for
the
the
job
label
look
up
shipping
that
gets
me
a
set
of
time
series
for
each
time:
series
I,
look
it
up
for
the
right
time,
boundary
and
I'm
the
last
hour
or
the
last
week
or
whatever
that
gets
me
a
set
of
those
chunks,
I
bring
the
chunks
in
and
unpack
those
into
the
individual
samples.
A
A
A
One
little
wrinkle,
the
you
know,
I
mentioned
that
the
the
samples
are
compressed
in
memory,
they're
kind
of
chunked
up
to
a
few
hours
in
memory.
So
we
also
need
to
go
to
the
in
jesters
to
get
the
that
last.
You
know.
Last
couple
of
hours
of
data
is
going
to
be
in
memory.
We
need
to
check
there
as
well
as
in
the
store,
and
then
we
do
like
everything
to
go
fast.
So
we
put
a
cache
on
there.
We
do
a
lot
of
caching,
we
cache
index
lookups.
We
cache
chunk
lookups.
A
We
cache
partial
results
with
cache
lot
of
things
and
and
to
stress
the
the
the
micro
service
nature
of
this.
The
the
fact
that
we've
broken
it
up
into
pieces
lets
us.
Maybe
maybe
this
is
a
place
where
we
kind
of
innovate
a
bit.
We
get
creative
and
add
more
kinds
of
queriers
and
more
kinds
of
caches,
and
we
can
do
that
fairly
freely
because
we
it's
AA,
split
into
bits.
A
A
A
A
A
A
The
whole
thing
is
multi
tenant
front
to
back,
so
we
we
give
each.
We
call
it
an
instance.
We
you
know
think
in
terms
of
you,
as
a
user
of
the
system
might
have
your
dev
instance,
your
staging
instance,
your
production
instance,
or
you
know
whatever.
However,
you
want
to
think
of
these
things
separately.
A
So
an
instance
is
not
necessarily
one
customer
could
have
multiple
instances,
but
each
one
has
a
key
as
an
ID
that
ID
is
is
carried
through.
The
system,
the
ID
forms
part
of
the
index,
so
we
can
look
up
queries
for
one
instance
at
a
time
and
the
whatever
is
is
on
the
query
side.
You
know
that
we
build
dashboards.
You
can
use
something
like
graph
Anna,
because
cortex
speaks
the
exact
Prometheus
API
that
that
Prometheus
does
so
anything
that
will
talk
to
Prometheus
can
be
used
on
the
on
the
query
side.
A
A
What
is
interesting
is
it's
kind
of
edging
I
mean
you
can
tell
from
the
from
the
picture
here
they
it's
edging
up
in
complexity.
They
certainly
have
a
more
components
than
they
started
with,
but
I
think
what
is
good
is
to
look
at
the
similarities.
You
know
its
they're.
Both
both
projects
have
a
huge
reuse
of
prometheus
code.
A
A
They
both
have
this
split
between
recent
data
in
in
the
Thanos
case.
The
recent
data
is
just
in
the
Prometheus
and
and
Prometheus
writes
it
to
disk.
They
take
those
disk
blocks
and
ship
them
off
to
long-term
storage
like
like
an
s3
same
kind
of
thing,
but
there
is
this
split
same-same
in
court,
excellent,
arrows,
there's,
there's
cloud
buckets
like
s3
and,
as
I
said,
they
do
have
multiple
components
in
their
architecture.
A
So
yeah
a
lot
of
lot
of
similarities
between
the
two
projects.
I'll
just
point
out,
the
big
difference
is
multi-tenant
right.
Sex
is
a
multi-tenant
system.
We
needed
that
we
were
building
a
cloud
prometheus
as
a
service
and
we
didn't
want
to.
We
didn't
want
to
kind
of
install
a
new
Prometheus.
Every
time
a
customer
turned
up
so
cortex
is,
is
multi-tenant
front
to
back
cortex.
Does
this
automatic
sharding,
based
on
the
the
distributed
hash
table?
A
Thanassis
is
kind
of
starting
from
the
standpoint
you
you've
mark
manually,
divided
up
your
you.
You
have
these
Prometheus
running
separately
to
begin
with
differences
in
the
storage
format.
So
cortex
does
what
I
call
query
sharding.
So
if
you
do
a
30-day
query
will
actually
run
31-day
queries
in
parallel
to
speed
that
up.
That's
that's
a
feature
that
US
doesn't
have
Fallas
does
down
sampling,
so
they
can
take
your
old
or
data.
Maybe
you've
sampled
it
at
a
five-second
interval
and
they
can
down
sample
it
to
a
one-minute
interval.
A
I
said
on
the
description
of
of
the
talk
that
I
talk
about
our
experiences
in
production.
We
have
been
running
it
nearly
two
years,
so
one
of
the
things
is
that
that
anyone
can
sign
up.
You
know
if
you
show
up
you
get
a
free
trial,
so
I
knew
anyone
can
kind
of
I.
Imagine
some
of
you
are
in
this
room,
pointing
your
notebooks
at
my
service
and
and
trying
to
break
it
so
yeah.
A
A
A
This
is
where
I'm
talking
about
something
like
big
table
or
or
DynamoDB,
Amazon
and
Google,
and
so
on.
They
run
that
at
a
huge
scale,
so
so
on
on
our
side
on
the
code,
we
want
to
talk
to
it
as
many
times
in
parallel
as
possible.
However,
on
the
flip
side,
you
need
to
batch
up
your
your
requests.
Otherwise,
you,
the
power,
call
overhead,
is
very
highs
and
those
things
are
pulling
in
opposite
directions,
so
kind
of
kind
of
tuning
and
tweaking
that
is
a
lot
of
the
work.
A
That's
gone
into,
cortex
also
tuning
the
schema
hot
spots
of
you.
If
you
write
to
the
same
place
in
the
in
while
in
terms
of
the
underlying
data
store,
if
you
write
to
the
same
place
too
often
at
the
same
time,
it'll
choke
we're
sharin
Version
nine
of
the
schema
of
exactly
how
we
do
that
and
and
hilariously
we
have
to
maintain
all
the
code
for
all
the
previous
eight
schemas
is
all
still
in
there,
because
we,
you
know,
we
actually
used
this
thing
in
production.
We
still
still
reading
some
of
that
data.
A
A
So
it
says
in
the
Prometheus
documentation
that
100,000
should
be
no
problem
and
that's
true
enough.
How
do
you
get
to
those
numbers
that
basically,
by
putting
something
like
your
client
IP
in
as
a
label,
you
go?
You
know
if
you
have
a
million
customers
actually,
even
if
you
only
have
a
thousand
customers
but
you're
running
your
service
on
a
thousand
machines
and
that
multiplies
up,
so
you
have
a
million
time
series
now.
So
this
is
that
kind
of
effect
the
the
labels
multiply
and-
and
we
basically
that's
still
a
challenge
at
some
point.
A
So
that's
that's
the
the
flipside
of
when
I
said
earlier.
I
just
hope
that
all
three
of
them
don't
crash
at
once.
If,
if
we
have
a
right
head
log,
then
that's
less
of
a
danger,
it
is
a
very
complicated
beast
cortex
because
of
all
these
multi
processes,
microservices
pretty
complex
to
run
so
Tom's
been
working
on
a
simpler
config.
For
that,
oh
yeah,
we
run
rules
like
alerts
and
things
that
that
is
not
currently
sharded.
That
needs
to
be
charted
for
scalability.
We
might
add
downsampling,
but
anyway,
I
hope
we
get
more
users.
A
A
B
A
Yeah,
so
that
so
the
I
guess
the
point
you're
making
is
is
that
on
the
scraping
side,
there's
a
single
point
of
failure
and
yeah
in
the
Thanos
project.
The
way
you
deal
with,
that
is
by
running
two
of
them
scrape
everything
twice
and
then
deegeu
and
we
do
not
have
that
feature
in
cortex
I
mean
basically
basically
we're
not
putting
any
we're,
not
we're
not
getting
that
process
to
do
any
work
in
the
in
the
cortex
model.
It's
just
scraping
and
immediately
sending
the
samples.
A
So
so
it's
more
important
that
it's
a
che
in
in
the
cortex
model.
The
the
originating
Prometheus
is
it's
kind
of
ephemeral.
You
know
if,
if
it's
sending
us
samples,
we've
got
them
mm-hmm.
If
it's
got
a
problem,
it's
not
sending
us
samples
so
yeah.
That
is
a
feature
we
could
add,
but
it's
less
of
a
sort
of
critical
point
in
cortex,
one.
B
A
A
C
A
A
So
in
in
DynamoDB,
well
up
until
about
two
weeks
ago,
you
had
you
got
to
pick
a
capacity
level.
You
paid
pre-selected,
the
capacity
level.
So
if
you
select
that
to
load,
then
it'll
just
back
up
in
memory
and
I,
I'm
I'm
sure
I
am
one
of
the
hundred
experts
in
the
world
on
scaling
DynamoDB,
because
I've
done
it
a
thousand
times.
I
are
also
one
of
the
top
100
people
in
the
world.
For
modesty.
Yes,
so
that's
one
reason.
A
Yes,
so
if
you,
for
whatever
reason,
if
you're
running
Cassandra
yourself
you'd
have
the
same
problem,
you've
picked
a
fixed
level
of
capacity
and,
if
you're,
if
you're,
trying
to
flush
chunks
faster
than
that,
so
that
what
is
nice
is
the
in
jesters
can
kind
of
grow
and
shrink
they.
Don't
they
don't
need
to
flush
immediately.
A
Then
it
so
that's
one
reason
just
not
enough
capacity
in
the
backend.
Some
of
that
stuff,
like
short,
live
time
series,
is,
is
a
surprising
problem.
You
know
like
I
mentioned
on
the
slide.
Apache
spark
will
like
run
pods
for,
for
10
seconds,
run
like
a
thousand
pods
for
10
seconds,
and
so
we
we
barely
get
one
Sam.
Actually
what
we
should
probably
do
is
throw
it
away,
because
it's
very
little
you
can
do
with
the
query,
but
but
that
has
been
a
case.
A
We
we
have
limits
in
there.
Now
we
have.
We
have
a
bunch
of
kind
of
soft
limits
in
the
software
to
kind
of
avoid
just
getting
taken
down
by
someone
who
has
an
exciting
kind
of
data,
but
that's
that's
another
another
kind
of
reason.
If
it's
just
a
lot
of
tiny,
tiny
chunks,
then
then
it's
going
to
need
a
lot
more
resource
than
you
were
thinking
of
we're
at
time.
Okay!
Well,
thank
you
very
much.
I'm
here
all
week
come
to
the
weave
Works
booth.