►
From YouTube: Cortex 101: Horizontally Scalable Long Term Storage for Prometheus - Chris Marchbanks, Splunk
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Cortex 101: Horizontally Scalable Long Term Storage for Prometheus - Chris Marchbanks, Splunk
Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for Prometheus metrics, and a horizontally scalable, Prometheus-compatible query API. Cortex allows users to deploy a centralised, globally aggregated view of all their Prometheus instances, storing data indefinitely. In this talk we will discuss Cortex's history, Cortex's architecture, and how to get started with Cortex. Cortex is a CNCF sandbox project.
https://sched.co/UaiH
A
B
A
Am
a
maintainer,
both
Prometheus
and
cortex,
currently
working
at
Splunk
building
internal
observability
platform
for
our
cloud
systems,
I'm
passionate
about
all
things
observability
and
in
my
free
time,
love
getting
outdoors
skiing,
uphill
hiking.
Currently
it's
super
beautiful
back
in
Boulder,
so
doing
some
scrambling
two
days
ago.
I
think
this
is
when
that
picture
was
taken
fun
times,
start
off
with
show
of
hands
this
audience
who
uses
Prometheus.
A
Hopefully
everyone,
if
you're
in
a
cortex,
talk
great
perfect,
who
is
multiple
kubernetes
clusters
or
environments
where
they
deploy
Prometheus
into
also
almost
everyone
so
who
runs
cortex
now
a
handful
sounds
about
right:
that's
what
I
expect
for
an
intro
class
so
start
with
a
little
bit
kubernetes
and
just
Prometheus.
What's
awesome!
First-Class
support,
Prometheus
is
really
awesome
in
kubernetes,
all
of
cabernet
xapi
eyes
are
instrumented
using
it
and
prometheus
has
support
for
community
service
discovery
makes
it
really
powerful
huge
community
of
users.
A
Basically
any
question:
you,
google,
it
and
there's
probably
five
blog
posts
that
show
up
about
how
to
do
this
in
Prometheus
and
kubernetes,
and
the
alerting
and
exploration
of
your
metrics
is
really
great
and
really
powerful.
Without
any
external
dependencies.
You
just
run
a
Prometheus
server.
It
works.
You
can
look
at
it,
you
can
get
alerts
I,
don't
you
don't
have
to
maintain
a
huge,
a
distributed
system
to
do
that,
but
there's
some
struggles.
First,
off
storage
is
limited
to
local
disk.
A
That
can
be
a
pain
that
can
be
painful
sometimes,
and
it's
really
hard
to
query
and
alert
across
multiple
clusters:
multiple
systems.
How
does
this
cluster
compared
to
this
cluster
over
here?
How
much
CPU
am
I
using
across
my
entire
fleet?
Those
are
hard
to
answer.
You
have
to
go
out
to
you
know,
potentially
dozens
of
Prometheus
instances
and
compile
all
this
data
together.
It's
not
great.
So
these
are
some
of
the
issues
that
cortex
looks
to
solve.
A
A
So
here's
some
practical
use
cases
so
that
actually
mean
in
systems
you're
going
to
use
one
long-term
storage
of
all
or
a
subset
of
metrics
cortex
just
uses
Prometheus
remote
right.
You
can
send
it
all
your
metrics,
you
can
send
it
only
the
metrics
you
care
about
whatever
you
feel
like
they're
centralized
system
for
aggregating
data
from
many
isolated
clusters
all
the
times
that
cluster
a
doesn't
have
any
communication
with
the
cluster
B
and
it's
nice
storing
in
this
third
place
somewhere
over
here,
Prometheus
as
a
service.
A
So
that
would
be,
for
example,
at
spawn
cloud.
We
use
cortex
to
provide
Prometheus
as
a
service
to
a
bunch
of
dev
teams.
This
can
be
internal
or
external
uhm,
just
depending
on
what
your
companies
are
doing
and
I'm
sure
there's
a
bunch
of
other
use
cases.
I
haven't
even
mentioned
here,
be
creative.
There's
some
super
cool
stuff
out
there
I'm
sure
you
could
do
here's
a
handful
of
companies
that
I
know
of
that
use.
Cortex
I'm,
starting
that
upper
left.
A
We
works
big
user,
that's
where
cortex
originally
came
out
of
they've
super
interesting
data,
and
then
they
let
anyone
on
the
internet
go
sign
up
and
send
them
terrible
terrible
data.
So
that's
pretty
fun,
then
kevanna
labs,
Aspen,
mesh
EA,
the
gaming
Electronic
Arts,
is
a
super
interesting
one
powers
all
of
their
internal
metrics
a
whole
lot
of
other
companies.
My
cortex
has
been
running
for
about
three
years
in
production
and
I.
Believe
you
can
still
query
data
from
pretty
much
three
years
ago.
A
In
those
we've
works
clusters,
I'm
just
super
cool
and
we've
just
started
with
regular
releases
recently
they're
all
zero
point.
Something
I
think
we're
zero
point.
Four
is
soon
somewhere
in
there,
but
it
gives
a
bit
more
community
in
a
bit
more.
You
can
see.
What's
changing,
how
does
cortex
do
it?
The
cortex
is
built
on
the
exact
same
code
as
Prometheus.
We
just
take
the
Prometheus
code
vendor
it
in
and
use
it.
A
That
means
prom
QL,
exactly
the
same
you're
used
to
and
it
works
and
then
and
similarly
alert
rules
recording
rules
all
the
same
syntax
you're
already
used
to
using
in
your
in
your
clusters.
In
addition
to
that,
we
make
cortex
horizontally
scalable
highly
available.
Instead
of
using
local
storage,
you
have
no
sequel
and
blob
storage
for
the
data
persistency
and
then
we've
embed
multi-tenancy
throughout.
So
you
can
isolate
data
between
different
customers,
different
internal
products.
A
However,
you
see
fit
cortex
the
CNCs
sandbox
project
and
you
can
find
out
that
github
account
hopefully
website
soon
need
to
work
on
that
Sam
start
Intuit
global
view
associate
cortex
is
a
central
system,
key
difference
between
it
and
Thanos,
which
sure
many
of
you
have
also
heard
about
you'll
be
able
to
send
data
from
many
different
H,
a
pairs
of
Prometheus
to
one
centralized
cortex.
They
tape
airs
in
prometheus
world.
A
If
you
want
high
availability,
you
run
to
cortex
will
do
the
deduplication
for
you
and
all
of
that,
and
then
you
can
graph
it
into
your
chosen.
Graphing
technology
works
pretty
well.
So
now,
let's
dig
into
what
does
that
little
blue
circle
actually
contain
here
at
the
beginning,
the
simplest
way
is
just
a
single
binary.
It
actually
is
mostly
that
blue
circle,
you
send
your
data
and
your
dashboards
point
them
to
cortex.
You
run
one
command,
cortex,
pass
it
a
config
file.
There's
an
example
config
file
in
our
github
repo.
A
Typically,
in
metric
systems,
you
have
a
way
way
more
rights
than
you
ever
do.
Reeds
and
that
paths
you
will
have
to
scale
very
differently
compared
to
your
query
path.
You
can
kind
of
think
in
cortege.
There's
three
different
subsystems,
they're
very
helpful.
First,
one
ingest
getting
data
in
pretty
key,
can't
do
much
else.
If
you
don't
have
any
data
and
then
you
can
query
getting
data
out
and
then
finally,
the
rule
rule
an
alert
evaluation
subsystem
jump
into
each
of
those,
so
starting
with
and
ingest.
A
A
The
first
one
is:
it
will
D
duplicate
samples,
so,
rather
than
persisting,
data
for
multiple
for
all
of
the
data
will
actually
only
take
samples
from
one
of
those
high-ability
have
a
lability
pairs
of
Prometheus
instances,
drop
the
other
ones
and
send
sind
off
one
set
of
data
and
that
how
that
is
calculated
that
sends
off
the
data
to
the
investors,
based
on
a
consistent
hashing
of
a
specific
series.
You
always
know
the
data,
for
one
series
will
go
to
the
same
investors
in
this
case.
It's
usually
a
set
of
three
investors.
A
Again
we
just
bender
in
the
same
code
and
it's
based
on
the
gorilla
paper
from
Facebook.
The
advertised
data
is
that
you
can
take
a
timestamp
value
which
would
be
usually
about
16
bytes
of
data
and
on
average
it
gets
compressed
at
one
point:
three:
seven
bytes
per
sample.
So
pretty
awesome.
That
is,
as
you
can
imagine,
quite
a
big
cost
savings
compared
to
just
storing
everything
into
something
like
sequel
and
then
each
chunk
is
sort
of
capped
at
a
limited
size.
A
So
by
default
it's
or
twelve
hours,
you
can
have
more
than
twelve
hours
of
data
in
one
chunk
and
that
will
get
flushed
and
then
for
an
idea
of
scale.
There
will
typically
be
millions
of
chunks
in
memory
and
your
investors,
you
can
have
billions
of
chunks,
billions
of
series
in
the
long
term
store
and
each
chunk
will
be
a
few
kilobytes
in
size
when
everything's
done.
A
So
that's
the
interest
path.
Next
up
is
querying,
as
you
can
see.
There's
a
couple
shared
components
here
at
CD
and
console
are
still
used
and
the
investors
are
used
for
that
recent
data.
So
those
are
sort
of
the
shared
things.
You'll
see
first
off,
who
always
here
at
Toms
last
talk
about
query
front-end
few
people.
A
It
does
things
like
splitting
queries
by
day,
so
if
you're
doing
month-long
queries
even
longer
than
that
which
is
pretty
a
common
thing,
if
you
want
a
long
term
stories,
you're
going
to
be
looking
at
long
data
ranges,
it'll
split
those
queries
up,
so
they
are
executed
and
much
much
faster
than
if
it
was
just
a
single
thread
and
it'll
do
things
like
query.
Caching
and
it'll
fairly
schedule
queries.
This
is
a
multi-tenant
system.
You
don't
want
one
of
your
tenants
just
abusing
the
system
and
shutting
everyone
else
out.
A
So
it'll
fairly
schedule
queries
across
the
different
tenants
and
then
we
move
into
the
query
or
component
the
career
component
it'll
go
sit,
go
the
crew,
front-end,
say
yep
I
need
to
do
this,
query
gets
the
data,
gets
the
query
and
I'll
actually
do
the
prom
QL
evaluation,
this
embeds,
the
prom
QL
engine
from
Prometheus,
and
it
reaches
out
first
to
the
investors
for
any
recent
data,
as
sayings
are
six
to
twelve
hours
of
data.
Is
there
so
you
need
to
reach
out
get
the
recent
data
pull
it
back.
A
It'll
also
reach
out
to
the
store.
So
that's
for
any
data.
That's
are
being
flushed
series
that
have
gone
stale
things
like
that.
It'll
pull
all
of
that
data
back
as
well.
Merge
those
sets
together
if
there's
any
sort
of
duplicate
data
which
can
happen
in
some
situations,
it'll
merge
it
cleanly
and
send
it
back
to
the
query.
A
So
yet
again
have
a
component
for
that
multi-tenant
and
it
works.
You
have
a
ruler
component
and
again
will
reach
out
to
the
in
jesters
via
the
distributed
hash
table,
that's
stored
in
EDD
and
console
look
at
recent
data
similar
the
queer.
Similarly,
it
also
can
reach
out
to
the
store
and
get
any
data.
Most
rules.
Only
look
at
you
know
five,
ten
fifteen
minutes
of
data,
but
there
are
some
that
need
to
go.
A
The
store,
especially
if
you
look
trying
to
like
look
into
the
future
on
disk
space
usage
and
things
like
that
after
the
rule
is
evaluated.
It
will
then
write
back
in
today
in
jesters,
so
so
right,
the
result
of
your
recording
roll
into
the
in
jester
and
at
which
point
it
can
be
queried
using
the
query.
Any
alerts
that
go
off
are
sent
to
and
the
alert
manager
component
within
cortex
alert
manager,
like
all
the
other
components,
just
vendored
in
upstream
Prometheus
alert
manager
with
some
multi
tenant
wrappers.
A
You
can
use
the
same,
alerting,
rule
definitions.
You
are
used
to
and
configuration
that
you
are
used
to
to
send
off
your
alerts
to
Vic
drops
pager
duty,
slack
email,
whatever
your
alerting,
whoever
wakes
you
up
in
the
middle
of
the
night
and
then
serve
us
a
little
bit
of
recommendation
here
is
that,
depending
on
your
use
case,
I
really
like
keeping
the
alerts
in
the
native
Prometheus
still
and
then
use
cortex.
A
Where
do
you
actually
put
this
data?
I've
just
been
saying,
store
a
lot
so
there's
a
few
different
stacks
available,
first,
sort
of
AWS
stack
of
demo,
DB
and
s3,
and
second
BigTable
and
Google
Cloud
Storage
for
the
Google
stack
and
then
you
can
also
have
Cassandra
for
if
you're
not
running
in
one
of
those
cloud
environments
or
just
more
comfortable,
running
Cassandra,
there's
also
both
DB
for
local
development
options
would
not
recommend
using
it
for
any
data.
A
You
want
to
persist
a
long
term,
but
it's
there
and
it's
great
for
just
spinning
up
quick
instances
and
playing
around
with
cortex
and
then
you'll
notice,
like
each
of
these,
both
had
like
Google
has
both
a
no
sequel
database
and
a
blob
storage.
So
reason
is
cortex.
Has
the
idea
of
it
separates
data
from
indexing
data
and
chunk
data?
A
So
you'll
see
situations
where
you
wanna
store
that
index
data,
always
in
something
like
time:
ODB
BigTable,
these
use,
no
sequel
stores
Cassandra,
but
the
trunk
data,
especially
if
it's
three
or
four
kilobytes
the
economics
might
be
a
bit
different.
If
you're
storing
it
for
long
amounts
of
time
it
is,
it
can
be
cheaper
to
start
into
the
blob
storage
like
s3,
the
Google,
Cloud
storage
and
said
it
all
works.
In
addition,
we
provide
some
tools
for
automatically
scaling
dynamodb,
so
as
load
changes
throughout
the
day.
Your
dynamodb
scaling
will
change
with
it.
A
Just
some
nice
tools
like
that
included
within
cortex
yeah.
So
what's
new
in
the
in
cortex
world-first
super
exciting
couple,
new
experimental
features
in
jesters
now
can
create
and
ship
blocks
instead
of
those
individual
chunks.
It's
very
experimental
but
try
it
out.
I'm
super
curious
and
super
excited
to
see
where
this
goes
and
it's
a
great
collaboration
point
with
the
Thanos
community.
We
believe
it's
just
a
bunch
of
Thanos
code
vendored
into
cortex
at
this
point
and
you
can
do
the
same.
A
Hopefully,
at
that
point
one
less
one
last
system
you
have
to
deploy
to
deploy
cortex
and
just
work
internally
experimental
right
now
then
try
it
out
have
a
good
time
and
then
some
and
then
some
upcoming
work
first
piece
that
I'm
very
excited
I
was
right
ahead
log
for
in
jesters
right
now,
all
of
the
ingest
er
data
is
stored
in
memory,
which
means,
if
you
say,
one
of
them
ohms
or
something
like
that.
That
data
is
lost,
give
three
of
them.
A
It's
pretty
rare
that
you
will
lose
all
of
them,
but
it
can
happen,
and
this
right
ahead.
Log
won't
make
recovery
way
better
today,
two
resiliency
way
better
and
just
a
little
bit
less
stressful
about
losing
single
and
jesters
and
finally,
query.
Parallelization
I
would
again
refer
to
you
to
tom's
talk
on
blazing
prime
QL
in
the
previous
session,
for
more
in-depth
into
this,
but
being
able
to
issue
out
queries
that
you
can
break
down
individual
queries
into
many
different
parts.
That'll
really
speed
up
these
big
queries
that
commonly
gets
into
a
core
tech
system.
A
B
A
A
It's
going
to
depend
oh
yeah,
so
his
question
was
we
I
mentioned
a
couple.
Experimental
features,
experimental
features
when
will
those
become
stable
features?
There's
no
formalized
process
for
making
of
taking
a
feature
out
of
experimental
right
now.
They'll
mostly
depend
on
how
many
people
actually
use
it
and
when
it
becomes
stable
enough
that
a
couple
of
the
big
companies
use,
big
companies
that
are
deploying
cortex
feel
comfortable
using
it
instead
of
console
instead
of
at
CD
things
like
that.
What's
that
yeah
yeah
test
it
out
for
us
and
we'll
be
ready,
faster.
C
A
Are
multiple
companies
involved,
I
would
say,
I,
don't
know
10
or
10,
or
so
people
with
reasonable
contribution
right
now
and
maybe
half
of
those
core
fauna
specific
and
then
there's
still
we've
works.
You've
got
Brian
over
here.
He
does
a
fair
amount
of
work
from
we
works.
We've
had
some
work
from
digitalocean.
Recently
I
met
Splunk
dr.
fauna,
so
I
I'll
do
a
bit
of
work
as
well.
So
there's
a
variety
of
companies
across
yeah,
yay,
yay,
well,
I
guess
he's
Microsoft.
Now
few
different
companies.
B
Thank
you,
so
cortex
is
built
with
the
multi-tenant
casing
in
mind
right,
but
if
you
take
that
out
of
the
equation-
and
you
were
just
have
like
you're
just
thinking
of
a
single
tenant
solution,
what
would
your
preference
be
and
I'm
sure
this
isn't?
The
first
time
we've
gotten
this
versus
Thanos
I?
Think.
A
It
depends,
it
depends
on
your
system,
so
yeah,
let's
get
rid
of
multi-tenancy.
My
first
question
that
I
look
at
is:
are
you
aggregating
data
from
many
different
locations
so
like?
If
you
are
sending
from
many
different
kubernetes
clusters
that
they
don't
have
an
obvious
network
path
between
them?
It
can
be
a
lot
of
times
simpler,
just
to
send
them
to
a
central
store,
and
that's
a
pro
that
cortex
s.
A
The
second
thing
to
look
at
is
sort
of
costing
and
the
system
you
want
to
deal
with
Thanos,
you
spin
it
up
and
you
just
have
to
store
it
into
blob
storage,
but
it
is
missing.
It
is
missing
the
index.
It's
missing
that
index
lookup,
which
can
really
select,
which
can
make
Thanos
queries
slower
than
cortex
your
bear.
Your
mileage
may
vary
on
that.
A
That's
just
a
bit
of
my
experience
when
I
have
spun
up
both
of
them,
and
the
third
thing
is-
and
this
is
something
like
Fanus
is
probably
going
to
be
working
on-
is
right
now
Thanos,
especially
for
long-term
storage.
If
you
have
somewhat
abusive
cardinality
cases
where
you
say
had
a
point
in
time
where
there
was
just
a
ton
of
you,
had
some
carnality
disaster
or
something
like
that
pitch
for
Brian
in
my
talk
tomorrow,
we're
doing
we're
presenting
a
few
carnality
disasters
and
what
happens
Thanos
on
startup
for
their
store
component.
A
You
have
to
load
all
of
these
symbol
tables
and
this
metadata
into
memory,
and
eventually
that
can
become
very,
very
painful
and
I.
Think
we,
when
we
spun
it
up,
we
were
giving
it
two
or
three
hundred
gigabytes
of
RAM
and
it
wasn't
loading
our
data
after
a
while-
and
that
was
a
point
when
we
couldn't
use
that
us
anymore.
So
there's
a
few
things
to
look
at
it's
a
relatively
small
cluster.
A
From
Prometheus
2
cortex,
without
like
long-term
data
loss
for
data
currently
in
in
Prometheus
I,
don't
know
of
an
easy
way
to
do
that.
I've,
you
could
this
isn't
easy,
but
you
could
effectively
like
you,
could
write
historical
data
into
cortex
to
be
a
remote
write.
Protocol
like
that
could
work,
but
I,
don't
know
of
any
tools
out
there
that
just
do
it
for
you.
D
A
Yeah,
like
that
seems
like
it
seems,
technically
very
possible,
no
easy
way
to
do
it
now
and
it
seems
more
possible
with
the
workaround
just
using
blocks
rather
than
chunks,
because
then
you
could
potentially
just
upload
a
block
and
put
some
data
put
some
metadata
around
it.
Yeah
does
not
exist
now.
Oh
it's.