►
From YouTube: Scaling Prometheus: How We Got Some Thanos Into Cortex - Thor Hansen, HashiCorp & Marco Pracucci
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scaling Prometheus: How We Got Some Thanos Into Cortex - Thor Hansen, HashiCorp & Marco Pracucci, Grafana Labs
Cortex is a long term storage for Prometheus, designed for scalability, multi-tenancy and high-availability. It can reliably ingest and query millions of time series per second with sub-second latency. The current storage design uses a NoSQL store to index series and an object store for compressed time series data - two dependencies, and one with significant cost implications. In this talk we will show the new experimental Cortex blocks storage, based on Thanos and Prometheus TSDB, aiming to reduce the Cortex operational cost without compromising scalability and performances. We’ll cover the trade-off between the standard chunks storage and the new blocks storage, and share lessons learned running Cortex at scale. Cortex is a CNCF sandbox project.
https://sched.co/Zeuw
A
Hello,
welcome
to
the
talk
scaling
prometheus,
how
we
got
some
tunnels
into
cortex.
In
this
talk,
we
are
going
to
show
cortex
and
specifically
a
new
storage
engine.
We
have
built
into
cortex
leveraging
containers
before
that.
Let
me
do
a
quick
introduction
about
cortex
cortex
is
a
distributed
time.
Series
database
built
on
top
of
prometheus
cortex,
is
horizontally
scalable
and
highly
available
and
offer
a
durable,
long-term
storage.
For
your
time
series
data
cortex
supports
multi-tenancy,
and
the
typical
use
case
is
to
have
a
global
view
across
multiple
prometheus
servers
over
the
years.
A
A
You
have
a
prometheus
configured
to
remote
right
to
the
distributor
and
the
distributor
is
the
ingress
for
the
right
path
and
is
responsible
to
shard
and
replicate
your
series
across
a
pool
of
ingestors
injesters.
Keep
they
receive
the
serious
samples
in
memory
and
periodically
flush.
There's
a
series
to
the
long
term
storage
under
the
foot.
The
long
term
storage
is
actually
composed
by
two
different
data
stores,
an
object
store
and
an
index
store.
A
In
a
cortex
cluster,
the
query
frontend,
the
main
purpose
of
the
query
frontend,
is
to
offer
result
caching
and
also
employs
some
optimization
techniques
which
we
will
see
in
details
later
on,
which
allows
to
parallelize
the
query
execution
across
multiple
query
nodes.
So
the
query
or
a
splitted
query
that
can
be
splitted
by
the
query.
Front-End
is
actually
executed
by
a
pool
of
queries
which
will
fetch
the
most
recent
samples
from
the
ingestors
and
all
their
data
from
the
long
term
storage,
and
then
we'll
run
the
promql
engine.
A
On
top
of
this
now,
this
slide
shows
you
the
microservices
architecture,
but
you
are
not
required
to
deploy
cortex
in
microservices
architecture,
because
cortex
actually
support
a
second
operational
mode,
which
is
the
single
binary
mode,
and
it's
the
easiest
way
to
deploy
a
cortex
cluster
today.
So
when
you
deploy
cortex
in
single
binary
mode,
cortex
is
running
as
a
single
binary
with
a
single
configuration.
A
This
architecture
over
the
time
proved
to
work
and
scale
very
well.
We
have
seen
cortex
clusters,
ranging
from
few
tens
to
100
million
active
series
and
in
the
typical
use
case,
we
see
the
99.5
percentile
query
latency
below
2.5.
Second,
however,
for
many
users
requiring
both
an
object,
store
and
an
index
store,
introduce
extra
operational
complexity
and
if
you
run
in
the
cloud
like
big
table
or
dynamodb,
also
extra
costs
to
run
a
cortex
cluster.
So
almost
an
year
ago,
we
started
brainstorming
on
the
idea
to
remove
the
index
store
at
all.
A
A
Currently,
in
the
experimental
phase
we
supporting
cortex,
so
you
can
deploy
cortex
with
the
chunk
storage,
which
is
the
architecture
I
just
showed
you
or
you
can
deploy
cortex
using
the
new
and
experimental
block
storage.
In
this
talk,
we
are
going
to
cover
how
the
block
storage
work
under
the
food-
hello,
everyone.
My
name
is
marco:
I'm
a
software
engineer
at
grafana
labs,
I'm
a
cortex
maintainer
and
I
recently
joined
tino
spontaneous
as
well.
B
Hi,
I'm
thor
hansen,
I'm
a
software
engineer
at
hashicorp.
I
got
involved
with
cortex
in
my
previous
role,
where
we
serve
metrics
and
alerts
for
many
customers.
We
were
a
small
team
and
needed
the
scalability
of
cortex,
as
well
as
the
multi-tenancy,
but
didn't
want
to
have
to
manage
a
separate
index
store,
which
is
where
this
all
started.
For
me.
B
B
B
B
B
B
We
implemented
most
of
the
existing
cortex
features
like
rate
limits
and
operational
tooling,
and
we've
made
many
optimizations
and
bug
fixes.
Today,
the
cortex
block
storage
is
still
marked
experimental,
but
at
kerfano
labs
they're
already
running
it
at
scale
in
a
few
of
their
clusters,
and
we
expect
to
mark
this
feature
as
stable.
Soon.
B
So
let
me
show
you
the
current
state,
the
cortex
architecture
doesn't
change
much
between
the
original
trunk
storage,
which
we'll
still
continue
to
support
and
the
new
black
storage
writes,
still
are
distributed
and
replicated
to
ingestors
and
then
uploaded
to
object.
Storage
reads
still
go
through
the
queries
which
read
from
the
ingestors
to
get
recent
metrics.
B
B
B
B
B
B
So
here's
a
closer
look
at
the
right
path.
A
given
metric
is
replicated
to
multiple
ingestors
for
each
user,
so
you
can
see
here.
The
same
metric
is
written
to
three
separate
blocks
and
three
separate
ingestors.
These
blocks
are
written
to
an
adjuster's
local
storage.
Every
two
hours
and
a
long
running
process
called
the
shipper
will
discover
newly
written
blocks
and
upload
them
to
long-term
storage.
B
This
poses
an
interesting
scaling
problem
in
regards
to
how
many
blocks
are
created.
The
number
of
blocks
grows
quickly
with
the
number
of
tenants.
A
cluster
has,
with
a
thousand
tenants
and
a
small
scale
of
50
ingestors,
we'll
be
creating
600
000
blocks
a
day
which
equates
to
200
million
blocks
a
year.
B
B
This
results
in
one
block
per
day
per
tenant
when
we
originally
had
12
times
the
number
of
ingestors
pretended
and
importantly,
the
compactor
can
horizontally
scale
to
support
a
large
number
of
tenants.
Now.
The
previous
solution
reduces
the
footprint
after
compaction
in
the
object
storage,
but
we
are
still
shipping
one
block
per
adjuster
per
tenant
every
two
hours,
as
well
as
opening
a
tsdb
on
every
ingester
for
every
tenon,
which
meant
the
memory
footprint
did
not
scale
with
the
number
of
nodes.
B
A
So
we
have
seen
that
we
can
efficiently
he
ingests
a
large
amount
of
samples
per
second
with
a
pretty
low
latency.
But
the
real
question
is:
how
do
we
query
back
this
data?
One
in
one
of
our
clusters?
We
are
running
at
grifon
labs.
We
have
a
tenant
with
30
million
active
series
running
on
the
block
storage,
which
means
that
we
store
about
200
gigabyte
of
blocks
per
day
after
compaction.
A
If
you
project
this
to
one
ear
retention,
it
means
that
we
need
to
be
ready
to
query
back
this
data
about
across
a
storage
of
about
70
terabytes,
to
understand
how
the
the
read
path
work
for
the
block
storage.
We
have
to
do
a
little
step
back
and
focusing
on
the
query
front-end,
which
is
the
ingress
on
the
read
path.
A
Now,
as
previously
mentioned,
the
query
frontend
provide
two
main
features:
query
execution,
parallelization
and
result.
Caching,
the
basic
form
of
the
query,
execution
parallelization,
is
based
on
time
splitting
when
the
query
frontend
receive
a
query
spanning
over
a
large
time
range,
a
time
range,
which
cover
more
than
one
day,
the
query.
Frontend
split
this
query
into
multiple
queries:
each
one
covering
one
single
day,
aligning
the
timestamp
of
the
splitted
query
to
midnight
utc.
A
So
if
we
receive
a
query
spanning
over
the
last
three
days,
this
query
will
be
actually
executed
will
be
actually
split
into
three
different
queries.
Each
query
will
cover
a
different
day
and
there's
three
splitted
queries
will
be
executed
concurrently
by
the
queries
and
their
results
will
be
then
merged
by
the
query
frontend
before
sending
back
the
response
to
grafana
or
your
quilling
tool.
A
We
have
seen
that
this
allows
for
a
better
parallelization
when
you
run
queries
over
a
large
time
range.
The
querier
then
execute
this
one
day.
Query
now
the
query
periodically
discover
new
blocks
uploaded
to
the
storage
running
a
scan
over
the
bucket,
and
it
keeps
in
memory
a
map
of
all
the
known
blocks
in
the
storage
for
each
block.
We
just
need
few
information,
which
is
the
tenant
id
the
block
id
and
the
minimum
and
maximum
timestamp
of
samples
within
the
specific
block.
A
And
then,
when
the
query
receive
a
query,
it
look
for
all
the
block
ids
containing
at
least
one
sample
within
the
query
start
and
end
time
and
based
on
the
filtered
block
ideas.
It
computes
the
set
of
store
gateways
that
should
be
queried.
In
order
to
query
back
the
data
from
desk
blocks,
the
query
will
fetch
the
most
recent
data,
which
has
not
been
flushed
to
the
storage
yet
from
the
ingestors,
and
we
will
query
the
blocks
stored
in
the
object
store
through
the
store
gateways.
A
This
means
that
we
can
horizontally
shard
the
blocks
across
the
pool
of
store
gateways
and
for
each
block
belonging
to
the
chart
of
one
specific
store
gateway,
the
store
gateway
loads.
The
index
editor
the
index
adder
is
a
small
subset
of
the
entire
block
indexer
in
in
our
clusters.
We
see
that
the
index
editor
is
in
the
order
of
two
percent
of
the
index,
so
it's
a
little
part
of
the
index
and
it's
used
to
speed
up
the
index
lookup
at
query
time.
A
The
query:
query:
the
blocks
through
the
minimum
set
of
store
gateways
holding
the
required
blocks.
So
when
the
query
receive
a
query,
the
query
computer,
the
list
of
block
ideas
which
should
be
queried
and
then
look
up
the
ash
ring
which
we
use
to
as
the
baseline
technology
to
implement
the
sharding
and
the
replication
to
find
the
minimum
set
of
store
gateways
holding
the
blocks
that
needs
to
be
queried
and
concurrently,
query
does
blocks
through
the
store
gateways.
A
Now
we
actually
want
to
lower
as
much
as
possible
the
calls
to
the
object
store
both
because
of
performance
reasons,
and
also
because
of
costs
given
most
of
the
object
store.
Pricing
is
based
on
a
mix
of
data
stored
in
terms
of
gigabyte
and
number
of
api
calls.
You
run,
and
we
have
introduced
three
layers
of
caching,
the
metadata
cache
the
index
cache
and
the
chunks
cache.
A
The
metadata
cache
is
used
by
the
blocks
discovery
mechanism,
which
is
running
both
inside
the
query
and
the
store
gateway,
while
the
index
cache
and
the
chunks
cache
is
only
used
by
the
store
gateway.
The
index
cache
is
a
caching
layer
in
front
of
the
posting
and
serious
lookup,
and
the
chunks
cache
is
a
cache
in
front
of
the
fetching
of
the
chunks
containing
the
compressed
samples.
A
A
We
have
done
an
atypical
setup,
so
we
are
running
two
identical
clusters
since
few
months
by
identical,
I
mean
identical
in
terms
of
version
of
cortex
that
we
run
and
scale
same
number
of
cortex
nodes,
ingesting
the
same
exact
series,
roughly
10
mega
10
million
active
series,
and
then
we
have
introduced
a
proxy
which
is
called
query
t
and
we
open
source
as
well,
which
mirror
every
single
query
we
do
receive
to
both
clusters
and
we
make
the
two
backend
clusters,
one
running:
the
block,
storage
and
another
one
running.
The
chunk.
A
A
We
are
currently
working
on
many
things,
but
to
mention
the
most
important,
probably
we
are
continuously
working
to
improve
the
query
performances.
This
is
an
endless
work
and
we
are
doing
many
high
iterations
over
it.
We
have
several
ideas
we
still
want
to
to
experiment
and
that
they
may
lend
into
upstream
changes,
but
yeah
keep
in
mind.
We
are
spending
a
lot
of
time
on
continuously
improving
performances.