►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
When Prometheus Can’t Take the Load Anymore - Liron Cohen, Riskified
Riskified started from using a pair of Prometheus servers in each of its clusters, but soon enough, Prometheus couldn’t take the load anymore. Once it happened, the SRE team started to check what is the best tool for Multi, HA, long-term Prometheus. They decided to check Thanos, Cortex, and M3. In this session, Liron will share her outtakes of the different tools - which tool can provide the best performance and High Availability, the most cost-effective, and the easiest to deploy and operate.
By the end, you’ll get a better understanding of the different tools and which one is the best solution for your use case.
A
Before
we
start,
I
have
been
a
devops
and
site
reliability
engineer
for
the
past
seven
years
before
my
current
job
as
an
sre
at
riskyfied,
I
was
a
devops
consultant
in
multiple
companies
and
when
I'm
not
working
or
studying,
I
enjoy
traveling,
diving
and
basically
anything
water
related,
as
you
can
probably
tell
by
my
happy
face
in
this
photo
so
before
we
will
dive
into
the
different
solutions
we
decided
to
check.
Let's
briefly
talk
about
the
issues
we
had,
so
what
we
started
with
was
an
architecture
used
by
a
lot
of
people
and
companies.
A
A
A
There
is
no
centralization,
no
global
view
of
data.
If,
for
example,
we
want
to
query
multiple
clusters,
we
can't-
and
there
is
no
long
term
storage
of
data-
we
can
configure
a
retention
of
multiple
years
as
an
example.
This
way,
so
we
knew
we
needed
a
solution
for
all
of
these
issues
and
we
knew
that
there
are
some
tools
that
can
help
us.
A
Eventually,
we
were
left
with
three
potential
tools.
We
decided
we
wanted
to
choose
from
number
three
cortex
and
tongues.
It's
important
to
note
that
I'm
not
representing
any
of
these
projects,
I'm
not
part
of
the
maintainers
of
none
of
them.
We
simply
wanted
to
examine
their
different
architectures
to
understand
what
each
tool
can
offer,
what
are
the
advantages
and
disadvantages
of
each
soul
and
how
they
differ
from
each
other.
A
They
are
all
written
in,
go
open,
source
projects
that
are
compatible
with
prometheus
and
they
all
offer
us
a
solution
that
can
suit
the
issues
we
just
talked
about.
All
these
projects
offers
a
long-term
storage
for
our
metrics,
a
global
view
of
data.
They
are
horizontally,
scalable
and
making
sure
our
metrics
are
highly
available
and
durable.
A
A
A
I
will
start
by
reviewing
each
one
of
the
architectures,
so
we
can
understand
what
are
the
pros
and
cons
of
each
and
be
able
to
compare
all
of
them
by
the
end.
Eventually,
I
will
tell
you
what
we
chose
to
use,
but
please
remember
that
our
choice
was
made
before
some
new
features
were
added
to
this
project.
A
M3
is
based
on
a
push-based
model,
meaning
the
prometheus
from
servers,
use
remote
right
api
to
push
data
to
m3.
So,
let's
start
by
understanding
how
the
right
path
works.
First
of
all,
prometheus
scrapes
metrics.
As
we
know,
then
the
m3
coordinator,
which
can
be
deployed
as
a
sidecar
alongside
prometheus,
gets
promethia's
remote,
write
request.
It
can
also
get
prometheus
remote,
read
request,
as
we
will
see
in
a
moment,
as
its
name
implies.
A
The
duration,
the
data
will
remain
in
memory,
depends
on
the
configured
block
size,
which
is
the
duration
of
time,
dictating
how
long
new
writes
will
be
compressed
in
memory
before
being
flushed
to
the
disk.
For
example,
block
size
can
be
set
to
2
hours.
There
is
also
xcd,
which
stores
the
metadata
used
by
each
of
the
components.
A
A
A
Well,
under
it
path,
grafana
or
other
api
client,
sends
its
query
to
amp's
required.
This
component
is
responsible
for
exposing
the
metrics
and
metadata
of
the
time
series
stored
in
m3db,
so
for
writes
to
m3db.
We
use
a
dedicated
deployment
of
m3
coordinator
instances
and
then
for
queries.
We
can
use
a
dedicated
deployment
of
m3.
Query
instances
note
that
it's
also
possible
to
use
just
the
m3
coordinator.
A
So
what
are
the
advantages
of
m3?
First
of
all,
data
resides
within
the
cluster
on
disk
and
replicated
between
availability
zones
and
or
regions.
There
is
no
use
of
cloud
storage
service,
such
as
s3
or
google
code
storage,
which
means
that
a
bandwidth
costs
are
relatively
low.
So
if
you're
running
things
both
on
prem
and
on
cloud,
it
might
be
a
good
idea
for
you
to
use
m3
since
cloud
bandwidth
cost
might
be
high.
A
When
moving
data
between
the
cloud
and
on-prem
data
centers
and
b,
it
means
lower
latency,
which
of
course
means
a
better
performance.
Fetching
data
from
an
object
store
like
amazon
s3,
might
be
slower
than
using
local
disk.
It's
also
a
push-based
model
system.
Prometheus
uses
remote
right
api
to
send
data
to
m3,
which
also
have
some
advantages,
in
particular
in
use
cases
where
you
have
ephemeral,
cluster
or
in
terms
of
availability.
If
the
prometheus
server
becomes
unavailable,
all
data
up
to
that
point
is
still
available
for
queries.
A
A
A
A
A
A
A
The
distributors
are
responsible
for
validating
the
samples
they
get
and
can
also
deduplicate
incoming
samples
from
multiple
replicas
of
the
same
prometheus
servers.
In
order
to
coordinate
which
replica
is
currently
elected.
As
the
leader,
the
only
replica,
the
distributor
will
accept
samples
from
it's
needed
to
have
a
key
value
store
where
the
data
about
the
elected
replica
is
saved
and
it
can
be
console
or
at
cd.
A
A
A
The
chunks
are
indexed
by
time
range
and
labels
in
order
to
provide
a
fast
lookup
across
chunks.
The
index
will
be
kept
in
a
key
value,
store,
amazon,
dynamodb,
google,
big
table
or
apache
cassandra,
and
the
chunks
will
be
kept
in
object
store
such
as
amazon,
s3,
dynamodb,
google,
cloud
storage
or
microsoft.
Azure
storage.
A
A
The
query,
front-end
stores
the
query
into
in-memory
queue
and
then
the
query
or
component
picks
it
up
and
executes
it.
The
query
are
fetch
samples
from
the
in-memory
series,
samples
in
the
investors
and
the
long-term
storage,
while
executing
query,
because
the
ingesters
hold
the
in-memory
series
that
have
not
yet
been
flushed.
A
Finally,
courier
sends
results
back
to
the
query
front-end,
which
then
forwards
it
to
the
client.
The
query
front-end
is
an
important
component,
thanos
also
added
to
their
architecture
lately
because
of
some
important
features.
Splitting
the
query.
Frontend
split
queries
of
multiple
days
into
multiple
single
day
queries.
It
gives
us
the
ability
to
execute
queries
in
parallel
on
downstream
queries,
which
means
a
faster
query,
execution
and
b.
It
prevents
out
of
memory
issues
when
executing
large
multi-day
queries.
A
A
A
The
in-memory
samples
in
the
ingestors
will
be
flashed
when
a
new
tsdb
block
is
created
in
default
to
two
hours
block
range
periods
to
an
object
store
such
as
amazon,
s3,
google
cloud
storage,
etc.
In
this
architecture,
there
are
two
additional
components
that
are
based
on
thanos
components,
as
we
will
see
soon.
A
First,
the
store
gathway
that
query
blocks
from
the
object
store
and
is
used
by
the
querier
at
query
time.
It
also
uses
index
cache
ton,
cache
and
metadata
cache
in
order
to
speed
up
queries
and
reduce
the
number
of
api
calls
to
object,
storage,
and
there
is
also
the
compactor
which
is
responsible
for
reducing
the
number
of
blocks
stored
in
the
long
term,
storage
by
merging
and
duplicating
smaller
blocks
into
larger
ones,
and
by
that
also
making
them
query
more
efficiently.
A
A
It
gives
us
the
ability
to
use
chunk
storage.
In
case
we
are
putting
performance
at
the
top
of
our
priorities.
Chunk
storage
is
faster
than
block
storage.
We
can
also
decide
to
use
block
storage
when
we
are
putting
simplicity
and
cost
reduction
at
the
top
of
our
priorities.
Clock
storage
is
eventually
cheaper
than
chunk
storage.
A
There
is
also
the
query
front
end
which
allows
query,
parallelization
and
results,
caching,
which
also
have
which
also
have
a
great
impact
on
performance
and
the
push-based
model
that
cortex
is
based
on
also
offers
multiple
benefits
in
terms
of
performance.
Prometheus
push
data
to
cortex.
So
if
the
cluster
you
are
collecting
the
matrix
from
and
the
cluster
where
the
query
is
are
far
away
geographically,
keeping
all
the
data
in
cortex
will
decrease.
A
Query
latency:
there
won't
be
any
gaps
in
the
graphs
caused
by
prometheus
restart
because
the
pushes
happen
as
soon
as
the
data
is
scraped.
Even
if
one
of
the
leafs
of
prometheus
server
is
down,
you
will
not
see
gaps
in
the
data
and
again
in
terms
of
availability
when
an
edge
location
becomes
unavailable.
All
data
up
to
date,
point
is
still
available
centrally.
A
Also
it's
a
great
option.
If
you
don't
want
to
enable
ingest
to
your
clusters
and
basically
it
offers
all
the
advantages
of
the
push-based
system.
We
talked
about
earlier
when
we
talked
about
m3,
so
we
can
really
see
how
cortex
has
been
designed
with
focus
on
high
performance
of
long-term
storage.
But
there
are
some
disadvantages
as
well.
A
Secondly,
in
order
to
use
all
of
its
features
such
as
the
duplicating
you
rely
on
external
dependencies,
such
as
at
cd
or
console,
eventually,
those
are
more
moving
parts
to
deploy,
operate
and
monitor
the
push-based
model
that
we
mentioned
already
also
have
some
cons.
As
we
mentioned
before
extra
complexity
and
resources,
you
need
to
manage
separate,
cortex,
cluster
and
storage
on
top
of
your
prometheus
deployment.
A
Lastly,
the
different
storage
types
have
some
trade-offs
as
well.
Chunk
storage
gives
us
great
performance,
but
is
expensive
when
comparing
to
the
block
storage
and
it's
an
additional
resources
that
need
to
be
taken.
Care
of
block
storage
is
cheaper
and
simpler,
but
its
performance
is
lower,
so
we
saw
that
cortex
can
offer
us
multiple
ways
to
deploy
it.
Each
has
its
own
trade-offs,
as
I
mentioned
before,
the
cortex
team
worked
closely
with
the
palestine,
so
both
cortex
and
thanos
added
components
and
abilities
that
eventually
improved
their
solution
and
added
more
options
and
abilities.
A
A
A
Its
purpose
is
to
upload
data
tsdb
blocks
and
to
an
object:
storage
such
as
iws,
s3,
google,
cloud
storage
or
microsoft,
azure
storage
and
to
give
other
tennis
components
access
to
this
time,
series
data
in
prometheus,
as
we
will
see
when
we
will
talk
about
the
query
path,
sidecar
uploads,
the
tsdb
blocks
to
an
object
storage
as
prometheus,
produces
them
every
two
hours.
This
gives
us
the
ability
to
configure
prometheus
servers
to
run
with
relatively
low
tension
notice.
A
That
using
this
model
means
prometheus,
cannot
be
fully
stateless
if
it
crashes
or
restarts
the
last
two
hours
of
metrics
will
be
lost,
so
persistent
disk
for
prometheus
is
still
needed
using
prometheus
remote
right.
Api
gets
you
closer
to
stateless
prometheus,
but
it
still
won't
be
fully
stateless,
so
it
is
still
always
recommended
to
have
a
persistent
disk.
A
We
can
see
how
the
right
path
here
is
super
easy.
You
can
actually
only
add
the
channel
sidecar
to
your
prometheus
pods
and
configure
your
object.
Storage
in
order
to
save
long-term
data.
What's
great
about
thanos
is
that
features
can
be
deployed
independently
of
each
other.
You
can
start
by
only
deploying
a
side.
Car
and
gradually
add
other
components
to
use
and
deploy
other
features.
A
A
A
The
query
front-end
that
can
be
put
in
front
of
thanos
queries
to
improve
the
read
path
in
order
to
have
some
important
features
like
splitting
and
results.
Caching,
it
is
based
on
the
cortex
query,
front-end
component,
that
we
mentioned
before
note
that
at
the
moment
only
range
queries
can
be
splitted
and
cached.
A
A
A
A
A
The
pool
based
model
gives
us
simplicity
and
it
makes
the
right
path
more
efficient,
because
it
ships
full
compressed
blocks
every
two
hours
by
default,
which
can
also
prevent
data
lost
in
case
of
network
issues.
That
last
more
than
few
minutes
and
lastly,
it
its
recently
added
query
front
end,
can
improve
performance.
A
The
block
storage
is
slower
than
the
alternative
storage
solution
that
cortex
and
m3
offers
the
same
cons
of
the
push-based
model
that
we
talked
about
also
apply
here,
but
there
are
also
trade-offs
when
we
talk
about
pool
based
model,
it
means
that
the
data
from
the
last
two
hours
is
less
durable.
Samples
aren't
being
saved
immediately
through
remote
storage,
which
means
we
cannot
access
data
from
the
last
two
hours.
If
there
are
network
issues
and
we
can
even
lose
it
if
prometheus
goes
down,
it
might
also
have
worse
latency
for
the
query
path.
A
A
A
There
are
other
aspects
that
can
be
compared
that
we
didn't
talk
about,
but
the
bottom
line
is
that
I
think
all
of
these
solutions
are
great.
I'm
pretty
positive.
You
will
be
satisfied
with
with
whichever
solution
you
choose,
but
there
are
some
differences
that
might
make
one
of
these
tools
and
better
match
with
your
needs
and
architecture.
A
There
are
also
other
aspects
according
to
which
we
can
compare
the
solutions,
for
example
from
ql
compatibility
according
to
chromelab's
latest
test.
This
is
the
chrome,
ql
compatibility
of
m3
tunnels
and
cortex
official
documentation.
As
I
said,
I
do
find
m3
official
dogs
lacking,
and
they
also
mentioned
in
the
official
docks
that
additional
work
is
still
needed.
A
I
personally
enjoyed
cortex
to
fermentation
the
dust,
but
thanos
also
has
grey
ducks
we're
talking
about
installation
via
hound
charts.
You
can
find
helm
charts
for
all
of
these
tools,
but
take
note
that
cortex
does
mention
that
their
official
hound
shot
still
needs
work,
that
the
m3
offers
only
a
home
chart
for
the
m3db
operator
and
the
sanos
does
not
have
an
official
hunter
but
does
have
multiple
community
home
charts
options.
A
If
you
are
using
the
q
prometheus
stack,
helm,
chart
the
prometheus
community
chart
that
many
use,
including
us
by
the
way,
to
install
prometheus
operator,
prometheus
rafana,
alert
manager
and
more.
You
already
have
built-in
integration
with
thanos
and
you
can
configure
the
standard
sidecar
very
easily.
A
So
at
this
point,
I'm
quite
sure
you're
asking
yourself:
okay,
so
what
did
you
choose?
We
eventually
decided
to
go
with
thanos.
We
wanted
to
keep
it
simple
fast
offers
a
good
enough
performance
for
our
use
case.
Its
costs
are
relatively
low.
The
cube
prometheus
stock
helm
chart,
which
we
use,
has
ability
and
support
to
install
the
standard
sidecar.