►
From YouTube: Thanos: Highly Available, Pluggable, Long Term Metric Storage for Everyone!- Wiard van Rij, Fullstaq
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Thanos: Highly Available, Pluggable, Long Term Metric Storage for Everyone! - Wiard van Rij, Fullstaq
Prometheus was initially made for short metric retention to answer questions on “what is happening ‘now’”. It is a strong project that solves certain problems really well, but still as a monolith when doing so. Thanos has been made to enable scaling, highly available setups and long term (cheap) storage for Prometheus. Everyone could leverage Thanos for these features. It does not stop there; Thanos has multiple components that could be used for multi-cluster telemetry, remote writes, and multi-tenancy. We want to introduce everyone to Thanos. Explaining the use-cases and how it could benefit your stack now observability becomes such an important factor in tech.
A
A
A
A
A
A
So
it's
important
to
get
a
top-down
view
of
committees
as
it's.
Well,
it's
doing
quite
a
lot
of
things,
but
the
things
that
you're
doing
is
it's
doing
really
well,
so
in
the
middle
we
have
to
promote
your
server
and
we
can
do
the
retrieval
of
those
metric
pages.
So
basically,
that's
the
retrieval
part.
We
are
pulling
metrics
from
those
metric
endpoints
and
we
can
do
this
in
a
certain
interval
which
is
just
the
scraping
interval.
A
But
how
do
we
discover
these
targets?
Well,
that
can
be
done
via
multiple
methods.
If
you
check
out
the
prometheus
website,
you
have
all
kinds
of
surface
discovery
methods
I've
listened
to,
which
is
the
community
service
discovery
and
the
file
service
discovery,
and
basically,
what
we
often
do
with
the
file.
Discovery
are
static
targets,
so
we
input
the
host,
which
is
just
a
static
target
and
prometheus
will
try
to
scrape
that
endpoint.
A
Then
we
also
have
this
http
server
and
well.
This
allows
for
program
well,
which
is
the
query
language
for
prometheus,
to
allow
you
to
query
your
metrics
over
time
and
prometheus
does
have
its
web
user
interface,
which
is
also
really
nice,
but
perhaps
a
more
common
user
interface
is
corfana,
which
is
really
nice
for
the
visualization,
and
you
can
do
so
much
more
with
grafana,
but
that's
out
of
scope
for
right
now.
A
A
For
example,
as
you
can
see,
the
tsdb
is
storing
its
data
on
the
node
itself,
on
disk
or
on
a
volume
depending
on,
if
you
run
on
cubic
meters,
which
is
fine,
if
you
have
a
few
surface
that
you're
scraping
and
you
have
a
data
retention
of
30
days
that
that
that
works
perfectly,
because
prometheus
is
able
to
scrape
a
lot
of
targets
and
is
able
to
retain
a
lot
of
data
but
the
retention
period.
I
haven't
seen,
for
example,
a
retention
period
of
two
years
for
a
ten
thousand
of
servers
in
one
single
prometheus.
A
That
can
be
quite
tough
for
prometheus.
It
wasn't
built
for
a
very
long
retention
period.
So,
let's
question
prometheus
for
a
little
bit.
I
don't
want
to
say
anything
bad
about
just
because
I
really
love
prometheus
and
social
too,
but
there
are
a
few
limitations.
For
example,
I
have
a
single
node
with
prometheus
on
it.
I
have
a
second
note,
with
four
surfaces
on
it
and
wow
prometheus
have
discovered
my
four
surfaces
and
it's
all
running.
A
Well,
it's
scraping
my
four
surfaces
and
I
can
query
the
data,
but
what,
if
note
1
crashes
well,
this
might
pose
a
problem
depending
on
your
use
case,
of
course,
but
at
the
moment
it
isn't
scraping
my
four
services
anymore,
and
I
also
don't
have
an
opportunity
to
query
my
data
from
the
period
that
committees
was
running
and
obviously,
if
you
are
known
to
committees,
we
can
just
add
replicas.
We
can
just
add
another
prometheus,
which
is
what
we
can
do
so
here
we
have
two
nodes
and
both
nodes
are
running
prometheus.
A
A
A
There's
just
a
little
downside
to
this,
because
both
nodes
have
the
same
time
series
but,
as
we
have
discussed,
a
prometheus
will
scrape
on
a
certain
interval,
and
this
isn't
a
fixed
interval
in
time.
So
it
isn't
scraping
at
zero,
zero,
zero,
zero,
it's
just
scraping
every
15
seconds
and
it
really
depends
per
prometheus
when
it
is
scraping
it.
So
it
could
be
the
case
that
both
prometheuses
have
the
same
raw
time
service
data,
but
the
time
series
would
most
likely
be
the
same.
However,
the
samples
would
be
different
to
the
different
scraping
times.
A
We
are
missing
data,
so
if
our
query
hits
node
1,
we
still
have
this
gap
in
our
data,
and
this
is
fine
for,
for
example,
alert
manager
because
alert
manager
is
aware
of
this
behavior
and
it
can
just
filter
and
deduplicate
the
data.
But
if
you
are
talking
about
making
queries
with,
for
example,
grafana
and
making
dashboards,
it
could
be
that
if
you
just
hit
the
refresh
button,
you
hit
the
prometheus
that
was
offline
for
five
minutes
and
you
have
a
gap
in
your
data.
A
A
There
are
a
few
downsides
to
this.
Well,
first
of
all,
you
need
to
have
access
to
your
external
cluster,
because
it
is
gathering
the
information
about
your
services
via
the
api
server.
So
you
will
need
to
have
access
to
your
other
server,
and
this
guide
this
might
be
just
a
little
bit
annoying
for
you.
A
Perhaps
another
solution
which
I
personally
don't
recommend
is
using
federation.
So
federation
is
just
a
native
solution
within
prometheus
which
allows
you
to
scrape
different
prometheus
surface.
So
what
we
could
do
is
implement
a
prometheus
server
on
your
external
clustering
and
just
expose
your
prometheus
and
let
it
be
scraped
by
a
central
cluster.
A
This
works,
but
I
wouldn't
recommend,
because
it
requires
a
lot
of
data
to
be
sprayed.
There
can
be
massive
amounts
of
metrics
on
your
kubernetes
cluster
and
if
this
grows,
this
solution
doesn't
scale
that
well.
Another
solution
which
I
just
added
for
completion
is
expose
your
metric
endpoints
in
another
cluster.
A
A
I
normally
assume
that
you
want
to
have
all
your
metrics
well
gathered
from
your
clusters
and
not
just
one
single
point.
But
if
this
is
your
use
case,
you
could
do
this
and
just
define
your
target
as
a
static
target.
So
I've
talked
a
little
bit
about
some
form
of
limitations
and
prometheus,
but
it
doesn't
make
prometheus
bad
or
anything.
In
fact,
we
actually
really
love
prometheus,
and
this
is
where
the
talk
of
dance
is
about
about
extending
prometheus,
because
we
like
prometheus
and
there's
a
lot
of
love
between
our
communities
and
yeah.
A
Let's
get
started
about
talking
about
tunnels,
first
of
all,
our
community.
I
think
we
are
really
proud
about
everyone
involved
in
the
dallas
community.
We
are
aware
fully
sourced
from
the
start,
and
well
it's
been
a
few
years.
We
started
in
november
in
2017.,
myself,
not
included.
I
just
came
on
later
yeah
and
we
are
on
cncf
incubating
project
and
personally
for
myself.
This
is
really
important.
A
We're
also
hitting
almost
ten
thousand
github
stars,
which
would
be
another
milestone
for
this
project,
and
we
have
so
many
contributors
and
slack
users
are
really
active
and
I
really
love
the
vibes
on
our
slack.
So
please
feel
welcome
to
join
us
on
cncf
slack
on
darno's
channel,
happy
to
see
you
there
and
feel
free
to
ask
questions
there.
A
Don't
always
take
my
only
word
for
it.
We
have
numerous
companies
running
thousand
production
to
ensure
they
have
reliable
and
scalable
monetary
platform.
I'm
also
really
really
happy
that
those
companies
are
willing
to
add
themselves
to
the
adoption
page
on
our
github
and
also
if
you
are
listening
right
now
and
you're
using
downloads,
please
feel
free
to
add
yourself
to
the
adopter
list.
A
A
So
we
can
have
a
multiple
clusters
with
prometheus
and
we
can
add
them
all
to
a
unified
way
to
query
them
all,
which
is
really
awesome,
and
I
will
get
back
to
on
how
this
works
and
how
it
would
look
like
it
also
unlimited
retention-
and
perhaps
this
is
weird
I
mean
you-
can
have
a
limited
retention
for
everything.
But
what
thomas
incorporates
is
storing
the
data
on
the
object
store,
so
it's
really
cheap
and
efficient
and
the
skills,
and
that
makes
it
the
unlimited
retention
and
thanos
is
also
prometheus.
Compatible.
A
A
A
A
A
A
We
also
have
a
generic
monitoring
cluster
and
we
have
two
query
components,
and
these
are
stateless
and
you
can
run
multiple
replicas
of
this
and
we
have
grafana
which
hooks
into
the
query
component.
So
we
are
querying
grafana
to
the
query
component
and
the
query
components
are
doing
their
queries
to
our
sidebars
and,
as
you
can
see,
cluster
2
has
multiple
replicas,
and
the
great
part
of
this
is
that
the
query
component
does
deduplication
on
the
fly.
A
So,
even
if
in
cluster
2,
one
node
goes
down
or
prometheus
goes
down,
we
can
still
have
our
data
and
well
when,
for
example,
the
prometheus
is
back
online.
We
used
to
have
this
gap
in
our
data.
If
we,
for
example,
hit
exactly
that
prometheus.
That
was
offline
for
a
certain
amount
of
time
or
the
query
components
just
prevents
that
as
it
does
the
duplication
of
the
data,
so
you
always
have
the
same
unified
way
to
displace
your
data.
A
A
But
we
can
do
a
lot
of
different
things.
As
you
can
see
here,
we
have
one
cluster
with
only
a
side
card
and
we
have
another
cluster
with
a
query
component
and
store
component
and
the
sidebar,
and
this
works
just
as
well.
So
in
a
monitoring
cluster,
we
have
a
unified
way
to
query
all
our
clusters,
but
we
only
have
one
cluster
with
a
long-term
metrics,
and
this
works
just
fine,
and
this
is
really
awesome.
A
And
we
can
go
just
further.
We
can
add
another
profana
with
multiple
query
components
on
our
cluster
one,
and
this
might
be
really
useful
if
you
are
working
with
different
teams
wanting
their
own
grafana
handsets
that's
possible.
We
can
just
have
this
one
cluster
atom
profana
at
the
queer
components,
and
we
still
have
our
monitoring
cluster,
in
which
we
can
see
all
our
clusters.
A
A
For
example,
if
we
have
a
third
cluster
and
we
can
go
basically
bananas,
this
is
a
draft
that
I
made
a
while
ago,
and
I
just
figured
to
include
this.
This
is
just
having
various
components
and,
as
you
can
see,
everything
is
just
chainable
plugable
and
you
don't
have
to
go
into
details
of
this
graph.
What
I'm
trying
to
say
with
this
is
that
it's
really
well
to
just
plug
and
play
to
make
changes
to
switch
things
up
and
to
move
things
through
various
pipelines.
A
A
I
will
upload
these
slides
and
you
can
check
all
the
links
for
yourself,
but
basically
artana's
website
has
lots
of
information
and
again,
if
you
are
missing
anything,
please
create
an
issue
or
a
pull
request
happy
to
improve
this.
I
also
want
to
give
a
shout
out
to
the
prometheus
operator,
which
is
really
awesome,
to
have
like
a
unified
way
to
set
up
your
prometheuses,
and
it
also
allows
you
to
incorporate
down
sight
r
with
the
primitive
operator
there's
also
the
the
cube
tunnels
elm
chart.
A
If
you
think
this
talk
was
really
nice,
please
do
check
it
out
and
we
also
have
more.
So
we
have
our
slack
it's
on
the
cncf
slack
and
we
have
our
channels
there
feel
free
to
join,
and
we
also
have
questions
and
discussions
on
our
github
and
obviously
our
get
up
in
general
for
poor
question
issues
and
again
check
out
our
website,
and
I
would
like
to
thank
you
all
and
our
community,
everyone
involved
in
tunnels
and
cncf,
and
thanks
so
much
for
having
me
here.