►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Absorbing Thanos Infinite Powers for Multi-Cluster Telemetry - Bartlomiej Płotka & Kemal Akkoyun, Red Hat & Frederic Branczyk, Independent
Thanos is an open-source, CNCF’s Incubated project that horizontally scales Prometheus to create a global-scale highly available monitoring system. It seamlessly extends Prometheus in a few simple steps and it is already used in production by hundreds of companies that aim for high multi-cloud scale for metrics while keeping low maintenance cost. During this talk, core Thanos (and Prometheus) maintainers, will briefly introduce basic ideas behind Thanos and deployment models and use cases. After that, to satisfy more experienced users, they will explain more advanced concepts, tips for running on the scale, and the latest shiny usability improvements. Thanks to the growing community there is much to talk about!
https://sched.co/ekHk
C
A
A
But
then
the
second
half
is
kind
of
what
happened
in
the
recent
past
in
the
tunnels
project
kind
of
as
an
update,
so
that
everybody
doesn't
have
to
necessarily
follow
the
github
repos
and
follow
what
what's
happening,
but
can
just
every
now
and
then
join
some
of
these
sessions
and
and
kind
of
see
what's
new.
So
that's
kind
of
what
we're
going
to
be
doing
today
and
I'll
kick
us
off
with
the
introduction.
A
Then
kemal
and
batek
will
take
over
with
the
with
what's
new.
So
when,
when
I
talk
about
thanos,
I
often
like
to
refer
to
it
as
kind
of
distributed
prometheus
plus
plus.
What
I
mean
by
that
is
prometheus
is
kind
of
intentionally
a
monolithic
application
right.
The
storage
is
in
the
same
binary.
The
scraping
is
the
querying.
Is
everything
is
intentionally
monolithic
to
kind
of
increase
reliability
that
we
can
have
on
this
process?
A
That's
really
great
for
prometheus,
but
it
kind
of
limits
prometheus
in
a
little
bit
in
terms
of
like
horizontally
scaling
prometheus,
because,
obviously
you
can
only
scale
as
much
as
a
single
machine
can
and
that
that
isn't
to
diminish
the
need
for
prometheus
you'll
later
see
how
prometheus
and
tanos
harmonize
really
nicely
together.
But
thanos
is
kind
of
the
additional
bits
and
pieces
to
make
prometh
turn
prometheus
into
a
global
scale.
A
Monitoring
system,
and
so
a
couple
of
things
that
thanos
provides
on
top
of
prometheus
is
a
global
view,
so
that
you
can
query
data
across
all
of
your
prometheus
instances,
long-term
storage,
so
that
you
don't
have
to
only
rely
on
the
local
disk
that
the
prometheus
has
available
and
then
a
couple
of
other
features
that
are
neat,
probably
the
most
interesting
one
being
down
sampling,
which
is
kind
of
related
to
long-term
storage.
Because
when
we
talk
about
storing
long-term
metrics
data,
we
tend
to
also
query
that
data
long
term.
A
But
when
we
scrape
data
or
collect
data
at
15
or
30.
Second
intervals,
but
query
it
over
a
year
we're
not
actually
interested
in
that
super
high
resolution
data.
We
don't
even
have
enough
pixels
on
our
screen
to
show
all
of
this
data,
and
so
downside
thing
comes
in
really
handy
in
those
kind
of
situations,
because
if
we
query
data
over
a
year,
it's
plenty
enough
to
query
data
at
a
one
hour
resolution,
for
example,
and
still
get
an
accurate
picture.
A
A
What
I
also
like
to
say
is
that
thanos
essentially
pulls
prometheus
apart
into
its
individual
modules,
so
the
query
layer,
the
rule
evaluation,
the
storage
et
cetera
and
puts
these
into
individual
components
so
that
we
can
then
horizontally
scale
those
and,
as
a
matter
of
fact,
thomas
tries
to
implement
as
little
as
possible
and
make
as
much
reuse
as
possible
from
prometheus.
So
thanos
tries
to
just
kind
of
fill
the
gap
and
not
reinvent
things,
but
build
on
top
of
giants
essentially
being
prometheus
here.
A
This
is
just
just
a
list
just
so
you
can
refer
to
this
list
later
once
we
go
through
examples
of
uses
of
all
of
these
components,
so
let's
say
dive
right
in
and
get
to
the
first
kind
of
example,
usage-
and
this
is
probably
the
most
common
thing
that
I
see
people
starting
out
with
with
tunnels
and
that,
if
you're
just
starting
out
this
is
what
I
would
probably
recommend
you
you
trying
out.
So
you
may
be
just
be
running
prometheus
already
and
you
want
to
run
prometheus
in
high
in
highly
available
manner.
A
Obviously,
because
this
is
your
monitoring
system,
you
kind
of
are
relying
on
your
monitoring
system
to
monitor
the
rest
of
your
infrastructure.
So
you
want
that
to
be
available
right
and
then
the
typical
thing
when
we're
not
using
tunnels
just
with
prometheus.
What
we
tend
to
do
is
we
put
a
load
balancer
in
front
of
prometheus,
and
we
query
our
two
prometheus
servers
through
that
load
balancer
now
this
is
a
little
bit
problematic,
but
why
the
problem
is
because
prometheus
is
a
pull-based
model.
A
The
prometheuses
prometheus
servers
that
we're
seeing
here
are
scraping
the
same
targets,
but
it's
slightly
offset
at
intervals.
So
that
means,
while
for
alerting
purposes,
they're
close
enough
when
you're
querying
data
and
graphing
it
over
time.
This
can
lead
to
inconsistencies,
and
so
this
tends
to
be
problematic
or
at
least
confusing
to
users.
This
gets
more
problematic.
A
It
queries
both
prometheuses
for
all
the
data
they
have
available
for
a
particular
query
and
then
merges
these
using
a
deduplication
algorithm
and
presents
you
with
one
consistent
result
and
the
way
that
this
integration
kind
of
works
is
through
the
tanos
sidecar.
The
sidecar
is
really
just
a
shim
between
prometheus
and
thanos,
and
it
really
just
converts
thanos
api
calls
grpc
calls
into
prometheus
native
api
calls,
and
this
is
already
a
really
really
useful
example
of
how
people
can
make
use
of
towns,
and
it's
a
very
common
thing
that
people
do.
A
And
so
this
is
why
why
I
often
refer
to
thomas
as
a
toolkit
right,
because
we
can
pick
and
choose
and
architect
our
monitoring
system,
to
exactly
the
way
that
our
organization
needs
and
only
to
the
extent
that
our
organization
needs,
because
we
could
build
an
arbitrarily,
complex
and
arbitrarily
featureful
monitoring
system.
But
it's
so
much
better
if
people
can
actually
just
have
the
complexity
that
they
truly
need
right,
and
so
I
think,
that's
what's
really
powerful
about
the
tunnels
project.
A
A
Long-Term
storage
in
tanos
always
revolves
around
object,
storage,
and
so
the
way
that
that
happens
is
essentially
that
these
side,
cars
that
I
talked
about
earlier
kind
of
converting
the
store
api
calls
to
prometheus
native
api
calls.
It
actually
has
a
second
kind
of
responsibility,
which
is
it
whenever
prometheus
produces
data
on
disk.
It
takes
that
data
and
uploads
it
to
object.
Storage,
now
pretty
much.
Every
object,
storage
provider
from
any
like
well-known
cloud
provider,
is
supported
by
thanos
and
many
of
the
kind
of
the
ones
that
you
can
run
yourself
as
well.
A
Basically,
anything
that's
amazon,
s3
compatible
works
and
then
the
way
that
you
can
actually
query
this
long-term
data
is
by
using
the
tanostore
component.
This
component
again
implements
the
tunnel
store
api,
but
instead
of
interfacing
with
a
prometheus
server,
it
reads
that
data
from
object,
storage
and
then
provides
it
to
the
courier.
Whenever
you
query
a
promptql
query
and
then
in
this
kind
of
scenario,
the
last
component
that
you
would
make
use
of
is
the
compactor
if
you're,
maybe
already
familiar
with
database
technologies.
A
So
that's
how
we
do
long-term
storage,
and
one
thing
that
I
forgot
to
mention
is
the
compactor
also
takes
care
of
down
sampling.
Remember
when
I
said
that
down
sampling
is
really
useful
for
querying
long-term
data.
Well,
this
down
sampling
actually
needs
to
be
computed
somewhere
right,
so
the
compactor
is
one
of
those
things.
What
is
the
component?
That
does.
Does
that
so
these
are.
A
You
can
start
with
just
the
querier
and
the
sidecars
right,
but
if
you
want
local
storage
well,
then
you
add
object,
storage,
the
store
component
and
the
compactor,
and
just
like
that,
you
can
kind
of
upgrade
your
monitoring
system
based
on
your
actual
organization's
needs
and
not
just
throw
a
bunch
of
processes
into
your
into
your
organization
that
you
may
not
even
need
right.
So
I
I
think
it's
always
important
to
kind
of
evaluate
what
your
organization
really
is.
A
Being
when
you
have
a
kind
of
service
relationship
with
your
prometheus
server,
so
you
may
have
totally
remote
prometheus
servers
that
may
be
on
edge
infrastructure
or
something
like
that,
and
you
want
to
push
that
data
as
opposed
to
having
all
the
tunnels
components
pull
data
whenever
it
wants
to
query
something
in
this
kind
of
scenario,
you
may
want
to
push
this
data
so
that
it's
available
at
a
low
latency,
essentially
and
what
we
have
for.
That
is
what
we
call
the
receive
component.
So
this
component
implements
the
remote
write
protocol
from
prometheus.
A
So
with
that,
this
is
kind
of
this
kind
of
concludes
the
architectures
that
I
wanted
to
present
today
as
an
introduction
to
tanos
and
just
to
reiterate-
and
I
hope
this
has
kind
of
become
clear
to
everyone.
Thomas
is
really
a
toolkit
that
you
can
use
to
build
exactly
the
monitoring
system
that
your
organization
needs,
and
you
can
pick
and
choose
from
all
these
wonderful
components
to
build
exactly
that
and
at
heart
the
thing
that
really
powers.
B
B
We
have
the
ability
to
concurrently
execute
select
queries,
so
this
would
really
be
helpful
for
the
queries
that,
for
the
complex
queries
that
you
have
with
multiple
select
statements
and,
moreover,
now
we
have
a
couple
of
like
multiple
layers
of
caches,
especially
using
memcache
d.
Using
this
functionality
for
the
stor
gateway
component,
we
can
actually
cache
the
metadata
files
and
chunks
itself,
and
this
kind
of
helps
us
to
reduce
the
traffic
and
latency
between
the
object
store
and
your
like
data
cluster.
B
B
One
of
the
cool
features
that
we
have
recently
implemented
is
the
enabling
this
store
filters
with
this
you
can
actually
select
which
store.
Do
you
want
to
query
to
debug
that
particular
story?
Api
component-
and
this
is
how
the
new
bucket
viewer
looks
like
you-
can
see
your
blocks
with
the
different
intervals
and
different
sizes
of
blocks,
and
you
can
see
their
metadata
information
and
you
can
actually
download
the
meta
json
file
itself.
B
If
you
want
to
further
debug
things
with
that,
let's
get
to
the
demo,
and
in
this
part
I'm
gonna
like
we
will
talk
about
a
simple
architecture
which
we
use:
sidecar
fan
of
sidecar
instead
of
prometheuses
and
for
each
for
our
each
cluster.
We
will
deploy
prometheus
and
we,
then
we
will
for
to
have
a
global
overview.
B
B
From
now
on
we
are.
We
will
deploy
the
thanos
query
for
like
global
overview
for
sake
of
demo.
We
are
also
deploying
a
siteguy
proxy
to
kind
of
inject
some
latency
to
the
querier,
because
we
don't
have
enough
data
to
actually
create
and
load
in
in
this
environment,
and
now
we
can
actually
access
our
query
content
and,
for
the
last
part,
let's
deploy
thanos
query
front-end,
for
that
we
have
a
configuration
for
the
cache
which
tells
us
to
actually
cache
everything
in
memory,
and
we
kind
of
deploy
it
now
and
let's
go
to.
B
Yes,
okay,
I
hope
you
can
see
that
this
is
a
thanos,
queer,
frontend
ui,
it's
actually
the
same
with
the
thundersquare
here.
So
when
we
execute,
let's
make
it
a
bit
bigger
and
when
we
actually
execute
a
query,
a
query
to
be
specific,
a
range
query:
it's
supposed
to
get
more
than
five
seconds,
because
we
also
yeah
injecting
some
latencies
yeah.
For
this
query,
it
takes
over
five
seconds,
but
now
it
should
be
cached,
and
then
we
execute
this
again.
B
B
C
C
So,
first
of
all,
we
kind
of
because
well,
thanks
of
the
of
the
fact
that
we
that
we
used
promote
use
code,
it
was
as
easy
as
upgrading
the
the
few
dependencies
to
get
shiny,
new
psdb
isolation
mechanism,
which
allows
well
kind
of
weapons
and
queries
to
be
kind
of
isolated
and
and
and
that's
pretty
sweet.
Furthermore,
we
are
active
on
the
analytic
api
side,
so
we
are
actually
with
the
sick,
observability.
So
a
special
interest
group
on
the
cncf
site.
C
We
are
collaborating
on
kind
of
exploring
the
use
cases
and
and
apis
that
an
integration
that
would
allow
us
to
better
leverage
metric
data
for
analytic
use
cases.
So,
for
example,
this
is
like
our
poc
opslytics
project,
which
allows
you
to
convert
prometheus
and
thanos
metrics
into
the
packet
file.
So
this
is
pretty
convenient.
We
are
planning
to
add
more
more,
you
know
apis
and
formats
like
apache
arrow
or
maybe
arrow
itself
and
and
pandas
and
stuff.
So
all
of
this.
C
Well,
if
you
have
any
ideas,
please
and
and
feedback
and
want
to
help
and
join
us
here,
yeah
please
visit
this.
This
repo
we
are,
we
are
your
help,
is
welcome.
One
thing
that
I
want
to
focus
as
the
last
thing
for
this
talk
is
kind
of
multi-tenancy
aspect,
because,
especially
when
building
centralized
monitoring
system
like
tunnels
with
long-term
storage
retention,
you
really
think
about
like
long-term
use
cases,
when
more
teams
will
use
the
same
system
and
you
know
how
they
can
collaborate.
C
Do
I
need
to
create
another
thanos
cluster,
just
for
a
separate
team.
What
if
you
are
a
sas
provider-
and
you
have
you-
know,
customers
that
are
not
part
of
the
same
custom
organization,
but
actually
you
know
they.
Their
data
have
to
be
securely
isolated
right.
We
actually
built
thanos
in
mind
with
that,
obviously,
with
the
multi-density
features
being
like
in
more
advanced
part
of
the
story
of
thanos,
but
well
here
we
are
after
third
year
of
of
thanos
products.
So
I
want.
B
C
And
and
kind
of
showcase
you
a
way
of
making
tunnels
a
multi-multi,
multi-tenant
system,
a
configuration
and
deployment
model
describing
that
so
let's
go
for
that.
So
I
would
describe
what
we'll
see
in
the
demo
so
first
we'll
introduce
kind
of
two
set
of
let's
say:
penance
prometheuses,
tenants,
data,
we
call
first
theme
fruit
and
second
vegetable
veggie.
So
you
can
see
they
have
separate
collection
path.
C
They
already
have
set
up
tunnels
with
sidecar,
so
they
can
query
their
own
data
and
because
you
have
the
separate
kind
of
infrastructure
for
both
team
for
each
team,
you
kind
of
expect
the
isolation
and
so
without
there's
no
multita
multi-tenant
infrastructure.
It's
rather
a
separate
infrastructure
for
each
tenant,
but
yeah.
Technically
it
works
right.
But
there
are
problems
with
that,
and
especially
with
bigger
bigger
systems
you
get
introduced,
I
I
we
call
it
tomato
problem
tomato
because
if
you're
aware
this
is
actually
both
fruit
and
vegetable
in
the
same
time.
C
C
Something
you
cannot
easily
achieve
with
this
with
this
particular
deployment.
Additionally,
what?
If
you
have
more
teams
right
and
how
do
you
scale
if
you
need
to
set
up
a
separate
cluster
or
separate?
You
know
kind
of
infrastructure
for
each
tenant
that
doesn't
scale
well,
so
ideally,
you
want
to
raise
more
and
have
some
multi-tenancy
system
and
thanos
was
definitely
thinking
about
that
use
case
for
a
long
time.
So
let's
do
first
step.
C
Obviously
we
can,
you
know,
join
this
data
within
single
global
view,
so
put
like
multi-time,
querier
and
and
allow
accessing
the
data
from
both
to
team
fruit
and
to
veggie.
However,
because
we
believe
in
tunnels,
in
the
unix
philosophy,
where
you
are,
you
know
doing
one
thing:
one
functionality
and
doing
it
good,
so
you
don't
want
to
spread
your
focus.
That's
why
there
is
no,
like
you
know,
direct
alf
or
airbag
system
built
into
the
tunnels.
C
However,
we
integrate
with-
and
we
kind
of
build
ecosystem
on
top
of
that,
so
to
achieve
this
red
hat
actually
built
prom
label
proxy
project,
which
is
part
of
the
product
use
community
org.
So
you
can
add
that
as
a
side,
car
and
this
properly
understand
promoters
api,
so
actually
thanos
apis
as
well
http
ones
for
for
querying
and
accessing
metadata
and
all
other
source
stuff
and
injects
the
proper
tenant
label
of
your
choice
into.
B
C
Query
system
to
ensure
the
data
isolation.
The
critical
path
here
is
kind
of
how
we
separate
tenants
between
themselves,
how
we
identify
them
and
we
do.
We
use
the
same
mechanism
for
as
for
anything
else.
As
for
series,
as
for
tsd
blocks,
we
use
labels,
so
tenant
is
just
another
label
for
us
and
you
will
see
that
on
the
demo,
so
things
of
chrome
label
proxy
and
some
authorization
proxy
of
your
choice,
because
we
don't
want
to
you
know,
force
you
to
use
odc
or
whatever.
C
C
For
this
we
I
also
use
katakura.
Hopefully
we
can
expose
that
after
kubecon
for
the
for
the
public
use.
So
let's
go
for
that
and
let's
start
with
just
starting
those
produces.
We
have
probabilities
for
team
fruit,
let's
copy
configuration
for
that
and
two
replicas
for
teamveggie,
let's
copy
those
files,
and
now,
let's
prepare
a
directory
for
for
our
prompt
users.
Let's
start
the
promptus.
C
It
might
take
some
time
actually
well.
We
started,
let's
create
a
sidecar
after
that,
let's
now
create
team,
veggie
promises
and
sidecar
and
another
pro
materials
and
side
car.
As
you
can
see,
I
can
I'm
using
just
docker
machines
for
for
for
simple
showcase,
and
let's
start
you
know
with
no
data
tenancy
model,
so
this
means
that
we
start
a
separate
querier
for
veggie
and
for
fruit
teams,
and
so
it
should
look
like
I
presented
before.
C
A
B
C
C
Problem
as
we
as
we
describe,
and
we
have
also
reused,
infra
so
problem,
so
let's
try
to
build
a
multi-tenant,
read
path
for
tunnels.
So
let's
stop
our
aquarius,
because
we
want
to
set
up
a
one
multi-talent
querier
that
will
be
safe
to
use.
Let's
start
that,
and
the
difference
here
is
that
we
just
point
to
all
the
stores,
so
all
the
sidecars
we
have
for
both
mv
veggie
and
team
and
team
fruit
without
anything
else
as
we
describe.
This
is
like
single
purpose
thing.
C
So
as
long
as
you
have
all
stores
store
apis,
so
two
side
cards
from
veggie
and
one
from
fruit,
we
will
have
all
the
data
right,
so
this
is
kind
of
our
admin
tomato
view.
Whatever
you
want
to
call
yeah,
we
see
all
the
data,
which
is
not
great
if
you
want
to
ensure
some
isolation.
So
let's
let's
actually
go
for
prom
label
proxy
project
right,
so
it
is
as
simple
as
yeah.
C
We
can
kind
of
isolate
tenants
based
on
that,
so
once
we
create
the
proxy,
let's
leverage
this
proxy
with
some
kind
of
alf,
and
I
choose
the
caddy
server,
which
is
like
a
fancy
engines,
and
there
is
some
configuration
very
simple
configuration
which
exposes
two
ports
39091
for
team
fruit.
So
in
this
part
it
will
just
append
the
the
parameter
to
the
url
portion
with
tenant
equal
steam
fruit,
and
there
is
a
second
part
which
will
inject
tenant,
costing
veggies
veggies.
C
So,
depending
on
what
part
you
are
using,
it
should
inject
the
correct
parameter
and
also
it
points
to
the
prom
label
proxy
that
understand
this
parameter
and
kind
of
knows
how
to
inject
that
into
query
into
other
critical,
prompt
use
apis.
So
once
we
start
that
we
should
have
a
query
for
fruit
team,
which
is
which
should
actually
give
us
yeah
only
team
fruit
data,
even
though
this
kind
of
query
have
access
to
all
of
the
stores,
right
and
kind
of
the
same
will
be
for
veggie
team.
C
C
Of
this
demo,
let's
go
back
to
slides
and
I
think
we
don't
have.
You
know
a
long
time
here
to
describe
all
of
this
and
hopefully
maybe
on
the
next
crypt
comes.
We
can
demo
further
multi-tenancy
parts
because
we
talked
about
you
know,
read
path
of
the
multi-dance
multi-tenant.
You
know
reits
isolation,
but
there
is
much
more.
We
already
solved.
You
know,
storage,
soft
tenancy
where
you
maybe
include
upload
order
blocks
all
the
data
into
the
same
bucket.
You
can
totally
do
that.
It's
still
multitenant.
C
You
can
have
separate
buckets
for
each
tenon,
that's
okay!
We
call
it
hard
tenancy
and
the
same
for
receiver.
We
can
we
build
receiver
with
multitenants
in
mind,
so
you
can
have
soft
tenancy
where
you
have
just
same
ingestion.
Receiver
notes
used
for
multiple
tenants
building.
You
know
separate
blocks
still
but
same
ingestion
path,
but
you
can
have
heart
tendency
where
you
distinct
injection
nodes
to
make
sure
it's
it's
just
much
safer
and
you
can
have
greater
sll
slo
on
that.
C
So
that's
it
from
the
multi-tenancy,
and
I
know
many
people
really
looking
forward
to
that
and
and
we
are
leveling
we
are
leveraging
that
at
redcot
as
well.
So
this
is
pretty
exciting
to
actually
yeah
prepare
a
demo
so
to
showcase
and
showcase
this,
for
you
all
right
and
last,
but
not
the
least,
we
are
not
stopping.
We
have
lots
of
lots
of
stuff
to
do
quickly,
mentioning
you
know
bucket
viewer.
We
want
to
contribute
up
to
problem
cues.
C
C
So
it's
pretty
amazing
and
yeah
some
different
cashback
and
supports,
and
I
want
to
also
mention
the
mentorship
like
we
spent
the
amazing
couple
of
months
of
monitoring
like
multiple
amazing
people
and
some
of
them
are,
you
know,
becoming
tunnels,
maintainers
and
or,
like
you
know,
helping
us
with
the
community
actually
starting
some
cncf
meetups
as
well
so
yeah.
If
you
want
to
mentor
someone
or
mentored
on
be
mentored
yeah,
please
be
yeah,
follow
our
twitter
and
and
promote
use
twitter
to
to
get
more
info
yeah.