►
From YouTube: Intro to Thanos: Scale Your Prometheus Monitoring With Ease - Lucas Serven & Dominic Green
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro to Thanos: Scale Your Prometheus Monitoring With Ease - Lucas Serven, Red Hat & Dominic Green, Improbable
Thanos is an open-source CNCF Sandbox project that builds upon Prometheus components to create a global-scale highly available monitoring system. It seamlessly extends Prometheus in a few simple steps and it is already used in production by dozens of companies that aim for high multi-cloud scale for metrics while keeping low maintenance cost. During this talk, core maintainers of Thanos will explain basic concepts behind the project, its use cases, and tradeoffs. You will learn where to start and how to quickly deploy Thanos on Kubernetes without impacting your existing Prometheus setup. This talk is recommended for beginners that want to know more about running highly available Prometheus setup at scale with potentially unlimited metric retention with the lowest possible effort and cost.
https://sched.co/UaiW
A
B
A
We
got
fire
and
it's
not
system
outage.
It's
fire
warmth
light
it
could
even
be
prometheus.
However,
we're
going
to
refer
to
it
as
a
signal
here
today
and
we
can
put
it
on
top
of
a
watch
our
to
signal
something's
happen,
maybe
an
oncoming
attack.
We
can
then
use
these
watch
towers
to
help
defend
our
town,
I
work
for
a
games
company,
so
I
like
to
use
pixel,
art
and
games
when
we're
describing
staff
as
an
attacker
approaches,
we
can
light
the
signal
fire
to
tell
our
town
that
an
oncoming
attack
is
coming.
A
However,
as
we
grow
and
our
city
spreads
out,
we
need
more
towers
to
signal
and
an
attacks
coming,
so
we
build
out
more
towers
and
now,
when
an
attack
comes,
the
first
light
goes
up
to
signal
that
attacks
coming
it.
Then
it
propagates
back
to
tell
the
town
that
the
attack
is
coming.
However,
in
this
point
we
tend
to
lose
some
resolution
and
the
three
signals
at
the
bottom
have
now
just
become
one
here.
We
see
a
goblin
that
man's
the
middle
tower,
and
this
is
David,
however,
he's
quite
forgetful.
A
He
only
ever
remembers
the
past
three
days
of
when
attacks
have
happened.
This
isn't
great.
Also,
we
sometimes
have
sneaky
enemies
whereby
they
come
in
and
in
stealth
and
take
our
tower
and
we
don't
actually
know
an
attacks
coming
at
that
point
now.
This
can
be
thought
of
as
signals
being
passed
around
by
watchtowers
back
to
our
town,
or
it
could
also
be
thought
of,
as
Prometheus
recording
metrics
back
up
to
something
like
Ravana.
A
Now
all
these
problems
in
the
tower
defense
or
a
lot
like
we
could
have
with
this
Prometheus
setup,
I'm
Dominic,
Greene
I,
am
a
software
engineer,
improbable
I
work
in
a
number
of
open-source
projects
such
as
Thanos
goji,
OPC
middleware,
and
go
HTTP
wires
and
also
organized
a
number
of
meetups
in
London.
My.
C
Name
is
Lukas
Edwin
marine
I'm,
a
software
engineer
at
Red
Hat
by
way
of
core
OS
I'm
working
on
the
observability
team
on
all
things,
Thor
knows
Prometheus
and
prometheus
operator
and
without
further
ado
today,
we're
introducing
Thanos.
This
is
an
open
source
project
dedicated
to
providing
a
global
query
view
of
metrics,
providing
a
scalable
and
highly
available
metric
system
and
also
long
term
retention
with
unlimited
storage.
So
a
little
bit
about
the
Thanos
community.
C
C
Since
then,
it's
also
been
adopted
by
a
bunch
of
different
companies.
So
don't
just
take
Tom's
word
for
it
or
my
word
for
it
we're
running
it,
but
so
are
a
lot
of
other
people
and
today
we're
actually
going
to
start
our
discussion
not
with
Sano's
but
with
prometheus.
We're
gonna
specifically
look
at
some
key
features
of
how
Prometheus
works.
A
Okay
start
we
kind
of
want
to
have
a
shared
understanding.
Now
this
is
a
quote
from
the
Google
sre
book.
It
signifies
that
monitoring
is
that
are
collecting
aggregating
and
displaying
real
time
quantitative
data
about
a
system
so
Prometheus
and
by
extension,
Thanos,
are
monitoring
systems.
We've
built
Thanos
on
top
of
the
shoulders
of
giants,
so
we've
used
all
the
components
Prometheus
to
build
out
a
metric
system
in
a
distributed
way.
A
If
you
can
put
your
hands
up
if
you've
heard
of
Prometheus
before
fantastic
and
keep
your
hands
raised,
if
you
use
an
introduction,
amazing
and
anyone
using
thon
awesome
production-
oh
cool
nice,
a
lot
more
than
I
expected
so
with
Prometheus
on
your
client
side,
you
instrument
a
number
of
metrics.
You
expose
these
via
the
Prometheus
server,
which
is
an
HTTP
endpoint,
which
then
Prometheus
can
grab
these
metrics
from.
C
Cool
all
right,
so
toss
begin.
Let's
just
take
a
look
at
a
really
basic
prometheus
deployment
in
kubernetes,
where
we
have
a
number
of
micro
services
being
scraped
by
one
prometheus
server
and,
let's
think
about
what
happens,
for
example.
If,
for
any
number
of
reasons,
our
Prometheus
goes
down
what
happens
to
our
entire
metric
system
and,
let's
think
about
it
more
specifically,
what
were
to
happen
if
Prometheus
went
down
right
in
the
middle
of
an
incident,
we're
having
a
production
outage
and
right
when
we
need
metrics,
most
Prometheus
is
down
at
the
heart
of
it.
C
Obviously
one
of
the
results
of
this
we're
running
two
Promethea
we're
also
using
twice
resources,
CPU
memory
disk,
but
keep
this
in
mind
because
we're
gonna
come
back
to
this
set
up
a
few
more
times.
Another
limitation
of
Prometheus
today
is
a
storage
limitation,
so
by
default,
Prometheus
has,
as
we
all
know,
a
retention
of
15
days
right.
C
So,
if
I
ingest
some
metrics,
these
samples
are
kept
in
TS
DB
for
15
days,
and
the
question
remains
what,
if
I
want
to
query
data
from
several
months
ago
or
a
year
ago,
and,
moreover,
what
happens
if
something
happens
to
my
disk,
what
are
my
fallacies
and
got
scrubbed
if,
for
any
enum
reasons
and
I
lose
all
my
metrics
now?
What
do
I
do?
C
A
A
At
a
start,
you
can't
get
the
fine
granular
details
that
you
can
do
from
your
Prometheus
in
your
leaf
clusters,
and
so
at
times
your
engineers
end
up
going
from
the
main
cluster
at
the
top
out
into
the
the
leaf
clusters
to
try
and
get
more
detailed
information
and
then,
as
we
scale
up,
maybe
you
want
the
Federation
to
have
a
che
as
well.
This
point
we're
now
scraping
double
amount
data
from
the
clusters
at
the
bottom,
so
actually,
and
then,
when
they
go
hey
che
as
well
you're
doing
two
four
eight
x
amount
data.
A
Normally
you
have
Federation's
so
that
you
can
propagate
queries
and
use
it
with
your
profile
setup.
This
is
relatively
okay
with
three
clusters.
You
can
pull
the
data
into
memory.
It
works
fine
Federation.
What
effectively
does
is
pulls
all
the
data
from
those
leaf
fermitas
into
memory
to
then
be
served
again,
however
ones
when
we
scale
up.
If
we
have
then
n
number
of
clusters,
that's
all
that
memories
coming
into
that
federated,
federated
endpoint
I'm,
going
to
blow
up
that
instance.
If
we're
not
careful.
C
Right,
so
this
is
where
Thomas
comes
in.
The
whole
point
of
Thanos
is
to
try
to
combat
some
of
the
issues
that
we
see
when
we're
stretching
Prometheus
to
this
really
global
scale.
When
we're
scraping
from
hundreds
thousands
of
different
clusters,
the
question
is:
how
can
we
make
Prometheus
scalable
and
highly
available
so
at
its
core
Thanos
consists
of
a
number
of
composable
components
that
we
can
use
to
bring
scalability
H
a
long
term
retention
on
top
of
any
existing
Prometheus
setup.
C
With
that
in
mind,
the
goals
of
Thanos
is
number
one
to
be
Prometheus
compatible.
This
is
not
just
something
that's
convenience.
This
is
actually
strategic.
We
want
Thanos
to
be
built
on
top
of
all
the
great
design
principles
of
prometheus
to
leverage
all
of
the
knowledge
and
all
the
learnings
from
the
last
several
years
of
running
Prometheus
and
to
leverage
the
Prometheus
ecosystem
number
two.
We
want
to
have
a
global
view
of
metrics
a
query
time.
We
want
to
be
able
to
see
metrics
from
all
over
different
clusters.
C
We
want
to
have
long-term
retention,
so
we
want
to
be
able
to
see
historical
data
from
several
years
ago.
From
all
of
our
clusters,
but
also
we
want
to
be
able
to
see
data
down
to
the
individual
sample,
even
if
it's
from
like
two
years
ago-
and
that
brings
us
to
the
fourth
point,
which
is
we
want
to
provide
down
sampling
and
compaction
of
this
data.
So
we
can
make
our
queries
efficient.
C
So
let's
do
a
thought,
experiment
and,
let's
think
back
to
the
federated
Kriya
setup
that
we
were
just
discussing
earlier
at
the
heart
of
it.
One
of
the
issues
with
this
setup
is
the
fact
that
we're
ingesting
data
from
and
different
clusters
into,
one
single
one
single
instance,
and
this
can
bring
scalability
issues,
but
let's
just
think
for
a
minute.
What
could
we
do
if
we
invented
a
new
component
where,
instead
of
ingesting
data
from
every
single
cluster
at
the
leaf,
we
actually
query
data
at
runtime?
C
So,
for
example,
this
querying
service
receives
a
request
from
a
client.
It
proxies
it
down
to
the
leaf
clusters,
joins
the
responses
and
returns
that
to
the
client.
Let's
imagine
something
like
this:
suddenly,
our
service
for
querying
becomes
somewhat
more
scalable
right.
We
can
actually
were
not
ingesting
data
from
every
single
cluster,
so
we
are
just
dealing
with
queries
and
this
becomes
somewhat
tenable.
C
There
is
one
limitation
here.
One
big
problem
with
this
design
is
that
every
single
time
that
we
get
a
request,
we
have
to
proxy
it
to
every
single
cluster.
Imagine
that
we
are
querying
for
data
for
a
time
series.
It's
only
on
cluster
1
we're
nevertheless
gonna
proxy,
that
query
to
every
single
other
cluster
and
that's
really
inefficient
right.
C
So,
let's
imagine
for
a
minute
if
we
could
instead
add
yet
another
component,
and
what
this
did
was
a
side
card
that
was
sitting
next
to
Prometheus
and
just
announced
hey
these,
the
metrics
on
my
teas
to
be
on
this
local
cluster
now
Sano's
or
this
new
component.
We're
matching
this
query
component
can
know
ahead
of
time.
Hey
if
I
get
a
request
for
this
metric
I
know
which
cluster
I
have
to
hit
and
we're
no
longer
having
to
make
requests
for
every
every
single
other
cluster
and
we
can
make
requires
more
efficient.
C
We
can
optimize
this
a
little
bit
more.
If
you
recall
here
we're
exposing
to
NPA
two
API
endpoints,
one
for
the
Prometheus
query,
MPI
and
one
for
this
new
imaginary,
announcing
metrics
API.
If
we
combine
these
into
one
single
API
that
does
both
querying
and
announcing
labels,
we
can
also
take
advantage
of
this
moment
and
say
hey
rather
than
exposing
the
the
query
data
the
results
using
JSON.
C
What
did
we
did
it
using
protobuf
in
this
way
we
reduced
the
load
at
the
query
side
on
the
query
component,
because
we
no
longer
have
to
do
all
these
extra
allocations,
CPU
blah
blah,
and
we
can
also
reduce
our
bandwidth
usage.
So,
let's
expose
one
G,
RPC
API
that
does
both
querying
and
announcing
to
make
this
more
efficient.
A
So
at
this
point,
we're
effectively
designed
the
sanest
query
component
and
the
final
sidecar,
the
query
component
pans
out
and
calls
to
each
side
car,
which
then
does
a
remote
read
to
the
local
Prometheus,
exposing
the
information
about
the
data
that's
stored
on
that
node.
Now
this
communications
are
all
done
by
G
RPC
and
one
of
the
most
important
parts
of
tunnels,
in
my
opinion,
is
the
story.
P
I,
so
this
API
is
what's
exposed
is
a
well
as
a
subset
of
the
RPC.
So
it's
exposed
by
all
the
things
within
Thanos.
A
So
here
you
can
see
the
info
endpoint,
which
gives
us
the
metadata
about
the
time
and
the
kind
of
data
that's
on
that
node,
as
well
as
the
series
which
gives
back
a
streamed
response,
allowing
us
to
query
they'll,
send
that
data
more
efficiently
to
to
get
more
information
about
the
story.
Pi,
Bartok
and
Frederick
are
going
to
be
doing
a
deep
dive
tomorrow
so
head
over
to
that
and
to
really
dig
into
the
store
API.
A
So
as
it
turns
out,
we've
also
solved
the
h,
a
problem,
at
least
for
the
federated
instance.
Here
now
we
have
graph
on
a
calling
in
I
can
round-robin
between
to
query
components
in
that
monitoring.
Cluster
means
that
if
one
goes
down,
the
other
one
can
still
serve
traffic
and
it's
just
proxy
requests
through.
We
can
also
use
the
same
thing
to
do
hey
che
Promethea
in
the
leaf
clusters.
Here
we
can
see
that
the
story
pi
now
knows
about
many
of
the
Prometheus's.
A
However,
we've
just
got
the
same
problem
as
we
had
with
the
federation
earlier.
We
now
have
to
get
back
all
the
data
from
both
of
those
side,
cars
and
the
Prometheus,
and
within
combining
it
in
memory
in
the
courier
and
send
me
back
to
crew
fauna,
one
if
we
have
a
component
that
could
combine
multiple
streams
of
data
and
feed
it
back.
What
we
do,
that's
the
query
component.
So
what
we
can
effectively
do
now
is
put
a
query
component
down
in
those
leaf
clusters,
along
with
the
the
side
cars.
A
This
means
that
when,
when
we
call
into
the
query
er,
we
can
then
merge
the
two
streams
and
send
it
back.
As
I
said
before,
the
story,
pi
is
really
important.
In
turn.
Us
and
this
the
query
component,
along
with
a
sidecar,
implement
that
API,
so
we
can
just
switch
them
in
when
we
need
to
so
here
we've
made
a
lot
of
progress
towards
a
scaleable
meteor
system,
but
what
about
storage?
We
said
we
also
wanted
to
keep
the
data
for
longer
right.
C
So
we
already
know
of
one
really
obvious
solution
for
storage
and
that's
object.
Storage.
This
has
the
benefit
of
being
both
really
cheap
and,
on
the
other
hand,
super
flexible
and
scalable
right.
Most
cloud
providers
don't
require
you
to
pre,
allocate
some
amount
of
storage
for
your
object
storage,
so
we
can
actually
upload
blocks
as
we
need
and
the
storage
will
just
grow
with
us.
C
So
we
one
solution
that
we
can
do
is
say:
hey,
let's
make
our
tano
sidecar
upload,
TS
DB
blocks
as
Prometheus.
Produces
these
TS
DB
blogs
these
immutable
blocks.
We
could
have
the
sidecar
upload
them
to
object,
storage,
and
now
we
suddenly
have
a
way
to
keep
long-term
data.
So
let's
make
sure
that
you
know
if
the
storage
on
the
clusters
died,
we
don't
lose
any
of
the
blocks.
C
However,
one
of
the
things
that's
missing
here
is:
we
need
a
way
to
expose
the
long-term
store
data
to
the
rest
of
to
the
query
API.
So
one
thing
that
we
could
do
is
imagine
yet
a
new
component
that
is
a
gateway
to
to
this
data
stored
in
object
storage.
We
already
have
a
waiting.
We
already
have
an
API
that
we
defined
for
querying
and
announcing
the
metrics
that
are
stored
in
some
storage
and
that's
the
story
API.
C
Now
there
is
we've
solved
the
question
of
how
do
we
store
long
term
storage,
how
we
store
long
term
metrics,
and
also
how
do
we
query
for
long
term
metrics?
But
one
of
the
issues
that
comes
with
this
great
new
power
is
is
the
following.
Imagine
the
week
query
for
two
years
worth
of
data
of
some
time
series
that
sample
that
15-second
interval?
This
will
be
hundreds
of
thousands
of
data
points,
and
this
presents
a
number
of
problems.
C
Number
one
we're
doing
a
ton
of
disk
reads:
we're
using
a
ton
of
bandwidth
and
then
we're
returning
more
data
points
that
we
probably
have
pixels
on
our
dashboards.
So
we're
doing
all
this
work
for
no
reason,
one
thing
that
would
be
great
is
hey
if
we
query
for
a
ton
of
time
for
a
really
long
time
range.
Let's
only
use,
maybe
one
sample
every
hour,
so
we
want
to
do
is
down
sampling
of
our
data.
C
A
Cool,
so
we've
seen
how
we
started
out
with
a
federal
federal
rate
of
Prometheus
we've
gone
from
there
to
add
the
query
component
to
allow
us
to
proxy
traffic
out
to
multiple
side
cars,
the
side
car.
Then
it's
taking
those
requests
and
proxxon
it
to
a
Prometheus
with
the
remote
Reid
and
uploading
blocks
into
object.
Storage.
At
this
point,
we've
added
in
the
store
gateway
which
can
read
data
from
the
object,
storage
and
serve
it
back
using
the
same
story
pi
and
finally,
we've
had
the
compactor,
which
can
dance
sample
and
compress
data.
A
C
That's
unsafe
and
number
two,
even
for
us
that
would
be
really
cumbersome
because
we
have
to
manage
configuration
for
every
single
customer
cluster
as
it
goes
up
and
down.
So
this
this
model
just
simply
won't
work
for
us.
What
we
need
to
do
is
switch
from
a
model
where
we're
pulling
metrics
from
all
of
our
customers
clusters
to
one
where
the
customers
clusters
are
pushing
metrics
up
to
us.
Luckily,
such
a
model
already
exists
in
the
previous
ecosystem,
and
this
is
called
the
Prometheus
remote
right
API.
So
let's
do
one
more
thought.
C
One
thing,
obviously
we're
going
to
need
to
do
to
make
this
component
useful.
Is
it
we
need
to
expose
the
same
store
API
that
we
implemented
in
the
sidecar
and
the
query
and
the
store
gateway?
And
this
way
we
can
query
the
metrics
that
are
stored
in
the
local
API
or
in
the
local
TS
dB.
So
so
far
so
good.
We
now
no
longer
require
our
customers
to
open
up
a
port
right.
They
just
remote
write
their
own
checks
directly
to
us
and
we
no
longer
have
to
manage
configuration
about
information
about
all
the
customers.
C
Query,
endpoints.
So
so
far,
we've
made
a
lot
of
progress,
but,
as
we
start
scaling
this
up
from
a
few
clusters
to
n
clusters,
we're
gonna
run
into
an
issue.
This
is
looking
a
lot
like
the
same
Prometheus
Federation
setup
that
we
had
at
the
beginning,
where
we
have
n
clusters
of
thousands
of
clusters
that
are
all
writing
data
into
one
single
process.
That's
ingesting
it!
This
at
one
point,
is
going
to
hit
a
limit
and
it's
going
to
break
for
the
same
reasons.
So
we
need
to
do
something
to
make
the
system
scalable.
C
One
thing
the
one
way
that
we
can
solve
this
is
by
turning
our
new
component
into
a
hash
ring.
So
let's
say
we
make
this
component
a
scalable
hash
ring
that
we
can
grow
up
and
down
to
suit
our
needs.
You
know
as
load
grows
and
these
components
will
take
care
of
forwarding
requests
to
each
other
and
we
can
load
balanced,
all
incoming
requests
among
them.
C
One
detail
that
we
have
to
make
sure
when
we're
creating
this
hash
ring
is
that
we
want
to
make
sure
that,
on
all
all
samples
from
one
given
time,
series
always
end
up
in
the
same
replicas.
So
that
way,
the
time
series
is
is
a
is
coherent
in
one
single
TS
DB,
and
this
will
solve
our.
This
will
solve
the
scalability
question,
because
now
we
can
grow
this
hashing
as
we
need,
but
notably
this
doesn't
yet
solve
our
high
availability
issue.
So
imagine
that
one
of
the
replicas
in
this
hash
ring
goes
down.
C
This
means
that
all
time
series
that
are
destined
for
that
replica
will
no
longer
be
able
to
be
accepted.
So
clearly
we
need
to
do
something.
One
way
that
we
can
address
this
issue
is
by
implementing
replication
and
we
can
say,
for
example,
hey
all
incoming
time.
Series
have
to
be
replicated
to
three
different
replicas
and
unless
they're
accepted
by
two
replicas,
we
don't
mark
the
requests
as
successful.
Now
we
can
suddenly
tolerate
replicas
failures
and
we
can
adjust
the
replication
settings.
Obviously
we
can
say
replicate
to
five.
C
It
must
be
at
least
three
bla
bla
bla,
and
this
way
we
have
both
scalability
and
high
availability.
So
we've
addressed
on
two
of
the
concerns.
We
still
have
a
question
about
storage.
How
can
we
make
the
data
that's
stored
on
how
the
metrics
that
are
stored
in
these
new
components
that
are
receiving
metrics?
How
can
we
make
this
also
long-term
and
durable?
Luckily,
we've
already
addressed
storage
before
with
a
thumb
of
sidecar
and
what
we
can
do
is
say:
hey,
let's
make
the
same
coin:
upload
all
the
data
to
object,
storage,.
A
So
yeah
here
at
this
moment,
we
can
then
have
that
component
upload
its
blocks.
Much
like
the
sidecar
does
and
then
add
back
in
the
store
component
that
we
had
earlier.
So
this
store
gateway
will
proxy
traffic
and
the
query
can
only
query
the
hash
ring
where
it
can
query
out
to
object
storage.
This
sounds
like
it
would
work
so
now
we
can
use
that
to
get
the
historical
data
as
well,
and
this
effectively
is
a
Prometheus
remote
receiver.
So
this
allows
us
to
use
the
remote
write
to
store
data
and
upload
it
to
object.
A
A
A
So
the
last
thing
that
we
haven't
talked
about,
though
in
the
four
quadrants
we
saw
earlier,
was
the
fact
that
we
could
send
alerts
and
do
encoding
rules
at
some
point,
you
may
want
to
record
over
multiple
clusters
a
time
series,
for
instance,
the
maximum
number
of
players
in
a
game
server
across
multiple
clusters.
This
is
where
we
can
use
something:
a
lot
like
Prometheus
recording
rules,
so
we've
taken
those
and
put
them
into
the
runoff
rule.
A
However,
alerting
rules,
we
should
make
sure
we
push
right
down,
stack
more
close
to
where
we
actually
take
the
data.
So
at
this
point
again,
we
started
out
with
Prometheus
in
almost
like
a
federated
way.
We've
now
added
the
query
endpoint,
which
will
query
data
anywhere
we've
added
the
remote
receiver
that
we
can
do
remote
writes
to
again.
We've
got
the
object,
storage
satisfied
and
then
we've
added
a
rule
component
to
allow
us
to
add
multiple
metrics
from
clusters
together
or
send
alerts
to
alert
manager.
A
C
Right
so,
let's
take
one
last,
look
at
our
functional
diagram
for
Prometheus
and
see
how
this
diagram
changes
now
that
we're
adding
in
Thomas
to
the
mix.
So
the
first
thing
we
did
was
we
decoupled
querying
from
scraping.
We
saw
also
that
by
running
a
stateless
query
component,
we're
able
to
provide
a
highly
available
querying
service
next.
We
saw
that
by
exposing
metadata
about
what
time
series
are
stored,
where
we're
able
to
make
this
querying
more
efficient,
and
we
also
saw
that
we
can
provide
an
additional
way
to
ingest
metrics,
namely
with
remote
right.
C
We
saw
how
we
can
query
the
data
to
sort
an
object,
storage
using
the
store
gateway
and
also
how
we
can
make
this
query
more
efficient
using
a
compactor.
And
finally,
we
briefly
mentioned
how
we
can
use
the
Thanos
ruler
to
provide
a
learning
rule,
sorry
to
provide
recording
rules
for
the
data
that's
being
received
from
all
of
the
different
story.
Api's,
and
by
doing
all
of
this,
we
just
built
Thanos.
A
If
you
want
to
give
Stannis
to
try
yourself
and
we've
actually
uploaded
a
snorer
to
cata
Koda,
so
you
don't
actually
have
to
download
it
and
install
it
and
run
it.
You
can
go
to
this
website
and
use
the
online.
However,
it
is
relatively
easy
to
download
and
run
yourself
and
with
that.
Thank
you
very
much.
C
C
A
E
A
The
moment
correct
me
if
I'm
wrong,
I,
don't
think,
there's
anything
that
automatically
and
expired
storage.
For
you,
however,
because
it's
in
blob
storage
is
literally
just
folders
and
files
in
object
storage,
so
you
can
very
quickly
purge
the
their
old
ones
if
you
wanted
by
our
command
I
think
it's
something
that
the
team
are
looking
into
for
the
future,
but
it's
definitely
not
there
at
the
moment.
Right.
C
It's
also
really
easy
to
clean
up
blocks
that
are
stored
in
object,
storage,
because
the
way
that
the
Prometheus
he
has
DV
format
works
is
that
all
blocks
are
named
with
a
ul
ID.
That's
a
universally
LexA
graphically
sortable
ID
and
the
first
few
bits
of
it
are
a
date.
So
you
can
just
like
sort,
even
an
AWS
or
anywhere.
You
want
and
say,
hey
anything.
That's
before
this
name
essentially
like
this
lexographic
like
sorting
just
delete
it
or
something.
B
C
So
when
Prometheus,
when
Prometheus,
has
remote
writing
to
fanos,
this
is
actually
just
tailing
the
right
ahead
log.
So
it's
not
it's
not
like
remote
writing
the
data
that
is
already
written,
two
blocks
on
TTV.
It's
actually
remote
writing
what
it
has
in
the
right
ahead.
Log,
what
else
anything
any
red
or
any
right
remote
write,
requests
that
fail
will
get
buffered
in
memory
and
preemie.
This
will
keep
trying
to
retry
those
remote
write
requests
so.