►
From YouTube: Thanos Deep Dive: Inside a Distributed Monitoring System - Bartlomiej Plotka & Frederic Branczyk
Description
Speakers: Bartlomiej Plotka, Frederic Branczyk
Thanos is an open-source CNCF Sandbox project that builds upon Prometheus components to create a global-scale highly available monitoring system. It seamlessly extends Prometheus in a few simple steps and it is already used in production by dozens of companies that aim for high multi-cloud scale for metrics while keeping low maintenance cost. During this talk, Frederic Branczyk and Bartek Plotka, core maintainers of Thanos and Prometheus projects, will explain advanced concepts behind the Thanos project. This talk will cover: - Possible deployment models - Integration points with other systems - Important advanced features e.g discovery, multi-label HA, query load balancing - Example solutions for multi-tenancy, authentication and cross-cluster communication in Thanos. Join this session to learn about advanced concepts and operational models of Thanos!
A
Alright,
let's
get
started
after
some
technical
difficulties:
Thank
You
Lily
for
lending
us
your
laptop
okay,
so
welcome
to
the
Thomas
deep
dive
before
we
jump
into
the
presentation.
Let's
give
it
a
quick
show
of
hands
who
here
is
already
using
Thomas,
Wow,
very
cool
who
is
planning
on
using
tonneaus?
A
A
Hopefully
we
can
raise
your
confidence
in
this
system
or
make
your
about
running
it
already
or
hopefully
we
can
increase
your
confidence
about
what
you're
about
to
be
using
so
I'm
Frederick
I
work
at
Red,
Hat
I
work
on
pretty
much
everything
around
observability
at
Red,
Hat
I'm,
a
prometheus
maintainer
I'm,
a
tano's
maintainer,
that's
kind
of
why
I'm
here
and
I
also
lead
second
servant.
Asian
within
kubernetes
and
I
have
partek
with
me
today.
Yeah.
A
So
we
already
had
our
tano's
introduction,
but
let's
recap
a
couple
of
things
just
so
we
make
sure
we
are
on
the
same
page.
So
thomas
is
obviously
part
of
the
CN
CF.
That's
where
we're
here.
It's
a
CN,
CF
sandbox
project,
and
we
have
all
these
wonderful,
open-source,
repo
stats
that
you
can
read
here,
a
bunch
of
stars
and
so
on.
Obviously,
the
most
important
metrics
out
there
and
more
largely
we
operate
within
the
Prometheus
ecosystem
and
I
think
something
that's
really
important
to
internalize
about
tunnels.
Is
we
don't?
A
We
want
to
not
reinvent
the
wheel,
so
we
want
to
make
use
of
everything
that
is
already
existing
in
the
Prometheus
ecosystem,
and
if
it's
not
good
enough,
we
improve
that
as
opposed
to
reinventing
everything
new
yeah
and
something
that
I
personally
think
is
super
exciting,
which
is
completely
non
technological.
Is
we
recently
set
up
like
neutral
governance
for
the
project,
so
that
all
of
you
can
continue
to
enjoy
this
project
and
can
trust
that
no
one
company
will
ever
take
over
this
project?
A
So
I
think
that's
really
awesome,
but
yeah
not
now,
let's
jump
back
into
technical
things.
So,
let's
really
reiterate
on
a
couple
of
architectural
things
about
tano's,
so
that
we
can
deep
dive
into
some
of
the
cooler
things.
So
this
is
probably
one
of
the
most
common
setups
that
you
see
running
tunnels.
A
They
also
configure
the
sidecar
to
upload
this
data
that
Prometheus
periodically
flushes
to
disk
to
object,
storage
and
then
eventually,
you
also
not
just
have
this
be
a
backup,
but
you
actually
put
the
ton
of
store
gateway
in
front
and
actually
can
query
of
this
long-term
data.
So
whenever
we
do
this,
the
when
we
now
insert
a
courier
a
query
into
our
courier,
our
nice
expression
browser
right
that
we're
we
know
and
love
from
Prometheus
already
we
have
that
in
Thomas
as
well.
A
So
when
we
do
that,
the
courier
won't
only
fan
out
to
the
side
cars
to
get
the
live
data,
you
will
also
get
the
historic
data
that
is
loaded
from
object,
storage,
so
I
think
this
is
a
probably
a
very
typical
typical
use
case
that
we
see
in
the
evolution.
Some
people
don't
even
go
to
this
next
step,
and
what
we
see
here
is
an
additional
component
called
the
ruler
and
what
this
allows
you
to
do
is
to
do
alerting
and
rural
evaluation
on
a
global
level.
A
Most
of
the
time
alerting
on
the
leaf
prometheus
is
just
fine
and
you
should
always
continue
doing
that.
But
there
are
things
like
trend,
analysis
or
like
cloth
cluster
analysis
that
you
want
to
do
where
the
ruler
comes
in
handy,
so
you
can
also
make
use
of
all
the
rule
evaluation,
things
that
were
used
to
under
leaf
Prometheus's
and
then
going
even
further.
As
I
said.
All
of
these
components
all
expose
the
story
API
and
we
don't
stop
at
the
querier.
A
The
courier
also
exposes
this
thing,
which
I
think
is
really
cool,
because
now
this
allows
us
to
layer
the
system
in
a
hierarchical
manner
and
build
truly
global
systems.
So,
as
I
was
going
through
all
of
these
architectures,
if
you
paid
attention
and
I
hope,
everyone
did
I
know
it's
late.
But
what
was
common
in
all
of
these
architectures
is
one
really
crucial
component
and
this
is
the
store
API,
and
this
is
one
of
the
things
that
I
want
to
deep
dive
a
little
bit
more
into
today.
A
So
literally,
every
component
either
exposes
or
somehow
interacts
with
this
API
within
tonneaus,
so
really
the
sidecar
the
store,
the
ruler,
the
the
receive
component.
If
you
went
to
some
of
the
other
tunnels
talks,
there
were
a
couple
of
mentions
about.
This
is
still
an
experimental
component
and
you
should
totally
evaluate
whether
you
actually
need
that
component,
but
this
also
exposes
it
and
the
courier
itself
exposes
it.
A
So
there
was
a
research
group,
for
example,
that
implemented
an
adapter
for
the
that
kind
of
turns
the
store
api
calls
into
calls
against
open,
TS,
dB,
and
suddenly
people
could
query
all
this
data
that
they
had
historically
always
had
in
their
open,
TS,
DB
and
query
it
with
the
Prometheus
query:
language,
for
example.
That's
awesome.
We
never
even
thought
of
that.
So
I
think
this
is
one
of
those
things
where
having
such
a
common
language.
A
This
is
the
actual
raw
data,
where
the
courier
grabs
time
series
from
a
store
to
process
in
a
query,
and
then
the
other
two
are
not
quite
as
obvious
I
think,
but
once
you
know
what
it's
for
I
think
it
is
more
obvious
switches,
label
names
and
label
values
is,
whenever
you
type
an
expression
into
the
expression
browser
in
tano's
or
intermediate.
Even
this
originally
is
an
API
that
existed
or
exists
in
Prometheus
is,
and
this
is
for
autocomplete.
So
when
you
start
typing,
you
get
a
drop-down
in
the
expression
browser
that
tells
you.
A
These
are
all
the
metrics
that
are
available.
It
does
autocomplete
essentially,
and
that's.
This
is
how
it
works
internally.
So
let's
walk
through
the
entire
workflow,
essentially
from
the
courier
to
how
it
gets
all
of
this
data.
So
really
like
primitive
kind
of
setup
could
be
that
you
have
your
autonomous
courier
and
you
literally
just
say
this
is
the
address
for
my
store
that
I
want
you
to
talk
to,
and
this
is
perfectly
fine.
A
The
time
range
that
this
particular
store
can
serve
the
what
kind
of
store
it
is,
as
well
as
a
particular
set
of
labels
that
identify
all
the
data
in
that
in
that
store
and
why
this
is
important
is
because
now
we
can
make
true-
matches
of
our
queries.
So,
let's
see
how
that
works
and
what
that
means.
So,
whenever
we
do
a
query,
this
is
a
very
typical
query,
for
example,
that
you
see
in
Prometheus
world.
Everything
in
this
query
gives
us
hints
as
to
how
we
can
optimize
this
query
in
antennas.
A
So,
for
example,
we
have
the
metric
name
and
we
have
some
label
selectors
on
this
in
this
brewery,
and
so
this
label
selector
can
tell
us,
because
some
of
our
stores
are
exposing
this
labels,
this
particular
label
name.
So
we
can
make
sure
we
know
because
of
our
query.
We
are
selecting
region
us.
What
did
I
do?
A
There
are
really
typical
Innis
in
a
distributed
system
like
this
Cassandra,
for
example,
does
a
really
a
couple
of
really
cool
things
with
bloom
filters,
where
we
can
also
do
true
negative
matches,
so
we
know
that
we
never
need
to
fan
out
to
a
particular
set
of
notes.
So
those
are
things
that
we
want
to
explore
that
we
haven't.
Really.
There
is
still
a
bunch
of
experiments
that
we're
running
in
order
to
figure
out
is
fanning
out
to
a
store.
Actually
that
expensive,
with
a
with
an
empty
result.
A
So
those
are
some
of
the
questions
that
were
that
we're
still
exploring,
but
just
we
thought
in
a
deep
dive,
it's
kind
of
cool
to
share
like
the
thoughts
that
we're
going
through
as
we're
working
and
optimizing
the
system.
So
I
talked
about
this
all
of
this
in
the
sense
of
the
querier,
but
this
is
actually
an
abstraction
that
we
have
antennas,
that
we
call
it
a
proxy
store
and
why
I
bringing
this
up
is
because
this
is
not.
This
is
not
just
the
querier.
A
This
is
the
concept
that
we
can
now
reuse
and,
as
a
matter
of
fact,
I
won't
dive
dive
into
where
we
reuse
this
in
tonneaus,
because
actually
it
hasn't
been
implemented.
But
it's
being
discussed
but
I
think
it's
just
really
cool
that
we
have
these
abstractions,
that
we
can
talk
about
in
the
Tanners
project
now
and
we
can
think
about
a
more
abstract
level.
B
You
Frederick
before
we
start
I
would
like
to
ask
two
questions.
Can
you
please
raise
a
hand
if
you
are
now
storing
and
have
access
to
years,
metric
years
worth
of
metrics
in
your
monitoring
system?
Anyone
have
access
to
years
the
layers
of
the
data?
Okay,
okay,
so
it's
pretty
rare
right
because
it's
a
hard
problem
to
solve
and
now
please
raise
a
hand
if
you
would
like
to
have
an
access
for
the
years
of
the
data.
B
If
you
would
like
to
have
this
feature,
okay,
it's
like
half
of
the
of
the
room
and
actually
it's
it
makes
sense,
because
it's
super
useful
to
have
a
very
long
time,
metric
retention
for
your
metrics.
Now
this
is,
you
know
crucial,
because
you'd
like
to
have
you
know,
data
the
access
to
the
to
the
to
the
data
in
the
past,
and
you
could
have
you
know,
perform
long
term
characteristic,
detection
or
like.
Maybe
you
want
to
report
your
service
level
indicators
and
you
want
to
check
something
in
the
past.
B
Maybe
you
know
you
missed
some
some
detail
during
incident
response
or
something
like
that.
So
having
this
data
available
in
your
storage
is
the
key
and,
as
we
learned
yesterday
from
Lukas
and
Dominique
on
the
Eternals
intro
and
from
whether
fredericka
adjustment
ago,
tennis
is
designed
to
have
virtual
due
to
be
able
to
store,
in
the
query
virtually
unlimited
retention
of
your
metrics.
Now
in
this
part,
I
would
like
to
focus
and
spend
time
to
dive
into
challenges
that
you
might
have,
while
querying
here's
of
the
data
in
actually
any
Prometheus
based
system.
B
So
the
system
that
we'll
use
you
know
use
prom
QL
as
a
query
layer
and
all
starts
with
a
dynamic.
Where
were
the
favorite
resolution
right
well
think
about
graphing?
The
metrics
on
your
monitor,
for
example,
in
prefer
a
dashboard
or
prom
queue.
Sorta
know
see
why
having
dynamic
query
resolution
is
very
important.
This
is
because
you
know
the
data
and
the
storage.
B
There
are
probably
scraped
with
high
resolution,
like
relatively
high
resolution,
like
15
seconds,
something,
for
example,
rendering
all
of
those
samples
during
query
means
you
no
longer
time
ranges
like
one
month
suddenly
needs
to
display.
You
know
two
hundred
thousand
of
samples
and
that's
way
beyond.
You
know
number
of
pixels
we
have
in
our
monitors.
Even
if
someone
has
like
a
you
know,
4k
widescreen
match
screen
so
well.
How
prom
we
could
prom
QL
engine
solves
that
problem.
B
That
limitation,
right
and
proper
engine
is
something
that
many
many
projects
were
use
right
now
like
panels
from
to
use
cortex
even
right.
So
this
applies
to
all
of
them
for
each
query
range
request
from
QL
defined
certain
step
and
from
quell
then
evaluates
the
given
query:
every
defined
step,
independent
the
sample
stored
in
your
storage,
so
we
can
efficiently
present.
You
know
larger
time
range
we've
read
of
being
here,
accurate
enough,
so
how
it
works
in
practice.
B
In
this
example,
we
have
one
five
hour
time
range
query
which
we
render
where
we
render
like
250
samples
per
series
on
the
screen,
but
we
actually
work
on
1000
series.
You
know
on
the
storage
level
and
while
this
works-
and
you
can
have
dynamic
query
the
resolution-
there
are
some
limit
to
it,
notably
like
you
know,
maybe
when
we
extend
the
time
range
to
to
the
mouth
of
the
move
for
time
range.
Suddenly
we,
we
are
still
displaying
the
same
resolution
thanks
to
the
dynamic
step.
B
B
We
have
right
now,
because
for
that,
as
you
might
probably
from
here,
we
are
storing
samples
in
a
compressed
chunks,
which
is
essentially
a
gorilla
compression,
algorithm
structures
where
we
store
all
our
samples
in
this
reduces
by
the
size
of
the
overall
storage
like
11
times
in
average,
and
allows
us
to
save
disks
IO
and
network
bandwidth
in
terms
of
tano's
in
case
of
tano's.
But
there
is
trade
of
right
and
one
of
the
trade-off
is
that
the
compressing
takes
time
and
for
one
samples
it
can
be
no
up
to
40
nanoseconds.
B
So
this
means
like
that,
it's
actually
fair
latency
for
smaller
time,
rich,
arrange,
requests
times.
Shorter
time
range
queries.
However,
you
know
when
we
increase
that
when
you
start
to
query
like
month
of
the
data,
it's
actually
pretty
significant
amount
with
time.
So
you
know
within
if
we
increase
that
even
two.
C
B
Gets
to
extreme
because
we
suddenly
have
to
fetch
and
present
like
two
over
two
million
billions
to
billions
of
samples
which
already
takes
you
know
over
one
minute
of
the
time
of
the
latency
just
to
decompress
those
samples,
not
mentioning
index
lookup
and
other
latencies
as
well.
So
what
do
we
know
about
like
what
you
now
at
this
point?
First
of
all,
even
compress
data
for
one
year
is
is,
is
quite
heavy
on
the
disk.
B
B
This
is
where
tunnels
down
sampling
camps
very
handy,
and
this
is
why
tunnels
can
design
down
something
from
the
very
beginning
of
the
project
and
the
something
is
this
idea
of
reducing
resolution
or
reducing
the
resolution
of
the
picture
of
make
maybe
metric
data,
while
preserving
the
majority
of
the
accuracy
of
the
result
now
how
we
achieve
that
inside
tunnels?
So,
as
you
may
be,
familiar
channels
are
reused
at
the
same
TSD
before
matters
perfused
us,
so
everything
is
stored
in
the
t's
DB
blocks.
B
B
This
allows
to
overall
reduce
massively
the
chunk
size
thanks
to
the
lower
resolutions,
but
let's
look
on
how
we
do
this,
while
still
presently
at
preserving
accuracy
right
because
that's
important,
we
don't
want
to
lose
any
information
for
each
row
chunk.
We
have
to
compute
the
various
aggregations
right
so
technically
for
each
for
one
chunk,
we
have
five
smaller
aggregation
chunks,
so
we
have
count
which
you
know
have
number
of
samples
for
the
current
window.
We
have
some
which,
which
aggregates
the
sum
of
the
values
in
the
current
window.
We
have
minimum
maximum.
B
We
have
at
the
end.
We
have
also
a
counter
which
maintains
the
state
of
the
counter
and
make
sure
we
are
accounting
for
the
counter.
Resets.
Obviously,
as
well-
and
this
is
important-
because
this
will
be
used
against
queries-
that
requires
rate
functions
or
rate
based
offenses,
like
rate
increase
irate
now,
we
also
can
combine
two
of
those
aggregation
chunks
into
together
to
calculate,
for
example,
average.
B
So
that's
pretty
handy
as
well,
so
how
it
works
in
practice
right
we,
we
know
how
it
looks
on
the
storage
level,
how
essentially
from
ql
knows
what
chunk
to
use
and
if
you
stir
something
down
sample
to
resolution
at
all
right.
So
let's
take
this
example
from
ql
query,
where
we
have
like
write
a
rate
over
a
counter
of
the
others
for
step
10
seconds,
which
is
very,
very
high
resolution.
B
What
is
happening
is
that
prom
coil
ask
the
storage,
which
is
tano's
in
this
case,
for
example,
and
it
gives
very
a
lot
of
details
in
the
in
the
request.
We
have
you
know:
label
matures.
You
have
time
range
requested.
You
have
a
step
and
hints
for
for
the
functions
that
will
be
used
over
this
data,
so
because
this
is
high
resolution
that
the
requested
data
is
requested
to
be
in
high
resolution.
We
as
a
ton
of
weavers
or
we
are
respond
with
their
own
chunks.
B
Now,
let's
imagine
that
we
are
actually
extending
time
range
to
in.
Maybe
you
know
month
of
the
data.
Suddenly,
the
step
is
automatically
calculated
to
be
the
safe
half
an
hour
and
and
then
Thanos
has
to
decide
what
resolution
teachers
on
the
storage
level
and
it
does
it
by
asking-
can
I
fit
at
least
five
samples
within
this
within
this
interval,
and
the
answer
is
for
half
an
hour.
Yes,
we
can
use
five
minute
resolution
because
there
there
will
be
potentially
six
samples
within
that
window,
so
yeah.
This
is
how
this
tano's
decides.
B
How
to
what
resolution
to
use.
However,
as
you
remember,
we
have
like
five
those
aggregation
chunks,
so
this
is
where
the
hints
comes
handy
from
the
from
the
from
ql
request,
where
it
gives
us
it
advices.
What
function
will
be
used
in
this
case
right,
so
we
know
what
aggregation
should
be
used
on
the
storage
level
and
present
it
to
the
to
the
well
for
evaluation.
B
Now,
when
we
change,
for
example,
query
to
the
average
of
certain
gouge
for
active
others,
for
example,
then
you
know
hint
changes
as
well
average,
and
this
is
how
we
know
what
aggregation
chimes
to
use,
for
example,
salmon
count
in
this
case
now.
Okay,
at
this
point,
we
know
how
ton
of
them
samples
the
data,
how
he
chooses
what
our
chance
to
use
over
on
the
query
time.
But
let's
look
on
the
reservoir,
we
fix
what
we
fixed,
what
we
solved.
So
this
is
a
you
know.
B
If,
instead
of
the
high-resolution
data,
I
will
use
them
sample
data
for
the
month
or
one
year
queries
we
suddenly
can
achieve
much
much
more.
You
can
be
user-friendly
experience
right
and
and
and
lower
load
on
the
panel
system
and
performing
those
long
term
that
long
time
range
queries
are
much
much
known,
much
more,
cheaper
and
faster,
which
was
our
initial
goal.
Now
there
are
like
few
things
that
you
need
to
be
careful
when
using
down
sampling,
and
this
is
why
down
something
was
was
previously
released
as
a
experimental
feature.
B
First
of
all,
some
us
are
choosing
the
step
in
a
different
way.
For
example
for
me
to
use
you
I
and
tunnels,
you
I
are
having
semantics
of
specifying
you
know
interval
in
seconds.
However,
when
is
graph
Anna,
there
is
this
thing
called
resolution,
which
is
essentially
a
ratio
of
samples
per
pixel.
Secondly,
using
great
with
non
sampled
data
might
be
tricky,
so
people
used
to
have
just
raw
data
and
and
used
to
hard
code.
B
This
five-minute
orange
selection-
and
there
is
very
important
thing
about
it
because
to
forget
to
work
correctly,
you
need
to
have
at
least
two
samples
within
this
range
selector
and
now,
when
you
query
the
data
that
is,
maybe
you
have
lower
resolution
so,
for
example,
for
five
minutes,
you
might
have
been
a
one
sample.
It's
it's
in
this
case,
you
need
to
kind
of
increase.
This
range
select
range
selector
to
the
larger
values
for
this
function.
B
B
Have
you
know
all
those
order,
resolutions
available
in
your
system
which
allows
flexibility,
because
you
can
query
a
long
time
record
by
ranges
as
well
as
zooming
into
you,
know
what
happened
one
year
ago
now
there
is
a
trader
obviously-
and
the
trade
of
here
is
that
that
it
increases
the
size
of
the
store
data
in
the
object
storage.
However,
we
choose-
and
you
know
obvious
large,
because
it's
cheap,
so
this
is,
might
might
be
worth
to
consider.
B
It's
also
morph
to
mention
that
that
the
prom
to
you
second
system
is
looking
for
some
down
sampling
solution
right
and
and
oh
they
try
to
explore
if
it's
useful
and
if
yes
like
what
implementation
should
be
should
be,
it
should
be
done,
should
be
choosing
and
Isetta
knows
like
we,
we
choosing
some
implementation
and-
and
we
are
already
discussing
and
collaborating
with
other
system
to
make
this
than
something
a
standard
right.
Maybe
to
have
this
in
the
prompt
to
use
itself.
B
So
we
are
looking
forward
to
you,
know
collaborate
even
more
with
with
projects
like
cortex
or
MVB,
where
we
can
have
a
comment
on
something
logic
that
we
can
already
an
improvement.
Now,
that
was
for
done
something
right,
but
like
unrelated
now,
unlimited
retention,
like
obviously
there
are
some
consequences
on
like
you
need
to
scale
the
long-term
storage
width
as
well
right.
So
this
is
something
we
were
improving
over
the
last
month
by
an
eyeblink
like
a
couple
of
new
mechanisms
for
scaling,
the
store
gateway
essentially
and
the
store
gateway.
B
Is
this
component
that
is
deployed
on
the
objects
nicknamed
against
the
object,
storage
and
allows
you
to
browsing
Broz
those
altmetrics
and
overall
tours
great.
However,
with
large
huge
number
of
objects
in
the
object
search
you
you
can
expect
some
kind
of
slowdown,
so
you
need
to
be
able
to
have
like
a
scale
of
platform
for
this
case,
and
one
of
the
mechanism
here
is
time
partitioning.
B
So,
in
the
current
version
of
the
of
the
newest
version
of
the
terrace,
we
allowed,
to
you
know
kind
of
tell
the
store
gateway
to
be
responsible
only
for
a
subset
of
the
blocks
for
a
certain
time
range.
So,
with
this
configuration,
if
the
query
will
I
mean,
if
the
query
will
touch
the
certain
time
range,
it
will
go,
for
example,
one
year
or
like
half
a
year
ago
or
before
it
would
go
to
the
upper
store
gateway.
B
If,
if
it
will
be
a
for
afresh
data,
it
will
go
to
the
another
one
second
mechanism
to
to
scale
out
store.
Get
you
a
two
different
nodes
is
a
block.
Sharding
block
starting
is,
is
again
a
way
to
tell
store,
get
way
to
be
responsible
only
for
a
certain
blocks,
for
example,
for
blocks
from
a
certain
region
like
in
this
example.
B
Thanks
to
that,
and
actually
you
do
that
by
a
kind
of
relabel
in
configuration
that
is
exactly
the
same
as
rebuilding
doing
scrape
scraping
the
data
on
from
queue
size
or
service
discovery,
and,
and
you
essentially
tell
what
blocks
to
choose
based
on
external
label
as
well.
So,
thanks
of
that,
when
you
have
a
query
against
you
know,
maybe
certain
of
selecting
that
is
selecting
certain
you.
B
Overall,
let's
sum
up
what
we
learned
today,
we
explained
our
store
API,
and
you
know
what
role
it
has
in
our
system
and
why
it
keeps
flexibility
in
the
way
we
dive
into
down
sampling
and
and
why
it's
important
and
what
we
actually
are
solving
here
and
last
but
not
least,
we
touch
them
ways
to
further
scale
out
the
read
path
on
the
tunnels,
long-term
storage.
So,
with
this
being
said,
we
are.
Thank
you.
Thank
you
for
listening
to
us
and
yeah
we're
happy
to
take
some
questions
and
we
might
have
four
minutes
for
that.
B
D
Sounds
good,
so
I
had
a
question
about
the
read
hints
so
I
assume
you're
picking
apart
the
prompt
QL
and
you
see
certain
functions
and
you
like
a
list
identify
like
a
read
hint.
Does
that
mean
you
have
to
have
a
read
hint
hard-coded
in
Thanos
somewhere
for
every
bronchial
function
in
order
to
handle
it
efficiently,
yeah.
B
So
you
say
so,
but
there
was
only
like
a
six
of
those.
To
be
honest,
so
you
know
we
care
about
minimum
maximum.
Yes,
we
need
to
much
to
the
aggregations.
We
care
about
rate
based
functions
and
count,
and
some
so
there
is
like
not
many
of
those
and
kind
of.
Fortunately,
because
prom
coup
is
very
stable
in
term
of
functions.
This
works
well,
but
yes,
overall,
that's
the
idea.
Thank.
D
E
Thanks
for
great
I
had
a
question
about
how
to
layer
like
would
you
recommend,
like
what's
a
reasonable
number
of
meteors
instances,
to
have
one
tightness
behind,
and
when
do
you
want
to
sort
of
add
another
query
on
top
of
it
like?
Do
you
just
do
it
on
a
DC
and
if
you'll
have
every
service
in
there
or
Denis,
who
say
like
oh
well,
500
services
per
query
or
basic
visit,
yeah.
A
So
the
good
and
bad
answer
here
is
Thomas
is
almost
a
toolkit
right.
So
there's
no
one-size-fits-all.
So
that's
part
of
the
sad
answer,
because
there
there
is
no
one-size-fits-all,
but
it
also
means
that
you
can
optimize
exactly
for
your
use
case.
So
yeah
there's
no
there's
no
real
answer
to
this,
but
most
people
the
way
that
we
see
it
is
that
people
start
layering
or
people
start
with
a
sidecar.
They
use
the
Site
tour
as
the
upload
mechanism
and
it's
really
just
a
backup
right.
They
add
the
query.
A
A
To
add
a
new
prometheus,
for
example,
is
more
less
of
a
question
of
a
proton,
owes
more
how
large
of
a
machine
are
you
willing
to
pay
for,
for
that?
Prometheus
is
more
more
the
question
here,
because
at
the
end
of
the
day,
you
always
will
need
to
ingest
all
of
that
data
right
and
if
you
charted
across
multiple
Prometheus
instances
or
have
it
in
one,
doesn't
really
matter.
A
E
The
layering
of
panels
where
we
had
like
multiple
queries
like
you'll,
have
one
/,
DC
and
then
you'll
have
one
global
one
sort
of
so.
You
have
two
layers
of
thanas,
but
would
it
be
sort
of
should
you
have
good
in
the
DC?
Maybe
two
query:
query
instances
sort
of
that
takes
care
of
different
services,
yeah.
B
That'll
make
sense
so
yeah
you're,
essentially
asking
if
like
when
to
use
for
the
rated
queries
and
stuff
like
that
like.
Essentially,
there
is
no
recommendation
from
the
like
panel
side,
like
all
in
terms
of
efficiency.
There
is
not
much
of
the
difference,
however,
it's
just
you
know
the
user
experience
difference.
Maybe
you
want
to
expose,
you
know,
be
able
to
just
allow
users
to
query
for
just
one
cluster
site
or
or
like
something
that
is
a
that's
very
popular.
F
B
I
mean
yeah,
so
I
think
this
is
like
a
very
important
like
thing
to
remember,
like
down.
Sampling
is
meant
to
improve
your
query
time
right.
So
it's
not
compressing
here.
Your
data
that
much
like,
essentially,
yes,
technically,
the
raw
blog-
and
you
know
the
five
minute
resolution
will
probably
provide
from
into
position,
will
be
a
bit
smaller.
However,
it's
not
you
know
focused
on
being
like
compress
on
a
disk
like
we
were
optimizing
for
the
query
latency
for
a
long
time
ranges
so
yeah.
That's
why
you
should
not.
B
You
know
if
you,
if
you
just
want
to
choose
one
resolution
to
store,
maybe
even
a
roll
resolution
make
sense
more
because
people
tend
to
just
use.
You
know
users
tend
to
use,
you
know
just
use
down
sampling
and
just
remove
everything
else.
Just
okay,
we
have
one
year,
don't
sound
data
for
a
long
time.
Yes,
but
then
you
cannot
zoom
in.
However,
you
could
just
switch
this
and
essentially
have
a
maybe
use
exactly
the
same
space,
but
just
use
roadblocks,
and
maybe
this
will
fit
your
use
case,
smaller
surgery.
It.
A
You
yeah
I,
think
something
important
to
remember
here.
Is
that,
like
in
abling
down
something
actually
increases
the
size
of
the
data
that
you
store,
but
it
in
like
reduces
the
latency
of
your
curries,
and
if
you
don't
care
about
the
high-resolution
data,
then
you
can
have
a
low
resolution
on
that
and
keep
the
lower
resolution
data
for
longer,
but
that,
as
partic
said,
means
that
you
can
zoom
in
and
I
get
the
high
resolution
data
so
again
like
choose
what
suits
your
needs.
Yes,
here.
C
Just
a
question
with
down
sampling
and
and
querying
down
sampled
data
or
making
queries
that
include
down
sample
data
and
very
recent
data
that
might
be
sitting
in
a
journal.
Does
the
journal
inhibit
the
performance
of
a
lot
of
queries
or
do
you
include
the
journal
information
in
in
in
the
downsampled
output?
Or
do
you
exclude
it?
So
you
don't.
B
We
are
talking
about
one
year
of
their
of
their
time
ranges
month
for
the
time
ranges
where
wall
is
like
two
hours.
So
it's
not
a
significant.
You
know
kind
of
so
you
know
when
you
curl
one
year
and
maybe
yes,
two
days
or
like
you
know,
even
even
a
week,
it's
not
done
sampled
here
and
that's
true
for
the
fresh
data.
This
is
still
okay
like
we
are
talking
about.
You
know,
lower
resolution
for
majority
of
your
requite
of
your
time
range
right.
They
are
requesting
on.
G
This
is
regarding
down
sampling
again,
so
the
thing
is
like
you,
guys,
are
showing
the
certain
standard
aggregations
right
like
count
some
main
max.
How
would
it
actually
work
for
percentage
like
let's
say
certain
metrics
I?
Don't
care
about
average
I,
only
care
about
like
percentiles?
How
would
I
actually
down
sample
a
core
is
any
plans
for
you,
an
adding
support
for
percentile
aggregation
for
down
sampling
data,
so.
A
Percentile
or
I
I
assume
you
mean
hit
like
history
or
title
or
something
like
that
right.
So
we
would
do
the
calculation
for
that,
based
on
down
sampled
data,
as
vatic
showed
like
we
retain
the
counter
hit.
Remember
a
histogram
is
just
in
composition
of
multiple
counters,
and
so
we
down
sampled
those
counters,
and
then
we
can
still
do
the
histogram
quantile
aggregation
on
that
down
sampled
data.
It's
just
a
lot
less
data
that
we
have
to
process
at
this
point,
because
it's
downsampled,
okay,.