►
From YouTube: Turn It Up to a Million: Ingesting Millions of Metrics with Thanos Receive - Lucas Servén Marín
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Turn It Up to a Million: Ingesting Millions of Metrics with Thanos Receive - Lucas Servén Marín, Red Hat
Thanos is an open-source CNCF Sandbox project that builds upon Prometheus components to create a global-scale and highly available monitoring system. In this talk, Lucas Servén presents a solution for creating a multi-tenant horizontally scalable metrics ingestion system using the newest addition to the Thanos toolset: the Thanos Receive component. The talk considers the motivations for building a system capable of ingesting metrics from thousands of clusters, including: multi-cluster monitoring and cluster telemetry. Lucas discusses how Thanos Receive is able to satisfy these requirements and how its hash ring design allows it to scale and maintain ingestion availability even during upgrades. Finally, the talk demonstrates the practice of running an automatically scalable hash ring by leveraging the Thanos Receive Controller, Horizontal Pod Autoscaler, and the Prometheus Adapter.
https://sched.co/ZejZ
A
Hi
everybody
welcome
to
turn
it
up
to
a
million
ingesting
millions
of
metrics
with
thanos
received.
I
hope
that
everyone's
doing
well
at
home-
and
I'm
happy
to
be
here
with
you,
so
I
originally
proposed
this
talk,
because
I'm
excited
to
share
some
of
the
work
that
we've
been
doing
inside
of
red
hat
on
the
observability
team
and
also
inside
of
the
thanos
project
around
ingesting
large
amounts
of
time
series
in
a
horizontally
scalable
way.
A
I
am
a
twin,
and
I
love
penguins
and
I'm
also
a
principal
software
engineer
at
red
hat,
where
I
work
on
the
observability
platform
team,
where
we
primarily
work
on
building
distributed
systems
for
monitoring
openshift,
I'm
lucky
that
at
work
get
to
contribute
a
lot
to
open
source
projects,
and
I
get
to
work
as
a
maintainer
of
thanos,
the
prometheus
operator
and
a
bunch
of
other
things,
and
you
can
find
me
on
github
as
at
squat
okay.
So
this
talk
assumes
some
familiarity
with
thanos.
A
So
I
recommend
that
you
take
a
look
at
the
talk
that
my
fellow
thanos
maintainers
are
giving
at
this
year's
scoop.
Con
called
intro
to
thanos,
we'll
also
talk
a
bit
about
it
right
now,
so
the
goal
of
the
thanos
project
is
to
provide
a
global
query
view
of
all
of
our
data
to
be
compatible
100
with
prometheus,
to
provide
long-term
retention
so
that
we
can
query
data
from
months
or
even
years
ago,
and
also
to
provide
compaction
down
sampling.
So
we
can
speed
up
our
queries.
A
A
We
have
one
central
monitoring
cluster
with
a
thanos
querier
and
maybe
a
grafana
next
to
it,
and
then
we
have
a
bunch
of
different
clusters
or
environments
with
prometheus
servers
that
we
want
to
query.
So
we
want
to
from
one
single
location,
query
the
data
from
all
these
different
prometheus
servers.
A
When
we
go
ahead
and
type
a
query
into
the
thanos,
ui
thanos
pulls
the
data
that
it
needs
from
the
prometheus
servers
by
making
a
grpc
request
to
the
thunderside
cars
in
each
cluster
and
the
thundersidecar
in
turn
proxy
that
requested
their
prometheus
api
all
right
so
far,
so
good.
But
what
happens
if
we
don't
have
ingress
access
to
the
clusters
that
we
want
to
query?
A
One
option
we
have
in
this
case
would
be
to
switch
from
a
pull-based
model
to
a
push-based
approach.
It
turns
out
that
prometheus
has
an
api
exactly
for
this
use
case.
It's
called
the
prometheus
remote
right
api
and
the
implementation
is
really
nice.
So
when
prometheus
is
configured
to
remote
right
to
an
endpoint,
it's
essentially
tailing
the
right
ahead
log
for
the
database
and
sending
all
of
the
time
series
to
this
remote
endpoint.
A
The
api
has
built
in
retry
logic,
request,
batching
and
back
offs,
and
a
lot
of
configurable
features
that
make
it
really
good
for
for
replicating
data.
So
the
thanos
receiver
essentially
answers
the
question
of.
How
can
we
build
a
thanos
component
to
accept
metrics
sent
over
the
prometheus
remote
right
api?
A
This
component
should
be
able
to
ingest
metrics
that
are
sent
from
lots
of
clients
into
one
local
tsdb.
So
far
so
good,
but
imagine
what
happens
when,
instead
of
having
two
or
three
or
four
or
just
a
few
clients,
we
go
to
having
an
arbitrary
amount
of
clients.
So
one
single
component,
ingesting
metrics
from
thousands
of
clients,
is
going
to
eventually
break.
We
need
a
way
to
make
this
component
scalable
and
that's
exactly
what
this
talk
is
about.
A
Okay,
so
how
do
we
make
the
thunder's
receiver
scale?
The
approach
that
we
took
was
to
design
the
receiver
as
a
hash
ring
of
many
replicas
collaborating
to
ingest
data
together.
So
when
prometheus
remote
writes
data
into
the
hash
ring,
the
request
gets
load,
balanced
and
initially
lands
on
one
of
the
replicas.
A
This
replica
hashes
the
received
time
series
and
their
labels,
and
it
finds
a
corresponding
replica
that
should
handle
this
request
and
it
forwards
the
request
to
it.
So
this
internal
forwarding
of
requests
is
what
allows
us
to
distribute
load
equally
across
all
of
the
replicas
and,
more
importantly,
it
also
allows
us
to
scale
the
ingestion
stack
horizontally.
A
If
we
know
that
one
of
the
hash
rings
has
to
serve
a
tenant
that
has
a
lot
of
clients
and
ingests
a
lot
of
data,
then
we
can
scale
that
hashing
independently,
for
example,
hash
ring
b
might
have
to
serve
thousands
of
clients.
It
needs
to
have
bigger
a
bigger
hash
ring
with
beefier
machines.
We
can
accommodate
that.
On
the
other
hand,
if
we
know
that
hash
ring
c
only
has
to
serve
a
few
different
tenants
that
are
sending
very
few
metrics,
then
we
can
scale
that
hashing
down.
A
But
what
happens
now
if
one
of
the
replicas
is
experiencing
problems
for
any
reason,
maybe
we're
rolling
out
a
new
version
or
we're
adding
more
memory
to
it.
Whenever
this
replica
is
down,
the
hash
ring
has
an
entire
set
of
hashes,
so
an
entire
set
of
time
series
and
labels
that
it
can
no
longer
forward
anywhere.
A
That
means
that,
for
example,
if
we
choose
to
replicate
our
data
always
three
times
when
we
receive
a
request,
the
initial
replica
will
forward
this
request
to
three
different
nodes
in
the
hash
ring
and
as
long
as
two
of
the
nodes
respond
with
yes,
I
was
able
to
write
this
data.
Then
we
return
back
to
the
client,
a
200,
and
we
say
this
was
written
successfully
into
our
tsdb.
A
If
we
want
to
tolerate
having
more
nodes
at
any
given
time,
all
we
have
to
do
is
increase
the
replication
factor.
So,
for
example,
if
we
change
the
replication
factor
from
three
to
five
replicas,
then
we
can
tolerate.
Having
two
nodes
down
at
any
given
time
now,
we've
designed
our
thunders
receiver
to
be
both
scalable
and
highly
available.
We're
able
to
ingest
data
from
thousands
of
clients
into
our
tsdb,
but
since
we're
ingesting
so
much
data
into
our
tsdb
storage
is
likely
going
to
become
an
issue
on
these
nodes.
A
We're
going
to
be
creating
so
many
tsdb
blocks
with
so
many
samples
and
time
series
that
it's
going
to
occupy
a
lot
of
space,
so
storage
is
going
to
become
a
bottleneck.
The
way
that
we
solve
this
inside
of
thanos
is
by
uploading
all
completed.
Tsdb
blocks
into
object,
storage,
object,
storage
is
cheap
and
essentially
infinitely
scalable.
So
we
can
upload
all
the
data
we
want
there.
We
later
make
this
data
queryable
by
using
a
component
called
the
thanos
store
or
the
thunderstor
gateway.
A
A
Until
now,
we've
seen
the
purpose
that
the
thunders
receive
component
has
inside
of
the
thanos
project.
We've
also
seen
from
a
high
level
how
its
hashing
design
allows
us
to
run
it
as
a
scalable
and
highly
available
metrics
ingestion
system.
But
how
do
we
actually
run
such
a
system
and
how
do
we
automate
it
so
that
it
scales
horizontally
in
a
platform
like
kubernetes?
Let's
take
a
look
at
the
details
we
mentioned
earlier
that
the
thunder's
received
component
can
be
configured
into
hash
rings.
A
Today,
this
configuration
works
by
using
a
json
file
to
explicitly
map
the
addresses
of
the
different
replicas
to
distinct
hash
rings.
Unfortunately,
json
is
not
a
very
human-friendly
language
and
we
definitely
don't
want
to
have
to
write
json
by
hand
every
single
time
that
we
need
to
scale
our
hash
ring
up
or
down.
However,
thanos
needs
this
configuration
file
so
that
the
replicas
can
discover
each
other,
because
thanos
doesn't
know
anything
about
kubernetes
or
any
other
cloud
or
container
orchestration
system.
A
Fortunately,
we
have
an
effective
pattern
for
solving
these
types
of
problems
in
the
kubernetes
community,
namely
controllers
and
operators.
At
red
hat,
we've
written
a
small
project
called
the
thunder's
received
controller
for
the
sole
purpose
of
automating.
The
configuration
of
our
hash
rings.
This
kubernetes
controller
works
by
watching
all
the
staple
sets
that
match
a
specific
label
selector.
Whenever
the
replica
count
for
one
of
these
staple
sets
containing
thumbs,
receivers
changes
the
controller
regenerates,
the
hash
ring
configuration
json
file
and
updates
a
configmap
holding
it.
A
The
thumbnails
receive
replicas
in
turn,
watch
the
file
system
for
changes
to
their
configuration.
So
when
the
kubelet
updates
the
volume
out
for
the
configuration
file,
the
thunders
receive
replicas
reload
and
update
their
in-process
representation
of
the
hash
rings
to
reflect
the
new
state
of
the
world.
A
A
A
Now,
let's
try
to
do
exactly
this
in
a
quick
demo.
Let's
deploy
this
super
simple,
automated
thought
and
receive
hash
ring
onto
a
kubernetes
cluster.
We'll
then
run
one
process
next
to
this,
that
produces
metrics
in
the
remote
write
api
format
and
sends
them
to
the
hash
ring
for
ingestion
by
the
way,
all
of
the
commands
instructions
and
manifest
that
I'm
using
these
demos,
for
my
talk
can
be
found
on
my
github
in
a
repo
called
kubecon
eu
2020.
A
What
we're
going
to
do
is
deploy
authors,
receive
hash
ring
and
also
a
thunderous
receive
controller
along
with
it
to
configure
the
hash
ring.
So,
let's
begin
by
creating
the
namespace
for
our
cluster,
this
is
going
to
be
the
founder's
namespace
and
now
we're
going
to
create
all
the
manifest
that
we
have
in
the
directory
for
our
first
demo.
A
All
right
great
now,
let's
take
a
look
at
what
manifests
we
actually
deployed.
We
have
a
bunch
of
manifest
corresponding
to
the
thunder's,
receive
controller
itself
and
then
some
manifest
corresponding
at
the
bottom
to
the
thunder,
receive
hash
ring
for
our
default
hash
ring.
Here
we
have
a
staple
set
with
one
single
replica.
So
when
we
take
a
look
at
the
pods,
we
have
one
single
thunder's
received
pod
in
the
default
hash
ring
and
we
also
have
the
funnels
receive
controller
pi.
A
A
A
I
think
that
these
kind
of
easy
wins
are
oftentimes
underrated,
in
fact,
inside
a
red
hat,
we're
running
essentially
exactly
this
same
stack
as
part
of
the
openshift
monitoring
platform.
One
of
the
services
that
we
run
is
a
thunders,
receive
hash
stream,
with
six
replicas
that
ingest
metrics
from
on
the
order
of
ten
thousand
openshift
clusters.
A
Of
course,
since
we're
the
observability
team,
we
closely
monitor
all
of
the
services
that
we
run.
So
I
was
pretty
easily
able
to
get
some
statistics
around
the
amount
of
data
that
we're
ingesting
as
of
late
july.
When
I
recorded
this
talk,
we're
regularly
handling
around
10
million
active
time
series
and
we're
ingesting
on
the
order
of
30
000
samples
per
second.
A
A
This
is
possible
because
we
have
a
good
idea
of
how
many
openshift
clusters
there
was
going
to
be
in
the
wild,
and
we
know
exactly
which
time
series
these
clusters
will
be
sending
and
how
often,
but
nevertheless,
every
few
months
we
need
to
resize
our
cluster
resize.
The
hash
ring
in
order
to
accommodate
the
increased
load,
which
means
that
we
have
unnecessary
toilet
for
engineers,
and
this
problem
is
even
more
pronounced
for
workloads
that
are
spikier
in
nature
than
ours
are.
A
So
imagine
that
the
amount
of
time
series
and
samples
that
a
hashrate
needs
to
ingest
are
varying
wildly
from
peak
to
trough.
This
kind
of
volatility
means
that
it's
going
to
be
really
difficult
to
appropriately
size,
an
ingestion
service,
so
without
good
automation.
In
these
cases,
the
best
option
that
we
have
is
to
over
provision.
A
In
the
case
of
the
thunders
receive
controller,
we
can
use
the
horizontal
pod,
auto
scaler,
to
scale
the
hash
ring
up
when
the
hash
ring
is
swamped
and
using
too
much
cpu,
and
then
we
can
use
the
thunder's
receipt
controller
to
automatically
regenerate
the
configuration
for
this
hash
ring
and
then
have
everything
working
for
us
just
out
of
the
box.
Let's
try
running
this
exact
configuration
of
an
auto
scaling
thunders
receive
hash
ring
in
a
new
demo.
A
This
is
going
to
look
a
lot
like
the
first
demo,
where
we
had
the
thunder
receive
controller
generating
the
configuration
for
authorized
hashing,
but
with
the
added
change
that
the
hash
ring
is
going
to
scale
automatically
when
we're
increasing
the
load
on
it.
Using
the
horizontal
pod
autoscaler,
the
horizontal
pod
autoscaler
in
turn,
is
going
to
depend
on
the
availability
of
the
resource
metrics
api
in
the
cluster.
A
This
hash
ring
a
little
bit
more
to
its
limit
and
see
how
it
stabilizes
and
how
it
performs
all
right,
let's
begin
again
by
creating
the
thunder's
namespace
and
then
we're
going
to
deploy
all
the
resources
in
the
directory
for
our
second
demo,
and
you
can
see
a
lot
of
these
look
like
before.
But
let's
take
a
look
at
the
few
things
that
are
different
so,
namely,
let's
look
at
the
thunder
receive
staple
set,
and
one
of
the
things
we
can
see
here
is
that
we're
deploying
a
new
sidecar
inside
of
the
thunder
received
pod.
A
This
is
a
side
car
called
configmap
to
disk,
and
the
purpose
of
this
is
to
automatically
synchronize
the
configmap.
That's
in
the
api
to
a
phylon
disk.
Instead
of
avoiding
the
long
resync
loop
that
it
might
take
when
the
kubelet
has
to
update
the
volume
mount
this,
sometimes
when
a
big
amount
changes
in
the
api
can
take
a
long
time
for
it
to
be
reflected
on
disk,
so
here
we're
just
trying
to
have
the
hashing.json
configuration
file
update
a
little
bit
faster.
A
The
next
thing-
that's
different
in
this
demo,
is
that
we're
deploying
a
horizontal
pod,
autoscaler
resource,
and
here
we're
setting
the
minimum
replica
count
to
five
replicas.
This
means
that
we're
always
going
to
have
at
least
some
ingestion
capacity
in
our
hash
ring,
and
that
means
at
a
minimum
five
replicas.
The
other
thing
is
notable
here
is
that
we're
scaling
based
on
the
average
cpu
utilization
of
the
hash
ring.
So
when
the
average
utilization
of
all
the
pods
in
the
hash
ring
reaches
50
of
the
cpu
that
they
requested,
then
we
scale
up.
A
A
All
right,
this
looks
good,
so,
let's
run
the
second
terminal
of
process
is
just
watching
the
horizontal
pod,
auto
scaling
resource.
So
we
can
see
the
decisions
that
it's
going
to
be
making
in
real
time
and
now
in
one
last
terminal,
we'll
run
a
process
to
generate
load
against
our
hash
ring,
in
this
case
we're
going
to
be
using
fresh
traxxas
avalanche
project,
which
is
a
binary
that
generates
remote.
Write,
requests
against
some
endpoint
that
you
choose.
A
Another
thing
to
note
here
is
that
we're
going
to
be
deploying
a
thousand
metrics
each
with
100
different
time
series
pretty
soon
the
horizontal,
auto
scaler
scaled,
the
hash
ring
up
to
seven
nodes,
and
when
this
occurred,
the
process
is
generating
the
load
against
our
hash
ring
started
logging.
A
lot
of
errors,
this
means
that
our
hash
ring
is
in
some
kind
of
unstable
state.
A
Soon,
after
that,
the
pot
auto
scaler
scales
once
again,
the
hash
ring
up
to
the
next
10
minutes
of
this.
The
next
10
minutes
of
the
demo
are
pretty
uneventful.
The
hash
ring
is
super
stable
at
nine
pods
and
we're
not
generating
any
errors.
So,
let's
fast
forward
a
little
bit
in
the
top
screen,
we're
now
running
cube,
ctl
top
to
see
in
real
time
how
much
cpu
and
memory
each
of
the
thunder
c
pods
is
using.
A
One
thing
that
we
can
see
is
that
whenever
the
process
that
generates
load
changes,
the
time
series
that
it's
sending
the
characteristics
of
the
cpu
utilization
change
as
well.
The
reason
for
this
is
because
every
single
time
that
we
change
the
time
stages
that
we're
sending
we
also
change,
which
final
thunder
receive
replicas.
These
hash
rings
end
up
on.
So
it
means
that
the
characteristics
of
which
pods
receive
how
many
metrics
and
which
pods
have
to
forward
metrics,
to
which
other
pods
changes.
A
So
we
might
in
sometimes
have
a
very
smooth
distribution
of
metrics
and
other
times
we
might
have
very
lumpy
distribution
of
metrics,
where
a
lot
of
time
series
get
clumped
up
on
one
node
now
at
around
23
minutes
the
avalanche
process
reconfigures
itself,
and
it
seems
that
the
distribution
of
time
series
in
this
case
is
causing
a
really
uneven
distribution
of
metrics
and
increasing
the
cpu
utilization
of
the
hash
ring.
This
is
causing
the
hashing
to
become
unstable
and
is
going
to
require
a
scaling
event.
A
A
Around
the
31
minute
mark
the
hash
ring
finally
stabilizes
and
the
cpu
utilization
will
begin
to
drop
now.
The
separation
will
keep
dropping
so
much
that
the
horizontal
power
auto
scaler
elects
to
scale
the
hash
ring
all
the
way
down
to
12
replicas,
where
the
hash
ring
seems
to
be
pretty
stable
and
at
some
point
we
just
quit
the
process,
because
this
has
been.
This
has
given
us
already
a
lot
to
think
about
compared
to
the
first
exercise
where
everything
was
rainbows.
This
demo
didn't
quite
go
as
smoothly
as
we
would
have
hoped.
A
Even
though
these
experiments
don't
go
100
smoothly,
they
provide
a
super
good
insight
that
helps
us
understand
the
systems
that
we're
designing.
In
the
case
of
the
thunders
receiver,
we
can
begin
to
ask
questions
like.
Why
is
this
hash
ring
behaving
so
erratically,
or
is
it
so
unstable
one
of
the
ways
we
can
imagine
the
system
is
that
we
can
describe
it
as
being
an
unstable
equilibrium.
A
In
fact,
this
chaotic
state
that
we're
landing
in
is
some
kind
of
positive
feedback
where
we're
generating
more
instability,
the
more
unstable
that
we
become
to
help
understand
the
instability
of
the
system.
Let's
think
about
some
of
the
instances
where
we
had
long
spans
of
time
with
a
lot
of
errors.
So
one
of
the
notable
instances
was
when
the
horizontal
pod
auto
scaler
scaled,
the
hash
ring
up
to
20
pods.
A
A
A
Another
issue
that
we
spoke
about
earlier
is
that
the
hashing
configuration
loading
can
take
a
really
long
time
and
is
non-deterministic,
so
some
of
the
pods
might
load
it
before
others.
This
means
that
we
can
have
a
split
brain
in
our
hash
ring
where
some
of
the
pods
have
the
old
hashing
configuration
and
some
of
the
pods
have
the
new
hashing
configuration.
The
way
that
we
chose
to
address
this
was
to
have
a
side
card
that
automatically
loads.
The
new
hashing
configuration
whenever
the
configma
updates
in
the
api.
A
However,
this
has
some
other
downsides
to
it,
namely
when
all
of
the
pods
update
the
configuration
at
exactly
the
same
time.
There
is
an
instant
of
time
when
none
of
the
pods
are
available.
This
is
because,
when
the
hashing
configuration
changes,
the
thanos
receiver
takes
itself
offline
temporarily
when
it
flushes
its
tsdb.
A
A
However,
the
downside
of
this
is:
we
have
a
longer
amount
of
time
where
we
have
this
kind
of
split
brain
effect,
where
some
of
the
pods
in
the
beginning
have
already
up
to
the
configuration
and
the
pause
at
the
end
haven't
at
the
same
time,
whenever
the
pods
are
updating,
they're
still
going
to
be
getting
traffic
forwarded
from
the
other
pause
in
the
hash
ring.
This
is
because
the
final
receive
controller
doesn't
consider
the
readiness
or
unreadiness
of
a
pod
when
it
produces
the
endpoints
array.
A
Even
though
unready
pause
are
taken
out
of
the
load
balancing
group
for
requests
they're
coming
at
a
service
ip,
these
pods
can
still
get
requests
forwarded
directly
to
them
from
pods
in
the
hash
ring.
So
when
a
pod
is
temporarily
crash,
looping-
or
maybe
it's
temporarily
offline,
because
it's
flushing
its
tsb,
the
other
pods
in
the
hash
ring-
are
still
going
to
be
making
grpc
requests
to
it
and
causing
extra
load
on
this
pod
and
causing
more
trouble
for
this
pod.
A
That's
already
struggling
one
way
we
can
address
this
issue
is
by
using
back
off
logic
and
in
fact
we
actually
already
have
a
pr
in
the
thanos
repo
for
implementing
exactly
this.
Maybe
by
the
time
you
see
this
talk
it's
already
merged.
A
further
improvement
would
maybe
be
for
the
thunder
receive
controller
to
consider
pod
readiness
when
generating
the
endpoints
array.
Finally,
one
big
issue
that
we
have
in
thumbnails
receive
is
the
way
that
we
forward
requests
in
the
hash
ring.
So
normally
remote.
A
When
there's
enough
time
series
in
the
remote
write
requests,
statistically
it's
very
likely
that
some
time
series
will
be
destined
for
every
node
in
the
hash
ring.
So
we
might
have
one
incoming
request
and
it
can
result
in
15
outgoing
requests
inside
of
the
hash
ring,
even
though
the
final
receiver
batches
requests
that
are
destined
for
the
same
node
all
together,
we
still
end
up
having
a
lot
of
traffic
inside
of
the
hash
ring
for
one
single
request.
A
This
means
that
each
incoming
request
is
multiplied
by
the
number
of
nodes
in
the
hash
ring,
although
each
request
is
proportionally
smaller,
minus
some
gains
that
you
would
have
had
from
compression.
So
the
larger
the
hash
ring,
the
more
intra-hashing
traffic
is
generated
for
each
request.
Right
now,
we're
in
the
process
of
investigating
some
different
solutions
for
this
and
we're
specifically
looking
into
having
tiered
layers
of
hash
rings
kind
of
the
same
way
that
the
alert
manager
project
has
tiered
routing
trees.
A
So,
what's
the
deal
with
the
thunder's
receiver,
is
it
stable
or
is
it
not
stable?
Can
we
use
the
hashing
stuff?
The
answer
is
that
yes,
but
it's
complicated,
as
we
saw
earlier,
when
we
have
a
stable
sized
hash
ring,
we
can
happily
ingest
all
the
data
that
we
send
to
it,
but
it's
when
we
have
a
scaling
event
that
we
trigger
some
instability
that
leads
to
more
instability.
A
This
means
that
using
automated
tools
like
the
horizontal
pod
autoscaler
can
lead
to
unpredictable
results.
Yes,
it'll,
probably
stabilize
at
the
end,
but
we're
not
exactly
sure
we
can
partially
mitigate
some
of
these
uncertainties
by
implementing
policies
scaling
policies
in
the
horizontal
autoscaler,
which
is
something
that's
new
in
the
v2
scaling
api.
A
Another
thing
we
can
do
is
we
can
start
to
work
on
a
lot
of
the
improvements
that
address
the
problems
we
identified
earlier.
One
of
the
things
that
we
could
do
just
off
the
bat
is
eliminate
the
need
for
downtime
whenever
a
hash
ring
change
occurs,
as
we
mentioned
earlier,
whenever
the
hash
ring
is
reconfigured,
the
thunder
received
pods
temporarily
take
themselves
offline,
while
they
flush
our
tsdb.
A
Another
thing
we
could
do
would
be
to
use
consistent
hashing.
So
one
of
the
problems
that
we
see
when
we
have
the
split
brain
effect
on
our
hash
ring
is
that
one
group
of
pods
that
receives
a
request,
things
of
the
time
series
contained
in
it,
should
be
forwarded
to
another
set
of
pods
with
a
different
view
of
the
hash
ring.
These
paws,
on
the
other
hand,
think
that
the
time
series
should
go
to
the
first
set
of
paws
and
they
forward
to
each
other.
A
Infinitely
using
consistent
hashing
will
allow
us
to
partially
mitigate
this,
because
we
could
reduce
how
much
keys
get
redistributed
when
a
new
node
is
added
to
the
hash
ring
all
right.
So
that
gives
us
a
lot
of
ideas
for
ways
that
we
can
improve
the
thanos
project.
So
hopefully
some
of
you
feel
inspired
to
contribute
and
I
hope
to
see
you
all
on
github
thanks
a
lot
for
watching.
That's
what
I
have
for
you
today
and
I'll
stick
around
for
questions.