►
From YouTube: High Available + Scalable Prometheus with Thanos in Alibaba - Guo'an Qin, Alibaba & Tao Li, Alibaba
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
High Available + Scalable Prometheus with Thanos in Alibaba - Guo'an Qin, Alibaba & Tao Li, Alibaba
Alibaba Group is using Kubernetes to support the world's largest e-commerce business. With the respect of the availability and scalability, how to provide reliable fine-grained monitoring and alerting services is a indeed challenge. In this talk, we'll share the experiences in developing a fine-grained monitoring system with high availability and scalability based on the open source project Prometheus and Thanos. This system mainly supports Alibaba's cluster management system, which has 4 million TPS and 10K requests per-second. We will have a discussion in following topics. 1) How to support a large-scale scenarios using Prometheus? 2) How to solve data query problem caused by multiple Prometheus instance with low query latency using Thanos? 3) The lessons we learnt from Prometheus and Thanos's configuration, such as target discovery and management of recording rule and alerting rule.
https://sched.co/NroK
B
A
A
A
A
B
A
A
A
A
A
Data
to
solve
our
problems,
you
may
see
that
all
those
you
may
know
that
problems
are
larger
scale,
so
we
have
those
problems
on
some
version
of
and
the
time
series
database
have
these
kind
of
abrogates
capability
and
it
can
be
stored
in
the
database.
It
not
only
includes,
but
also
include
event.
Okay,
so
does
qubit
nature's
system.
Metrics
are
also
installed.
B
A
C
B
B
A
B
B
A
B
B
B
B
B
A
There's
no
accident,
never
we'd
start
to
the
system
and
jiggle.
Second
is
high
performance,
that's
also
a
importunity
principle,
and
we,
if
we
have
like
a
policy
engine
that
needs
to
use
our
data
to
adjust
much
and
we
need
to
have
this
high
performance,
local
query
capability
and
a
future
prediction
probability
and
to
have
this
kind
of
little.
Real-Time
query
this
kind
of
future
prediction
and
the
third.
A
B
B
A
C
Collect
the
data
from
different
data
source
and
they
are
four
components
underway:
first,
a
meter
which
is
mainly
responsible
for
okay.
So
many
of
the
data
into
a
new
one,
for
example,
to
encounter
to
I
will
get
into
a
gauge
one.
For
example,
you
have
many
containers,
you
want
to
do
an
average
of
them.
We
can
do
them
in.
This
is
mainly
used
for
our
centralized
storage,
so
focus
of
promises.
So
our
sanctuary
stories
can
get
the
data
easily
and
the
query
API
can
just
check.
B
C
C
B
C
Later
on
and
they
can
be
defined,
export,
for
example,
I
have
got
20
of
that
CPU
and
only
2
to
3
of
them
can
be
used
today.
So
these
are
the
two
parts.
One
is
our
main
process,
and
second,
is
that
plugins
how
to
use
so
when
the
main
person
knows
to
wizard
plugins
and
the
plugins
can
be
started,
and
once
they
have
finished
eliciting
and
is
plucking
switches
to
be
closed,
you
can
imagine
that
every
time
and
to
be
used
and
once
I
have
operated
this
function,
and
this
can
be
closed.
B
C
C
C
C
B
C
C
B
C
Of
the
implementations
that
we
are
using
and
lastly
I'd
like
to
appreciate
our
open
source
committee
and
also
all
our
engineers
contribute
to
this
open
source
community.
So
since
we
can
achieve
this
kind
of
fast,
developing
and
aggregation,
because
we
have
understanding
a
very
high
standing
point,
we
have
never
Easter
advantage.
We
have
used.
B
B
C
In
these
lines,
I
would
like
to
briefly
introduce
that
many
of
you
today,
you
shouldn't
have
data
locally,
so
what
is
the
meaning
of
it,
and
the
discards
can
briefly
introduce
that
actually
enjoy
how
to
adjust
a
queue
as
to
guarantee
the
stability
of
the
business.
I
can
see
that
actually
police
the
response.
Time
of
these
businesses.
B
C
B
B
B
B
B
C
C
From
different
clusters
and
scheduled
a
different
resource
for
them-
and
this
is
useful
suitable
for
the
scenario
that
we
have
thousands
of
machines
and
this
kind
of
working
model-
don't
need
to
go
to
the
public
network
and
can
reduce
the
unnecessary
overhead
in
a
network.
Also,
this
kind
of
deployment
master
can
satisfy
the
demand
of
many
SMEs,
but
in
a
scenario
of
Alibaba,
we
have
several
dozens
of
clusters
and
we
have
our
hundred
thousands
of
the
parts.
So
we
cannot
maintain
it
through
this
way
and
also
very
big
problem
of.
C
B
C
B
C
A
pro
meters
for
every
application,
so
if
your
application
is
also
enlarging,
then
this
message
is
not
because
remember:
we
have
our
tens
of
thousands
of
applications,
so
this
method
cannot
be
used
in
this
scenario
also
for
promises.
In
order
to
solve
this
kind
of
general
view
problem,
they
have
used
a
Federation
and
they
can't
just.
B
C
So
if
you
want
to
see
the
requests
of
every
second
per
second
in
Emory
machines,
and
we
can
just
collect
the
data
all
of
the
data
in
this
machine
in
this
room
and
also
in
the
core
machine
room,
we
can
just
deploy
promises
and
to
gather
data
after
the
application.
So
Senna's
can
check
the
relevant
data
from
the
general
promises
and
to
get
the
sum
and
to
know
the
requests
per
second
of.
C
B
A
The
tax
and
the
cost
will
be
high
under
there
will
be
some
reliability
issue.
So
when
we
were
doing
this,
so
we
were
talking
about,
we
need
to
lower
the
dependence
on
the
outside
and
if
we
introduce
another
Prometheus,
then
there's
another
point
that
might
have
a
failure.
So
if
we
transfer
data
on
the
public
websites,
their
risks.
C
B
A
The
data
you're
poor
might
running
behind
and
you
might
do
so
data
point
instance.
If
you
use
public
network
and
it
ran
out
of
telling
the
data
might
be
lost
and
when,
if
it's
just
a
demo,
you
can
tolerate
this
kind
of
error.
But
if
you
pour
out
all
the
data,
the
arrow
will
be
magnified
and
this
is
not
acceptable.
B
A
A
Room
we
have
like
tens
of
thousands
of
like
this
kind
of
similar
structures,
so
the
issue
would
be
quite
complicated.
So
if
we
want
to
achieve
this
kind
of
like
high
availability,
you
can
see
that
in
our
government
in
every
minute
we
cannot
accept
it.
This
kind
of
like
status
because
we
need
to
we
need
a
higher
availability.
A
The
biggest
issue
here
is
that
for
decay,
each
case
that
you
run
is
independent.
There
will
be
a
latency
in
each
case
and
there
will
be
minor
differences
if
we
use
this
kind
of
like
a
liberal
view,
and
to
have
this
kind
of
a
data
Priory,
there
will
be
some
difference,
maybe
well.
For
instance,
if
you
refresh
the
page,
it
turned
out
to
be
different
and
we
need
to
align
this
kind
of
like
Prometheus
somewhere
to
connect
the
different
information.
Yes,
send
the
require
or
send
the
query
to
seyn
prometheus.
A
But
this
also
brings
another
problem,
so
this
will
waste
the
resources
of
other
Prometheus
and
order
requiring
can
only
go
to
one
Prometheus
and
other
Prometheus
is
only
responsible
for
pulling
out
data
and
the
resources
are
not
fully
utilized.
So
Prometheus
actually
has
a
lot
of
tools
that
can
be
used
to
hold
scalability,
but
it
doesn't
have
a
lot
of
problems
and
in
a
community
would
you
have
solutions?
So
in
the
end
we
decided
to
use.
Venice
Venice
actually
have
these
kind
of
a
compatible
HDI.
A
B
A
B
A
A
B
A
Of
the
instability
of
Federation
and
these
actually
effectively
soft
problems
of
availability,
and
it
can
help
us
to
get
all
the
data
and
I
also
want
to
add
that
when
you,
when
we
use
Prometheus,
there
was
one
problem
that,
if
you
have
like
kind
of
like
a
service,
needs
to
work
across
like
different
segments
and
the
functions
that
we
had
could
not
work.
In
that
case
of
scenario.
But
fanners
could
work
and
that's
kind
of
a
global
view,
deployment
that
we
have
so
in
our
mechanical
room.
A
We
have
these
higher
e
deployment
and
it
connects
to
two
different
choir
resistance
and
forms.
This
entire
system
and
order
data
will
be
sent
from
the
main
cluster
to
the
sub
clusters
in
different
mechanical
brooms
and
will
be
returned
to
you.
The
client
and
this
actually
helped
you
address
alibaba's
issue
in
most
of
the
cases
and
Sana's
could
help
us
to
support
our
data
storage
requirements.
B
A
Some
other
works
that
we
did
and
customer
target
discovery.
So
you
can
see
that
because,
for
instance,
we
have
not
enough
data
storage,
some
other
issues,
so
we
need
craziest
just
stop
automatically
pulling
our
data
so
for
cube
inators
and
this
kind
of
like
discovery
system
so
actually
centers
could
help
us
to
address
the
issue
it
supports
kubernetes
and
in
Alibaba,
because
our
business
and
our
other
historical
issues,
because
of
all
those
elements.
So
we
have
these
Cuban
ADEs
and
our
other
discovery
system.
So
we
developed
this
kind
of
like
component.
A
The
custom
eight
target
discovery,
so
it
can
verify
order
data.
It
will
work
with
the
same
format
same
as
Prometheus
and
Prometheus
can
then
answer
our
requirements
accordingly
and
also
configuration
management.
So
if
it's
just
one
or
two
Prometheus
you
can
manager
to
manually,
but
we
have
so
many
Prometheus
and
also
we
have
this
kind
of
like
change
of
configurations
of
primitives
and
there's
no
centralization
tool
for
Prometheus,
so
we
actually
developed
a
tool
or
self.
A
So
we
have
these
kind
of
like
central
management
and
users
could
actually
send
this
kind
of
a
change
of
configuration
and
we
use
sidecar
to
put
in
on
the
side
of
the
Prometheus,
and
it
will
synchronize
all
the
data
and
the
pooted
locally
and
prometheus
can
load
the
data
when
you
send
it
request,
so
it
actually
hold
us
to
address
the
issue.
Last
part
is
thinking
about
the
future.
So
when
we
use
it's
a
nose
and
a
Prometheus
of
all
like
it's
been
a
while
now,
so
we
have
been
thinking
that
what
are
the
enhancement?
A
What
would
what
are
things
of
like
they
can
improve?
So
we
think,
like
the
most
important
thing,
is
authentication,
because
security
is
a
priority
and
it
is
quite
important
for
us
to
use
this
kind
of
a
goth
education
process
and
help
us
to
authenticate
who
are
the
users
that
we
have
kinda
to
have
these
authorized
request
and
that
she
authorized
them
to
only
use
part
of
the
data
and
have
these
kind
of
like
a
targeted
management
methods
and
to
have
a
more
precise
targeting
and
a
flow
control
and
a
second.we
about
your
query.
B
A
That
we
need
you
for
the
time
knots
and
every
data
we
need
to
return
the
data
and
then
it's
a
story
locally.
So
we
can.
We
go
back
to
the
data
quickly
and
third
base.
Query
safeguards,
I,
think
Sana's
and
Prometheus
both
have
this
kind
of
function,
but
we
need
to
do
more
precise
management.
We
want.
B
A
To
be
integrated
with
Apocrypha
link,
so
the
calculating
capability
of
us
can
be
enhanced,
so
we
can
integrate
with
a
link
and
support
more
calculating
capabilities.
There's
a
problem
with
Prometheus
ASAT,
it's
a
single
like
it's
a
single
host
and
then,
when
you
run
it,
they
will
have
some
button
eggs
in
terms
of
the
calculating.
So
this
is
all
we
want
to
share.
Thank
you
very
much.
A
I,
do
you
have
a
question
yeah
because
we
have
monitoring.
So
what
is
one
of
those
strategies
where
you
monitor
the
cold
and
warm
and
hard
data,
and
what
is
the
differences
between
online
and
offline
data
so
for
the
first
row
question?
So
actually
what?
What
is
to
the
strategies
that
we,
what
a
strategist
that
we
use
actually
for
for
the
data
that
we
have.
A
B
A
It's
just
not
like
one-time
data
loading
data.
If
it's
like
long
time
away
from
this
OMB
we
have
compactable
sample
and
the
retention,
then
we
will
delete
the
data,
so
actually
the
strategies
are
quite
complicated.
So
if
you
have
specific
requests
a
we
can
talk
later
in
details
because
I
cannot
summarize
it
in
one
or
two
sentences.
The
second
question:
I
can't
remember
it.
Could
you
please
repeat
it
so.
The
second
question
is.
B
B
B
B
C
A
B
A
In
I
noticed
that
you
didn't
use
object,
storage
by
furnace,
so
you
actually
locally
store
the
data
by
Prometheus,
so
for
every
data
center.
How
do
you
store
the
data
right
like
by
segments
or
I
mean?
How
is
that
deployment
or
kubernetes?
Oh,
is
it
on
the
cuban
ATIS
physical
virtual,
physical
virtual.