►
From YouTube: Sharded And Federated Prometheus Servers to Monitor Distributed Databases - Jun Li & Viswa Vutharkar
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Sharded And Federated Prometheus Servers to Monitor Distributed Databases - Jun Li & Viswa Vutharkar, eBay
At eBay we have developed a geo-distributed transactional document store called NuData. It is deployed on Kubernetes. The current deployment has thousands of pods across three datacenters, and is monitored by Prometheus. For scalability, our Prometheus cluster has sharded servers to monitor individual infrastructure components and federation servers to retrieve aggregated metrics from sharded servers. For high availability, each sharded/federated server is configured with an active/standby pair over its load-balancer. A routing map ( a time series) is automatically constructed by each shard server and assembled by the federation server, to direct Prometheus queries to the right servers. Today we have over 90 Prometheus servers on two datacenters to collect over 11 M metrics/60 seconds on 400 metrics (and 900 rules), to support health monitoring and performance debugging of NuData.
To learn more: https://sched.co/GrXO
A
So
over
the
last
three
years
we
have
developed
geode
each
with
a
today's
in
eBay
called
new
data,
and
today
we
have
deploy
the
new
data
in
eBay,
the
internal
data
data
centers
with
thousands
of
posts
and
over
the
community's
infrastructure.
So
we
use
the
parameters
2.3
from
metric
monetary
and
we
use
this
metric
monitoring
system
to
help
us
so
for
the
system
operation
and
continue
the
system
development.
So
just
to
give
you
a
brief
overview,
this
system
at
the
high
level.
So
we
had
this
current
library
that
is
a
attach
to
the
database
application.
A
They
will
rather
request
to
the
service
tier
that
service
tier
then
end
up
into
the
packet
data
based
in
the
dead
bag
and
a
rapist.
We
had
the
proxy
to
process
the
the
query
request
and
then
the
request
persist
and
then
resources
to
the
back-end
storage
engine,
and
it
also
can
create
a
indices
where
necessary
and
push
the
index
to
the
gobo
secondary
in
the
store.
A
So
just
give
you
a
little
bit
detail
about
these
database
framework
in
order
to
set
the
context
for
the
rest
of
presentation,
so
the
customer
can
create
a
multiple
key
spaces
and
each
key
space
is
considered.
Multiple
sha
and
each
jar
consider
multiple
replicas.
The
replicas
can
be.
The
master
can
be
the
second
le
and
can
be
the
co
hidden
and
hidden
is
I
say
for
the
purpose
of
Baker
in
recovery,
and
so
the
older
replicas
in
within
a
single
sha
is
scattered
across
multiple
data
center.
A
So
in
this
database
brain
work,
we
capture
throughput
latency,
Ella,
the
queuing,
the
saturation
and
in
the
states
the
state
can
be
live,
Master,
replica
and
master.
Secondly,
or
can
be
the
permit
ease
up
and
down,
and
we
actually
learned
Kubrick
to
expose
these
OS
metrics
and
that
kubera
is
hosted
in
each
of
the
host
machine.
Also,
we
develop
and
install
the
custom
metric
exporter
to
capture
the,
for
example,
I
or
thesis
metrics.
There
is
a
much
richer
compared
to
the
default
cube
red
os
matrix
with
respect
to
the
this
operation,
and
so
in.
A
Overall,
we
capture
about
20
million
matrix
instances
per
square
interval
in
the
script
in
to
go
today.
We
define
assess
one
minute
so,
each
day
in
each
data
center
we
capture
about
195
gigabytes
of
the
fanciest
data
and
we
had
the
two
data
center
to
host
this
a
promises
cluster
and
we
had
little
time
to
set
the
seven
days
today.
A
So
the
integration
of
the
Prometheus
with
the
Cuban
net
is
a
cioppino,
not
a
nice
feature.
For
example,
you
can
ultimately
script
the
target
in
you
know
in
a
cluster.
So
whenever
you
had
a
civil,
bring
it
up
the
automatic
s
Square,
and
then
we
also
can
filter
the
server
the
target
based
on
the
auto
discover
the
labels,
then
also
we
can
have
indeed
automatically
the
matrix
labels
into
the
matrix
based
on
the
label
they
specify
in
the
prospect
so
now
with
the
label.
Y,
nothing
that
we
can
describe
is
a
hierarchy.
A
So
there's
two
kinds:
hierarchy
that
we
describe:
one
is
the
physical
hierarchy,
so
you
can
describe
the
pot,
the
host,
the
wreck,
the
data
center
and
then
a
logical
hierarchy
is
the
one
that
I
described
to
you.
That
you
can
have
a
replica,
you
can
have
a
shower
another
key
space.
So
all
these
labels
is
automated
in
jetty
into
the
matrix,
a
based
on
the
post
met.
So
we
don't
need
to
write
the
code
to
autumn
to
manually
inject
these
labels.
A
So
with
the
with
the
labels
to
describe
the
hierarchy,
then
now
we
can
aggregate
matches
across
the
hierarchy.
So
this
is
key.
We
example
on
the
body
be
processed
total,
which
is
a
counter
and
we
discounted
a.
We
can
compute
the
transition
rate
at
the
replicas
level.
Then
we
can
sum
the
transition
rate
at
the
star
level
and
then
at
the
key
space
level
and
finally,
at
the
namespace
level.
A
So
on
the
right-hand
side,
which
you
can
see
is
actually
you
can
aggregate
in
the
matrix
across
around
the
physical
hierarchy,
namely
that
you
can
get
the
host
level
metrics.
You
can
have
the
host
label
transaction,
the
red
level,
congestion
and
in
the
zone,
level,
congestion
and
then
finally
reach
the
same
namespace
level.
A
A
Many
of
these
that
we
could
you
know
is
because
that
each
each
metric
try
to
many
dashboard
and
each
dashboard
is
trying
to
aggregation
hierarchy
that
I
described
to
you
earlier
in
the
previous
line
and
also
try
to
turn
it
up
that
in
our
previous
instances,
the
SIP
consumption
that
that
devoted
for
this
metric,
aggregation
or
recall,
recording
to
evaluation.
It's
a
much
much
different
compared
to
the
amount
of
the
CPU
that
you
square
they
spend
on
the
metrics
crappy.
So
just
you
cannot
just
okay,
I
have
a
scraping
and
done
now.
A
B
Hi,
my
name
is
Vishal
Thakur
I
work
on
metrics
monitoring
and
alerting
in
the
data
infrastructure
team
at
eBay,
so
I'm
going
to
spend
the
next
few
minutes
talking
about
how
we
set
up
a
sharded,
federated
Prometheus
cluster
and
deployed
it
to
production
at
eBay
to
monitor
the
geo
distributed
database.
That
Jun
talked
about
so
I'm,
showing
here
a
simplified
example
where
the
target
space,
the
thousands
of
parts
across
hundreds
of
shards
that
we
have.
B
You
cannot
monitor
with
the
single
Prometheus
because
of
the
amount
of
matrix
that
we
collect
and
the
rules
and
all
of
that
that
June
mentioned.
So.
The
logical
thing
is
to
try
to
add
more
Prometheus
servers,
but
in
such
a
way
that
we
can
scale
the
Prometheus
set
up
in
proportionate
to
how
the
zero
distributed
database
is
also
going
to
scale.
B
So
in
this
simple
example,
if
you
add
one
more
Prometheus
and
take
half
of
the
parts,
for
example,
the
parts
which
have
even-numbered
shard
IDs
and
the
parts
which
have
the
odd-numbered
shard
IDs
and
get
them
scraped
by
two
different
Prometheus
servers.
You
already
kind
of
reduce
the
load
by
50%.
So
we
take
this
concept
generalized.
It
parameters
has
this
nice
hash
mod
function
and
you
can
apply
strategically
to
certain
set
of
labels
and
then
achieve
your.
Your
targets
can
be
grouped
into
more
than
two
groups
n
number
of
groups.
B
However,
you
want
it
like,
you
can
cut
and
slice
them
right.
So,
let's
look
at
how
it
looks
like
so
at
the
top.
You
see
the
gray
box,
which
is
our
Prometheus
ya,
mall
template.
So
we
have
templates
for
the
Prometheus
job,
for
recording
rules
for
alerting
rules
for
our
secrets,
config
Maps,
all
of
the
kubernetes
artifacts,
so
they
go
into
our
automated
script
set
up
and
we
turn
out
the
actual
deployment,
artifacts
and
secrets,
and
so
on.
So
here,
if
we
take
the
example
of,
we
want
to
split
the
target
space
into
two
groups.
B
You,
the
automated
scripts,
substitute
the
template
variables
and
you
can
see
in
this
example.
The
modulus
remains
two,
but
the
regular
expression
zero
one,
two
three
four
files
based
on
how
many
you
want
right
so
by
using
the
key
space
and
the
shard
ID
of
the
target
part
as
the
input
to
the
hash
mode
function.
We
also
have
this
nice
side
effect
that
all
the
parts
belonging
to
a
particular
shard
of
a
particular
key
space.
They
get
scraped
by
the
same
Prometheus
and
it's
it's
a
little
bit
key.
As
you
will
see
later.
B
It
helps
us
in
writing
aggregation
in
a
much
easier
way.
So
we
talked
about
the
scaling
of
the
Prometheus
in
proportionate
to
the
growth
of
the
datastore.
The
next
challenge
to
tackle
is
the
high
availability
and
that's
pretty
straightforward
of
taking
the
same
example,
Prometheus
1
and
Prometheus
2.
B
So
the
next
way
to
scale
within
a
target
space,
so
we
said,
will
scale
along
the
short
boundaries
so
using
shot
as
the
dimension.
You
get
a
number
of
from
edges
clusters,
so
we
expanded
that
concept
and
gave
ourselves
another
way
to
scale
and
also
isolate.
So
we
have
several
categories
of
parts.
If
you
will
we
have
the
database
parts,
we
have
indexing
parts,
we
have
service
parts,
we
have
cubelets
which
expose
the
operating
system
metrics
and
they
get
scraped
by
a
independent,
Prometheus
cluster.
B
So
we
map
this
categories
to
Prometheus
clusters,
so
that
gives
us
another
way
to
scale
the
Prometheus
independently.
If,
if
the
OS
metrics
are
getting
huge,
then
we
just
have
to
scale
the
Prometheus
cluster
corresponding
to
the
OS
category
and
so
on,
and
all
of
these
from
these
clusters
are
again
mirrored
in
the
remote
DC
and
you
can
see
here.
We
also
have
all
these
Prometheus
endpoints
defined
as
data
sources
in
graph
honor
and
when
I
get
to
the
graph
on
slides.
B
That
I
will
describe
how
the
query
happens
and
how
we
have
made
it
convenient
to
the
end
user
as
well.
So
the
next
challenge
after
scaling
and
high
availability,
obviously
is
how
do
we
federate
now
now
that
we
split
our
metrics
across
several
different
Prometheus
clusters
and
Prometheus
shards?
If
you
take
an
example,
you
want
to
know
total
inserted,
docume
count
for
now.
Your
data
is
split
across
to
prometheus,
so
what
we
do
is
we
define
using
recording
rules,
aggregated
metrics
at
the
Prometheus
shard
level.
B
So
in
the
example
that
I
gave,
if
you
aggregate
in
Prometheus
DB
one,
you
get
the
count
for
the
total
inserted
documents
in
your
odd-numbered
parts
and
then
on
the
other
side,
you
get
the
total
count
up
to
even
numbered
parts,
and
then
you
agree
ate
it
up
and
at
the
Federation
level,
and
you
get
the
total
count
across
the
entire
key
space.
Now
we
use
deliberately
a
prefix
level
zero
for
all
these
aggregated
metrics
that
occur
on
the
Prometheus
char
level,
and
that
makes
us
easy
to
define
the
Federation
job.
B
B
Again,
the
Federated
set
up
itself
is
highly
available.
A
we
have
a
mirrored
set
up
in
a
remote
data
center,
which
scrapes
the
same
set
of
all
the
permits.
Instances
now
one
thing
I
did
not
talk
about
the
Prometheus
instance
and
it's
mirrored
instance.
We
don't
directly
go
scrape
them
either
from
Garifuna
or
from
the
Federation.
They
are
fronted
by
a
load,
balancer
whip
and
the
Federation
goes
talks
to
the
load
balancer
web.
So
if
the
primer
is
down,
the
mirror
can
provide
the
data.
B
B
So
we
have
several
categories
of
Prometheus
clusters
and
each
category
can
have
n
number
of
shards
and
each
shard
is
nothing
but
an
H,
a
pair
with
an
active
in
one
data
center
and
and
the
standby
in
another
data
center
and
all
the
traffic
is
pulled
through
the
load,
balancer,
the
query
traffic
and
the
Federation
traffic.
This
entire
setup
is
deployable
within
a
couple
of
seconds
through
our
automation,
scripts
and
it's
fully
deployed
out,
including
the
rules
and
recording
rules
and
secrets
and
any
other
things
that
are
needed
for
the
set
up
to
work.
B
So
from
the
Prometheus
side.
If
we
go
on
to
the
query
side,
the
graph
on
a
side
so
I
mentioned,
we
use
the
hash
mode
function
strategically
on
a
certain
set
of
labels.
So
now
you,
our
data,
is
distributed
and
the
user
wants
to
query,
not
the
federated
rolled-up
metric
but
say
the
document
count
at
a
shard
level.
So
now
does
the
user
have
to
know
about
which
Prometheus
script,
your
data
shard?
So
that's
the
challenge
and
we
try
to
solve
it
with
couple
of
techniques.
B
One
technique
we
are
calling
it
the
federated
lookup
and
routing
table
on
the
parameter
side
and
then
on
the
graph
fauna.
We
make
use
of
the
graph
on
a
template
variables
and
template
data
sources,
and
I
will
go
into
details
so
before
that,
how
it's
going
to
look
like
for
an
end
user
on
the
graph
on
our
dashboard.
So
they'll
come
to
this
dashboard
and
they'll
see
the
key
space
field
and
a
nice
drop-down
list
of
key
spaces
that
they
can
select
from.
They
don't
have
to
type
they're,
not
to
remember
anything.
B
Once
the
key
space
is
selected,
they
then
select
the
shard
ID
that
they're
interested
in
and
the
data
source
field
is
automatically
populated
and
all
the
graphs
are
nicely
refreshed
from
the
context
that
they
are
interested
in
now,
as
I
mentioned,
that
is
possible
by
some
of
these
techniques,
so
the
technique
on
the
Prometheus
side,
we
built
this
special
metric
that
we
are
calling
the
routing
map
it's
built
on
the
OP
metric,
which
is
generated
by
default
on
Prometheus
for
every
target.
That
is
scraped.
B
So,
if
you
look
at
the
recording
rule
in
the
gray
box
there,
you
will
see
that
it's
nothing
but
count
of
up
and
aggregated
by
some
strategic
labels
like
key
space,
shard
and
zone.
So
the
instant
time
series
vector
examples
for
corresponding
to
that
metric.
You
can
see
the
two
blue
boxes
for
from
which
is
DB
one
and
DB
2.
You
will
see
key
space,
one
and
and
the
odd-numbered
shards
and
key
space
one
and
two
on
the
right
side,
with
the
even-numbered
charts
right.
B
So
this
is
part
it
is
kind
of
looking
like
already
a
partial
lookup
table
and
then,
when
you
federate
this-
and
you
pull
these
into
your
Federation
server,
you
can
inject
other
external
labels
like
data
source
name.
So
on
every
one
of
our
Prometheus
instances.
We
added
data
source
name,
which
corresponds
to
the
name
that
we
give
to
that
endpoint
in
graph
honor
when
we
define
the
data
sources
in
graph
honor.
So
what
that
gets
injected
and
you
look
at
the
the
bottom
yellow
time
series
list
of
values.
B
Now,
if
you
have
a
key
space
name
and
a
shard
ID,
you
can
find
out
by
a
query
what
the
data
source
name
is
and
given
a
data
source
name,
the
ref
honor
can
go
pull
the
metrics
from
there.
So
with
that
routing
map
time
series
in
place
now
key
space
is
nothing
but
a
label
values
function
applied
on
that
metric
and
you
retrieve
the
label
values
for
the
key
space
dimension
and
similarly
for
char
ID,
you
retrieve
the
shard
ID
dimension,
values
for
a
given
selected
key
space
and
the
third
critical
one.
B
Once
you
know
the
key
space
and
shard
ID,
you
again
apply
the
label,
values
function
and
you
retrieve
the
data
source
name.
So
that's
how
the
in
the
dashboard
the
data
source
gets
populated,
there's
a
little
trick
here
in
graph
ahna.
The
templated
data
sources
cannot
be
of
type
query
type,
so
you
cannot
assign
a
label
values
function
to
it,
so
we
had
to
create
an
intermediate
hidden
variable
that
has
that
value,
and
then
the
data
source
templated
variable
just
points
to
that.
B
A
So
the
first
one
I
have
to
share
with
you
this
monitoring
on
the
monitoring.
So
basically,
once
you
place
a
big
cluster,
so
you
need
to
monitor
itself.
So
we
come
up
with
a
dedicated
Prometheus
instance
called
safeml,
and
this
F
minus
HS
script,
all
the
metrics
that
expose
from
the
permittee
instances.
So
by
having
all
the
instances
related
metrics
to
be
into
a
single
fmcs
database
name.
We
can
so
example
to
compare
the
performance
that
behavior
that
manifested
by
all
the
instances.
A
For
example,
we
can
see
that
whether
the
low
that
manifested
by
all
the
firmaface
chars
for
the
for
the
purpose
of
scraping
the
always
metric,
whether
they
have
the
even
role
or
not
I,
saw
this
and
I
J
attached
a
prior
to
this
as
a
bomb
monitoring
instances.
So
just
give
you
that
the
dashboard
that
we
have
so
the
first
one
which
we
saw
that
well,
how
many
fanciest
instances
that
we
capture
for
each
individual
instances
at
this
time
so
there's
a
first
dashboard.
A
The
second
why's
that
well
how
many
instances
or
parameters
in
this
entire
Custer
has
been
done.
Has
a
country
down
right.
Third
was
a
little
bit
more
involved,
which
is
just
try
to
show
you
that,
well
with
it
a
standard
deviation
of
the
scripting
rate
for
each
instances.
Alright.
So
what
is
the
scraping
rare
instances
ativan
so
here
is
a
defined
at
the
rate
of
the
limit
is
a
particular
matrix
that
expose
Mike
Ramirez
instances.
A
It's
about
the
appending
up
to
the
Pentheus
database
and
we
compute
the
rate
and
we
over
the
razor
the
read
there
is
a
prior
to
the
the
same
matrix
that
computed
for
the
last
30
minutes
30
minutes
ago,
and
so
they
will
give
you
the
deviation
and
if
you
see
the
deviations
get
job,
they
know
that
there's
something
wrong
about
this
scrapping
performance,
and
so
we
can
attach
the
a
little
to
this,
a
self
monitoring.
So,
for
example,
we
can,
we
can
have
a
stress
photos
to
capture.
A
How
many
permit
instance
is
a
talent
is
time
and
then
whether
the
square
instances
for
its
premiere
is
a
chaser
Tameka
job
and
also
we
can
monitor
the
with
rich
build
here
from
the
federation
server
perspective,
and
so
this
is
actually
the
values
where
they
are
allowed
to
share
with.
You
is
actually
how
we
do.
The
troubleshooting
use
the
metrics
place
monitoring
system
right,
so
we
we
aggregated
at
the
one
of
the
dashboard
so
that
the
total
service
error
rate
of
a
costing
tax
service
cluster
with
hundreds
of
the
service
port.
A
So
now
it's
a
well.
If
you
see
the
service
rep
bump,
where
how
do
you
find
out
that
what
is
exactly
the
the
port
that
you
will
need
to
go
inside
to
inspect,
for
example,
using
the
the
local
law
and
then
for
the
troubleshooting?
So
this
the
the
one
of
the
query
that
we
use.
Basically,
it's
actually
try
to
compute
accumulated
error
for
each
service
port
and
we
compute
a
profile
of
there
and
then
because
it,
this
query
is
actually
only
invoked
or
each
Charlotte
committee
server
describe
the
this
particular
service
metric.
A
So
we
had
to
apply
in
another
profile,
another
top
five
aggregation
across
this
Charlotte
permitted
shoddy
parameters
per
militia
in
order
to
get
that
global
top
five
worst
performance,
and
after
that
you
can
get
into
the
pole
and
inspector
the
local
law
for
the
trap,
Oh
Sookie.
So
this
is
why
example
using
the
the
top
five
top
k-
another
one
is:
actually
we
can
use
them.
A
We
can
compute
the
error
rate
and
that
actually
will
allow
in
this
Prometheus
consult
or
we
saw
all
of
the
port.
The
service
port,
the
transients,
and
we
found
out
that
she's
permit
is
one
single
wet
cancer.
Is
you
can
Prada
hundreds
of
these
fancies
and
can
humanely
humic
and
operator
or
developer?
Can
visually
identify
that?
What
are
the
worst
performance
of
the
of
this?
That
produced
the
highest
service
error
rate
and
we
had
to
do
it
for
all
these
shaadi
server?
A
Another
one
is
actually
about
aggregation
of
the
database
matrix
I
do
the
aggregation
we
need
is
a
replica
Shah
and
in
the
key
space,
however,
the
foundation
is
not
possible
to
aggregate
the
kubera
expose,
always
metrics
the
reason.
Why
is
because
the
pot
name
expose
only
the
pot
name
and
then
all
these
annuity
fan
or
this
database
associated
hierarchy?
A
Key
special
and
replica
is
not
capture,
so
fortunately
we
have
to
expose
the
internal
information,
which
is
that
the
pot
name
is
actually
based
on
the
concatenation
of
the
key
space,
try
D
and
replica
ID,
so
we
use
the
per
movies
config
bag
and
then
have
the
regular
expression
to
attach
these.
These
always
metric
and
we
use
a
regular
expression
to
a
strata.
These
are
the
kiss
by
kiss
Bayside,
Ashanti
and
replica
Heidi
and
injected
into
the
committee's
metric
as
the
additional
labor.
A
Once
we
have
this
label
hierarchy,
then
H
we
can
sum
over
the
container,
in
always
matrix,
or
example,
the
this
region,
it
so
memory,
and
there
is
some
over
the
container.
We
get
the
replica
level
matrix
and
we
from
the
replica
we
get
a
shaft
on
the
shower.
We
get
the
key
space
and
also
we
can
get
an
intern
named
specificity
Cerie
so
that
this
kind
of
always
aggregation
is
justin,
also
be
very
useful
for
us
to
do
the
perform
to
traverse
a
to
disk
or
resilience.
Resiliency
testing
also
endurance
testing
day.
A
We
test
the
system
for
over
the
one
week,
two
weeks
three
weeks,
because
when
you
see
the
weather
memory
leak
has
happened,
whether
the
CPU
consumption
has
a
suddenly
change
or
not.
So
this
turns
us
was
very
useful.
The
last
one
is
we
want
to
show
it
to.
You
is
about
the
summarization,
so
Prometheus
comes
to
be
the
manager,
but
eliminate
mostly
is
just
communicate
to
you
whether
it
has
happen
or
not.
So
there's
no
much
summarization
capability
purine.
So
what
we
have
he
said.
Well,
we
explore
these
the
knowledge
of
the
permission.
A
We
just
comes
with
a
lot
of
label.
The
labels
come
from
the
recording
loo
that
we
evaluate
this
alert.
It
comes
with
the
recording
loose
label.
So
now
we
can
summarize
all
these
alert
with
this
label,
so
in
particular
we
can
have
a
severity.
So
we
can
see
summarize
over
the
critical,
critical
or
high
or
warning
there's
a
three
level,
the
similarity
that
we
define
in
our
system
and
we
can
aggregate
it
also
over
the
logical
hierarchy
to
see
that
well
in
between
each
data
center.
A
How
many
all
happen,
all
the
the
logical
hierarchy,
for
example,
can
see
the
how
many
has
happened
within
this
key
space
for
the
last
three
hours
right
so
well,
and
then
we
also
have
the
status,
which
is
that
liquid
is
active
or,
as
we
have
been
resolved.
So
this
is
a
another
label
that
we
can
attach
to
also
passive
state
operation
would
not
be
able
to
solve
this
problem,
so
we
actually
learned
the
unit
research
that
we
turned
that
we
persist.
This
permits
a
lot
into
this,
alas,
research
and
near
are
not
accessible
index.
A
The
this
a
lot
with
with
the
label
that
we
specify.
So
this
is
outcome
of
the
Cabana
despot,
so
I
mean
not
easy
to
see
from
the
front
of
from
audience.
You
have
to
far
away,
but
then
we
can
see
that
the
world
you
can
have
a
logical
hierarchy,
physical
hierarchy,
that
you
can
do
this
future
linear
equation.
You
can
less
severity
and
you
can
have
a
file
in
a
key
file.
A
You
know
reads
or
not
so
there's
one
point
is
when
I
think
one
thing
that
would
happen:
it
is
a
chi
there's
a
ton
window
right
if
you
can
specify
the
time
window
in
the
last
1
minute,
2
minutes,
then
that
you
can
get
a
real-time
dashboard.
If
you
studied
like
one
hour
two
hours
three
or
one
day,
then
they
will
give
you
the
historical
atlas
summarization.
So
that's
another
very
nice
for
the
photo
uncork
a
person.
A
So
with
that
right
to
conclude
our
presentation,
so
a
premiere
is
if
download
on
the
Internet
is
actually
the
staying
along
single
process
instance.
But
in
this
presentation
was
sorry
to
you
that
we
can
create
a
horizontally
scalable
and
Charlotte
federated
server
cluster
without
modifying
the
Prometheus
ausco,
and
that
with
this,
a
scalable
caster
really
to
handle
the
root
and
already
in
a
root
and
dashboard
with
this
hierarchical,
aggregated
metrics.
So
with
that,
we
like
to
can
come
up
here
and
he
said
thank
you
for
your
attention
and
we
can
have
thank
you.
Q&Amp;A.
B
So
we
have
a
web
hook,
alert
manager.
You
can
write
customer
books,
so
we
have
a
web
web
hook
that
intercepts
alerts,
and
then
we
send
it
to
our
internal
long-term
retention
store
as
well
as
to
elasticsearch
and
in
future.
We
have
plans
to
send
it
to
Thanos
and
have
global
query
as
well
as
long-term
retention
through
Thanos
yeah.
B
B
A
Excellent
question:
okay,
so,
okay!
So
let's
we're
going
to
go
to
this
a
walking
table,
so
I
didn't
I!
Think
they
emphasize
that
routing
table.
It's
a
ton
base,
gobo
local
paper,
so
I
say
ten
base.
What
does
it
mean?
So
this
routing
information
where
you
had
to
sha
that
rotating
table
information?
We
call
it
at
a
time
and
then,
if
you
know
the
switching
time
so
then
you
know
that
before
that
point
it
has
to
server.
A
For
after
that,
you
have
three
server,
then
I
treat
that
with
that
you
can,
you
can
had
crea
bundle
E,
and
then
you
can
do
that.
So
we
could
continue
shouting
so
with
this
mechanism.
What
we
just
need
to
have
time
to
put
it
about
a
great
deal
in
raise
this
question
say
we
can
perform
the
continuous
study
in
this
initially
set
up.
A
B
Yes,
yeah
so
because
it's
a
drop-down
list
when
the
user
selects
a
key
space
and
a
shard.
Obviously,
as
I
mentioned
earlier,
because
of
the
way
we
picked
the
hash
mode
function,
the
shards
data
is
in
a
single
promise.
Yes,
so
one
parameters,
data
sources,
all
we
need
to
display
all
the
shard
level
data.
B
Yes,
so
that's
where
the
global
query
part
comes
in,
so
when
we
started
this
out,
we
were
actually
keeping
an
eye
on
thin
O's
and
cortex
and
all
these
other
products.
So
that's
where
the
global
query
part
comes
in.
If
you
want
something
in
between
fully
federated
aggregated
versus
single
shard
level,
and
you
want
something
in
between.
Obviously,
you
need
global
query.
There.
A
Okay,
so
when
you,
when
you
aggregate
to,
shall
you
had
some
cooper's
in
your
mind
right,
so
maybe
you're
going
to
aggregate
a
based
on
some
label?
Okay,
so
maybe
based
on
I,
want
to
aggregate
that
to
shatter
its
has
manifested
a
super
ll
read
we
turn
up
or
something
so
you
can
constrain
with
the
label
at
this.
B
Yes,
yeah
so
right
now,
like
I
said
we
have
templates
for
recording
rules
and
the
Prometheus
jobs
and
so
on,
and
our
automation
scripts
just
take
the
templates
and
then
based
on
the
input
they
generate
it
for
n
number
of
Prometheus
instances.
It's
I
mean
it's
sufficient
for
our
purposes
right
now,
it's
not
very
complicated
just
with
a
couple
of
scripts.
A
B
Yes,
so
we
have
deliberately
limited
to
only
the
Federation
server,
only
scrapes,
the
already
aggregated
metrics
from
the
children
Prometheus,
and
that's
why
our
recording
rules?
We
have
hundreds
of
recording
rules
for
aggregating
it
on
several
different
ways
and
several
different
combination
of
labels
that
we
are
interested
in.
B
Or
definitely
definitely
yeah.
We
actually
have
only
one
single
Federation.
So
far
we
didn't
find
a
need
to
have
more
than
one
Federation,
because
all
the
Lord
like
June
mentioned
earlier,
goes
down
to
the
recording
rules
and
we
have
so
many
recording
rules.
We
have
all
possible
kind
of
aggregations
that
we
need.
The
Federation
is
just
pulling
them
and
aggregating
some
of
them
to
a
level
one
where
needed,
but
in
most
cases
is
just
aggregating
them.
A
We
weren't
away
so
the
do
not
really
come
from
the
prometheus
and
we
remove
the
and
then
go
to
a
rep
manager.
Let
me
go
to
the
way
hook,
I
mean,
and
then
we
had
a
weapon
called
the
slack.
So
that's
a
real
time.
I
live
in
is
a
cold
sue.
This
permit
is
Custer,
and
what
we
have
is
that
is
a
loot
is
the
alert
summarization.
This
is
a
held
on
call
people
to
digest
how
that
continues.
A
lot
string
comes
for
the
last
one
hour.
A
B
Also,
there's
one
more
use
case
whenever
we
have
any
productions
incident
when
we
want
to
do
root,
cause
analysis,
people
want
to
go
back
and
look
at
what
else
happened
at
that
time?
How
did
we
miss
it?
How
did
the
on-call
miss
it
and
so
on,
and
the
elasticsearch
helps
us
to
you
know,
look
at
the
history
and
try
to
roll
it
up
in
several
different
ways:
yeah.
B
So
in
this
case,
the
active
and
the
standby
both
send
alerts,
and
we
actually
have
not
one
but
two
alert
managers
and
two
different
data
centers
right,
so
they're
both
are
sending
to
both
alert
managers
and
we
have
some
logic
within
the
web
hook
that
does
the
dedupe
based
on
alert
signature.
So
if
you
look
at
alert
coming
from
two
different
Prometheus's,
almost
all
the
information
would
be
similar
except
the
start.
Time
might
be
slightly
different
because
they
might
be
scraping
at
slightly
skewed
times
and,
of
course,
the
external
label.