►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
M3 and Prometheus, Monitoring at Planet Scale for Everyone - Rob Skillington, Uber
For the past few years Prometheus has solved the monitoring needs of many and it is exceptional at what it does. Prometheus has exploded in popularity and now many wish to store more metrics, at longer retention and establish a single pane of glass on top of Prometheus for their monitoring needs across regions. M3 is an open source metrics platform that you can deploy and run using Kubernetes and Helm that integrates with Prometheus. It can store petabytes of metrics data with replication for high availability in a cost efficient manner, with compaction averse time series storage and index that can efficiently index and run dimension based regexp queries on billions of metrics. Using a real world example we will cover in this talk how to deploy M3Coordinator and M3DB using the M3 Kubernetes operator and connect your Prometheus instances together into a single global monitoring system.
https://sched.co/MPbX
A
Hello
everybody
good
morning
my
name
is
Tom
I'm,
just
hosting
such
doing
a
quick
introduction
for
Rob,
hopefully
everyone's
in
the
right
room.
There's
been
a
few
room
swaps,
so
if
you're
not
here
to
talk
about
m3
I'm
from
atheists,
you're,
probably
looking
for
app
dev,
which
I
have
no
idea
where
it
is
nobody's,
leaving
so
we'll
get
on
with
it.
A
So
today
we're
going
to
hear
from
Rob
about
m3
and
Prometheus
I
was
on
the
program
committee
and
was
super
keen
on
this
talk
mainly
mean
I'm,
one
of
the
Prometheus
developers,
but
I
also
develop
a
competing
system.
M3
and
I
really
want
to
hear
about
how
they've
solved
some
of
the
same
problems.
We've
had
to
solve
so
you're
a
round
of
applause
for
Rob.
B
Thanks
Tom,
so
yes,
this
talk
is
called
interim
prometheus
monitoring
at
planet
scale
for
everyone
and
other
than
that,
clickbait
title
that
I'm
gonna
talk
a
little
bit
about
being
wondering
and
increasing
number
of
things,
so
yeah,
there's
a
few
people
in
this
space
and
technology
products.
Now,
as
Tom
kind
of
mentioned,
metrics
is
being
used
as
a
platform.
B
More
than
ever,
it's
becoming
really
a
foundational
building
block
similar
to
how
databases
of
applications
for
products
metrics
are
being
essentially
a
building
block
for
building
reliable
systems
of
scale
and
delivering
high
level
levels
of
service
quality,
and
it's
also,
you
know,
being
used
for
order
remediation
or
to
rollback
it's
being
used
for
what
are
balancing
stateful
systems.
It's
it's
becoming
pretty
prevalent
and
like.
B
If
you
look
back,
you
know
five
to
ten
years,
I
think
logging
was
which
I'm
a
far
more
popular
than
the
metrics,
and
that
probably
still
is
the
case,
but
metrics
now
has
a
pretty
strong
foothold
in
the
in
observability.
So
you
know,
I
also
want
to
talk
a
little
bit
about
operating
in
in
many
different
regions.
Many
different
environments
that
that's
becoming
increasingly
more
common.
B
These
days,
especially
a
because
it's
easier
to
do
with
the
cloud
and
B,
because
a
lot
of
it
makes
a
lot
of
sense
for
a
loved
businesses
and,
of
course,
you
know
that
the
forth
talk
itself
is
going
to
focus
on
n3
and
Prometheus.
So
Who
am
I
an
Australian
monitoring,
node,
so
self-proclaimed,
of
course,
and
I've
commonly
found
that
I'm
drowning.
B
So
Prometheus
just
as
a
reminder.
This
is
the
first
talk
of
the
day
and
I've
ever
told
about
m3,
which
is
a
much
solving
different
things.
Prometheus
is,
is
what
most
organizations
will
reach
for
and
should
reach
for
when
they
begin
wanting
to
monitor
their
their
stack
and
then
starting
to
build,
alerting
and
other
kind
of
order,
monitoring
systems
that
they're
at
their
companies
and
organisations.
B
B
It's
usually,
of
course,
the
single
promethease
instance
can
only
be
in
a
single
availability
zone,
because
it's
a
single
since
itself
and
it
basically
monitors
your
application,
as
it
runs
on,
say,
kubernetes
or
an
AC
to
the
ec2,
Amazon
platform
or
GCE.
So,
basically,
the
richer
service
discovery
layer
allows
it
to
kind
of
understand
how
and
where
your
application
is
running
very
easily
and
basically
discover
and
collect
those
those
metrics
make
them
accessible
to
you
for
visualization,
using
profiler
and
also
you
know
using
alert
manager
or
some
other
alerting
engine
so
m3.
B
What
is
that
that
is
it's
a
project
from
uber?
It
was
began
in
early
2015,
so
kind
of
six,
nine
months
or
so
officer
Prometheus
itself,
and
it
was
built
primarily
to
scale
monitoring
horizontally.
So
it
was
kind
of
orthogonal,
at
least
in
the
early
days
of
what
I
was
trying
to
do,
and
and
pretty
much
still
is
again
about
prometheus
is
what
every
organization
should
reach
for.
First
before
any
of
these
other
systems,
because
it
sells
so
many
of
your
your
problems
issues
and
is
interoperable
with
everything
out
there
from
day.
B
One
and
most
people
don't
have
that
many
metrics,
so
even
running
to
Prometheus
instances
will
serve
a
whole
bunch
of
organizations
just
fine
but
m3
for
those
that
it
doesn't
solve.
You
know
that
that
setup
doesn't
solve
so
well
is
essentially
yes
built
to
scale
your
monitoring
horizontally
and
in
a
very
cost-effective
nature.
So
that's
one
thing
that,
from
day
one
has
been
very,
very
important
for
uber
some
organizations,
it's
very
easy
for
it
to
happen.
It
happened
to
us.
B
It
happened
to
Netflix,
they
haven't
tons
of
other
companies,
you're
monitoring
and
observability
platform,
especially
when
engineers
start
to
instrument
absolutely
everything
they
possibly
can
begins
to
be
a
large
part
of
your
cloud
footprint.
Essentially,
it
ends
up
costing
you
a
lot
so
rubriz
being
able
to
push.
You
know
that
percent
of
spend
of
your
entire
stack
for
monitoring
down
to
roughly
two
percent
up
from
you
know
larger
organizers.
Sometimes
you
see
a
10
fifteen
20
percent
being
spent
on
monitoring
their
ability
and
reliability
at
companies
and
organizations.
B
So
about
eight
months
ago
my
Master's,
probably
wrong
in
that
one
long
flight
was
my
excuse
anyway,
is
basically
being
written
in
open
source
from
2016,
but
we
only
really
put
together
the
documentation
and
other
stuff
to
actually
run
it
with
integrations
with
things
like
prometheus
and
that
since
August
last
year,
so
we
have
a
monthly
community
meetings
with
attendees
from
small
to
large
organizations.
It's
released
every
few
weeks
and
also
last
night
as
I
was
basically
trying
some
things
out
that
I
realized
in
quite
work
right.
B
So
you
know
this
is
kind
of
what
it
looks
like
it's
basically
multi-region
aware,
so
it
can
basically
store
metrics
local
to
eight
each
region
that
you
run
in
and
then
essentially
fans
out
queries
when
you
need
to
join
data
across
those
two
whole
of
the
regions
that
you
run
in,
and
so
you
know
it
understands
Prometheus
and
understands
graphite
and
it's
kind
of
compatible
with
multiple
metrics
formats,
which
rubra
was
really
important.
When
we
got
started
because
we
basically
had
a
whole
bunch
of
legacy.
B
Applications
that
emitted
graphite
metrics
still
to
this
day
do
they're
becoming
a
smaller
percentage
of
the
overall
footprint
of
applications
adiru,
but
they're
still
there
and
we
didn't
want
to
turn
them
off
or
run
multiple
disparate
systems.
Just
for
the
different
types
of
metrics
formats
that
we
supported.
B
So
three
main
points
I
want
to
talk
about
in
three
I
just
want
to
basically
talk
to
how
you
know
it
runs
pretty
much
anywhere
these
days,
which
is
you
know,
great
for
I,
think
the
open
source
community,
whether
you're
running
on
premise,
whether
you're
running
in
the
cloud,
whether
you're
running
on
kubernetes
it
there
and
it
also
supports
monitoring
a
whole
bunch
of
different
applications.
It's
scalable
to
billions
of
metrics.
This
is
a
workload.
That's
been
run
in
production
uber
for
quite
some
time
now,
and
it
has
a
focus
on
simple
operability.
B
We
to
this
day
only
have
two
dedicated
s:
eries
on
the
m3
team
at
uber,
which
is
pretty
incredible.
Considering
it
went
from
you
know,
tens
of
storage
nodes
to
thousands
of
storage
nodes
today,
so
touching
on
runs
anywhere.
So,
as
I
mentioned,
you
know
similar
to
Prometheus,
because
it's
basically
compatible
with
Prometheus
as
sort
of
a
long-term
storage.
You
can
monitor
a
ton
of
any
application,
that's
cloud
native
or
has
a
service
discovery
layer
in
prometheus.
B
It's
runs
on
kuben,
AZ
or
on-premise,
as
I
mentioned
it's
multi-region
aware.
So
that's
that's
pretty
important,
I
think.
As
as
again
as
we're
kind
of
running
these
more
complicated
applications
everywhere
to
be
able
to
be
native
natively
multi
region
is
important.
So
why
would
you
want
to
use
m3
with
previous?
So
a
lot
of
a
lot
of
people
are
wanting
to
kind
of
establish
a
Saleh's
of
their
companies
and
organizations
in
a
way
that
lets
them.
B
You
know
really
be
data-driven
about
what
they're
monitoring,
how
they're
monitoring,
how
they're
performing
and
a
lot
of
the
time
that
means
storing
things
for
longer,
so
you
can
basically
do
month
or
month,
comparisons
or
even
you
know,
just
getting
started
week
in
week.
So
you
can
store
metrics
at
very
specific
retention
periods
and
you
don't
have
to
worry
about
losing
one
or
two
nodes
or
backing
up
or
restoring
data,
or
anything
like
that.
B
As
long
as
you
always
have
at
least
a
certain
number
of
replicas
available,
you'll
be
able
to
access
and
read
and
write
your
data
just
fine
so,
and
you
might
also
want
to
store
different
types
of
metrics
at
different
retentions,
which
is
becoming
increasingly
important,
and
you
can
scale
up
your
storage
by
just
adding
nodes.
So
this
is
again
what
Prometheus
may
look
like
with
your
Apple.
B
This
is
what
using
it
within
three
months
like
in
a
very
basic
manner,
so
we're
still
looking
at
a
single
region.
In
this
example,
essentially,
entry
coordinator
is
a
proxy
tier
that'll
and
Jess
Prometheus
remote
right
requests,
which
is
HTTP
one
snappy
encoded
per
debuff,
and
so
you
can
actually
put
an
HTTP
load
balancer
in
front
of
entry
coordinator
and
send
a
whole
bunch
of
different
Prometheus
metrics,
just
so
that
just
at
that
layer
of
entry
coordinators,
so
it's
kind
of
stateless
unless
you're,
aggregating,
metrics
and
what's
nice
about
that.
B
Is
that
that's
really
the
only
two
roles
you
need
to
to
run
to
get
started.
The
m30
big
cluster
itself,
which
you
can
manage
with
kubernetes,
is
the
second
role
here
and
that's
spread
across
availability
zones
so
as
soon
as
a
metric
crosses
a
threshold
from
Prometheus
into
the
entry
system,
its
replicated
across
multiple
availability
zones,
and
so
what
a
lot
of
people
of
course
are
interested
in
doing
is
viewing
kind
of
all
the
metrics
from
multiple
Prometheus
instances
across
either
in
a
single
region
or
across
multiple
regions.
B
And
if
you
want
to
do
that,
essentially
you
can
point
for
finer
or
your
learning
engine
directly
to
the
entry
coordinator.
It
does
speak
Prohm,
ql,
natively
and
essentially
can
join
all
that
data
that
you
sending
from
the
difference
between
with
use
instances
because
it's
stored
in
one
place,
so
you
can
do
a
sum
across
regions
or
sum
across
wherever
your
application
is
running
and
you
don't
have
to
worry
about
Oh.
Does
my
Prometheus
live
here
for
this
application
in
this
zone
or
region?
And
then
you
know
you
can
also
deploy
a
dedicated
entry.
B
Query
instance
why
you
might
want
to
do
this
is
so
that
you
basically
don't
interrupt
your
rights
when
heavy
read
queries
come
in,
so
you
know,
sometimes
people
like
to
look
at
thousands
or
hundreds
of
thousands
of
metrics
and
perform
the
crazy
aggregations
which
might
cause
a
problem
for
query
services
of
any
kind.
So
these
poison
queries
are
able
at
least
to
be
isolated
to
the
rate
path
and
don't
disrupt
your
collection
of
metrics.
B
If
you
deploy
in
a
dedicated
entry,
query
instance,
so
talking
a
little
bit
about
graphite,
we
support
basically
carbon
TCP
on
ingestion
and
graphite
functions.
We're
trying
to
work
with
some
of
the
booking
comm
engineers
to
see
if
we
can
basically
use
their
graphite
query
layer
right
now.
It's
essentially
something
that
is
home-baked
based
on
the
0.10
release
for
graphite.
So
it's
not
as
up-to-date
is
the
most
most
recent
graphite.
B
So
this
is
kind
of
what
it
looks
like
with
graphite
carbon
TCP
line
protocol
eyes
can
be
adjusted
by
the
coordinator
and
then
essentially,
you
can
store
graphite
and
Prometheus
metrics
in
a
single
cluster
side-by-side.
So
this
is
quite
nice
because,
again,
if
you're
coming
from
like
a
legacy
system,
you've
got
your
latest
and
greatest,
but
you
also
want
to
support
graphite.
You
only
need
one
single
system
at
least
a
store
and
query
that
data,
so
M
3
runs
in
multi
region.
B
I
went
in
a
multi
regional,
wear
fashion,
as
I
kind
of
mentioned,
essentially
the
the
there's
zero
cross
region
traffic
running
where
you
store
all
this
all
your
data,
basically
region
local.
This
is
pretty
important
because,
as
a
lot
of
you
know
like
cloud
bandwidth,
costs
are
very,
very
real.
It's
one
of
the
more
expensive
parts
of
actually
running
in
the
cloud
and
shipping
traffic
cross
region.
B
B
Local
deployment
of
m3
looks
like
this
is
essentially
a
Prometheus,
a
sentence,
remote
right
data,
two
in
three
coordinators,
which
are
stateless
in
your
availability
zones
and
then
the
entry
to
be
cluster,
as
I
said,
is
again,
is
spread
over
availability
zones
managed
by
kubernetes
and
then
the
global
query.
Tier
looks
something
like
this,
where
you
can
essentially
put
a
HTTP
load
balancer
in
front
of
all
of
your
regions
and
then
essentially,
queries
can
hit
either
any
region.
B
It's
a
let
builders
build
mentality,
essentially,
is
what
we
call
it.
So
there
there's
some
nice
things
that
comes
from
just
having
a
whole
bunch
of
network
service
and
making
it
easy
to
deploy,
but
it
is
definitely
not
Maritimes.
So
you
know
because
of
this.
You
know
it
forced
us
into
basically
make
systems
that
made
everything,
look
the
same
and
scale
out
basically
Methos
market
services.
B
In
a
very
horizontal
fashion,
so
so
one
of
the
benefits
of
this
charity
is
essentially
there's
no
one
boarding
to
whole
month
to
monitoring,
there's
no
provisioning,
a
service
for
a
team.
You
just
add
storage
nodes
as
required,
and
it's
used
for
a
whole
bunch
of
different
things.
So
at
first
you
know
it's
just
a
standard
kind
of
application,
metrics
latency,
request,
counts,
etc.
B
Finding
is
like,
but
then
we
started
realizing
that
developers
were
using
it
for
business
metrics
as
well.
So
you
know
as
anyone
that's
used
a
Prometheus
client
library
knows
very
easy
to
begin
to
just
drop
in
instrumentation
and
in
tons
of
your
code
pause,
and
so
it's
not
meant
to
be
relied
upon
for
business
making
decisions,
but
it's
very
easy
to
get
an
idea
of
how
your
application
is
basically
performing
at
a
different
market
segment.
B
If
you
can
support
high
cardinality
metrics
at
the
storage
layer,
and
then
you
know,
we
have
also,
we
also
collect
basically
Network
fabric
metrics,
a
bandwidth
and
latency
for
different
network
devices
and
routing
systems,
data
center
device
temperatures.
We
do
capacity
planning
off
these
metrics
for
basically
buying
hardware.
B
We
also
do
a
whole
bunch
of
stuff
that
I'd
never
even
knew
about
that.
There's
a
whole
bunch
of
people
using
us
as
a
platform,
basically
for
syncing
data
into
it
and
then
reading
that
data
out
load,
balancer
applications,
and
so
you
know
some
of
these
stateful
things
are
built
on
Apache,
helix
or
other
kind
of
infrastructure,
which
basically
means
they
move
data
around
as
workloads
move,
so
the
n3
platform
itself.
You
know
it's
obviously
has
to
support
a
large
large
amount
of
users.
B
We
basically
see
around
700
million
metrics
being
aggregated
by
the
system
at
peak
capacity,
and
then
that
translates
to
roughly
like
35
million
metrics
stored
at
any
given
time
to
storage,
and
this
is
across
thousands
of
in
30
V
nodes
and
there's
about
nine
billion
metrics
in
you
Sidhu
room.
So
it's
architected,
you
know
for
a
liability
scale,
that's
that's
really
where
we
started
and
why
we
started
there.
That
was
the
the
fundamental
underpinnings
of
this
project.
Each
component
is
designed
to
run
across
availability
zones
in
region
and
this
lower,
inter
region
network
bandwidth.
B
Queries
are
executed
and
distributed
in
parallel.
So
one
of
the
biggest
thing
when
you
start
to
get
to
the
billions
of
metrics
is
being
able
to
find
these
metrics
and
find
them
quickly
so
because
of
the
flat
collection
that
basically
any
metrics
platform.
Does
you
don't
write
a
schema?
You
don't
tell
your
metric
system
how
to
basically
store
and
index
different
types
of
metrics.
So
essentially
you
need
to
build
a
very
generous
indexing
system
and
when
you
have
flat
structure
like
that
being
able
to
find
metrics
at
at
very
large
high
cardinalities
is
becomes
difficult.
B
So,
basically,
here
you
can
see
that
we
distribute
the
work
of
at
least
finding
those
metrics
and
extracting
that
data
to
too
many
multiple
storage
nodes,
and
this
essentially
happens
in
parallel,
as
you
execute
a
query
so-
and
you
know,
as
you
kind
of
like,
continue
to
to
get
a
larger
amount
of
this
flat
structured
data
spread
across
everywhere
being
able
to
distribute
that
metric
lookup
starts
to
matter
a
lot.
So
this
is
a
post.
You
know
other
types
of
systems
that
basically
archive
data
to
long-term
storage.
B
You
know
they
might
off
load
it
to
s3
or
other,
like
slower-moving
dater
we're
houses,
and
this
is
actually
you
know
a
great
pattern.
I
think
that
for
some
people
works
really
well,
but
because
it's
you
know
a
little
bit
perhaps
easier
to
manage.
But
having
said
that,
you
know
kubernetes
and
communities.
B
So
you
can
imagine
that
you
know
once
you
get
to
a
certain
size
for
a
time
window,
a
time
slice
in
retention
that
you're
looking
at
that's
a
lot
of
data
to
pull
back
and
for
a
single
query,
note
to
basically
perform
that
lookup
and
and
find
the
data
you're
looking
for
and
so
the
index
you
know
talking
a
little
bit
more
about
that
is
basically
backed
by
FST
segments.
This
is
similar
to
what
Apache
Lucine
uses
to
index
documents
and
which,
of
course,
elasticsearch
is
built
upon.
B
It
essentially
allows
us
to
filter
and
regex
queries
over
billions
of
metrics.
We
are
using,
and
we've
made
some
upstream
changes
to.
Let
go
the
go
Couchbase
of
elam
library
which
provides
us,
so
we
didn't
build
this
from
scratch.
This
is
is
a
very
powerful
library
and
definitely
very
interesting,
so
basically
an
FST
segment.
That
kind
of
looks
like
a
try.
B
Basically,
the
union
of
all
of
the
matching
try
nodes
that
that
you
match
for
a
query,
and
then
we
have
a
you
know,
focus
on
again
as
super
low
probability
again,
there's
only
two
essays
on
the
entry
team
that
is
managing
all
of
this.
Basically,
you
know
can
run
premise,
but
also
an
MDB
and
and
what
makes
them
a
lot
of
this
able
to
be
run
easily.
B
Is
that
basically,
we
have
automation
that
that
can
replace
notes
for
you
add
or
scale
up
or
scale
down,
depending
on
what
you
need,
and
so
you
know,
we've
kind
of
pushed
for
these
fewer
roles
and
pushing
complexity
into
these
roles
themselves.
So
again,
you
only
really
need
two
roles
to
get
started
within
three.
We
don't
have
any
background
tasks
or
require
monitoring.
We
used
to
have
a
large
batch
down
sampling
that
we
would
do
where
you
basically
write.
You
know
raw
metrics
into
our
system
then
has
some
batch
process
come
collect?
B
Some
of
those
raw
metrics
can
pack
them
into
one
minute
or
ten
minute.
I'll
store
them
back
now.
So
when
you're,
storing
metrics
for
months
or
years,
just
collecting
about
ten
seconds
and
then
trying
to
like
read
all
that
ten-second
data
in
one
query:
it's
not
really
very
efficient
you're,
just
operating
on
gigabytes
and
gigabytes
of
data,
so
down
sampling
that
into
one
that
are
ten
minute.
Tiles
is
very
important.
One
thing
we
found
they
were
doing
it
in
a
batch
fashion.
B
Reading
aggregated
data
that
we
stored
and
then
writing
it
back
is
that
you
basically
have
to
babysit
this
process.
That
goes,
and
does
this
kind
of
like
large,
batch
processing
job
now,
so
we
basically
in
the
entry
coordinator
and
if
you're
running
in
a
more
scaled
up
fashion,
you
can
use
entry
aggregator,
which
is
a
distributed.
Aggregation
server,
but
basically
we
do
all
that
a
down
sampling
on
the
ingestion
path.
So
essentially
this
happens
in
real
time
as
you're
collecting
your
metrics.
B
So
as
long
as
you're
you
know,
instances
are
healthy,
you're,
fine
and
you
don't
kind
of
have
to
babysit
these
background
tasks.
So
that's
one
differentiating
factor,
I
guess
with
with
m3,
which
is
interesting,
the
you
know
and
again
the
koobideh
is
operator
can
handle
a
lot
of
the
replacing
and
scaling
up
is
scaling
down
of
resources
for
m3,
so
I
thought
that
we
might
have
a
quick
demo
of
at
least
some
of
what
it
can
do.
So
this
is
essentially
what
I'm
going
to
run
through
now.
There's
essentially
we're
gonna.
B
Basically
look
at
a
multi
region
set
up
in
three
and
you
can
try
and
leave.
It
should
be
fairly
reproducible
and
this
repository
that
I've
linked
here
basically
shows
a
readme
on
how
to
how
to
reproduce
this
benchmark.
But
essentially
we
basically
got
this
I
set
up.
That
looks
very
similar
to
this.
So
node
exporter
and
problem,
we've
kind
of
compressed
into
a
single
benchmark
or
utility
called
prom
room
by
bench
and
essentially
it
generates
node
exported
data
using
the
influx
DB
metrics
metric
benchmark
or
generator.
B
So
basically,
all
this
data
looks
like
no
T
supporter
coming
in
and
it
just
remote
right
for
that
using
the
Prometheus,
remote
right
format.
So
that's
protobuf,
encoded,
snapping,
snapping,
compressed
Oratia
p12
the
entry
coordinator
and
then
this
they
will
all
write
it
to
basically
local
regional
and
3
DB
clusters.
So
demos
always
fail
so
I'm
prepared
for
that
and
we'll
just
see
what
we'll
just
see
how
we
do.
B
In
and
see
basically,
okay,
so
I've
basically
got
ten
of
these
chrome
bench,
remote,
remote,
Prometheus,
bench
markers
running,
so
each
one
of
these
does
roughly
about
a
thousand
metrics
per
second
coming
from
node
exporter,
so
I'm
just
gonna
pull
it
forward
to
microphone,
or
instance,
running
here.
Okay,.
C
C
B
This
fluid
added,
so
you
know
I'm
kind
of
expecting
to
get
to
about
if
we
can
get
to
400k
that'll
be
nice
a
lot
a
hundred
percent
expecting
to
get
there
considering
I
was
playing
around
with
this
last
night.
So
this
is
basically
a
view
of
the
gke
google
kubernetes
engine
cluster
view,
and
so
this
is
the
utilization
across
all
these
13
nodes.
B
So
this
is
the
CPU
utilization
of
the
cluster
now
I'm
using
30
nodes
here,
but
there's
essentially
you
know
the
coordinate
is
running
on
top
of
those
DV
nodes,
as
well
as
the
problem,
remote
venture
and
and
some
other
things
in
the
cluster
like,
etc,
D
and
stuff,
like
that.
So
it's
not
just
in
3b
using
these
resources
either.
So
here
yeah
we
can
see.
Basically,
CPU
is
definitely
starting
to
spike
up.
C
B
B
This
dashboard,
as
well
by
the
ways
can
be
found
on
prefer
Norcom.
So
if
you
find
yourself
playing
around
with
them,
3
B
and
you
want
to
want
to
actually
see
some
of
the
stats
coming
out
of
it,
you
can
find
it
there,
and
so
yes,
now
we're
at
300k
doing
yeah.
Basically,
nine
million
actual
rights,
3
million
logical
rights-
you
can
see
the
you
know
the
service
side,
latency
creeping
up
here.
This
is
basically
a
p99.
B
C
C
D
A
E
E
B
So
the
reason
we
have
already
begun
talking
about
that
right
now
is
because,
yes,
it
does
understand
native
prong
qo.
However,
it's
never
going
to
be
one-to-one
as
accurate
as
default
Prometheus.
So
it's
because
it's
you
know
being
developed
recently.
It's
still
basically
being
worked
on,
but
it
supports
pretty
much
all
to
make
like
all
of
the
functions
information
today,
there
is
I
think
some
issues
with
binary.
F
Yeah
so
one
of
the
challenges
of
scaling,
promises
or
is,
of
course,
at
storage
level.
That's
a
great
solution.
I
encountered
issues
on
the
memory
level,
meaning
that
we
they're
not
a
huge
parameters
that
are
managing
like
two
terabyte
of
data.
I'm
exploding,
the
in
memory
so
I'm
wondering
how
which
solutions
you
see
do
implement.
Or
would
you
recommend
for
this
kind
of
yeah.
B
Great
question
so
I,
like
one
thing
is
essentially
memory,
is
always
gonna,
be
an
issue
and
all
of
these
systems.
Some
of
the
reason
why
we
get
such
high
compression
on
this
data
and
allows
you
to
really
reduce
the
cost
is
that
you're
basically
doing
very,
very
high
levels
of
runtime
length
compression
on
this
data,
which
is
a
lot
easier
to
do
when
it
stored
in
memory.
So
you'll
see
systems
like
basically
m3
has
similar
issue
and
uses
a
fair
amount
of
memory,
depending
on
the
block
size
that
you
choose.
B
You
can
make
that
block
size
smaller,
but
it
makes
your
queries
like
take
more
time.
This
is
basically
searching
more
blocks
on
disk,
but
essentially
we
would
like
to
get
to
a
point
where
we're
kind
of
like
not
really
doing
regular
compactions,
because
that's
why?
What
we're
trying
to
avoid
with
the
system
and
reduce
the
cost
overall
but
kind
of
this
kind
of
concatenation
like
compaction
process
with
m3,
so
hopefully
at
some
point
we'll
be
able
to
actually
unload
some
of
that.
That's
not
being
touched
very
frequently,
but
we're
not
there.