►
Description
Configuring Prometheus for High Performance [A] - Björn Rabenstein, SoundCloud Ltd.
A key to Prometheus's success is its operational simplicity. It takes only minutes to get a server with a meaningful configuration up and running from scratch. However, once you need to run your Prometheus server a bit hotter, you will quickly find yourself practicing the dark art of tuning Prometheus command line flags. This talk will shed some light on how to handle various scenarios: How to run Prometheus on a tiny machine or how to utilize a large amount of RAM. How to optimize for very high ingestion rates or for a lot of time series. How to get the most out of your old spinning disk or how to limit the wearout of your expensive SSD. There will also be a sneak preview of ongoing efforts to reduce the required amount of flag tuning.
About Björn Rabenstein
Björn is a Production Engineer at SoundCloud and one of the Prometheus core developers. Previously, he was a Site Reliability Engineer at Google and a number cruncher for science.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
I'm
beyond
I
want
to
talk
about
configuring.
Prometheus
for
high-performance
Who
am
I
written
on
my
github
profile
here,
I'm
a
permitted
development,
a
SoundCloud
engineer,
more
or
less
I
was
not
with
the
project
whenever
started,
and
not
one
of
the
legendary
founders,
but
I
have
been
there
for
long
enough.
That
I
must
record
before
was
cool
and
I
honestly
I
had
no
clue.
This
would
like
become
the
mainstream
monitoring
system
of
our
time,
so
something
it's
like.
A
It
has
many
entrance
barriers
in
a
way
kind
of
complicated,
completely
new
concept,
but
one
thing
and
I
think
this
is
one
of
the
reasons
why
it's
now
here
and
we
have
packed
rooms.
It's
operationally
really
simple:
it
takes
literally
minute
you
get
a
premiership
server
running
your
wallet
or
you're,
not
exporter
or
your
primitive
yourself.
I
do
this
literally
in
minutes
in
demos,
so
people
get
something
meaningful
doing
and
then
they
take
it
from
there
right
now.
This
is
not
an
introduction.
Torque!
It's
an
a
german
needs.
A
I'm
saying
about
in
english
means
advanced
I
guess
so
you
all
are
super
transformative
users-
and
you
are
all
here
to
know
about
operational
problems
if
you
want
to
get
really
high
performance
out
of
your
limited
resources.
So
I
don't
want
beginners
in
this
talk
because
they
would
be
scared.
It's
actually
quite
easy
in
the
beginning.
But
if
you
do
something
really
crazy,
then
it
becomes
more
complicated.
A
I
also
feel
you
know
all
the
best
practices
right,
don't
go
crazy
with
your
serious
cardinality,
like
SoundCloud
likes
to
put
the
user
ID
in
the
metric
than
your
350
million
metrics,
or
something
use,
recording
rules
for
fast
queries
and
all
that
to
do
that
right,
right,
okay,
so
this
is
about
how
do
you
tweak?
You
promised
this
configuration
to
get
the
best
performance
out
of
it?
Most
of
those
tweaks
are
just
command
line
flags
and
like
every
reasonably
behaved,
unique
command.
A
You
can
just
run
from
sssh
and
it
gives
you
the
flag
which
that's
not
even
all
of
them,
like
the
47
Flags
46
story,
46
lags
in
1.5
32.
Luckily
we
namespace
them
right.
So
it's
a
bit
weird,
but
it's
helpful.
If
you
have
many
of
them
and
most
of
the
tweaks
are
related
to
the
local
storage
and
if
you
color
them
vipin
flex
that
stops
with
your
local,
they
are
only
17.
A
So
apps
like
it's
like
shock
therapy
right
first,
you
see
46
minutes,
17
easy,
but
17
is
still
a
lot
and
I
mean
I'm
to
blame
for
that
also
they've,
guys
Julius.
He
is
one
of
the
real
founders
seen
in
the
room.
Perhaps
he
ran
away
I,
don't
know
first,
because
I
blame
him
now.
Okay,
so
back
when,
for
me
she
was
created
like
2012,
and
what
the
prototype
has
a
prototypical
storage
layer
storage.
We
won
if
you
want
and
Julius
realized.
A
This
will
not
pull
it
off
if
he
do
the
real
thing,
so
he
started
right,
storage
v2s.
He
call
it
now
so
he's
to
blame
for
the
fundamental
design,
I
kind
of
joined
the
effort
and
then
I
took
over
now
I'm
the
de
facto
maintainer.
Also
like
most,
if
you
like,
2013
storage,
2014,
perhaps
2015
was
public
launch.
Now
we
have
2017
so
most
of
the
lifetime
of
prometheus.
It
had
the
storage
it
kind
of
has
aged
and
not
always
gracefully.
You
run
into
more
more
corner
cases.
You
realize.
A
Okay,
we
have
to
optimize
that,
and
sometimes
you
cannot
just
clear-cut
uses,
make
it
one
way
you
have
to
ask
the
user
for
trade-offs
between
data
safety
and
performance
or
something
the
ones
when
I
came
to
another,
and
here
we
are
so.
The
best
way
to
understand
flags
is
from
first
principles.
That
means
you
read
the
code
and
you
understand
what
the
facts
are
doing
right,
if
you
don't
have
the
change
but
you're
like
half
an
hour
to
spare
watch
this
talk
by
this
person
in
the
other,
promises
also
admit
more
views
right.
A
So
this
will
explain
how
v2
works
in
principle
and
then
it
also
explains
the
flags
from
first
principle.
This
talk
is
not
the
other
talk,
so
this
is
more
like.
Let's
talk
about
practical
ideas,
how
to
treat
your
flags
there's
also
a
nice
page.
This
is
more
like
man,
page
style
or
cook
book
sell
on
our
awesome,
documentation
site.
We
get
lots
of
questions
about
storage
tweaking
on
the
list
or
the
memory
usage
and
things
like
that.
A
Okay,
let's
start
were
you
all
started,
you
spin
up
your
first
primitive
server,
you
let
it
run
and
since
you
all
advanced
permissive
users
and
general
like
yours,
the
right
operational
attitude,
you
monitor
your
primitive
user,
guess
what
monitoring
system
we
would
recommend
Prometheus
right.
So,
okay,
you
can
graph
things.
This
is
a
graph
of
the
memory
usage
of
a
permit.
If
you've
just
started
up.
There
are
many
lines.
The
green
one
is
artists
rather
than
segments
eyes
or
something
broadly
speaking,
kernel
developers
will
kill
me
now.
A
This
is
essentially
the
physical
memory
your
process
actually
takes.
Then
we
have
a
blue
line,
which
is
the
heap
space
as
seen
the
gold
heap
space
seemed
ago.
It's
objects
on
the
heat
that
are
still
in
use
and
objects
that
are
not
in
use
anymore,
but
not
yet
connected
by
the
garbage
collector
okay.
So
the
blue
line
is
super,
noisy
and
Brian
who
sit
as
told
you
something.
Yet
there
are
counters.
This
is
not
a
counter
at
the
gauge,
and
you
have
exactly
that
problem.
A
It
has
Peaks
all
the
time
and
even
Prometheus
doesn't
get
all
the
peaks
because
it
might
scrape
your
other
Prometheus
server
every
15
seconds
and
then
graph
anna
assembles
it
down.
So
there
is
a
Purple
Line.
This
is
max
over
time
for
five
minutes
and
the
other
probe
aligners
Miniver
type
Brian's
idea.
So
this
helps
a
lot
to
see
that
they're
actually
like
maximums
happening
all
the
time
in
reality,
Peaks
could
be
even
higher,
so
my
suspicion
would
be
that
we
have
a
huge
p
key
and
that
drove
up
the
RSS.
A
But
the
RSS
is
weird
right.
We
will
talk
about
the
lighter,
so
you
see
this
whatever
the
curve
is
all
goes
up
and
to
the
right.
So
this
is
clearly
memory
League
right,
so
you
wait
for
time
for
a
while
and
then
you
and
what
do
you
do
you
go
to
github
you
file
an
issue
and
say
yeah,
you're
kind
of
glorious
monitoring
system.
You
might
not
have
noticed,
but
it
has
a
memory
leak.
A
I
have
noticed
some,
perhaps
yeah,
perhaps
not
so
if
you
have
enough
memory
and
you
wait
for
a
while,
it
looks
that
so
you
get
a
steady
state
at
some
point.
I'm
cool
question
is
were
what
determines
where
the
steady-state
is
right
and
that's
actually
super
hard
to
answer.
There
are
so
many
moving
parts
in
a
premiership
server.
Even
things
like
service
discover,
you
might
be
quite
memory
hungry.
Then
you
have
to
retrieve
samples
from
the
discover
target.
You
have
to
ingest
them.
You
have
to
index
them.
A
Indexing
is
violetta
TV
for
us,
it's
just
a
fairly
established
library,
but
for
some
reason
sometimes
it
takes
weird
amount
of
memory.
Then
we
have
curious.
Of
course,
queries
sometimes
take
a
weird
amount
of
memory,
and
then
we
have
our
actual
or
simple
George
George
v2,
which
we
understand
quite
well,
but
which
also
takes
memory.
They
allocate
memory,
they
free
memory
and
for
some
reason
you
get
this
blue
curve
and
then
the
go
runtime
request,
memory
from
the
operating
system
and
the
operating
system
does
something-
and
you
get
this
weird
green
line
kind
of
hard.
A
But
there's
one
lever
you
have
to
move
at
the
user.
It's
your
responsibility
as
a
fan
teaser.
You
have
probably
seen
it
because
this
is
the
one
flag.
You
cannot
ignore:
storage,
local
memory
chunks.
What
does
that
mean
chunks?
The
memory,
the
the
wrong
table?
Sort
chunk,
so
we
have
one
kilobyte
fixed
size
chunks
where
we
put
the
assembles
in
with
a
compression,
so
we
get
many
there
and
then
a
series
of
chunks
is
a
time
series
and
we
write
them
all
to
disk
this
the
same
layout.
A
So
we
have
chunks
on
this
and
chunks
of
American
that
just
mirror
them
to
this
and
load
them
back
if
needed,
and
we
want
to
tell
Prometheus
how
many
chunks
can
be
memory-
and
this
is
this
number-
it's
like
two
to
the
20s
or
whatever,
like
it's
a
million
essentially
a
mega
chunk.
That's
the
default
value
for
bigger
service,
that's
not
enough,
and
for
smaller
service.
That's
too
much
and
you
will
just
zoom
by
default.
A
A
They
are
called
chunks,
waiting
for
persistence
and
then
their
chance
that
are
do
this,
but
still
lingering
a
memory,
because
caching
is
good
right,
okay
and
whenever
you're
number
of
chunks
the
memory
goes
beyond
this
number
primitives
will
kick
out
those
chunks
that
are
already
written
to
disk.
These
written
use.
How
caching
works?
Of
course
it
might
have
you
have
too
many
turns
are
not
yet
written
to
this
all
are
not
yet
completed
or
a
query
uses
them
at
the
moment.
A
This
is
also
a
condition
where
you
cannot
evict
them,
and
then
this
number
would
go
over
that
number
and
your
memory
would
explode
again,
but
promises
at
least
tries
to
protect
itself,
and
then
it
stops
ingest
it
you
don't
have
you
have
probably
into
this
okay,
so
how
to
avoid
stopping
ingestion?
You
want
to
tell
the
server
to
resist
chunks
more
quickly.
If
you
get
closer
to
that
point
and
that's
why
we
have
another
flag
at
some
point.
We
realized
internet
another
flags,
so
this
tells
the
premiership
server.
A
How
much
chunk
may
be
in
that
state
where
they're
waiting
for
persistence,
so
we
specify
that
so
that
prometheus
knows.
If
you
get
close
to
that
value,
we
have
to
speed
up
how
we
persist
to
disk.
Of
course,
you
might
ask:
why
are
we
speeding
up
persistent
to
disk
all
the
time
like?
Why
not
best
effort
just
write
to
this
as
fast
as
possible?
This
is
a
good
question.
It's
explained
a
detail
in
that
other
talk.
A
Just
very
short
discs
are
hard
right
and
spinning
this
our
heart,
because
they
take
eternally
to
see
to
that
file
on
this,
where
we
write
our
chance
to
so
we
have
one
file
for
serious.
You
want
to
write
chunks
to
the
end.
You
see
there
takes
10
milliseconds
right,
10
milliseconds
is
like
endless,
and
then
you
write
one
chunk
and
then
you
seek
to
the
next
series
again
10
milliseconds,
to
write
a
single
kilobyte.
This
throughput
is
like
ridiculous
right
that
doesn't
work,
so
you
need
to
match
up
right.
A
You
need
to
to
save
a
few
chunks
in
memory
before
you
seek
to
that
point
on
the
disk
and
then
write
like
for
10
chunks
or
whatever.
Okay,
as
if
these
solve
all
problems
right,
so
they
don't
have
seek
time.
So
we
can
just
write
tiny
amounts
all
the
time
right,
no
from
be
core
ssds
hate.
If
you
write
a
single
kilobyte
like
at
least
I,
want
to
write
for
pillow
right,
bigger
SSDs
want
to
write
16
kilobyte.
This
is
their
page
size.
If
you
don't
do
this,
you
get
write
amplification.
It's
super
horrible.
A
You
also
burn
through
the
lifetime
of
your
SSD.
Yet
so
definitely
whatever
you
have,
you
want
to
match
up.
So
this
is
why
you
don't
want
to
write
at
full
speed
all
the
time
you
want
to
match
up
as
much
as
possible,
so
you
have
to
treat
those
numbers
so
Prometheus
has
a
nice
metric
called
the
persistent
urgencies,
gone,
that's
actually
exported
by
the
polity
server.
You
can
plot
it
in
prefer.
Now
you
can
alert
on
it.
So
this
is
essentially
how
quick
prometheus
wants
to
produce.
A
It's
calculated
from
all
those
numbers
that
are
also
exported
as
metric
so
much
chance
to
persist
is
the
limit
your
configured
chance
to
persist
the
actual
number
of
chunks
that
are
waiting
for
positions.
This
is
our
memory
churn
flag,
and
this
is
the
actual
number
of
memory
chance.
You
can
plot
them
all
nice,
curves
and
essentially
the
urgency
score-
is
chunks
waiting
for
persistence,
divided
by
the
maximum
allowed
number
to
wait
for
persistent,
and
sometimes
you
see
these
funny
messages
right.
A
George
has
entered
rational
chance
versus
peak
number,
something
something
big
number
big
number
something
right.
So
now
you
can
kind
of
guess
what
it
means.
So
we
have
all
those
numbers
here
and
you
see
chunks
to
resist
this
12
million
max
chunks,
which
is
50
million.
So
this
is
0.8.
You
can
see
how
this
works
and
what
is
rush
mode,
so
rush
mode
is,
if
the
urgency
score
ever
hits
0.8
then
Prometheus
thinks
it's
not
just
enough
to
be
gradually
faster.
A
We
have
to
do
like
desperate
measures
now,
so
it
does
things
like
now
we
go
as
fast
as
we
can.
We
also
don't
do
like
early
checkpointing,
which
is
a
different
concept,
and
we
also
don't
insist
on
F
thinking
every
single
file
anymore.
If
we
write
it
usually
I
mean
people
are
often
think
this
is
super
evil
and
they
may
like
write
FAQ.
So
right
server
goes
into
restaurant
all
the
time
or
can
I.
Do
it's
actually
not
super
evil?
If
it
helps
so
this,
you
can
plot
this
in
your
dashboard.
A
If
it
hits
0.8,
it
should
go
down
again.
So
at
some
point,
I
will
hit
0.7
and
then
you
leave
rush
mode
and
all
this
good
if
they
send
sometimes
or
even
periodically
fine.
What
should
never
happen
is
you
reach
open,
a
to
go
into
restaurant
and
the
urgency
score
goes
up
and
up
and
up
and
up
all
the
time,
then
you
are
in
trouble,
predict
linear
by
the
way
comes
in
handy.
A
If
you
want
to
alert
on
that
and
if
you
really
hit
one
something
that
happens
and
I
want
to
show
you
this
one
more
flag,
so
people
have
like
it
depends
on
context
and
also
religious
beliefs.
If
you
want
to
sync
files
or
not
so
this
is
also
configurable,
it's
a
trade
off
you
can
give
to
the
user.
The
default
of
this
flag
is
adaptive.
That
means
don't
think
in
rush
mode,
but
think
in
non
rush
mode.
A
You
can
also
say
never,
if
you
don't
believe
in
thinking,
because
you
want
to
use
the
page
cache
and
everything
say
never
and
if
you
think
you
should
always
EV
sing,
it's
really
vent
on
circumstances
longer
discussions
three
other
talks
could
be
given
about
that.
You
said
it.
You
always
okay,
so
this
has
happened
if
your
rush,
if
your
urgency
score
ever
hit
once
and
also
tells
you
the
reason
so
this
in
this
case
it
because
we
have
hit
the
max
change
to
persist.
A
It
could
also
be
the
other
way
that
your
memory
checks
go
up
too
much
perks,
because
you're
curious
that
in
too
many
or
other
reasons
that
I'll
talk
about
in
a
second
and
there's
a
even
tolerate
10%
more
than
the
configured
value,
but
then
it
also
gives
you
the
straighten
message
injection
throttle.
This
is
horrible
for
various
reasons.
We
don't
need
me
to
discuss
that.
You
should
definitely
avoid
that.
If
that
happens
ever
or
even
like
more
than
once,
you
should.
You
should
do
something.
Okay.
A
Naively,
you
might
think
okay,
a
chunk
is
a
kilobyte
I
said
that
so
Emil
is
a
gigabyte
easy
right.
So
let's
say
we
have
64
gigabyte
and
like
half
of
it,
for
charge,
32
million
memory
chunks
we
can
set.
This
done
right,
no,
doesn't
work
that
way.
Prometheus
is
doing
so
many
things
that
you
have
to
do
this
way.
Lower
and
yeah
I
mean
I
talked
only
talked
about
like
two
level.
A
We
curious
whatever,
although
things
take
memory,
so
this
is
way
lower
than
you
expect
manufacturers
in
the
equation,
and
if
you
look
back
at
this
thing,
so
the
green
line
should
never
cross
the
memory.
You
have
right.
So,
let's
whatever,
let's
say
that
our
real
memories
here
and
this
peak
was
already
quite
dangerous-
you
should
never
go
higher
than
that.
Now
this
line
is
kind
of
what
we
what
go
things
it
uses.
So
this
is
already
lower
and
even
this
ratio
is
quite
off
kind
of
hard
to
predict.
A
My
rule
of
thumb
is
this
is
probably
this
peak
is
probably
2/3
of
this
peak.
This
is
not
always
true,
but
it's
kind
of
a
conservative
estimate.
So
let's
say
this
line
is
2/3,
but
then
you
get
this
up
and
down
right,
that's
just
natural!
If
you
see
this
is
70s,
this
is
14
gig
here
this
is
a
factor
of
2.
This
is
not
coincident.
This
is
how
go
garbage
collecting
works.
A
You
have
the
heap
size
where
the
garbage
collector
was
done,
and
then
the
garbage
collector
rights
of
the
heat
to
double
in
size,
and
then
it
runs
in
your
garbage
collection.
Second
easy
right,
but
that
means,
if
this
is
the
line,
you
cannot
cross
it's
2/3
of
your
m
and
the
baseline.
You
have
that
your
garbage
collector
is
this.
This
is
1/2
of
2/3
is
1/3,
so
essentially
only
1/3
of
your
memory
is
available
for
all
the
chunky
of
allocating
all
the
time
and
all
the
other
things
that
might
be
allocated
all
the
time.
A
So
this
is
where
this
factor
of
3
from
the
documentation
side
comes
from,
but
this
is
really
a
lower
bound.
Some
people
try
that
and
then
they
so
what
we
do
it
sound
route.
This
is
from
our
original
chef
recipes
because
realized
old-school
and
deployed
primitives
on
their
metal.
This
chef,
so
this,
what
we
do
like
chef
has
memory
in
kilobytes
we
divided
by
6,
so
it's
6
of
our
storage
is
essentially
used
for
memory
chunks,
but
you
already
see
like
there's
a
comment
here.
A
This
is
a
rule
of
thumb
works
in
90%
of
the
cases.
Sometimes
it's
super
wasteful.
If
you
promise
it
to
us
actually
not
doing
so
much
else.
You
are
not
making
use
of
your
memory,
you're
wasting
a
lot
or
it's
the
other
way
round.
It's
actually
doing
a
lot
of
other
things
like
has
a
lot
of
series,
and
then
it
has
to
manage
individual
series.
Has
big
MVC's
has
the
experience
of
memory.
A
This
might
not
even
be
enough,
so
this
works
most
the
time
you
can
try
that
yourself,
but
sometimes
you
have
to
tweak
it,
which
is
not
a
good
situation
to
be
in
but
yeah,
manageable,
and
then
the
other
question
is
mark
strength
to
persist
or
the
good
value
there.
We
just
use
half
of
the
other
value
in
general
good
idea,
but
it
again
depends
on
things.
A
If
you
really
want
to
batch
up
a
lot,
then
you
want
to
have
it
as
high
as
possible,
but
on
the
other
hand
like,
if
you
have
many
time
series
and
every
time
she
has
this
head
showing
that
cannot
be
evicted
already.
You
don't
want
to
have
too
many
chunks
in
this
population,
because
then,
then
you
might
again
because
too
many
strands
here.
So
this
is
always
smaller
than
this
one.
Obviously,
but
the
numbers
are
smaller,
I
would
start
with
half
of
it
and
then
you
can
drink
it.
A
We
have
also
written
this
in
Georgia
recipe
that
people
can't
we
get.
This
is
all
valid
for
1.5,
2,
1
or
1.5
in
general,
but
the
other
one,
our
buggy,
don't
use
them
one
not
for
was
way
more
erratic
in
memory.
Consumption
animals
also
way
more
wasteful
for
various
reasons.
Sometimes
it
was
just
a
mistake.
Sometimes
was
a
trade-off
between
pure
performance
but
Brian,
who
I
mentioned
a
couple
of
times,
and
you
have
heard
a
key,
not
any
authenticity
in
the
first
row.
A
A
He
write
a
super
awesome
blog
post
on
his
company
text
block
at
like
fall
out
of
his
research,
and
this
is
funny
enough.
It's
exactly
the
opposite
question!
Then
we
have
a
tongue
clock
so
at
center
we
have
our
bare
metal
servers
and
you
think,
okay,
this
is
your
team
squeezy
server.
You
want
to
get
most
performance
out
of
it,
so
use
these
settings
and
tweak
at
will.
But
in
many
other
situations
it's
the
opposite.
A
You
know,
I
have
two
million
metrics
and
I
want
to
ingest
100,000
tables
per
second,
so
what
ec2
instance
do
I
have
to
order
or
what
container
settings
to
where
I
want
to
have
it?
I
run
my
promises,
server
like
on
cuneta
or
something
and
that's
all
explained
in
the
blog
post,
I,
wouldn't
even
go
into
all
the
details
because
it's
white
we
sophisticated,
but
you
can
try
this
out.
A
Like
of
course
again,
you
use
police's
to
monitor
promises,
and
these
are
the
queries
from
the
blog
post,
so
the
first
one
explained
that
the
blog
post,
just
broadly
speaking,
this
is
the
number
of
chunks
you
actually
create
in
six
hours,
because
six
hours
is
the
sweet
spot.
The
committee
is
once
for
King
up
the
right
right
and
you
can
do
this
with
a
few
factors
on
explained
in
the
approach-
and
this
is
just
example-
let's
try
it
out
on
a
PC,
Prometheus
or
SL
dot,
it's
6.8
million
chunks.
A
Do
you
want
as
chance
to
persist
to
a
six
hour
cycle
for
matching
up
right
and
then
the
heuristics
here
to
increase?
What?
How
many
more
memory
chance
do
you
want
is
actually
the
number
of
theories
rationale
is
kind
of
these
are
the
head
chunks
you
have
at
least
so
you
take
the
maximum
of
memory
theories.
You
have
it's
4.6
million,
pretty
beefy,
it's
already
kind
of
hard
to
run
a
parameter
server
that
hot
you
get
a
four
million
for
that
and
the
total
numbers
11
million.
With
our
divided
by
six
rule.
A
A
The
server
I
took
this
from
if
a
64
gigabyte
service
like
had
tweak
parameters,
these
are
actually
the
parameters
we
use
without
knowledge
of
write
blog
post,
so
you
see
they
are
way
lower
and
that's
because
we
on
that
server,
we
would
just
move
with
our
settings,
and
this
also
means
we
are
not
in
the
comfort
zone
that
Brian
Proctor's
put
your
insight
right
raw
process
like
yeah.
This
is,
we
will
probably
never
go
to
rush
mode.
You
have
lots
of
headroom
for
Peaks
and
stuff.
This
is
like.
A
We
run
it
as
hard
as
we
can
right.
This
will
go
to
rush
mode,
occasionally
it
kind
of
works.
This
is
a
production
server,
but
yes,
this
is
how
you
see
you
have
a
bit
of
leverage
between
super-conservative
and
run
it
leading
edge
all
right.
So
what
is
the
basic
problem?
We
happy
with
all
those
tweaks.
The
basic
problem
is,
we
have
all
those
factors.
They're
very
like
I,
talked
a
lot
about
the
number
of
C,
a
sea
of
calmly
memory,
and
you
have
so
many
things
that
might
or
might
not
take
memory.
A
So
yeah
it's
hard.
It
would
be
so
much
nice
if
you
can't
could
just
tell
to
me.
Yes,
please
take
that
amount
of
memory
and
do
the
best
like
yeah
the
one
optics.
We
should
have
totally
released
that,
on
the
first
day
of
the
conference
to
get
super
confused
with
kubernetes
106
being
released
at
the
same
time.
Anyone
they
set
a
new
flag.
It
will
be
released
next
week
or
something
so
you
can
actually
tell
Prometheus
the
target
heat
size.
A
You
can't
tell
infirmities
hours
s
because
go
doesn't
even
know
the
are
set,
but
the
upper
purple
line.
That's
essentially
what
you
can
clamp
or
mediator.
It's
not
rocket
science.
It
might
still
have
little
Peaks
going
over
that
1.9
go
on
and
I
might
help
us
to
be
this
to
implement
that
better,
but
that
works
right.
This
is
like
2
gigabyte.
You
can
now
tell
Prometheus
now
next
week
or
build
for
master.
Please
use
whatever
30
gigabytes
of
key
size
and
it
will
try
to
make
the
best
out
of
it.
A
You
can't
even
see
how
that
works.
I
screenshot,
my
mouth
cause
I'm,
sorry.
So
this
is
what
you
actually
get
125
million
times.
Do
it
usually,
but
then
the
crazy
team
that
has
so
many
instances
on
the
equivalence
class,
but
they
decided
to
really
blow
everything.
Suddenly
you
had
all
the
old
time
serious,
still
a
memory
lingering
somewhere
with
chunks,
and
you
have
through
more
time
serious
and
then
they
I
mean
this
is
actually
an
eight-hour
shift
and
it
already
goes
down
again,
but
often
you
have
this.
Oh,
my
god.
The
do
plug
was
wrong.
A
Let's
just
redeploy
everything
again
and
then
within
10
minutes.
You
have
like
triple
your
time
series
and
that's
usually
either
you're
super
conservative
with
your
settings
and
the
chunks.
They
are
the
same
in
1.5,
but
now
you
have
more
time
series
to
manage
you
either
room
or
your
settings
are
super
contoured
and
in
usual
cases
you,
you
don't
actually
use
your
memory.
This
is
now
the
new
thing
you
see
like
this.
The
blue
stuff
are
the
headshots
that
are
open.
A
The
old
stuff
are
the
chunks
that
are
waiting
for
positions
and
the
green
is
just
the
caching
okay
and
you
see
it
jumps
up
and
then
permute
is
kind
of
contacts
something
happen
here.
I,
don't
know
what
expensive
query
letter
D
be
compacted
or
I.
Don't
want
some
something
needed
more
memory,
so
permittees
reacted,
evicted
like
the
cash
junk
and
here's
the
other
jump,
which
apparently
was
heavy
on
the
memories
or
it
evicted
a
whole
lot
of
junk.
So
this
works
like
a
charm,
pretty
good
safety
lot
of
operational
trouble.
A
Okay,
what
it
doesn't
change
its
checkpointing
pattern
time.
Okay,
a
few
minutes
and
check
pointing
is,
is
my
my
next
biggest
enemy.
So
what
is
checkpointing?
If
you
watch
the
other
talk,
you
will
understand.
I
will
explain
it
super
quickly.
You
have
all
those
chunks
that
are
not
yet
written
to
this
thing
of
the
open
head.
Showings
and
now
you
want
to
shut
down
your
server.
So
where
do
they
go?
You
could
like
stop
ingestion
and
wait
until
they
have
been
all
written
to.
A
The
individual
files
will
take
like
six
hours,
but
yeah
I'll
have
so
much
more
for
shorter,
but
you
don't
want
six
hours
shutdown
time
right.
So
what
you
do
you
write
a
gigantic
block
to
this,
which
has
all
the
non
process
the
part
in
one
big
file,
because
writing
one
big
file
is
very
easier,
and
ideally
this
takes
like
15
seconds,
but
you
scale
up
your
primitive
server.
You
buy
more
RAM,
you
buy
bigger
this,
it's
all
possible,
but
what
never
scales
up
or
almost
not,
is
the
speed
you
write
to
this.
A
So
if
your
checkpoint
is
twice
as
much
try
to
speak
because
we
have
tries
as
many
serious
it
takes
approximately
twice
as
long
to
write
at
some
point
on
be
few
servers,
you
check
vertex
10
minutes.
15
minutes
gets
pretty
horrible
right
by
default.
We
checked
on
every
5
minutes,
not
only
on
shutdown
but
also
regularly
because
user
records
that
crash
right
and
you
don't
want
to
lose
too
much
data.
A
So
this
is
like
a
conservative
setting
don't
lose
more
than
5
minutes
of
data,
but
if
you
check
one
already
takes
50
minutes,
you
are
simply
check
waiting
all
the
time
that
totally
kills
you
or
your
disk
IO,
like
essentially
90%
of
GDP
I,
was
just
check
pointing
then
it's
super
horrible.
So
this
is
definitely
something
on
beefy
service.
Where
you
have
to
make
the
trade-off
you
just
don't.
Do
it
automatically
yet
per
suite,
we
will
didn't.
A
You
know
I'll
tell
you
other
stories
in
a
second,
but
this
is
definitely
something
on
beefy
servers
where
you
have
to
say:
ok,
I'm,
except
to
like
lose
50
minutes
of
data
or
half
an
hour
and
increase
that
we
do
50
minutes
of
Delta
by
default.
Also,
like
pressures,
don't
happen
too
often
and
yeah
just
metrics
right,
filling
data.
A
So
there's
this
other
flag
that
is
extensively
discussed
in
the
other
chart
and
even
in
the
discussion
it's
so
communicated.
I
won't
even
explain
it,
but
this
is
a
really
like
a
setting.
That
is
again
conservative
and
it's
only
makes
only
sense
on
spinning
this,
so
on
SSDs,
and
so
these
are
anyway
worse
in,
like
writing.
Checkpoints
because
they
really
hate
Satan.
They
hate
many
things,
they
hate
small
lights,
but
they
also
hate,
sustain
rights
or
approach
right.
So
on
spinning
this,
you
can
put
this
number
without
problems
too,
really
high
number.
A
You
should
do
that
on
sorry
on
SSD
on
spinning.
This,
it's
again
depends
on
your
sense
of
conservativism.
This
will
create
more
checkpoints,
so
a
complicated
issue
and
very
hairy-
and
this
is
Jack
said
very
high
of-
is
SSD
as
if
these
are
so.
This
is
like
what
you
can
get
with
a
really
nasty
upset
SSD.
So
the
yellow
curve
here
is
memory,
serious
maintenance.
So
this
means
whenever
or
that
often
a
memory
serious
was
looked
at
and
all
the
unprocessed
junks
were
written
on
to
disk.
A
So
this
is
should
be
a
flat
line
on
this
server.
It's
like
going
up
and
down
like
crazy
and,
like
the
naive
assumption,
would
be
when
it's
at
zero,
like
it's
doing
nothing.
This
is
when,
like
the
checkpoint
is
running
and
the
discount
do
anything
else
actually
wrong:
the
blue
shaded
area.
This
is
when
the
checkpoint
is
happening
and
at
the
end,
when
the
checkpoint
is
done
from
UTS
is
checked
on
done,
please
think
that
file
and
then
the
SSD
observes
like
thinks
the
file,
but
then
it's
super
upset
about
it.
A
So
this
is
for
ten
minutes.
That's
nothing!
It
can't
even
rename
a
single
file.
This
is
also
an
SSD.
This
is
nice,
which
is
90%
fooled
and
if
SSDs
are
almost
90%
full,
they
are
also
super
upset
so
like,
if
you
thought
as
a
piece
of
all
your
problems,
it's
yeah.
They
create
new,
interesting
problems
and,
like
some
people,
even
format
the
SSDs,
in
a
way
that
they
don't
even
partition
a
third
of
the
space
so
that
they
have
more
Headroom
to
breathe.
But
this
is
what
you
get
like
it
horrible
right.
A
A
He
locked
himself
in
for
three
months
and
brought
everything
new
with
all
those
things
in
mind
and
like
every
single
problem,
I
mentioned,
the
store
will
go
away.
I'm
sure
there
will
be
a
few
more
new,
more
interesting
problems,
but
the
problems
we
have
here.
They
are
also
it's,
of
course,
still
pre-offer,
but
watch
our
our
communication
channels.
There
will
be
another
soon
and
then
2.0
will
happen
sometime
this
year
and
everything
will
be
great
I'm
actually
not
spoiling
any
surprises,
because
farming
will
surely
give
very
insightful
talks
about
that
and
I
shouldn't.
A
Destroy
that
surprise.
Do
we
have
time
for
questions?
Yes,
five
minutes
edited
also
Julius
and
I
will
do
boost
duty
at
the
CN
CF
booth,
which
is
like
there
I
think
like
in
the
inner
ring.
It's
like
it's
difficult
to
find,
but
we'll
see
jewelry's,
Amin,
you're
kind
of
tall,
so
at
half-past
three
will
be
there
and
you
can
ask
questions,
but
we
also
have
five
minutes
right
now.
Thank
you.
A
A
This
is
all
men,
remote
search
is
always
meant
in
a
way
that
it
will
not
interfere
with
your
local
performance
and
I
mean
this
is
a
basic
idea
for
me
see
if
your
network
is
on
fire,
you
can
Samuel
whatever
you
say
this
to
doesn't
work
at
all,
but
you
sound
hosted,
tells
the
same
signal.
How
super
mediator
it
still
happily
collecting
metrics.
That's
the
idea
that
feel
still.
They
still
remain.
A
Okay,
let
me
say
one
thing:
this
is
like
I
mean
there
were
seven
seventeenth
like
slightly
covered
like
five
or
six,
so
the
above
is
kind
of
interesting,
but
the
lower
one.
This
is
something
I
should
have
done
way
earlier.
We
talked
about
how
the
go
garbage
collector,
like
needs
hundred
percent
heat
growth,
to
start
a
garbage
collection.
This
is
kind
of
based
on
a
heuristic
for,
like
essentially
all
of
the
heat
is
is
like.
If
you
know-
and
you
have
obvious
objects
coming
and
going,
then
it
makes
total
sense.
A
A
Setting
this
environment
very,
a
non
flag
environment
variable
go
GT
that
helps
you
how
much
your
heap
has
to
grow
to
trigger
a
garbage
collecting
cycle,
and
we
even
decided
to
put
this
by
default
to
40
in
row
1.6
and
that's
what
you
get
on
the
right.
So
this
is
the
usual
pattern.
You
know
then
I
restart
the
server
with
the
other
environment.
Variable
you
get
the
typical
cover
up,
and
here
your
steady
state-
and
you
see
this
bread
or
widths
or
whatever
height,
is
essentially
half
right.
So
this
line
for
exactly
the
same
thing.
A
You
need
it
like
twenty
seven
gigabytes
before
now.
You
need
the
free
memory,
upgrade
I
mean
not
completely
free,
because
you
do
more
garbage
collection
that
it's
another
call
or
something
right.
But
this
is
like
you
can't
even
do
this
with
your
1.5
server.
It's
like
I
mean
unless
you're
CPU
bound.
This
is
super
effective
and
will
be
default.
You
can
still
set
it
in
the
environment
variable,
but
this
will
be.
This
will
be
helping
a
lot
in
one,
not
six.
It
will
even
automatically
realize
oh
as
more
memory.
I
can
have
more
chunks.
A
We
see
new
heap
size.
So
that's
that's
pretty
pretty
nice
there's
no
verbs
right
in
robotics,
you
have
like
same
machine,
just
different
go
GC
setting
this
goes
up
to
ten
million
and
it
goes
up
to
60
million.
This
is
just
a
few
more
quality
born
for
garbage,
collector,
okay,
yeah
I
think
we
are.
We
are
out
of
time.
If
you
have
any
urgent
question,
raise
your
hand
now
or
come
to
the
booth,
say
us
a
hand.
A
Okay,
this
was
your
use
of
this
flat
right.
So
this
is
a
weird
implementation
detail.
In
this
storage
we
have
to
kind
of
lock
the
times
use
before
we
even
know
it's
fingerprint,
and
now
we,
you
know
whatever
it's
it's
much
better
than
that,
but
we
we
need
essentially
a
music
before
we
create
something
in
memory,
so
we
have
to
hospital
talking
or
we
have
a
limited
set
of
mutexes
and
round
robin
them
to
time
series
and
that's
by
default
for
thousands.
It
should
be
just
enough,
but
I
have
seen
scenarios
hi
ingest
container
over.