►
From YouTube: 73,000 Pods a Day, Lessons From Misadventures In Multi-Tenant - Shane Corbett & Wil Reed
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
73,000 Pods a Day, Lessons From Misadventures In Multi-Tenant - Shane Corbett, Amazon Web Services & Wil Reed, Acquia
We spent over two years pouring through 800 page linux kernel performance books, tweaking obscure control plane settings, and developing detailed custom monitoring dashboards so you don’t have to! We found there is a large delta between what we learned in CKA training, and the layer upon layer of hard fought knowledge it takes run a large scale multi-tenant application in production. Join us as we take you through real world findings that took months of research to fully understand, and provide evidence that some of the things we were convinced were best practices, were the very things holding us back the most.
A
Gang
thanks
for
coming
today,
we've
got
a
packed
agenda,
I
think
we're
clocked
in
at
like
34
30.,
so
we're
going
to
start
right
at
the
pen.
We
can't
take
any
questions
to
get
through
all
of
it,
but
I
tell
you
what
we
will
stay
as
long
as
it
takes
to
answer
all
your
questions,
so
don't
think
that
we're
fleeing
the
scene
or
anything
like
that,
I
promise,
we'll
be
with
you
and
I-
think
we're
about
two
minutes
out.
Thank
you
very
much
for
coming
in.
A
A
A
To
go
I'm
good!
Sorry,
ladies
and
gentlemen,
my
name's
Shane
this
is
my
good
friend
will
and
this
whole
week
at
kubecon
you're,
going
to
hear
amazing
success
story
after
amazing
success
story
delivered
by
some
of
the
most
brilliant
people
in
our
industry,
so
will
and
I
we're
going
to
give
you
a
break
from
all
that
right
now
we're
going
to
tell
you
about
a
misadventure.
A
We
had
running
a
large-scale
kubernetes
cluster
and
the
reason
why
we
think
you're
going
to
be
interested
in
this
is
We
Believe,
that
many
of
you
in
this
room
right
now
are
on
this
misadventure
with
us.
It's
just
no
one
told
you
our
misadventure
begins.
We're
all
great
misadventure
begins
a
misunderstanding.
You
see,
we
took
the
things
that
we
knew
to
be
true
in
kubernetes
and
misapplied
all
those
Concepts
to
things
that
were
actually
governed
by
the
Linux
performance
rules,
and
it
turns
out
that's
kind
of
a
big
mistake
to
make.
A
A
Linux
has
an
abstraction
layer
and
it
thinks
in
time
not
cores
now
I
get
it
I'm,
probably
sounding
like
the
crazy
guy
on
the
kubernetes
subway
right.
The
chords
are
a
lot.
So
let
me
explain
that
for
just
a
second
okay,
so
I'm
going
to
use
the
least
controversial
feature
in
all
of
kubernetes
to
explain
this
kubernetes
limits
now
I
know
I,
know
right.
A
Just
as
a
quick
review,
there
is
metadata
attached
to
a
container
a
specific
container
for
how
much
CPU
it
can
take
in
a
given
period
of
time.
That
period
of
time
happens
to
be
100.
Milliseconds
just
follow
that
number
away.
For
me,
real,
quick,
we'll
need
that
in
just
a
few
slides
and
every
period
we
reset
this
quota
of
limits
pretty
straightforward
right.
A
But
what
I
didn't
understand
is
when
I
was
configuring
cores?
What
was
really
going
on
behind
the
scenes
is.
It
was
converting
into
time.
Let's
explain
how
that
works
really
quickly,
see.
In
my
mind,
I
was
configuring,
One
Core
and
that's
broken
up
into
1
000
little
chunks
called
Millis
CPU
right
and
if
I
configure
10
Milli
CPU,
which
is
the
minimum
value.
A
What
I'm
really
saying
here
is
whatever
value
that
I
configure
divided
by
that
base
value
of
one
CPU
or
1
000
Milli
cores,
there's
a
percentage
but
a
percentage
of
what
it's
one
percent
of
this
100
millisecond
period
in
time
or
one
millisecond,
which
is
why
that's
the
smallest
value.
Now,
how
is
that
different?
A
A
Well,
you
see
I
didn't
think
to
have
the
good
sense
to
ask
them
anything
about
this
app
and
it
turns
out
with
our
fictitious
app.
We
have
four
threads
running
on
four
cores
and,
if
I
add
up
the
total
bill,
the
total
amount
of
CPU
utilization
in
this
100
milliseconds
of
time,
that's
passing
in
the
real
world.
A
I
get
400
milliseconds.
That
I've
just
said.
No.
That
was
nowhere
near
the
50
that
I
thought
when
I'm
thinking
a
time.
It
sounds
reasonable,
but
let's
take
a
step
back
right.
If,
if
can
anyone
tell
you
the
right
way
to
set
your
limits?
If
they
didn't
know
this
information,
how
many
threads
you're
running,
how
you're
gonna?
Oh?
How
many
parallel
processes?
A
I
promise
I'll
share
that
with
you
at
the
end,
but
when
we
measure
that
in
time,
what's
the
value
exactly
the
amount
that
we
needed
to
set
it
for
to
keep
it
from
throttling
did
I
need
a
Ouija
board
right,
a
dream
catcher
and
five
million
blogs
to
set
it.
No
I
just
needed,
like
a
five
second
chart
and
Prometheus
to
get
this
all
down.
Was
it
really
that
hard?
It's
not
when
you're
thinking
in
time,
okay,
Shane?
Is
it
really
that
easy
all
right,
I
left
a
few
things
out.
A
A
A
But
what
happens
when
I
go
to
production
and
I,
give
it
an
eight
core
box?
It's
going
to
go.
Oh
you
gave
me
eight
cores
great.
Is
the
number
for
performance
going
to
measure
out
to
be
the
same
if
I
have
eight
OS
threads?
In
fact,
it's
not
whoops
whoops.
So
a
few
things
that
we
have
to
think
about
when
we
do
this
sort
of
thing
cool.
A
So
you
get
this
wrong
like
I
got
it
wrong.
I
used
to
start
to
do
these
percentage
of
periods
and
all
this
type
of
stuff
and
I
wasn't
thinking
in
time.
So
I'd
go
to
Will,
all
judgy
and
stuff
right,
I'd
be
like.
Will
you're
99
throttled
on
your
application?
How
could
you
and
he'd
be
like?
Oh,
oh
dude,
dude?
What
do
we
set
it
for
and
I'd
be
like
I,
don't
know
right,
and
so
because
you
know
it's
just
some
random
percentage
up
there
and
he'd.
A
Look
at
me
all
disappointed
kind
of
like
you're.
Looking
at
me
right
now,
sir,
and
and
I'd
be
like
man.
There's
got
to
be
a
better
way
to
do
this
right.
Well,
you
see
I'd
go
back
and
I'd.
Do
my
engineering
right
because
I'm
an
engineer
I
got
this
right,
we'll
double
it
right,
he's
running
one
we'll
run
two
course,
because
that
sounds
good
to
my
ear.
Oh
you've
done
it.
Don't
you
judge
me
and
so
I
do
two
cores
now
right.
A
A
I
just
have
this
application
performance,
sorry,
buddy,
right
and
in
that
100
milliseconds
of
time.
If
I
didn't
have
another
application
that
needed
to
run
guess
what
goes
out
as
Idle
answer,
I
got
50
utilization,
but
I
got
a
huge
performance
problem.
That
would
be
a
monstrous
problem
to
try
to
figure
out
unless
we
were
thinking
in
time.
A
You
see.
Luckily
I
have
the
CPU
seconds
throttle
total
and
if
I
just
convert
that
to
periods
and
again
I'll
share
this
with
you.
Lo
and
behold,
what
number
is
it?
The
exact
number
I
need
to
raise
it
by
right.
I
had
200
I
needed
to
add
an
extra
200
to
get
it
to
400
the
exact
value
I
needed
to
stop
the
throttles,
stuff's.
Actually
quite
simple.
It's
just
two
little
charts
I
need
to
look
at
for
five
seconds
and
I
didn't
need
all
this
stuff.
A
That's
been
going
on
for
years
right,
but
I
know
exactly
what
you're
thinking
right
now,
you're
thinking,
shame
you
didn't
even
cover
per
CPU
slice
allocation
on
the
global
container
for
the
metadata
of
the
container.
What
kind
of
amateur
night
are
you
running
up
there?
Man,
that's
a
little
harsh,
but
I
got
you
fam.
Don't
worry!
A
I
wrote
a
whole
blog
post
going
into
the
brain
damage
details
of
all
this
now
I
know
the
the
Grand
Master
of
kubernetes
himself,
Tim
Hawkins
tried
to
get
us
not
to
shoot
ourselves
in
the
foot
years
ago,
but
there's
actually
some
cool
stuff.
Coming
out.
You
see
in
a
new
version
of
the
Linux
kernel,
that's
coming
there's
rollover
and
burst
values
for
the
stuff,
so
limits
might
be
cool
again,
but,
more
importantly,
you're
in
a
multi-tenant
talk
right
now
and
us
as
multi-tenant
operators.
A
We
don't
get
to
control
what
people
do
in
our
containers.
We
need
a
safety
net
and
if
you
need
a
safety
net,
this
stuff's
actually
not
too
bad.
If
you
think
in
time
now
someone
if
not
all
of
you
are
going
to
send
me
this,
oh,
but
you
can
do
all
this
stuff
with
requests.
It
turns
out,
that's
actually
a
more
fascinating
topic
and
might
not
work
quite
like
you
think,
to
discuss
that.
Let's
go
explore
that
Universe
for
just
a
bit.
A
Now
some
mathematicians
at
the
time
were
pretty
bummed
because
their
whole
life's
work
was
on
the
universe
was
static
right
overnight,
just
completely
bad.
This
makes
me
feel
a
lot
better,
because
all
my
math
and
all
my
performance
calculations
on
Linux
were
wrong
because
I
thought
the
Linux
Universe
was
static.
A
Now
I
don't
want
to
brag,
but
when
I
was
in
the
United
States
Marine
Corps
I
was
so
good
at
basic
math.
They
gave
me
a
special
box
of
crayons
to
do
hard
things
like
ratios
right.
So
when
it
came
to
take
my
cka
course,
I
was
feeling
pretty
confident.
I'm
like
I
got
this
now
tell
me
if
you
learned
the
same
thing
right,
so
one
request
equals
10
24
shares
and
if
I
got
double
those
amount
of
requests,
I
get
double
the
amount
of
shares
and
just
like
shares
at
a
company.
A
A
I'm
read
my
hero.
Brendan
Gregg
right,
who
makes
me
feel
like
I,
probably
should
be
working
at
Best
Buy
and
not
doing
this,
but
he
says
Hey,
crayon,
eater,
guess
what
it's
only
the
busy
shares
that
count.
Wait.
What
so
you
mean
if
this
container
is
Idle,
it
doesn't
count
in
the
performance
calculation.
It's
just
these
two
that
completely
changes
my
performance
ratio
on
this
node.
Oh,
my
goodness,
but
wait
a
minute
wait!
A
minute.
A
I
have
hundreds
of
PODS
forget
containers,
I
got
startup
containers
and
I'd,
rather
try
to
figure
out
the
heat
death
of
the
universe
than
try
to
figure
out
like
what
container
is
going
to
be
busy
at
any
given
time.
If
I
could
do
that
kind
of
math
I
would
have
gotten
to
ml
right,
and
would
it
be
the
same
if
I
could
calculate
it
on
a
node?
Would
it
be
on
all
thousand
nodes?
A
No,
it
wouldn't
oh
I'm
getting
a
little
anxious
now
right,
but
is
this
practical
you
see
when
I
was
thinking
in
core
as
a
developer
come
up
to
me
and
like
can
I
have
a
core
sure
buddy?
Here
you
go
right,
but
what
was
actually
happening,
let's
say
just
to
stretch
a
point
we
were.
We
were
working
in
stage.
Would
all
the
containers
be
busy
in
stage
like
they
are
prod?
No,
so
I
do
some
performance
crushing
you
all
do
performance
regression
on
your
apps
right,
I
know
I
know.
A
So
we
do
a
little
performance
regression.
What
happens
it
takes
up
that
entire
note,
because
nothing
else
is
going
on
right
if
I
was
thinking
in
cores,
I'd
be
convinced,
I
had
another
core,
but
I'm
actually
taking
up
the
whole
box
whoops.
What
happens
when
I
move
to
production
would
I
get
the
same
performance
for
fall?
Indeed,
I
wouldn't
I'd
be
like
what's
going
on,
and
would
that
be
the
same
across
all
1000
boxes
that
we're
running
and
I.
Think
no,
oh,
my
goodness!
A
Okay,
so
now
I'm
just
disillusioned
right,
I'm
screaming
at
the
lens
guys.
Why
does
this
have
to
be
so
brain
damaging
right
and
it
turns
out?
It
was
just
my
mental
model
was
all
wrong.
You
see
once
these
tasks
or
threads
get
put
on
a
particular
CPU
right.
Really.
What
we're
talking
about
this
weighted
share
system
is
who
gets
to
run
next
on
that
particular
CPU?
A
A
Okay,
to
really
understand
this,
though
we'll
have
to
do
the
completely
Fair,
scheduler
and
gang,
we
have
30
seconds
per
slide,
so
I
need
a
little
overconfidence
for
me
in
the
crowd
right
now
we're
going
to
learn
the
completely
Fair
scheduler
in
exactly
30
seconds.
Are
you
feeling
it
this
morning?
Are
you
ready
yeah?
That's
what
I'm
talking
about
all
right,
a
little
delusional,
overconfidence
right
all
right
gang
will
time
us
go
sweet
all
right
gang!
This
task
is
running
100
milliseconds
on
this
CPU.
It
is
hogging
up
all
the
CPU
time.
A
Meanwhile,
this
poor
task
is
waiting
patiently
for
that
CPU.
It
has
only
had
10
milliseconds
of
run
time
on
the
CPU.
That's
not
fair.
Linux
fare
scheduler
to
the
rescue.
It
sees
this
and
it
says,
wait
a
minute.
This
task
has
the
lowest
run
time.
Let's
put
that
on
CPU
once
these
two
a
tasks
have
the
same
amount
of
time
or
their
fair
share
of
the
CPU
time
bam.
That's
it,
ladies
and
gentlemen,
completely
fair
schedule
of
time.
39.
oh,
come
see
my
presentation
at
overconfidence
tomorrow,
I'm
going
to
kill
it
it'll
be
much
better.
A
I
promise
all
right
all
right
all
right.
So
why
did
I
take
you
through
that?
Okay,
what
is
a
request
at
the
end
of
the
day,
a
little
hand
wavy
here,
but
bear
with
me
all
right,
I
got
my
CPU
Hawk
is
back
run
time,
100
milliseconds
in
time.
What
I'm
actually
saying
here
with
request
is
one
is
divide,
this
runtime
by
the
number
of
shares.
A
Oh,
my
goodness,
we're
messing
with
time
again:
wow,
okay,
all
right!
So,
let's
think
about
this
for
a
minute,
but
am
I
the
Master
and
Commander
of
everything
that's
going
in
the
node
or
is
there
maybe
something
else
that
might
be
in
play?
We
call
that
the
witness
right,
okay,
okay,
here
we
go
all
right.
There
was
something
I
didn't
tell
you
in
all
this,
and
that
was
that
that
10
millisecond
task
was
waiting
on
IO,
and
so
what
was
happening
is
it
didn't
even
need
to
run
on
CPU?
A
A
That's
where
CFS
shines
right
so
I
want
to
let
the
CPU
Hog
Run
as
long
as
it
wants,
and
when
this
packet
comes
back,
I
set
a
flag,
saying:
I
need
to
run
and
I
want
it
to
run
it
as
fast
as
possible,
and
you
know
what
happens
from
here
right,
lowest
time
wins
right.
It
goes
in,
but
it's
an
I
o
task.
So
I
want
you
to
think
about
this.
It's
probably
going
to
run
on
CPU
for
very
small
and
then
go
back.
A
I
O
task
by
default
are
probably
going
to
be
the
smallest
time.
What
starts
happening
when
I
do
aggressive
requests
and
all
this
type
of
stuff
on
something.
That's
already
pretty
aggressive
right.
Oh
wait!
There's
a
lot
going
on
here,
evidently
cool.
So
how
did
I
get
this
fictitious
version
of
the
universe
right?
Well,
it
turns
out,
you
know,
I
did
CPU
stress
and
all
the
containers
at
the
same
time,
and
we
know
if
all
containers
are
busy
that
doesn't
happen
in
the
real
world.
Right
I
ran
the
note
at
100
busy.
A
Hopefully,
that's
not
happened
in
the
real
world
right
I
had
a
fixed
number
of
threads
when
they
could
be
dynamic
and
I
had
no
I
O
based
things.
So
I
got
this
like
fictitious
version
of
the
universe
and
how
that
worked.
Okay.
So
what
would
be
this
new
mental
model
that
I
would
have
to
pin
like
what
would
that
look
like
and
in
my
head?
It
looks
something
like
this.
So
I
have
two
containers,
we'll
call
them
task
groups
each
with
four
threads
or
tasks
in
them.
There's
a
million
places.
A
We
can
go
with
this,
but
I'm
going
to
focus
on
this
concept
of
saturation.
You
see
this
is
what
my
node
really
looks
like
and
on
this
CPU
I
have
one
task:
it's
all,
so
the
caches
are
all
warm
it's
at
its
Peak
Performance.
However,
this
one,
where
I
have
two
tasks
going
on.
It's
going
to
take
double
the
amount
of
time
right
to
get
that
same
resource
now.
What
I
didn't
know
is
if
the
CPU
wasn't
100
man
I
was
packing
that
note
as
dense
as
a
neutron
star
right.
What's
the
big
deal?
A
Let's
just
pretend
right
that
these
two
CPUs
are
100
utilized
I'm,
going
to
stretch
a
point
to
make
a
point
here
right,
but
this
task
is
actually
perfect
right
there,
it's
it's
100
utilized,
but
that's
fine
right
because
it's
perfectly
optimized,
but
you
let
me
bin
pack,
your
notes.
So
on
this
side,
oh,
my
goodness,
right
I've
got
so
many
tasks,
contexts
switching
I'm,
not
getting
any
real
work
done.
A
A
Now,
I'm
a
little
crestfallen
I'm
not
going
to
win
the
Nobel
Prize.
For
any
of
this,
it
looks
like
there
has
been
a
family
of
node
metrics,
called
pressure
style
information
that
tell
you
all
of
this
wonderful
data
right.
We
just
didn't
know
to
turn
it
on
right
and
what
we
did.
The
the
findings
were
pretty
crazy,
so
sure,
okay,
there
was
like
you
know,
50
percent
of
threads
waiting
and
all
that
type
of
stuff.
Maybe
we
should
use,
go
Max,
procs
or
something
to
limit
the
thread,
counts
and
stuff
like
that.
A
But,
interestingly
enough
that
wasn't
the
most
interesting
thing
that
we
found.
We
found
that
we
were
stalled
on
memory
up
to
10.
We
were
thrashing
the
memories
on
the
box,
they
were
saturated
and
the
most
fascinating
thing
was
IO
stalled
means
every
single
thread
on
the
box
was
stalled
up
to
35
of
the
time
whoops
because
we
weren't
even
looking
at
this
sort
of
thing.
A
B
I'm,
like
yeah
all
right
all
right,
Shane
click,
hey
babe,
I,
think
I,
think
Shane
hit
his
head
bungee
jumping
again,
but
in
the
morning
started
thinking
about
some
of
these
decisions
that
we'd
made
at
Aquia.
In
a
little
bit
of
a
Different
Light
see
we
were
scaling
our
customer
workloads
on
CPU
utilization.
As
a
percent
of
requests
and
I
mean
that's
a
kubernetes
default
right.
It
practically
comes
out
of
the
box.
So
could
it
really
bit
us
that
bad
well,
it
turns
out
it
could.
B
B
What
is
percent
of
requests?
I
never
really
thought
about
it
before,
but
when
Shane
tells
us
all,
this
about
requests.
I
mean
he's
talking
about
these
weights
right
and
like
the
Linux,
Fair,
scheduler
and
stuff,
like
that
he's
not
talking
about
like
using
a
certain
percent
of
your
weight
right.
What
does
that
even
mean?
B
B
Well,
not
something
actually
to
be
specific,
it's
all
the
processes
and
all
the
threads
in
that
container
added
up
on
average
per
second,
and
we
just
divide
that
by
the
request.
So
if
something
ran
for
500
milliseconds
per
second
on
average
on
the
CPU
and
it
requested
a
thousand
Milli
cores,
we
just
take
those
units
we
throw
them
out
the
window
and
we
do
500
divided
by
a
thousand
and
we
get
50
of
requests
right,
simple
enough.
B
It
shouldn't
surprise
us
at
this
point
that
on
a
four
core
node
that
first
container
is
going
to
get
one
CPU
second,
on
average
and
the
second
one
is
going
to
get
three
and
when
we
do
that
division
out
with
Shane's
special
box
of
crayons.
We
do
one
second
divided
by
one
core
and
we
get
a
hundred
percent
simple
enough
right.
I
mean
this
is
kind
of
intuitively.
How
we
would
expect
all
this
stuff
to
work.
B
B
B
Sometimes
that
comparison
is
a
perfectly
valid
one
to
make.
It
gives
us
a
lot
of
information,
but
by
no
means
does
it
give
us
a
complete
and
accurate
view
of
what's
actually
going
on
within
the
system.
Let's
look
at
a
more
extreme
example
of
this.
Let's
say
we
have
just
one
container
and
every
time
he
gets
a
request
like
a
web
request
or
something
it
has
some
work
to
churn
through
and
that
work
takes
about
eight
seconds
of
CPU
time.
B
But
let's
say
we
weren't
thinking
in
time.
Right,
let's
say:
okay!
Well,
that's
a
pretty
big
container
right.
It's
doing
a
lot
of
work,
I'll,
give
it
four
cores
right.
Four
chords
has
to
be
enough
right.
It
ran
on
a
four
core
box
before
four
cores
is
plenty
well,
what
happens
when
we
try
to
run
that
on
kubernetes
we
send
that
up
to
the
API
server
and
it
says
great,
you
want
four
cores
I'll
give
you
four
cores
here.
You
go,
but
let's
say
it
schedules
us
on
an
eight
core
node.
B
Well,
when
that
container
comes
up
on
Linux
and
there's
nothing
else
running
on
the
box,
it
can
really
spread
out
right
over
all
eight
cores,
which
means
it
can
get
its
eight
seconds
of
work
done
in
just
one
second
of
real
time,
which
is
wonderful
right,
I
mean
we're
getting
a
lot
of
work
done.
As
fast
as
we
can,
but
when
we
look
at
this,
when
we
look
at
our
metrics
we're
going
to
see
eight
seconds
divided
by
four
cores,
200
percent
of
requests-
oh
no,
we
say
right,
200
of
requests.
We
must
be
overloaded.
B
B
Let's
say
we
just
set
that
you
know
dash
dash
scale.
Replicas
two
done
next.
Pod
comes
up,
kubernetes
helps
us
out
right,
doesn't
want
to
waste.
Space
puts
us
on
the
same.
Node
gives
us
another
four
chords,
but
now
when
that
container
comes
up,
it's
not
a
free
range
anymore
right
Linux
has
to
constrain
both
of
those
workloads
equally
because
they
have
the
same
request
to
50
of
the
time.
B
But
when
we
look
at
our
percent
of
requests,
we're
going
to
say
okay,
four
seconds
over
four
cores
hundred
percent
of
requests
right,
hooray,
we
solved
the
problem-
we're
not
overloaded
anymore,
but
is
this?
Is
this
a
good
thing?
Did
we
help
I
mean
we
still
have
the
same
eight
seconds
of
work
right
that
didn't
change,
but
in
the
first
case,
when
we
have
one
container,
it's
getting
it
done
in
one
second,
and
by
adding
that
second
container
we're
doubling
the
amount
of
time
it
takes
to
get
that
work
done.
B
But
if
we
just
look
at
this
percent
of
requests,
which
is
all
the
HPA
is
doing
right,
we
could
be
misleading
ourselves
pretty
easily
to
think
that
we're
actually
helping
that
we're
actually
doing
something
good
right.
Now:
okay,
fine
requests-
maybe
maybe
it's
a
little
misleading
right.
Maybe
it's
a
little,
not
the
whole
picture
right,
but
maybe
we
should
be
asking
ourselves
a
little
bit
of
you
know
we
should
back
it
up
a
little
bit.
Maybe
we
should
have
some
more
fundamental
question.
B
The
one
on
the
left,
though
right
every
web
request
is
getting
it's
just
doing
some
data
processing
right
number
crunching,
something
like
that
that
all
shows
up
on
the
CPU,
wonderful
but
more
typical
web
applications
right.
They
don't
necessarily
do
that.
Let's
say
we're
waiting
on
a
database
or
maybe
we're
calling
out
to
some
external
thing.
It
doesn't
really
matter.
None
of
that
shows
up
as
CPU
utilization,
so
we
have
two
wildly
different
CPU
utilizations
and
the
same
saturation.
B
Well,
fine!
Okay,
at
this
point
you
know
I
give
Shane
a
call
back
and
I
say:
Shane
I,
don't
say
this
enough,
but
you
were
right.
I
don't
like
for
Center
requests
anymore
right,
it's
it's
not!
It
doesn't
I,
can't
trust
it
CPU.
You
know
it
doesn't
even
work
for
my
application.
What
do
I
do?
What
do
I
scale
on?
How
do
I
solve
this
problem
right,
and
he
looks
at
me
with
that
confident
glitt
in
his
eyes
that
39
second
confident
glint
in
his
eyes-
and
he
says.
A
B
Which
is
what
all
the
blogs
say,
right
requests
per
second
scale.
Your
applications
are
in
quest
per
second
because
of
that
specific
CPU
problem,
but
it
turns
out
requests
per
second
has
kind
of
exactly
the
same
problem
see
we
run
a
lot
of
different
applications
at
Aquia,
right,
some,
let's
say:
they're,
really:
Simple
Sites,
maybe
they're
serving
static
content
or
some
right,
not
a
complicated
request
to
fulfill.
So
that
can
take
a
lot
of
requests
per
second,
but
the
more
complicated
sites
right
spending
a
lot
of
time
in
the
database.
B
They
can't
take
as
many
requests
per
second
no
judgment
requests
per.
Second,
just
like
CPU
right.
It
tells
us
what
the
application
is
doing,
namely
how
fast
it's
processing
requests,
but
it
doesn't
give
us
that
context
to
tell
us
how
many
more
it
can
do
before
we
actually
need
another
replica
before
before
it's
going
to
fall
over,
so
we
really
sat
back
and
we
thought
about
it.
B
We
said,
okay,
what
do
we
want
to
scale
on
and
we
decided
that
any
metric
that's
worth
scaling
on
needs
to
at
the
very
least,
it
needs
to
be
highly
correlated
with,
what's
actually
going
on
inside
the
application
right.
At
the
very
least,
we
wanted
something
smooth
right.
We
didn't
want
these
big
spikes
up
and
down
and
wasting
all
that
time
scheduling
all
those
pods.
B
And
lastly,
we
wanted
an
early
enough
signal
that
we
could
actually
respond
to
it
right,
so
not
something
like
latency,
which
sure
is
very
highly
correlated
right,
but
if
you're
already
having
high
latency,
then
you've
already
failed
right,
you're
scaling
too
late,
see
every
application.
Has
this
kind
of
curve,
as
you
apply
load
when
the
loads
light
right.
B
Your
application
can
handle
it
no
problem,
but
maybe
it's
not
as
utilized
as
it
otherwise
could
be,
and
if
you
scale
at
that
point
right,
you
end
up
just
kind
of
scaling
empty
right,
we're
not
taking
as
much
as
a
good
advantage
of
the
resources
where
we
have
available
to
us.
But
if
you
push
that
load
too
far
right
you
end
up
introducing
latency
or
drops
or
errors
or
crashes
or
ooms
or
whatever
right
like
you
had
well,
hopefully
not
items,
but
you
end
up
with
issues
in
your
application
right.
B
What
we
were
looking
for
is
a
metric
that
told
us
where
that
inflection
point
was
right
right
before
we
start
introducing
issues,
and
we
wanted
to
set
our
HPA
to
a
little
bit
before
that
right,
the
idea
being
that
as
we
scaled
up
and
down,
we
would
be
holding
the
majority
of
our
replicas
in
that
optimized
Zone
right
now,
for
us
that
ended
up
being
essentially
the
number
of
threads
that
were
active
at
any
one
time
right.
It
didn't
tell
us
what
those
threads
were
doing
and
it
didn't
need
to
it.
B
Just
told
us:
hey,
they're
busy.
They
can't
take
another
request
right
and
when
you
have
a
fixed
number
of
threads,
like
Shane,
was
saying
you
know,
kind
of
how
much
of
that
you
can
take
until
you
need
another
replica
now
for
you.
This
metric
is
very
likely
to
be
something
very
different
right,
depending
on
the
application
and
the
language
and
all
that.
But
the
point
is
the
process
we
went
through
to
find
that
metric.
B
It
forced
us
to
learn
a
lot
about
our
application
about
what
happened
when
we
apply
load
to
it
and
what
it
looked
like
when
it
was
about
to
fall
over
right
and
I'm
happy
to
report.
We
started
rolling
this
out
to
production
and
we
drastically
reduced
the
churn
right
turns
out
when
you're
scaling
on
the
proper
metric.
B
Maybe
you
don't
need
to
be
running
73,
000
buds
every
single
day
right
now
in
our
customers
scale:
it's
because
they
actually
need
to
right
because
they
they
need
the
threads,
which
makes
us
really
happy
because
it
greatly
simplifies
our
cluster
setup
right.
We
don't
have
these
crazy
churn
and
all
the
problems
that
come
with
it
and
that's
a
whole
nother
talk
and
it
makes
our
customers
happy
right,
because
a
lot
of
customers
aren't
particularly
CPU
heavy
right.
B
A
Gang
our
time
together
is
that
and
then,
but
two
things
before
you
go.
You
see
we
got
caught
up,
that
what
we
needed
was
webassembly
based
injected
Envoy
BPF,
and
it
was
the
fundamentals
that
we
had
wrong
the
whole
time
and
when
we
focused
on
those
fundamentals,
that's
when
we
got
the
biggest
gain
and
we
thought
someone
was
going
to
come
on
a
silver
platter
with
this
best
practice.
A
This
thing-
and
it
was
always
wrong
because
we
had
to
measure
what
was
going
on
to
really
understand
what
the
best
practices
were
for
our
cluster,
and
it
turns
out
that
Prometheus
is
the
way
to
do
that.
But
it
turns
out
everything
we
knew
about
all
that
stuff
was
wrong
too,
and
that's
an
even
more
fascinating
universe,
but
that's
a
tale
for
another
day.
A
On
a
personal
note,
gang
I
just
want
to
give
you
a
heartfelt
I
got
rejected
for
kubecon
three
times
in
a
row,
and
this
is
like
my
last
big
thing,
and
so
the
fact
that
it's
like
standing
room
only
and
700
of
you
guys
showed
up.
I
can't
tell
you
what
that
means
to
us.
This
is
a
big
labor
of
love.
We
spent
three
months
on
it.
We'd
love
to
get
your
feedback.