►
From YouTube: Throttling: New Developments in Application Performance with CPU Limits - Dave Chiluk, Indeed
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Throttling: New Developments in Application Performance with CPU Limits - Dave Chiluk, Indeed
Are you seeing excessively long tail response times from your applications running on containerized clouds (Kubernetes, Docker, Marathon)? Have you ever seen an application be throttled even though it’s no where near its CPU limit? Up till now, the answer has always been simply turn off hard-limits, but that has potentially nasty performance implications in shared environments. Now there's another option! This session will explain the real root cause of what has been happening. We'll introduce the kernel mechanisms that Kubernetes and other Container Orchestrators rely on to enforce CPU limits. We'll then show how they were broken, how we fixed them, and what those changes mean for you and your clouds. By the end of this session you'll understand exactly what you are getting when you set the CPU limits on your pods.
https://sched.co/Uae1
A
All
right,
thank
you,
so
hello
I'm
the
track
host
for
this
session.
That
means
I
get
to
introduce
the
speaker
and
I
get
to
run
around
with
the
microphone
I
was,
my
name
is
Brian
Boehm
I
worked
for
a
company
called
weave
works
and
I
was
program
committee
member
actually
reviewing
the
performance
track
so
I'm
very
pleased
to
say
this
is
one
of
the
sessions
I
wanted
to
see
voted
for
in
the
the
selection.
It
is
performance
of
systems,
and
especially
kubernetes
very
dear
to
my
heart.
B
All
right,
so
this
is
a
throttling
new
developments
in
application
performance
with
CPU
limits.
As
I
was
introduced,
I
am
Dave
Cilic
I'm
a
Linux
platform
software
engineer
at
indeed
so,
first
off
what
in
the
world
does
that
mean?
That
means
I
work
on
the
OS
I
am
partially
a
kernel
engineer,
partially
an
OS
engineer
prior
to
indeed,
I
worked
for
a
company
called
canonical
which
owns
a
little
distribution
called
Ubuntu.
B
I
am
a
core
developer
for
Ubuntu
I
achieved
those
rights
there
and
I
still
maintain
those
rights,
but
I
was
also
attached
to
the
kernel
team.
There.
So
I
have
some
credibility
with
with
kernel
engineering
so
before
before
we
go
too
far.
I
do
work
for
indeed,
and
indeed
we
help
people
get
jobs,
we
have
250
million
unique
monthly
visitors.
We
have
about
25
million
jobs
on
our
site
and
we
have
1
million
employer-sponsored
jobs.
We
have
roughly
10,000
employees
at
the
moment
of
which
about
2,000
are
engineers.
B
B
How
you
set
that
in
kubernetes
you
go
into
your
pods
pod
definition
and
set
limits
under
resources
limits
CPU
you
can
set
it
to,
in
this
case
200
Mill
of
course,
or
requests
CPU,
134
ml.
Of
course,
I
divide
these
into
a
hard
limit
and
a
soft
limit.
That's
one
of
the
things.
I
kind
of
try
to
try
to
conceptualize
I
will
use
those
terms
interchangeably
when
I
say
hard
limits,
I
mean
hard
limit,
I
mean
limits
and
if
I
say
soft
I
mean
requests.
B
But
what
do
these
mean?
Well,
soft
limits
under
the
covers
are
using
C
group.
Cpus
years,
if
you
actually
navigate
to
the
CFS
sorry,
the
C
group
file
system
on
your
node,
you
will
see
under
sisyphus,
C
group
and
then
your
container,
a
file
called
CPU
shares.
If
you
cat
that
file,
you
will
see
that
it
has
the
usually
the
same
number
as
the
milk
or
is
that
you
requested
in
kubernetes
unless
you've
got
some
weird
mutating
Ingres
things
now.
What
does
that
mean?
What
is
that
number?
B
That
number
is
a
number
of
shares
that
you've.
Given
that
process
the
amount
of
CPU
time
you
get
for
that
process
is
proportional
to
the
amount
of
shares
for
all
containers
on
that
node.
So
if
you
take
the
shares
and
divide
it
by
the
sum
of
all
shares
for
all
of
their
products,
use,
that's
how
many
CPUs
you
are
guaranteed
to
be
able
to
use.
So
let's
do
in
a
little
bit
example
on
our
nodes.
We
have
88
course
88
core
node.
B
B
Now,
if
we
add
another
pod
to
that
node,
our
minimum
drops
to
44,
but
we're
still
possibly
able
to
use
all
88
course
do
that
again
with
to
another
mm
mm,
no
pod
and
we
drop
down
to
22
and
so
on,
get
down
to
11.
So,
basically,
you
can
consider
your
requests
as
a
floor
for
usable
CPU
for
that
pod.
Okay,
now,
let's
look
at
hard
limits,
hard
limits
under
the
covers
use,
C
group
CFS,
bandwidth
control.
B
If
we
go
back
into
that
directory
under
our
C
group
file
system,
we
see
that
there's
two
settings
for
this
CFS
period:
microseconds
and
CFS
quota
microseconds
for
the
period
it
is
by
default,
set
to
100,
milliseconds
or
100,000
nanus
microseconds
for
the
quota
that
is
equal
to
the
limit
that
you
set
for
your
pod.
In
this
case,
we
have
10,
milliseconds
set
or
100
mil
occurs
after
you've
used
all
of
the
quota
for
your
pod.
B
Cfs
bandwidth
control
gives
us
a
number
of
metrics
that
we
can
look
at
in
order
to
determine
how
our
pods
are
running,
and
if
we
look
again
in
that
seat
in
the
C
group
file
system,
we
have
a
file
called
CPU
stat.
It
provides
three
metrics
and
our
periods
and
our
throttles
and
throttle
time
we'll
start
with
throttle
time
because
I
think
it's
the
least
useful
throttle
time,
but
it's
probably
also
what
most
of
you
are
looking
at
throttle.
Time
is
the
sum
total
of
time
a
thread
in
AC
group
was
throttled.
B
What
does
that
mean?
So
that
means,
if
you
have
one
thread
in
your
application
and
it
was
throttled,
490
milliseconds
throttle
time
will
increase
by
90
milliseconds.
If
you
have
a
hundred
threads
in
your
app
in
your
pod
and
it
each
of
those
is
throttled.
490
milliseconds
throttle
time
will
increase
by
9000
milliseconds
every
hundred
milliseconds.
So
when
you're
looking
at
this
throttle
time,
it's
kind
of
useless,
because
you
have
to
also
know
how
many
threads
are
actually
running
in
your
pod.
B
What
I
prefer
to
use
instead,
which
is
more
agnostic,
is
an
hour
periods
and
NR
throttles.
So
an
hour
periods
is
the
number
of
runnable
periods
that
your
application
of
number
of
periods
that
your
application
was
running.
What
is
running
mean
that
means
actually
using
the
CPU
not
blocked
on
I/o,
not
blocked
on
network,
not
blocked
on
something
else,
not
sleeping
okay
and
our
throttled
is
the
number
of
those
periods
that
you
are
runnable,
where
you
actually
used
your
entire
quota.
B
Okay,
so
what
we
use
it
indeed
to
determine
how
badly
an
application
is
being
throttled?
Is
we
use
this
calculation?
So
we
take
the
change
in
NR,
throttle
and
divide
it
by
the
change
in
an
hour
periods,
and
we
call
that
throttle
percentage.
So
basically,
it's
the
percentage
of
time
that
you
are
running
that
you
got
throttled
regardless
of
how
many
threads
you
were.
You
had
all
right.
So
that
was
a
lot
of
words.
B
Let's
do
a
little
a
bit
of
an
example,
so
we
can
all
understand
a
little
bit
more
thoroughly
as
to
what
I'm
talking
about
we're
gonna
start
with
an
unconstrained
example.
This
little
application
requires
200
milliseconds
of
time
to
complete
a
request.
So
maybe
it's
a
big
application.
If
request
comes
in
at
time
0
we
would
run
for
200
milliseconds
and
they
move
to
respond
at
200
milliseconds.
If
we
were
unconstrained,
let's
now
apply
a
point
for
CPU
quota
or
a
400
mill
Accor
limit
to
that
application.
B
Again,
request
comes
in
at
time,
0,
but
now
it
runs
for
40
milliseconds
and
then
is
throttled.
Ok
and
it
does
that
again,
quite
a
few
more
times
before.
Finally,
before
it
finally
responds
at
440
milliseconds,
everyone
I
see
everyone
staring
at
the
slides
and
being
all
attentive.
This
is
great
all
right.
So
what
we
get
from
our
statistics
remember:
CPU
stat
from
earlier
is
we're
gonna
get
a
CPU
usage
of
200
milliseconds.
We
still
used
200
millisecond,
but
we
were
our
throttle.
B
Now,
if
we
were
monitoring
CPU
usage,
we
see
that
CPU
usage
is
actually
0.4
because
40
milliseconds
or
sorry
200
milliseconds,
divided
by
440
500
milliseconds,
is
0.4
CPU.
Okay,
that's
not
the
problem.
We've
all
been
seeing
right.
All
right,
I
see
some
nods
cool,
so
here's
the
problem.
This
is
what
the
problem
looks
like.
These
are
p95
response
times
of
one
of
the
applications
running
in
in
deeds'
infrastructure.
We
noticed
first
off
that
it's
a
bimodal
distribution.
We
have
some
really
fast
instances
and
we
have
some
really
slow
instances.
B
Those
slow
instances
are
having
two
plateaus
one
at
100
milliseconds
and
one
at
200
milliseconds.
If
you
ever
see
this
kind
of
behavior
in
your
applications,
you
should
immediately
think
throttling
because
of
those
100
millisecond
periods,
I
was
throttled
at
100.
Milliseconds
I
had
to
wait
for
that
hundred
milliseconds
to
expire
before
I
could
get
more
time
to
finish
up
the
request,
or
in
the
worst
case,
this
200
milliseconds
I
was
throttled
for
two
periods
before
I
got
got
runnable
again,
but
the
question
here
is
why,
in
the
world,
is
there
at
that?
B
Second,
that
bimodal
distribution,
we'll
look
at
that
in
a
little
in
a
little
bit?
First
thing
you
do
when
you
see
this
kind
of
a
problem.
Is
you
look
at
your
CPU
usage
and
that's
exactly
what
we
did?
We
see
that
there
we
have
a
limit
set
at
point:
six
CPU,
our
nodes
of
our
applications
are
very
dense,
but
even
looking
at
that,
the
average
utilization
on
this
machine
on
this
application
was
like
point.
B
Three
CPU
I
clearly
had
plenty
of
time
which
I
should
have
been
able
to
use,
and
this
is
the
problem
that
everyone's
been
seeing.
However,
if
we
look
at
throttled
percentage,
remember
that
bimodal
distribution
will
the
slow
nodes.
Those
slow
instances
are
being
throttled
80
to
90
percent
of
the
time.
The
fast
instances
worth
being
throttled
only
five
to
ten
percent
of
the
time.
Okay.
B
So
how
does
this
happen?
How
do
we
have
low
CPU
usage
with
high
amounts
of
throttling
anybody
all
right,
good
you're
in
the
right
place,
then
first
thing
we
did
is
we
increase
the
CPU
quota,
the
CPU
limits
for
those
applications
and
instantly
the
amount
of
throttling
that
both
of
those
those
slow
instances
received
was
dropped
to
almost
nothing
and
our
response,
our
p95
response
times
dropped
from
over
200
milliseconds
down
to
13
milliseconds,
just
like
it
should
have
been
okay.
So
that's
the
problem,
that's
what
we
wanted
to
go
about
and
fix.
B
Let's
recap:
what
do
we
know?
We
know
that
increasing
the
cpu
quota
mitigates
throttling,
but
we
also
have
all
these
questions.
What
are
the
root
causes
here?
When
we
were
looking
in
that
bimodal
distribution,
we
were
trying
to
figure
out
what
the
what
the
similarities
were
between
those
nodes
or
what
the
differences
were
between
those
nodes.
We
came
up
with
a
few
things.
The
slow
nodes
had
high
core
count.
The
slow
nodes
were
a
newer
CPU
architecture.
How
in
the
world,
could
a
newer
CPU
architecture
with
more
cores
result
in
lower
performance?
B
That's
exactly
what
we're
seeing.
The
other
thing
that
was
kind
of
unclear
is
that
there
was
a
mix
of
kernel
versions
in
there,
so
the
some
of
the
fast
nodes
were
running
newer
version,
newer
versions
of
the
kernel,
some
of
the
slow
nodes
were
running
newer
versions
in
the
kernel,
but
it
wasn't
really
quite
obvious
where
the
distribution
happened
and
the
other
thing
that
we
were
looking
at
is.
We
were
looking
at
spectrum
meltdown
in
mitigations,
because
this
was
around
the
time
that
all
of
the
spectrum
meltdown
stuff
was
coming
through
the
pipes.
B
It
would
aid
in
figuring
out
what
the
root
cause
of
this
was,
but
it
would
aid
in
proving
to
the
kernel
community
that
we
had
actually
solved
the
problem
with
whatever
change
we
came
up
with
so
I
went
through
and
through
a
number
of
things
that
probably
you've
all
tried
to
do.
You
run
a
be
against
your
application,
run.
Stress,
ng,
I,
read
I,
wrote
this
random
Bash
curls
of
bash
script,
wrapped
around
curl
in
asleep,
and
that
actually
kind
of
worked
like
half
the
time.
B
But
it
gave
me
lots
of
false
positives
and
I
really
figure
out
what
in
the
world
was
going
on,
then
will
my
co-workers
my
coworker
John
came
up
and
said
if
I
set
go
max
procs
on
my
going
application.
Two
three
on
these
nodes.
I
can
decrease
the
amount
of
CPU
usage
simultaneously
decreasing
the
amount
of
throttling
and
decrease
the
amount
of
response
times
for
that
application.
Well,
that
was
super
interesting
and
super
helpful.
B
So
let's
update
what
we
know.
We
know
that
increasing
the
CPU
quota
helps
us
mitigate
throttling.
We
know
that
decreasing
the
number
of
threads
now
in
our
application
also
helps
mitigate
throttling.
So
in
go
Lane,
you
said:
go
max
procs,
but
in
Java,
if
you
move
to
the
newer
JVM,
so
I
think
it's
you
191
or
newer.
Sorry,
eight
u-19,
not
191
or
newer.
There
actually
C
group
aware
which
helped
to
decrease
the
number
of
threads
they
spawn.
B
Java
is
still
really
bad
about
spawning
tons
and
thousands
of
threads,
even
after
that,
so
it
doesn't
completely
mitigate
the
problem.
The
other
thing
you
can
do
is
move
from
whole
from
fractional
to
whole
CPU
shares
in
kubernetes.
This
will
allow
you
to
use
this
will
allow
kubernetes
to
tasks
at
your
CPU
test,
set
your
pods
to
individual
CPUs.
That's
the
nuance
that
I
learned
separately.
That
I
just
happen
to
learned
around
this
time,
but
whatever
to
do
is
we've
got
to
create
that
reproducer
and
then
we're
gonna
probably
have
to
fix
the
kernel
scheduler.
B
So
first,
the
reproducer
I
wrote
this
thing
called
fib
test.
The
test
is
a
multi-threaded
calculation
of
the
fibonacci
sequence.
I
didn't
care
what
it
did.
I
just
didn't
want
it
to
be
optimized.
The
way
I
wanted
to
turn
through
CPU
time
and
tell
me
that
it
used
the
right
amount
of
time.
It
had
two
things
fast:
threads
and
slow
threads,
the
fast
threads
calculated
as
fast
as
possible.
That's
the
death
sequence
and
the
slow
threads
would
calculate
a
hundred
iterations
and
then
sleep
for
10
milliseconds
and
here's.
B
B
But
if
we
run
fib
test
here,
we
have
we're
running
it
with
one
thread:
one
fast
thread
and
the
rest
are
all
slow
threads,
so
one
thread
16
threads,
one
being
fast,
15
being
slow
and
in
the
worst
case,
88
threads,
1
being
fast
78
being
87
being
slow.
If
we
look
at
the
performance,
the
number
of
iterations,
this
application
is
able
to
perform.
B
We
see
that
in
the
single
threaded
case
it
performs
the
most
number
of
iterations,
which
is
kind
of
expected,
because
we
have
to
dedicate
time
to
context
switching
and
everything
else
to
those
for
that
slow
thread,
but
we
see
14
52
million
iterations
in
the
fast
case
over
one
second,
with
50
milliseconds
of
quota
and
in
the
slowest
case
on
88
course,
we
see
275
million
iterations.
So
it's
almost
5x
decrease
in
processing,
but
here's
the
key.
B
The
amount
of
time
used
dropped.
Incredibly,
if
we
look
at
the
single
threaded
case,
we
have
we're
using
539
milliseconds
in
that
period
right
in
that,
in
that
second
of
time,
50
milliseconds
times
10
periods
of
100
milliseconds
is
500
milliseconds.
So
we
were
expecting
so
a
little
bit
of
overhead
for
the
wrapper
script
to
do
some
calculations
and
bash
because
it's
slow,
but
you
know,
539
milliseconds
seems
reasonable.
B
On
the
single
threaded
case
on
16
CPUs
we're
seeing
530
milliseconds
of
usage,
that's
a
little
bit
less,
but
still
not
terrible,
but
in
the
88
core
instance,
we
see
only
183
milliseconds
of
CPU
usage.
Ok,
here's
our
problem,
we're
not
actually
getting
the
time
that
we
were
allocated.
We
were
expecting
that
88
core
example
to
get
500
milliseconds
or
more
all
right.
That's
actually
a
3x
difference
in
the
amount
of
CPU
you
were
able
to
use
on
that
hike
or
count
machine.
B
So
if
we
run
F
trace
on
this,
shout
out
to
my
good
friend,
Steve
Ross
stet,
who
you
know
inspired
me
to
take
photos
of
my
audiences,
he
actually
wrote
tracing
I've
traced
in
the
kernel,
but
he
also
wrote
kernel
kernel
shark.
So
this
is
a
picture
of
kernel
shark
run
against
fib
test.
We
see
on.
We
see
we
have
our
fast
thread
and
we
have
a
bunch
of
slow
threads.
Oh
sorry,
keep
in
mind.
This
is
on
the
417
kernel.
Okay,
this
is
the
kernel
that
doesn't
have
the
problem.
B
We
see
that
the
the
amount
of
time
between
the
start
of
our
fast
thread
and
the
start
and
the
restart
of
that
fast
thread
is
100
milliseconds,
which
is
exactly
equal
to
our
period,
and
we
see
that
that
fast
thread,
which
is
meant
to
chew
up
as
much
CPU
time
as
possible,
is
getting
48
milliseconds
out
of
that
50
millisecond
quota.
That's
exactly
what
we
want
to
see.
B
B
The
quota
I've
allowed
minus
the
number
of
CPUs,
so
if
I
had
had
I
had
50
milliseconds
of
quota
and
the
number
of
CPUs
was
16,
which
is
30
so
50
minus
16
is
34
kind
of
convenient
all
right,
that's
a
little
foreshadowing,
but
using
this
behavior
I
was
able
to
create
I
was
able
to
do
a
git
bisect
on
the
kernel
and
I
identified
the
causal
commit
and
that
causal
commit
is
5.
1,
2,
AC,
9,
9
9.
That
number
is
burned
into
my
head
forever.
B
It's
like
my
best
friend's
phone
number
from
one
as
a
child,
but
when
you
look
at
this
this
commit
it
is
a
fix
for
inadvertent,
throttling
due
to
clock
drift.
How
in
the
world
did
a
fix
for
inadvertent
throttling
due
to
clock
drift
caused
us
to
get
inadvertent,
throttling
okay,
so
before
I
go
on.
This
is
the
commit
that
a
lot
of
you
have
been
seeing
for
a
lot
of
for
the
last
few
years.
This
is
one
of
them.
This
actually
fixes
a
problem
that
a
lot
of
you
have
been
seeing
so
check
your
kernels.
B
Make
sure
you
have
this
fix,
but
that's
not
the
problem.
That's
not
the
fix
that
I
needed,
but
we
did
hit
that
fix,
because
there
were
still
some
older
nodes
that
hadn't
yet
been
updated
with
the
newest,
with
the
kernel
that
had
that
this
is
what
a
kernel
aim.
An
application
without
5-1
to
a
c99
work,
looks
like
the
CPU
usage
on
this.
Node
is
0.1
CPU.
B
Now,
if
we
looked
at
five
to
one
to
AC
ninety
nine
more
closely,
we
noticed
that
it
did
something
additional
to
just
fixing
that
inadvertent
throttle
throttling
due
to
clock
skew
it
also
fixed
per
CPU
quota
to
expire
on
the
period
boundaries.
Okay.
So
if
you
read
the
kernel
documentation,
which
is
amazing
by
the
way,
if
you
haven't
read
it,
it's
pretty
just
actually
pretty
darn
good.
B
The
last
few
years,
The
Seeker
CFS
bandwidth
control,
implies
that
you
only
are
allowed
strictly
to
use
the
amount
of
bandwidth
that
you
were
given
every
period,
no
more,
no
less
well.
This
fix
enforced
that
strict
nature.
Okay,
so
that
gave
me
a
clue
this
also.
This
really
is
what
required
me
to
figure
out
what
was
going
on.
B
So,
let's
update
our
real-world
verses
that
earlier
conceptual
model,
the
complications
we
have.
We
have
multiple
CPUs.
Now
we
often
now
have
multiple
threads
oftentimes.
Thousands,
if
you're
running
Java
and
cores
run
at
different
speeds.
I
haven't
touched
this
at
all,
but
this
is
something
that
you
should
also
take
away
from
this
talk
since
every
since
all
of
this
is
based
on
wall
time
and
cores
run
at
different
CPU
speeds.
B
If
your
core
is
down
clocked,
when
you
get
running
on
the
on
the
processor,
your
performance
is
going
to
be
negatively
affected,
because
five
milliseconds
running
at
800
megahertz
is
way
less
processing
power
than
five
milliseconds
running
at
3.6
gigahertz
or
whatever
else
your
cores
are
able
to
do
so.
One
of
the
things
we
did
was
made
sure
that
all
of
our
nodes
are
in
performance
mode.
B
Last
thing
is
that
schedulers
are
hard.
It's
a
little.
You
know,
I've
seen
a
few
slides
that
say
something
is
hard.
Schedulers
are
definitely
hard.
I
went
through
and
read
a
bunch
of
the
fare
scheduler
and
the
core
of
scheduling
bits
of
the
kernel
and
I
discovered
a
few
things.
The
quota
that
you
give
your
application
is
divided
into
five
millisecond
slices
and
then
assigned
it
to
individual
CPUs.
Here
we
go
we're
starting
to
figure
out
why
CPU
cores
matter?
Why
pinning
does
cores
matter?
B
So,
for
example,
if
you
gave
your
application
one
CPU
of
quota,
that
means
you
get
a
hundred
milliseconds
per
period
of
time.
That
means
five
milliseconds.
If
you
take
that
hundred
milliseconds
and
divide
it
by
five
milliseconds
to
get
the
number
of
slices,
you
have
20
slices
per
period
well
on
our
nodes.
We
have
88
course.
If
you
only
have
20
slices,
that
means
I
can't
use
68
of
my
cores,
because
I
can't
give
them
a
slice
right
off
the
bat,
and
the
last
thing
is
that
this
per
CPU
quota
will
expire.
B
If
not,
we
used
within
a
period
okay.
So
that
was
a
lot
of
words.
I'm
gonna
do
another
example
here
and
show
you
exactly
what's
going
on
in
the
kernel,
but
we're
gonna
use
a
two
core
machine:
two
cores
two
threads
to
separate
workers,
both
pinned
to
their
own
to
their
own
CPU
worker.
One
is
pinned
to
cpu
quota
to
CPU.
One
worker
two
is
pinned
to
CPU
two.
We
have
our
current
time
here
we've
given
our
application,
20
milliseconds
of
quota-
and
this
is
the
global
quota
bucket.
B
So
whenever
an
application
becomes
runnable,
it
has
to
go
to
the
global
quota
bucket
and
say:
hey
I
need
a
slice.
I
need
five
milliseconds.
In
order
to
run
that
quota
is
then,
when
it
becomes
running
that
quota
is
then
transferred
to
the
per
CPU
quota.
Okay,
right
now,
nothing
is
running.
Hence
there
is
no
quota
on
the
per
CPU
quote
of
queues.
B
So
if
we
advanced
a
little
bit,
we
see
that
worker
one
got
a
request
and
was
running
for
five
milliseconds.
It
had
to
transfer
five
milliseconds
from
that
global
quota
bucket
transfer
it
to
its
per
CPU
queue,
and
then
it
was
able
to
use
for
that
five
milliseconds
and
make
the
request
onto
the
request.
If
we
let
worker
to
do
the
same
thing,
great
we're
down
to
10
milliseconds
in
our
quote
in
our
global
quota
bucket.
But
now
what
happens
if
we
actually
had
a
smaller
request,
and
this
request
only
takes
one
a
complete?
B
Well,
we
still
transfer
an
entire
slice
to
c2
quota
to
CPU
one.
What
happens
to
that
last
four
milliseconds
that
gets
left
over
after
only
using
one
millisecond.
Well,
if
it
only
runs
for
one
millisecond
that
slice
the
leftover
bits
of
that
slice
are
gonna
persist
on
that
per
CPU
quota
after
five
milliseconds
there's.
This
thing
called
the
slack
timer,
which
is
going
to
return
everything,
but
one
millisecond
of
slack
back
to
the
global
quota.
Yeah
I
see
so
I.
B
Imagine
my
pain
when
I
discovered
this
for
the
first
time
and
I'm,
just
explaining
it
to
you
all
right,
but
what
happens
here?
We've
got
one
millisecond
on
a
C
hour
per
CPU
quota
that
is
not
being
used.
Okay,
great,
that's!
Fine!
Whatever!
Now,
if
worker
two
comes
in
and
gets
a
request
and
it
uses
it
keeps
using,
it
needs
a
lot
of
CPU
to
make
this
request.
It's
gonna
transfer,
eight
milliseconds
from
the
global
quota
bucket,
eventually
expiring
the
global
quota.
It's
now
exhausted
what
happens
now?
B
B
Well
that
nothing
happens
to
that
one
until
the
end
of
the
period
at
which
point
we
just
expire
it
and
throw
it
away.
Okay,
so
one
millisecond,
that's
where
all
of
our
problems
have
been.
It's
been
this
one
millisecond
think
about
the
impact
of
this,
though,
if
you
have
one
millisecond,
every
100
milliseconds
that
you
can
expire
and
you
have
an
88
core
machine.
That's
88,
CPUs
minus
1,
because
you
have
to
have
at
least
one
thread
that
needed
to
be
throttled.
B
That's
87,
milliseconds,
every
100
milliseconds
that
you
could
be
that
you
could
lose
due
to
these
1
milliseconds
just
being
left
over
on
these
per
CPU
queues.
That's
870,
Miller,
Coors
or
0.87
CPU
per
pod
per
container
in
each
pod,
even
actually
sorry
per
container
870
milk,
or
is
that
you
can
potentially
lose.
If
you
have
an
88
core
machine
per
container,
that's
a
lot
of
CPU
we're
losing
all
right.
So
what
are
the
solutions
and
workarounds?
B
This
is
why
you're
all
here,
some
people
have
not
gone
this
deep
and
you
know
they've
said
just
turn
it
off.
I'm
not
advocate
of
turning
it
off,
because
it's
now
fixed.
Let's
look
at
the
possible
solutions.
What
could
we
have
done?
First
thing
we
could
have
done
is
remove
5:1
to
$89.99,
but
that
would
have
broken
all
of
you
that
we're
hitting
that
due
to
clock
skew
that
was
not
going
to
be
slow.
That
was
not
going
to
be
an
acceptable
solution
to
the
carnal
community
and
I
have
higher
standards
than
that
as
well.
B
So
if
you
have
88
CPUs
all
trying
to
grab
that
lock
to
return
their
1
milliseconds,
you
cause
a
lot
of
waiting,
because
it's
all
synchronized
based
on
time-
and
the
last
thing
you
can
do-
is
remove
all
per
CPU
expiration
logic
and
that's
actually
exactly
what
we
did.
We
removed
all
the
per
CPU
expiration
logic.
B
That
has
resulted
in
these
two
commits,
and
these
are
now
applied
in
the
5
for
stable
5
for
kernel
and
I've,
also
backboarded
them
to
the
Linux
table
for
curls.
So
hopefully,
distros
are
now
pulling
this
in
Ubuntu.
Has
it
because
well
I,
went
to
court
of
rel
7
is
actively
testing
this
and
I
think
this
is
in
there
qn
of
QA
environments
and
relayed
to
has
it
as
a
work
in
progress,
and
it
should
be
available
hopefully
soon,
but
what
happened
to
fib
test,
because
that's
what
you're
all
wondering?
How
did
the
performance
change?
B
Well,
the
performance
changed
on
our
88
core
example.
We
went
for
50,
millisecond
quota
times,
10
periods.
We
got
previously
137
milliseconds
of
CPU
usage.
Now,
after
this
change
we're
getting
482
milliseconds
of
usage,
that's
almost
a
3x
increase
in
the
amount
of
CPU
time
that
application
can
use
across
all
ata
course.
B
So
what
are
the
takeaways
monitor
your
throttle
percentage?
That
is
the
most
application
agnostic
metric
that
I
like
to
use
my
math
doctorate
friend
likes
to
tell
me
that
it
doesn't
mean
anything
except
for
it
really
is
a
great
indication
for
how
badly
your
applications
are
being
throttled
and
how
badly
their
response
times
are
being
affected
number
to
upgrade.
Your
kernels
number
three
use
those
whole
CPU
quotas,
because
that'll
mitigate
this.
If
you
can't
get
the
your
underlying
platform
or
under
a
lying
cloud
to
upgrade
their
kernels.
B
C
C
B
So
the
question
is:
are
the
stats
exposed
in
kubernetes?
Unfortunately,
I
am
NOT
a
kubernetes
Pro
I'm,
actually
quite
the
kubernetes
new,
which
is
actually
why
I
was
able
to
do
this
because
I'm
much
more
experienced
in
the
kernel
I
fairly,
certain
couplet
exposes
these
metrics
via
the
API,
but
you
need
to
make
sure
that
you're,
monitoring
and
observability
is
actually
monitoring
those
those
those
bits
it's
actually
exposed,
as
in
our
periods
and
under
throttled,
is
what
it's
set,
what
it
sends
out,
but
you
have
to
do
the
math
on
it
yourself.
D
B
Okay,
so
that's
that's
actually
an
interesting
question.
Kubernetes
a
lot
of
the
people
have
gone
through
and
tried
to
work
around
the
five
one
to
AC
999
problem
by
changing
the
periods.
I
do
not
recommend
doing
that.
Ira
mentioned
very
much
so
recommend
getting
that
fixed
because
that
fix
actually
got
merged
with
roughly
six
months
ago
into
it
got
back
worded
into
all
the
Linux
table
kernels
six
months
ago.
B
If
you
are
not
able
to
get
these
patches,
which
they
really
are
pretty
much
available
everywhere.
At
this
point,
I
would
probably
just
increase
your
seat.
Your
CPU
limits
to
avoid
I
would
increase
the
B
CPU
limits
by
ten
times
the
number
of
CPUs
in
MillerCoors
did
that
make
sense.
Unfortunately,
I
don't
think
the
vertical
pod
autoscaler
is
going
to
behave
well
with
this
mechanism,
because
it's
gonna
be
like
oh
you're,
not
using
your
CPU,
but
you're
still
being
throttled,
I,
don't
know
if
they
use
throttling.
B
If
someone
that
knows
the
pod
autoscaler
better
can
clarify
that
I
would
love
to
hear,
but
I
would
suspect
that
their
pod
autoscaler
is
not
taking
throttling
into
consideration
or
the
fact
that
this
is
happening
other
than
that
yeah.
You
probably
would
want
to
turn
off
your
limits,
and
indeed
what
we
did
is
we
just
increase
the
amount
of
limit
that
the
applications
were
allowed
to
use.
B
Depends
well
that
that
really
depends.
It
doesn't
have
any
benefit
for
this
problem.
After
this
fix,
it
might
have
a
benefit
in
that
you
have
cash
locality.
You
know,
you're
gonna
have
warm
caches
on
your
cores.
You
are
possibly
not
going
to
context
switches
off
into
other
applications,
you're
not
going
to
have
contention
on
the
core
right.
B
C
E
C
B
So
the
reason
for
that
is
what's
going
to
end
up
happening,
is
you're
gonna
throttle
every
period
and
what
happens
is
you're
gonna
get
a
higher
average
and
best-case
response
time
with
a
lower
worst
case
in
p95
response
time?
Did
that
make
sense
you're
not
going
to
throttle
for
that
60
mill,
50,
milliseconds
or
something
in
the
period?
B
If
your
application
takes
in
his
case,
you
set
the
period
to
10
milliseconds
and
your
give
it
a
1/2
CP,
a
half
CPU,
that's
going
to
equate
to
5
milliseconds
every
10
milliseconds.
If
your
application
requires
20
milliseconds
to
make
a
response
now,
instead
of
using
20
milliseconds
to
make
that
response,
you're
gonna
run
for
5
milliseconds,
be
throttled
for
5
milliseconds
run
for
5
milliseconds,
be
throttled
for
five
milliseconds
burn
for
five
milliseconds.
We
throttled
for
five
milliseconds
a
few
more
times
and
then
you'll
respond
in
40
milliseconds
instead
of
20.
B
So
remember
that
per
CPU
quota
that
I
had,
if
you
use
whole
CPU
shares
you're
only
going
to
be
running
on
one
or
two
of
those
CPUs.
So
the
quota
is
going
to
be
allocated
to
your
CPU
that
you're
running
on,
and
so
all
of
your
threads
are
gonna,
be
using
from
the
same
per
CPU
quota.
So
it's
less
likely
that
you're
gonna
strand
milliseconds
on
random
CPUs.
So
what
happens
with
Java
is
it
tends
to
run
asynchronous
worker
threads
one
per
core
okay.
So
what
is
up
happening?
Is
you
do
an
I/o?
B
B
Threads
are
not
free,
don't
don't
treat
them
like
like
that,
but
it
won't
I!
Guess!
Yes,
if
you
don't,
if
you
can't
do,
if
you
can't
do
the
kernel
changes
decreasing,
the
number
of
threads
will
help
I.
As
the
kernel
engineer,
we're
fans
of
doing
things
efficiently,
spawning
4,000
threads
on
an
88
core
machine
is
not
always
the
best
strategy
right,
so
decreasing.
The
number
of
threads
will
increase
your
your
performance
of
your
application,
having
a
reasonable
number
of
threads
based
on
what
you're
doing
is
something
you
should
always
look
out
for.
E
B
Not
exactly
not
not
perfect,
not
not
not
strictly
depends
on
what
they
are.
It's
it's
very
case
dependent
right
so,
like
you
might
have.
If
you
allocate
4
CPUs,
you
might
want
a
KERS
asynchronous
buffered
I/o
and
you
might
want
also
Network
IO
threats
right.
You
might
want
to
have
four
of
each,
so
you
don't
want
to
have
you
know
just
one
of
each
for
each
core
right.
So
it's
like
it's
really,
depending
on
what
your
applications
doing.
Anyone
else
you.
G
B
That's
gonna
be
very
application,
dependent
again
what
I
tried
to
see
it
indeed
is
I
want
to
see
zero
to
ten
percent
throttled
time,
because
I
want
our
applications
to
be
sized
right
in
their
pod,
such
enough,
such
that
I
get
decent
amount
of
density
of
applications
of
pods
per
node
without
over
allocating
right.
So
it's
it's!
It's
really
a
it's!
A
value
judgment
do
I
care
about
the
density
of
my
applications
on
my
node
or
die
care
purely
about
performance.
B
If
you
care
purely
about
performance,
you
should
target
zero
percent
throttling
all
the
time.
If
you
care
about
density
on
your
nodes,
you
want
to
be
as
close
to
that
limit.
You
want
to
be
skirting
that
limit,
which
means
you're
gonna
hit
throttling.
You
know
every
little
bit
right
so
I
like
to
target
us
to
having
a
little
bit
of
throttling
on
all
of
our
applications,
but
nothing
excessive,
because
that
gives
us
a
lot
of
density.
B
So
if
it
speaker
off
feet,
we
actually
target
for
peak
peak
times,
but
that
is
a
work
in
progress.
That
I
think
it's
going
to
be.
My
next
task
to
work
on
so
I'll
probably
be
playing
with
the
pot
vertical
pot
autoscaler
in
the
next
year.
Maybe
maybe
some
other
things,
maybe
I'll
be
doing.
Another
talk
like
this
about
how
we
increase
the
density
by
awesome,
applying
the
vertical
pot
autoscaler.
So,
okay.