►
From YouTube: Staying in Tune: Optimize Kubernetes for Stability and Util... Randy Johnson & Koushik Radhakrishnan
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Staying in Tune: Optimize Kubernetes for Stability and Utilization - Randy Johnson & Koushik Radhakrishnan, VMware
Kubernetes provides a number of primitives to manage resource consumption. Implementing resource limits, requests and quotas are often the first steps taken to solve this problem at the pod or namespace level. However, the behaviour of an overall Kubernetes cluster as it nears capacity and the parameters available to tune it are often overlooked. To ensure optimal stability and utilization of a cluster, users must learn how to implement, test and manage these parameters over time. With their field engineering work done for healthcare and financial customers, Randy and Koushik have gathered valuable lessons on how one should approach this problem.This talk will illustrate how you should approach resource limits, resource requests, eviction policies and node allocatable constraints to get the most out of your Kubernetes clusters.
https://sched.co/UadR
A
A
You
are
all
the
real
MVPs,
so
we
are
here
today
for
the
last
session
of
the
performance
track,
staying
in
tune,
optimized
kubernetes
for
stability
and
utilization
by
Randy,
Johnson
and
couchy
credit
Krishna
from
VMware
I'm.
Here
to
remind
you,
there
will
be
a
feedback
forum
available
after
the
session.
Our
speakers
and
we
had
the
CNC
F-
would
really
appreciate
it.
A
B
Thank
you
so
much
for
being
here,
it's
the
last
session
like
we
mentioned.
So
this
is
we're
going
to
talk
about,
staying
in
tune,
optimizing
your
cumulus
clusters
for
stability
and
utilization.
It's
a
very
overlooked
topic
and
I.
Think
whatever
we
learnt
in
field
hearing
as
a
field
engineer
at
VMware,
we've
tried
to
put
it
in
this
topic
and
hopefully
you'll,
take
it
back
and
implement
and
tune
your
clusters
accordingly,
so
I'm
Kyle
shake
Radhakrishnan
I'm,
Randy
Johnson.
B
So
let's
talk
about
others.
I
mean
communities.
It
has
potential.
It's
gone
way
beyond
potential
right
now.
I
think
the
first
step
to
have
this
increase
beyond
its
potential
is
the
declarative.
Api
I
think
we
all
know
like
API
is
like
the
first-class
citizen.
It's
all
about
software
interfacing
with
software
and
then
the
next
important
aspect
of
this
is
the
control
loop
aspect.
I
mean
that's
a
self-healing
side
of
things,
autonomously
converging
on
building
blocks,
you're
having
rezulin
see
at
the
app
and
infra
layers.
B
So
these
are
kind
of
the
two
main
primitives,
but
for
a
talk,
you're
going
to
look
at
the
bin
packing,
as
which
is
very
important
and
we'll
be
discussing
a
lot
about
limits
and
requests.
It's
all
about
determining.
Where
is
the
best
part
placement
its
approximation
with
like
performance
guarantee
like
we
mentioned?
B
So
this
is
what
we're
going
to
be
looking
at
all
right,
no
talk,
so
we
have
to
break
these
invisible
walls
right,
I
think
we
always
have
these
default
parameters
that
we
set
up
and
we
tune
our
clusters
that
we
go
through,
but
we
need
to
break
these
walls
and
optimize
our
clusters
accordingly.
I
think
we
don't
know
where
that
ceiling
is
a
rather
invisible
Wallis
and
we
have
to
look
at
these
clusters
and
then
take
these
default
values.
That's
been
set
up
and
then
kind
of
change
them.
B
So
for
the
goals
of
this
talk
is
the
two
main
things
that
we're
trying
to
look
at
like
we
mentioned
its
stability
and
utilization,
they
always
seem
to
be
at
odds
with
each
other,
but
we
believe
that,
with
a
solid
understanding
of
these
stability
and
utilization,
we
can
achieve
both,
which
is
always
good
for
your
business
as
well.
So
what
are
the
challenges?
Let's
look
at
the
challenges
that
we
have
I
mean
Kuban.
It
is
isn't
configured
for
our
app
I
mean
it's
resources
burst
priority
you're.
B
Looking
at
a
lot
of
these
aspects,
for
when
you're
having
app
getting
deployed
onto
your
cube,
cluster
configuration
is
also
very
important
and
I
mean
when
we
have
your
app
deployed
on
your
cumulus
cluster.
Your
utilization
is
increasing.
The
behavior
of
clusters
become
even
more
complicated.
We
enough
in
a
talk,
we're
going
to
depth
about
how
these
behaviors
are
affect
us
or
affect
these
clusters,
and,
like
I
said,
I
mean
the
app
the
utilize
we
have.
These
apps
are
getting
configured
the
utilization
increases
whenever
utilization
increases,
we're
always
leading
into
disruptions.
B
There's
expected
disruptions
that
you
can
see
and
there's
an
unexpected
disruptions
that
you're,
probably
not
in
going
to
encounter
I
mean
you're
gonna
encounter
it,
but
you're
not
aware
of
so.
How
can
we
maintain
stability
when
you
have
these
disruptions?
So
we
always
believe
that,
in
this
model
that
we
talked
about
the
target
is
always
moving
you're.
Always
there
is
no
like
an
acid
state
you're
getting
to
to
be
state
where
you're
kind
of
the
target
is
always
in
a
moving
state.
B
Your
app
requirements,
change
the
underlying
infrastructure,
or
the
hardware
is
changing
the
only
constant
for
all
of
us,
which
is
why
we
are
all
here.
The
Kuban
it
is
is
constant
and
binds
the
app
and
hardware,
and
these
are
the
challenges
that
kind
of
like
we're
going
to
fix
it.
The
moving
target
that
we
talked
talking
about,
you're
going
to
be
optimizing,
your
app
based
on
that
target.
B
What
can
we
do
about
it?
Two
things:
the
limits
and
requests
we're
looking
at
scheduling
over
committing
and
eviction,
then
you're
also
looking
to
look
at
allocatable
capacity
and
we're
going
to
look
at
why
that
is
important
in
the
allocatable
capacity
you're
looking
at
defaults,
the
eviction
threshold.
Why
cube
system
our
cube
and
system
reserved
our
components
are
all
important
for
you
and
how
you
should
look
at
tuning
these
parameters.
C
Thanks
all
right,
so
we're
gonna
start
with
a
simple
example
and
introduce
limits
and
requests
and
go
over
some
of
the
problems
that
those
solve
and
then
we're
gonna
tie
that
back
to
some
of
these
other
behaviors
that
are
really
important
within
kubernetes
as
we
approach
utilization.
So
here's
our
example
node
and
its
really
simple
known
right
now.
C
It
just
has
one
replica
of
pod
a
deployed
to
it
and
there's
a
couple
problems
with
this,
the
first
of
which
is
that
when
the
scheduler
placed
this
pod
on
this
node,
it
didn't
do
that,
based
on
any
sort
of
information
about
the
resources
that
it
needs
and
another
problem,
that's
even
bigger
than
that
is
that
the
resource
consumption
of
this
pod
is
unbounded.
So
this
pod
is
free
to
consume
all
of
the
available
resources
on
this
node.
C
So
when
we
deploy
a
second
replica
of
this
pod
notice
that
the
CPU
utilization
is
split
evenly,
so
this
is
because
CPU
is
a
compressible
resource,
and
that
means
that
when
the
CPU
is
fully
exhausted,
each
of
its
consumed
murrs
will
be
throttled,
so
they'll
be
slowed
down.
But
memory,
on
the
other
hand,
is
incompressible.
So
when
we
run
out
of
memory,
it's
a
really
big
problem
and
we
can't
compress
it
in
the
same
way
that
we
could
CPU.
So
at
this
point,
there's
a
couple
things
that
can
happen.
C
Ideally,
one
of
these
pods
would
be
evicted,
but
since
the
known
is
fully
utilized,
it's
also
possible
that
the
eviction
process
doesn't
occur
fast
enough
or
that
it
doesn't
occur
at
all.
So
that's
problematic
because
at
that
point,
you'd
be
reliant
on
the
nodes
out
of
memory
killer
and
we're
not
going
to
go
into
the
details
on
that.
C
If
we
run
out
of
it
can
think
of
memory
as
being
incompressible
like
water,
so
if
you're
trying
to
compress
it
you're
gonna
have
a
bad
time
all
right.
So
what
do
we
do
about
this?
The
first
thing
we
should
do
is
set
up
limits
and
a
limit
is
just
a
way
to
specify
an
upper
bound
on
resource
consumption.
So
for
CPU,
which
is
a
compressible
resource,
the
limit
is
going
to
be
the
point
at
which
CPU
is
throttled.
It's
gonna
be
the
point
at
which
this
container
is
slowed
down.
C
C
So
it
turns
out
that,
even
though
these
are
really
simple
ideas,
just
specifying
an
upper
bound
and
lower
found
on
containers,
these
have
a
huge
effect
on
the
overall
behavior
of
our
cluster.
So
limits
and
requests
lay
the
groundwork
for
scheduling
it
affects
how
well
informed
the
scheduler
is
to
make
a
good
decision
on
pod
placement
it
affects
over
commit
and
over
commit
is
a
process
of
assigning
more
resources
to
containers.
Then
the
node
actually
has
available.
C
They
affect
eviction
and
eviction
is
the
process
of
kicking
pods
out
of
a
node
to
reclaim
resources
and
maintain
stability.
These
are
the
three
main
processes
that
effects
stability
and
utilization
within
a
cluster,
and
if
we
want
to
achieve
both
stability
and
utilization,
we
need
to
be
aware
of
and
understand
these
ideas,
alright,
so
back
to
scheduling.
So
in
kubernetes,
the
scheduler
filters
nodes
based
on
requests
and
based
on
the
request
here,
it
looks
like
another
pod
could
fit
on
this
node,
but
based
on
its
utilization
and
our
previous
example.
C
So
over
commit
again
is
the
process
of
assigning
more
of
a
resource
to
pods,
then
the
node
actually
has
available,
and
because
the
scheduler
only
considers
requests
not
utilization
or
limits.
We
implicitly
configure
this
with
the
difference
between
limits
and
requests
and
whether
or
not
over
commit
is
a
good
idea
for
your
app
is
really
up
to
you,
but
at
a
high
level,
without
over
commit
which
can
be
achieved
by
setting
requests
equal
to
limits,
your
utilization
will
be
a
little
bit
lower.
C
You
would
never
run
out
of
resources,
but
you're
basically
buying
stability
at
the
expense
of
utilization
with
overcommit.
If
you
allow
requests
to
be
less
than
limits,
you
can
achieve
higher
utilization,
you
might
run
out
of
resources
and
you're,
basically
buying
utilization
at
the
expense
of
stability.
So.
C
Quality
of
service
is
a
class
it's
assigned
to
pods,
based
on
their
requests
relative
to
limits
and
in
each
of
our
previous
examples,
we've
had
three
different
configurations
of
requests
relative
to
limit.
So
if
you
have
no
limits
or
requests,
that
would
correspond
to
a
best-effort
quality
of
service,
and
this
is
the
lowest
quality
of
service.
If
you
have
request
set
less
than
limits,
that
would
correspond
to
a
burstable
quality
of
service,
so
the
the
middle
quality
of
service.
C
So,
let's
go
over
an
example
of
eviction.
Again
eviction
is
the
process
of
kicking
pods
off
of
a
node
in
order
to
reclaim
resources
and
maintain
stability.
So
each
of
the
letters
here
corresponds
to
a
pod,
consuming
a
varying
amount
of
memory
relative
to
its
request
and
limit
and
we're
gonna
use
memory
here
as
an
example,
because,
as
we've
mentioned
earlier,
running
out
of
memory
is
the
biggest
risk
to
stability.
C
This
animation
shows
the
order
in
which
pods
from
the
previous
slide
are
prioritized
for
eviction,
so
notice
that
the
first
four
pods
all
have
a
quality
of
service
of
best
effort
and
the
third
and
fourth
pods
that
are
evicted
actually
have
the
same
priority.
So
when
this
happens,
the
quality
of
service
is
equivalent
and
the
priority
then
they're,
ranked
by
the
utilization
and
now
with
the
burst
of
all
pods,
because
we
have
a
request
and
that
we
compare
the
utilization
to
even
though
these
are
the
same
priority.
C
We
compared
the
utilization
relative
to
the
request
and
then
last
would
be
a
guaranteed
pod
and
again.
Guaranteed
pods
will
never
be
evicted
because
of
another
pods
resource
consumption.
But
if
system
Damon's
were
to
burst,
then
in
that
case
they
would
be
evicted,
and
in
that
sense
you
can
also
think
of
system
Damon's
as
having
a
guaranteed
quality
of
service.
C
So
by
default
limits
and
requests
are
optional,
and
this
is
kind
of
because
there's
not
really
a
sensible
default
for
these
values.
It's
entirely
up
to
us
to
set
these,
and
even
though
there's
no
error
or
even
a
warning.
If
we
deploy
a
pod
without
limits
and
requests,
they're
still
incredibly
important,
so
the
way
that
we
enforce
these
and
force
defaults
to
exist
on
pods
is
by
setting
up
a
limit
range
object
and
a
limit
range
object
will
allow
you
to
set
a
minimum
maximum
ratio
of
limit
to
request
or
default
values.
C
C
All
right
now
we're
going
to
get
into
allocatable,
and
this
is
a
really
important
concept
to
understand
and
I
think
it's
especially
important
that
we
as
consumers
of
kubernetes
think
of
nodes
in
terms
of
allocatable
capacity
and
not
just
total
capacity.
So
this
is
important
because
it's
how
the
scheduler
filters
nodes.
It
looks
at
the
remaining
allocatable
capacity
for
a
resource
and
compares
that
to
a
pods
request.
C
There
would
be
less
of
a
buffer
for
pods
to
burst
and
there
would
be
a
greater
buffer
between
the
point
at
which
eviction
begins
and
the
total
capacity
of
the
node,
and
this
would
make
it
less
likely
for
Damon's
to
experience
resource
contention.
So
likewise
the
opposite
is
always
true.
If
we
wanted
to
tune
for
utilization,
we
could
increase
allocatable
and
again.
C
This
would
just
give
more
capacity
for
the
scheduler
to
place
pods
more
of
a
buffer
for
pods
to
burst,
and
there
would
be
a
smaller
buffer
between
the
eviction
threshold
and
the
actual
capacity
of
the
node,
and
this
would,
of
course,
make
it
a
little
bit
more
likely
for
resource
contention
to
occur
so
overall,
as
allocatable
shrinks,
we're
tuning
for
stability
and
as
it
increases
we're
tuning
for
utilization.
So,
let's
talk
about
what
allocatable
constraints
are
because
that's
what
allows
us
to
tune
a
locatable,
alright.
C
So
to
start,
let's
take
a
step
back
and
take
another
look
at
what
nodes
actually
look
like
on
a
newly
installed
cluster,
so
by
default,
pods
and
Damons
consume
resources
and
we've
kind
of
been
glossing
over
this
fact
up
until
this
point.
But
now
everything
on
the
left
is
going
to
represent
utilization
of
pods
and
Damons,
and
everything
on
the
right
is
gonna
represent
allocatable
constraints
so
notice.
C
The
only
thing
that
we
have
on
the
right
is
an
eviction
threshold
for
memory,
and
this
threshold
applies
to
the
total
utilization
of
pods
and
Damons,
and
when
the
total
utilization
hits
this
point,
that's
when
eviction
begins.
So
one
problem
with
this
is
that
pods
and
Damons
are
actually
competing
for
resources.
The
scheduler
does
not
take
into
account
resource
consumption
of
Damons,
so
the
way
that
we
can
mitigate
this
is
by
setting
cube
and
system
reserved
which
are
actually
disabled
by
default.
C
So
cube
reserved
is
a
way
to
set
aside
resources
for
kubernetes
related
Damons,
like
the
cubelet
and
docker
system.
Reserved
is
for
everything
else
on
the
node
and
when
we
set
these
up,
these
are
factored
into
allocatable
constraints
and,
as
that
increases
allocatable
decreases
like
we
were
discussing
before.
C
So
this
is
how
we
take
into
account
daemon
resource
consumption
and
prevent
pods
and
Damons
from
competing
for
the
same
resources
and
again,
there's
not
really
a
default
for
this
either,
because
there's
not
really
a
sensible
default
to
set
kubernetes
is
unaware
of
the
underlying
hardware
and
our
app
requirements.
So
again,
this
is
something
that's
really
up
to
us
to
set
up
in
tune
and
just
to
recap,
for
CPU
the
scheduler
uses
allocatable
to
schedule
based
on
CPU
requests.
C
Allocatable
is
the
buffer
in
which
CPU
utilization
can
burst
and
when
utilization
exceeds
allocatable
pods
will
be
throttled
to
prevent
contention
with
Damons,
similar
story
for
memory
and
ephemeral
storage.
The
only
difference
here
is
that
we
include
the
eviction
threshold
in
node,
allocatable
constraints
and
quick
side.
Note
on
that.
C
That's
the
addiction
threshold
on
the
far
right
and
another
is
the
edge
of
allocatable,
and
this
one
is
triggered
by
the
consumption
of
pods,
and
although
we
use
CPU
memory
and
ephemeral
storage
as
examples,
those
are
just
what's
currently
supported
and
documented,
but
it's
also
important
to
note
that
the
concept
of
allocatable
could
could
potentially
be
generalized
beyond
these
types
of
resources
and
just
keep
this
in
mind,
and
we
could
extend
this
API
to
someday
account
for
any
type
of
resource
that
we
want.
Maybe
GPUs
example
all
right.
C
So
there's
some
key
questions
that
we
can
ask
ourself
with
this
to
inform
how
we
set
up
our
cluster,
the
first
of
which
is,
is
our
scheduler
prepared
to
make
informed
decisions
so
remember
by
default
limits
and
requests
are
optional.
The
way
to
enforce
that
is
by
setting
up
a
limit
range
object
once
we're
using
limits
and
requests.
One
of
the
most
important
things
to
consider
is
utilization
relative
to
requests,
and
this
cuts
both
ways
when
working
with
customers.
C
We
found
that
if,
for
example,
app
teams
consistently
overestimate
how
much
of
a
resource
their
app
needs,
the
scheduler
will
think
that
the
cluster
is
full
because
it's
comparing
the
pod
request
to
the
allocatable
capacity
of
each
node.
But
if
you
look
at
actual
utilization,
then
it's
gonna
be
hovering
around
like
five
to
ten
percent,
but
also
the
opposite
is
also
true.
C
Is
over
commit
configured
properly,
so
if
our
limits
are
too
high,
this
could
result
in
eviction
occurring
too
frequently,
and
that's
because
the
utilization
would
be
running
up
against
eviction
thresholds
if
our
limits
are
too
low.
This
could
result
in
containers
restarting
in
the
case
of
memory,
for
example,
because
those
containers
would
be
killed
and
restarted
or
throttling
in
the
case
of
CPU,
and
how
close
are
we
to
triggering
eviction?
C
So
this
question
is
a
little
bit
more
nuanced
than
just
looking
at
memory
and
available
memory
is
important,
but
that
tells
us
when
the
entire
node
is
at
risk.
So
because
we
understand
these
eviction
thresholds
and
these
values
are
well
defined
and
known
ahead
of
time,
we
can
be
a
little
bit
more
proactive
in
our
approach
to
monitoring
so
by
knowing
exactly
when
eviction
is
going
to
occur.
We
know
when
instability
at
the
app
level
is
going
to
occur,
not
just
instability
at
the
infrastructure
level.
So
again,
these
two
eviction
thresholds.
C
This
one
is
for
the
allocatable
capacity
of
pods,
and
this
one
is
for
the
eviction
threshold.
This
is
the
first
one
that
we
went
over
and
monitoring
these
two
eviction
thresholds.
It
can
be
really
powerful
because
it
informs
our
decision-making
process
moving
forward
for
how
to
in
our
clusters,
but
we
also
know
the
cause
of
eviction
and
that's
what
allows
us
to
tune
it.
So
we
can
build
upon
this
further
and
ask
ourselves
alright
what
if
we
frequently
encounter
eviction
caused
by
utilizing
allocatable
well.
C
In
that
case,
we
would
know
that
we
need
to
tune
our
limits
and
requests
to
adjust
our
overcommit,
and
if
we
know
that
we
are
frequently
encountering
eviction
caused
by
crossing
the
eviction
threshold,
we
know
that
we
needed
to
noob
and
system
reserved.
So
this
isn't,
because
we
haven't
set
aside
enough
resources
for
our
system,
Damons.
C
And
I
know
it's
late
in
the
day
and
I'm
throwing
a
little
bit
of
math
at
you,
but
this
will
be
easy,
I
promise.
So
in
all
of
our
examples,
so
far
we've
been
considering
allocatable
at
the
node
level
and
it
turns
out.
We
can
generalize
this
a
little
bit
further
and
think
about
the
allocatable
capacity
of
our
cluster.
So
this
can
help
us
be
a
little
bit
more
proactive
in
how
we
think
about
failover,
for
example.
C
So
it's
scheduling
is
a
bit
more
complicated
than
this,
but
the
point
is:
if
we,
if
these
values
are
well
defined
and
known
ahead
of
time,
why
not
plug
them
into
monitoring
tools
to
help
us
be
a
little
bit
more
proactive,
and
this
is
one
example
of
how
we
might
do
that
and
I
forgot
to
advance
the
slides,
but
we
could
generalize
this
for
an
arbitrary
number
of
failures.
So
imagine
if
you
know
you
had
two
nodes
and
one
failure.
C
This
would
be
50%
as
your
upper
bound
for
utilization
or,
if
you
had
10
nodes
and
one
failure,
this
would
be
your
upper
bound
for
utilization
would
be
90%
and
because
our
node
allocatable
constraints
are
well
defined
and
known
ahead
of
time.
We
can
take
that
into
account
as
well
and
be
even
more
precise
in
our
estimate.
C
These
are
some
of
the
defaults
that
we
think
it's
really
important
to
be
aware
of
so
eviction
hard.
This
is
the
default
eviction
threshold
that
we
talked
about
earlier.
Housekeeping
interval
is
the
interval
in
which
cubelet
takes
a
snapshot
of
resource
utilization
on
the
node,
so
the
cubit
doesn't
actually
have
a
real
time.
C
Understanding
of
resource
consumption
eviction,
pressure
transition
period
is
a
mouthful,
but
it
is
also
how
long
it
takes
for
a
node
to
be
able
to
schedule
pods
after
an
eviction
threshold
is
met,
so
I
think
that's
a
really
important
one
to
be
aware
of,
because
it's
five
minutes
so
taking
these
three
into
consideration.
Here's
what
a
worst-case
scenario
might
look
like
and
I
know
this
from
experience.
So
let's
say
our
workload
consumes
more
than
a
hundred
MIDI
bytes
faster
than
ten
seconds.
C
The
eviction
process
doesn't
begin
fast
enough
or
maybe
it
doesn't
happen
fast
at
all.
Now
we
rely
on
the
nodes
out
of
memory
killer
to
save
us
and
in
the
worst
case,
docker
is
killed
by
the
out
of
memory
handler
and
every
container
on
the
node
is
taken
down.
In
the
best
case,
we
can
reclaim
resources
on
the
and
the
note
is
stable
again,
but
we
can't
schedule
anything
on
that
note.
C
So
I
bring
this
up
because
at
the
very
least
you
might
want
to
increase
the
eviction
threshold,
the
default
there
is
only
100
megabytes
and
decrease
the
eviction
pressure
transition
period
unless
your
apt
and
tolerate
a
five-minute
downtime
and
for
a
docker.
Oh
quick
note
on
this,
so
I've
got
implied
defaults
up
here.
So
if
you
don't
pass
any
flags
in
the
implied,
defaults
or
effectively
what
is
passed
in,
but
I
included
the
flags
up
here
in
case
you
wanted
to
add
these
to
cubelet
and
tweak
them.
C
But
again,
if
you
don't
do
anything,
this
is
what
they
would
look
like.
So
before
we
talk
about
docker,
we've
got
flags
to
consider
so
cube
reserved
and
system
reserved.
We
talked
about
these
earlier.
These
are
how
you
set
aside
system
resources
for
danaans
and
eviction.
Soft.
This
one
is
similar
to
the
other
eviction
threshold
that
we
talked
about,
but
this
one
is
not
considered.
C
Well,
it's
not
accounted
for
in
allocatable
constraints,
so
it
will
not
affect
your
allocatable
capacity,
but
once
this
threshold
is
triggered,
there's
a
timer
that
starts
ticking
down
and
when
that
timer
hits
zero.
If
the
eviction
threshold
is
still
being
violated,
then
eviction
will
begin.
So
it's
kind
of
a
nicer
fiction
threshold
and
the
grace
period
below
is
how
you
would
tune
that
timer
that
we
talked
about
and
on
to
docker.
C
So,
as
we
were
saying
earlier,
by
default,
when
docker
terminates,
it'll
kill
all
of
the
containers
on
that
node,
and
there
is
a
feature
to
mitigate
against
this.
It's
called
live,
restore
again
implied
default.
If
you
don't
do
anything,
this
is
what
it
looks
like.
So
the
default
value
for
this
is
false,
but
if
you
do
want
to
take
advantage
of
that
feature,
you'd
have
to
set
this
flag
to
true.
B
Well,
that
brings
us
to
the
summary
again.
We
looked
at
a
lot
of
things.
One
is
the
limits
and
requests
and
allocatable
capacity
as
well
in
limits
and
requests.
We
went
to
scheduling
the
overcommit
eviction
and
we
also
looked
in
the
allocatable
capacity
defaults
eviction,
threshold
and
Kuban
system
and
how
you
can
go
about
tuning
those
parameters
for
your
own
clusters.
B
Again,
please
keep
in
mind.
We've
done
this
for
a
lot
of
customers.
We
every
cluster
is
different.
Every
app
that
you
deploy
in
a
cluster
is
different.
You'd
have
your
own
enrollments
to
think
about,
but
if
you
have
more
questions
to
ask
us,
please
feel
free
to
reach
out
to
us,
and
hopefully
this
was
insightful
and
how
you
can
optimize
the
clusters.
A
A
E
E
C
E
C
B
C
So
to
approach
that
I
would
probably
start
with
just
monitoring
tools
to
figure
out
what
our
requests
and
limits
should
be
for
the
app
so
that
we
know
what
appropriate
settings
are
for
that,
and
then
I'd
say
that
quota
is
really
just
gonna,
be
based
on
that.
What
do
you
think
is
appropriate
for
the
apps
in
that
namespace
again.
E
B
E
B
C
B
G
G
If
I've
got
an
application,
that's
got
a
long-running
priests
top
hook
and
the
note
goes
into
a
soft
memory,
condition
and
breaches
that
threshold,
but
doesn't
and
stays
there
and
then
other
pots
continue
to
consume
resources,
as
that
hook
terminates
and
if
you
then
breached
the
hard
eviction
limit
after
that,
how
the
sequence
of
events
would
play
out
if
you
guys
ever
encountered
that
and
the
work
that
you've
done,
I
have
a
general
sense
of
how
those
interacts
but
I'm
curious.
If
you
ever
ran
into
something
like
that,
so.
C
It's
not
something
that
I've
encountered
myself,
but
I
would
imagine
that
it
sounds
like
it
might
be.
A
good
use
case
for
the
soft
eviction
threshold.
I'd.
Imagine
if
you
had
a
brief
increase
in
utilization
and
then
maybe
you
could
adjust
that
grace
period
to
account,
for
that
might
be
something
to
to
think
about.
Yeah
I
hope
that
it
helps
answer
the
questions,
but
overall
no,
it's
not
something
that
I've
encountered
myself.
H
So
when
I
was
reading
up
on
how
kubernetes
handled
limits
and
requests,
one
of
the
things
I
read,
source
is
probably
relevant,
but
anyways.
It
sounded
reasonably
so
I
made
an
assumption.
That
was
true
is
that
when
you
set
a
limit
and
the
process
wants
to
go
beyond
that,
if
there
is
free
CPU
cycles,
kubernetes
will
allow
it
to
use
those,
so
it
allowed
to
exceed
the
limit.
However,
I've
sensing
several
cases
where
kubernetes
actually
throttled
on
the
limit,
even
when
CPU
was
idle.
H
F
Hey
so
thanks
for
giving
this
talk,
if
you
are
on
eks,
you
should
be
aware
that
the
newest
ami
they
just
released
on
Tuesday,
actually
sets
the
cube
reserved
and
the
daemon
reserved
by
default.
This
actually
caused
a
fire
for
us,
because
cluster
autoscaler,
which
will
add
more
nodes
to
your
ASG,
doesn't
take
into
account.
If
you
have
cube
reserved
actually
said
so,
pods
would
try
to
be
scheduled
onto
these
nodes.
It
didn't
do
the
math
thought
that
there
is
actually
a
larger,
the
normal
amount
of
memory
spun
up
another
node,
and
then
that
failed.