►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Restart-Free Vertical Scaling for Kubernetes Pods - Vinay Kulkarni & Peng Du, Huawei
Currently, vertical scaling of Kubernetes pod CPU or memory resources requires pod restarts. This is disruptive to services, and expensive for long-running applications or jobs. It is very important to have the ability to scale pod resources without restarts. In this talk, Vinay and Peng will present a design for restart-free scaling of pod resources, with container restart as a policy-controlled choice rather than a necessity. They will illustrate a customer use case, a gene-sequencing application where pod restart is expensive. They will go over the changes to Kubernetes components that made this feature possible. They will discuss how they addressed some unique challenges such as multiple scheduler race conditions, and respecting pod disruption budget if pod restart is needed. They will talk about how failures are handled via smart retries, and conclude with a demo.
https://sched.co/MPcJ
A
A
Besides
leading
this
effort
here,
I
work
with
container
networking
and
container
runtimes
over
there
and
unfortunately
my
co-speaker
hang
do
could
not
be
here
today
because
of
unforeseen
events
last
week,
and
but
he
did
a
great
job
helping
with
this
project.
So
what
are
we
going
to
talk
about
today?
We
are
gonna,
go
or
a
customer
scenario
which
motivated
this
whole
effort
for
us
and
we're
gonna
quickly.
A
Take
a
look
at
the
scaling
in
communities
today
and
how
pod
scheduling
works,
because
this
is
needed
for
the
design
and
we're
gonna
go
a
little
bit
deeper
into
our
solution
or
designed
for
the
vertical
scaling
of
community
spots.
We'll
look
at
some
of
the
policy
controls
that
we
implemented
as
part
of
this
effort
and
some
how
we
do
the
failure
handling
for
in
our
when
we
do
the
vertical
scaling,
then
we're
gonna.
A
Look
at
how
we
could
potentially
integrate
with
vertical
pod
autoscaler,
which
is
a
project
that
was
created
to
automate
the
process
of
scaling.
Your
pods
for
you,
nobody's
gonna,
sit
there
and
do
this
manually
so
and
then
we
had
another
issue
we
had
to
look
at,
which
is
how
to
handle
memory.
Spikes.
We
now
go
over
that
and
then
I'm
gonna
cross
my
fingers
and
do
a
demo
and
get
an
a
after
that.
A
So
this
is
this
is
a
biotech
industry
customer
and
they
are
currently
using
kubernetes
jobs
for
running
their
applications.
This
is
a
very
long
process.
What
I
understand
is
that
the
end-to-end
run
for
analysis
takes
almost
a
day
and
they
have
some
jobs
which
run
for
almost
four
hours.
So
if
your
pod
were
to
get
killed,
hurry
started
because
of
resources,
they
lose
a
lot
of
work.
Now
they
have
this
legacy
applications
and
they
could
potentially
rewrite
it
to.
A
You
know,
become
more
restart
resilient,
but
that's
not
a
priority
or
not
feasible
for
them
to
do.
It's
don't
fix
if
it
ain't
broken
kind
of
moto
there,
and
so
they
were
looking
to
get
this
feature
where
they
estimate
this
elastic
part
feature
would
save
them
about
fifty
to
sixty
percent
in
resource
utilization
cost
savings
which
otherwise
they're
resizing
for
peak
usage
and
they're,
not
utilizing
the
full
capacity
of
the
cluster
note.
A
A
Horizontal
scaling,
as
many
of
you
already
know,
it's
the
bigger
the
load
more
number
of
pods,
unique,
smaller
the
load.
You
reduce
the
pods
kubernetes,
the
forces
already
and
vertical
scaling.
This
is
gonna,
be
the
focus
of
our
talk
today.
This
is
where
we
increase
or
decrease
the
resources
given
to
a
part
that's
in
for
now
it
is
CPU
and
memory
for
our
top,
and
this
is
common
for
stateful
applications
and
the
job
that
I
just
described
with
our
customers
scenario.
A
The
scheduling
this
is
a
very
high-level
or
with
how
a
pod
gets
scheduled,
so
we
have
the
core
components:
the
API
server
scheduler
couplet
and
runtime
I
left
out
etcd
intentionally
for
space.
So
it's
knots
with
the
pod
create
we
put
the
part
in
the
API
server
and
the
scheduler,
which
is
watching
for
new
parts.
Its
job
is
to
find
parts
that
don't
have
a
node
and
find
it
at
home,
which
is
a
node,
and
to
do
that,
it
runs
through
a
bunch
of
predicates
and
priorities.
It's
a
non-trivial
process.
A
It
takes
a
little
bit
of
work
to
do
that
and
one
particular
predicate
that's
of
interest
to
us
here
is
called
pod
Fitz
resources.
This
predicate
finds
filters
out
the
nodes
which
can
accommodate
the
pot,
and
it's
once
the
scheduler
has
found
the
found
a
node
for
the
pot.
It
executes
sabayon
in
point
find
operation
and
the
pot
is
assigned
to
the
node.
It
essentially
sets
the
node
name
for
the
scheduler
in
the
pot
spec.
Now
the
couplet
goes
to
work.
A
It
does
another
check
to
see
if
the
pods
gonna
fit
on
the
node.
This
is
necessary
because
community
supports
multiple
schedulers,
as
you
may
know,
and
that
inherently
brings
in
potential
race
condition.
So
your
couplet
is
the
ultimate
gate.
It
gets
to
say
yes
or
no,
so
let's
assume
everything
goes
well.
Then
the
cubelet
now
moves
to
setup
the
pod,
which
is
today's
sets
of
the
C
groups,
and
then
it
calls
the
container
CRI
API
asking
the
runtime
to
run
the
set
of
the
start.
A
The
container
and
the
container
runtime
does
that
and
reports
back
status
and
cubelet
then
updates
the
parts
status
saying
okay,
the
container
is
ready,
country
is
running
and
it
had
everything
and,
let's
go
to.
When
we
started
working
on
this
project,
we
were
faced
with
a
couple
of
potential
options:
the
choices
we
had.
What
first
choice
was
to
directly
update
their
pod
resources
and
let
the
cubelet
and
scheduler
update
their
update.
They
did
I
in
parallel.
So
one
thing
I
forgot
to
mention
earlier
the
scheduler.
It
keeps
a
cache
of
the
node
available
resources.
A
It
does
this
for
performance
reasons,
I
suppose,
and
because
of
that
it
doesn't
have
a
live
view
of
the
node
available
resources,
and
if
we
went
with
this
option,
let's
say
that
cubelet
were
to
apply
the
resize.
It
says:
ok,
yeah
this.
This
capacity
request
for
the
from
the
part
I
can
do
that.
I
have
a
capacity,
but
at
that
very
moment
the
scheduler
might
be
working
on
assigning
a
new
part
to
this
node,
which
is
going
to
use
that
capacity.
A
So
when
that
new
pot
arrives
at
the
node,
it's
going
to
get
rejected
and
then
the
scheduler
will
have
to
go
back
to
work
all
over
again
from
the
top,
so
this
would
increase
the
overall
scheduling
load
for
the
system,
so
the
other
option
was
to
put
the
resize
into
the
workflow,
bringing
the
scheduler
into
the
decision-making,
just
as
it
would
if
it
were
a
new
part
coming
in.
We
looked
at
this
as
ok.
This
is
the
current
part
has
requested
X.
Now
the
it's
requesting
X
plus
y.
A
So
it's
as
if
it's
a
new
Delta
pod
of
size,
Y
on
the
same
node,
so
we
went
with
this
option
the
scheduler.
The
way
we
do
this
is
we
set
an
annotation
saying:
I
want
my
resources
are
X.
I
want
Y
and
schedule,
looks
at
this
and
looks
at
its
node,
available
resources.
Cache
and
says:
ok
looks
like
you're.
The
node
has
capacity
I'm
going
to
let
this
through
and
once
the
scheduler
lets
this
through.
It
does
so
by
updating
the
resources
section
the
couplet
takes
over
and
it
does
the
submission.
So
we
are.
A
We
are
focused
on
updating
the
resources
section
here
and,
as
shown
in
the
spec,
so
going
a
bit
more
deeper
into
this.
It
starts
with
the
API
caller,
which
could
be.
You
could
be
using
coop
CTO,
a
patch.
If
you
were
doing
it
manually
in
automation,
it
would
be
a
vertical
part
autoscaler
which
would
be
doing
this
for
you,
so
we
in
our
case
the
customer
wanted
jobs.
A
So
naturally,
we
started
with
job
controller,
but
this
could
be
done
for
any
kind
of
controller
or
it
could
be
done
for
a
pod,
standalone
pod,
and
so
what
we
do
is
we
patch
the
job
saying:
okay,
I
have
one
CPU
I
need
to,
and
then
the
API
server
does
some
basic
validation,
which
is
you
should
not
change
the
QoS
class.
By
doing
this,
and
once
it
sees
that
the
request
is
good,
there
is
no
errors
there.
It
updates
the
etcd
and
updates
the
job
spec.
A
Now
the
job
controller
here
is
washing
the
job
spec
it
sees.
There
is
a
difference
between
the
job
spec
and
what
the
running
bar
has,
and
it
says.
Ok,
this
is
a
case
of
a
resize.
We
need
to
bring
the
running
part
to
the
desired
size,
that's
in
the
job
spec,
so
it
sets
the
annotation
and
the
scheduler
which
is
watching
the
annotation
goes
to
work
here.
So
what
the
scheduler
does
it
interprets
that
annotation
as
a
resize
request
and
it
checks
its
node
resources
cache.
A
If
there
is
capacity,
it
will
say:
ok,
we're
going
to
go
to
step
6,
which
is
update.
The
pod
resources
now,
if
the
request
cannot
be,
cannot
be
filled
in
that
node,
then
it
looks
at
the
policy
if
the
policy
allows
ok,
if
I
can't
get
two
CPUs
on
this
node
put
me
on
some
other
node,
then
it
will
evict
the
pod
and
then
we
schedule
it
to
you
know.
If
the
policy
says
now,
don't
reschedule
me,
then
it
will
market
his
faith.
A
B
A
Like
it
needs
more
resources
or
less
resources,
and
it
runs
through
the
same
predicates
that
the
scheduler
is
looking
through
same
algorithm,
so
it
sees
if
it
fits.
This
is
again
with
the
multiple
scheduler
race
condition
in
mind,
so
assuming
that
it
fits,
then
it
will
set
the
resize
pod
condition
to
true.
If
it
does
not
fit,
it
would
set
the
resize
pod
condition
to
false.
Now
and
again,
if
it's
doesn't
fit,
it
will
check
policy
and
evict
if
necessary
or
Marquez
failed.
A
Now
we
go
to
the
next
step
where,
if
the
scheduler
is
watching
the
pod
status,
once
the
Keyblade
has
had
a
chance
to
update
this,
look
at
the
resize
request
and
update
the
pod
status.
If
the
couplet
was
successfully
was
able
to
successfully
resize
support,
accept
the
updated
request,
then
there
is
nothing
else
left
to
do
for
the
scheduler.
This
is
a
completion
of
a
successful
resize
operation.
A
However,
if
the
couplet
said
now,
I
couldn't
take
this
resize
because
there's
no
capacity,
then
the
pod
resize
condition
would
be
set
to
false
and
the
scheduler
will
roll
back
the
resource
update
that
it
did
in
step
six,
and
at
this
point
the
job
controller
will
come
and
take
a
look
at
it.
So
you
know
this
is
not
they
don't
match.
I'm
gonna
try,
but
this
is
a
failure.
A
I'm
gonna
come
back
later
and
try
again
when
I
might
be
supposed
successful,
so
we
had,
we
got
triggers
for
us
a
certain
degree
of
control
from
the
users
and
we
came
up
with
these
policies.
The
in
place.
Preferred
policy
is
where
you,
where
you
say:
okay,
if
I
cannot
be
resized
to
the
requested
capacity.
On
that
note,
we
scheduled
to
a
new
node
and
then
I'll
be
fine.
This
is
in
this
case
you
could
have
this
is
for
then
you
have
deployments.
This
would
be
the
kind
of
scenario
that
you
are
looking
to.
A
A
They
don't
want
they
if
the
CPU,
if
you
start
for
CPU-
and
you
can
potentially
continue
running
sub-optimally
so
you'd,
say:
okay
I'll
take
my
chances
I'm
going
to
continue
to
run
here
because
rescheduling
me
is
more
expensive
and
the
the
customer
whose
application
doesn't
wanna
be.
He
said
very
schedule
would
choose
this
policy
and
then
kubernetes
system
would
try
to
get
to
that
at
a
later
point
through
retries
restart.
A
This
is
an
interesting
one
where
there
are
some
applications
where,
even
if
you
were
to
successfully
resize
the
pod
all
the
same
node,
it's
not
able
to
use
that
this
one
example
that
I
have
for
this
is
Java
with
the
X
MX
flag,
where
you
specify
the
max
limit.
Even
if
you
increase
it,
you
have
to
restart
the
application
to
get
that
make
use
of
that
increased
memory
limit
and
failure
handling
the
scheme
came
out.
A
A
So
if
you
have
a
policy
of
increase
preferred-
and
you
failed
because
of
poor
desertion
budget,
when
you
get
the
notification
that
your
pdb
is
available,
you
try
that's
a
good
time
to
try
and
if
it
is
the
case
we're
in
place
only
and
you
have
you
don't
have
resources
on
the
node,
then,
when
other
parts
of
the
same
priority
are
hired,
they
leave
the
node.
That's
a
good
time
to
try.
So
we
we
would
do
this
based
on
events
besides,
in
addition
to
doing
based
on
time.
A
Vertical
power
arts
give
us
this
integration
with
VBA.
We
were
looking
into
how
to
do
this
because
eventually
we
will
be
using
VBA.
What
we
did
here
was.
This
is
a
very
brief
overview
of
how
BPA
works.
Here
you
have
the
recommender,
you
have
the
B
pod
vpa
object
where
the
recommendations
are
written
and
you
have
the
updater
which
does
which
takes
actions.
So
we
p8
the
way
it
comes
up
with
recommendation.
A
It
reads:
the
current
pod
usage
of
resources
from
the
metrics,
API
I
believe
API
server
has
this
metrics
API,
which
pulls
the
four
metrics
server.
You
have
to
run
that
component
and
then
you
have
Prometheus
cap
a
time
series
database,
which
stores
the
history,
historic
utilization
of
the
pod
and
BPA
consumes
these
two
in
pieces
of
information
and
the
recommender
comes
up
with
a
recommendation
in
this
case.
A
If
you
look
at
the
example
there
we
have
two
CPU
and
one
gig
of
memory,
maybe
VPN
says:
ok,
you
might
benefit
with
more
memory,
so
I'm
gonna
write
two
gig
of
memory
with
1.5
lower
bond
2.5
upper
bound.
This
is
a
very
abbreviated
example
of
what
a
recommendation
looks
like
and
once
we
piay
recomm
recommender
has
written
the
recommendation
to
the
VP
a
object
for
that
part.
Vp
a
updater
goes
to
work.
A
Vp
updater
is
watching
the
pod,
the
pod
VP
a
object
and
its
job
is
to
figure
out
how
to
give
this
pod.
The
new
recommended
resources.
Now
today
it
does
this
by
evicting
the
current
pod
and
then
the
controller
creates
a
new
part
to
replace
that
oh
I'm
missing
a
part.
So
it's
going
to
create
a
new
part
and
during
admission
control
it
rewrites
the
resources
section
with
the
new
recommended
resources
with
our
work.
That
is
unnecessary.
We
just
simply
update
it
and.
A
Coming
to
memory
usage
spikes,
so
even
with
this
feature,
you
could
have
your
pods
getting
out
of
memory
killed.
This
can
happen
for
a
few
reasons
you
have.
There
is
a
certain
sampling
interval
for
the
map
from
the
metrics
API,
which
is
I,
believe
30
seconds
and
VP
a
response.
There
is
a
lag
with
VP
as
well,
so
if
your
memory
is
really
spiking,
which
happens
at
the
start
of
the
application,
or
maybe
at
the
tail
end,
you
could
potentially
have
your
app
killed
before
the
VP
has
had
a
chance
to
resize
it.
A
So
we
looked
into
this
and
we
were
looking
for
a
solution.
We
looked
at
this
home,
kill
disable
flag.
That's
there,
the
C
group
and
I
believe
it's
kind
of
supported
with
different
runtimes.
Although
it's
hit
or
miss,
we
put
an
annotation
there,
which
the
uncle
disable
annotation,
which,
if
you
would
set
it
true,
we
disabled
the
home
killer
for
the
part
and
what
happens
in
this
case.
Your
pod
won't
get
killed
if
it
exceeds
the
memory
limit
that
you
are
asked
for.
A
Instead,
the
applications
in
the
pod
get
suspended
there
paused
and
then
until
VP
a
or
has
had
a
chance
to
look
at
the
usage
and
then
update
and
give
it
more
memory.
For
example,
and
once
that
happens,
the
application
becomes
runnable
again
and
its
resume
starts
running
again.
This
is
suitable
for
our
customers,
long-running
jobs,
where,
if
you're
a
for
our
job
at
3
or
15
minutes,
you
hit
this
condition.
You
do
not
want
to
be
killed
for
that,
and.
A
A
So
this
is
the
job
that
have
been
talking
about.
We're
looking
at
the
shop
would
run
a
part
that
contains
one
container
in
stress.
It's
your
Ubuntu
image
with
stress
app
installed
on
it,
because
I
want
to
run
some
stress
tests
on
this,
to
stress
the
memory
and
it's
going
to
ask
for
three
CPUs
and
one
gig
of
memory.
So,
let's
create
the
part.
A
Now
this
is
just
your
standard
cube,
CTL
described
part
output,
I
put
some
filters
and
there
to
filter
out
information
that
we
really
don't
care.
What
and
care
about
and
focus
on
what
we
are
interested
in
here.
So
you
can
see.
As
you
can
see,
the
pod
has
been
scheduled.
It's
running
on
node
one.
It's
got
this
container
ID
here
and
it's
got
three
CPUs
and
one
gig
of
memory.
Now,
let's
run
the
stress
application
asking
for
500
Meg.
A
A
Oops
well,
it's
expected.
You
have
one
gig
of
memory
and
you're
going
for
1.5
and
the
own
killer
says
no
so
signal
line.
As
you
got
terminated
today,
you
can
VP
a
would
potentially
do
this
fix.
It
would
take
care
of
this
for
you
by
analyzing
what
your
current
usage
is.
It's
growing
and
then
giving
you
more
memory,
but,
as
I
mentioned
earlier,
you
have
to
reschedule
the
part
you
have
to
evict
the
current
part
and
then
update
the
resources
of
the
new
part.
A
A
So
all
this
does
is
runs
that
cube
city
old
patch
command
that
I
described
in
the
earlier
slide.
So
we
did
that
and
if
you're
watching
the
blue
window,
you
see
that
it's
gone
to
do
gig.
So
the
container
the
pod
is
running,
the
container
is
running,
it
hasn't
been
restarted.
Let's
make
sure
that's
the
case,
so
we
just
run
this
again
and
see
it's
now.
Gonna
go
and
allocate
1.5
gig.
It
seems
to
be
getting
farther
along
down
the
road
and
exists
out
of
zero.
That's
a
good
thing.
A
So
last
year,
late
last
year,
as
we
were
finishing
up,
the
first
iteration
of
this
project-
JD
comms
modulator-
comes
engineers
reached
out
to
us
and
they
said
they
were
interested
in
trying
out
our
work
and
we
partnered
with
them.
They
needed
this
for
enable
this
for
deployment
parts.
So
we
worked
with
them
and
got
it
working
for
deployment
and
they're
reporting
that
it's
doing
fairly
well
in
its
in
pre-production
staging.
A
They
found
another
use
case
for
this,
where,
if
they
see
parts
that
are
not
scheduled,
they're
pending,
because
resources
is
not
available,
they
do
a
walk-through
and
see
which
parts
are
not
using
the
resources
that
they
have
been
given
and
then
try
to
resize
them
down
and
that
way
that
they
get
these
pending
pods
running.
So
they
get
more
work
done
and
the
the
last
slide
that
I
have
here.
These
are
the
resources,
it's
the
design,
doc
and
implementation.
These
are
pretty
much
like
I
see
at
this
point.
A
What
I
want
to
focus
on
is
the
latest.
The
third
bullet
point
over
there
based
on
there
was
a
another
design
that
was
proposal
that
was
preceding
us.
So,
with
Carol
and
viata
from
the
auto-scaling
team,
they
started
this
new
proposal,
which
merges
the
two
design
designs
and
then
it's
working
progress
at
this
point.
I
highly
recommend
that
you
guys
go
to
this
github
kubernetes
enhancement,
enhancements,
link
and
take
a
look
at
that.
A
If
this
was
you
before,
we
conclude,
I
have
a
few
asks
container
run
time:
growers
if
you're
in
the
audience,
please
ensure
good
support
for
update
container
resources,
API
and
consider
our
own
kill
disable.
We
kind
of
would
like
that
feature
to
be
enabled
and
work
well
across
different
runtimes
Kartik,
especially,
and
we
have
been
working
closely
with
signaled
on
this
new
cap
and
one
of
the
things
that
we
could
use.
A
How
but
signal
could
use
help
with
is
reviewing
code
PRS
and
looking
at
issues,
if
you
have
the
bandwidth
and
the
interest
we'd
really
like
you
to
do
that,
there
is
another
talk
right
after
this
talk
back
Carol
and
be
at
our
I.
It's
also
it's
going
to
talk
about
this
latest
cap
and
where
things
are
I
highly
recommend
that
you
attend
that.
So
with
that
I
conclude
the
presentation
and
open
up
for
questions
the
mic
here.
C
A
The
question
is:
do
you
recommend
manually
updating
the
resources
or
using
VP
a
I
believe
the
answer
is
to
use
VP
a
nobody
wants
to.
You
could
do
it
manually
for
experimenting
to
see
what
is
the
right
size
for
your
resources,
but
VP
a
has
a
mode
where
it
lets
you
do
that
for
more
details,
I
think
I
would
recommend.
A
Auto
mode
is
the
one
where
it
would
update
it
for
you
today.
Does
it
by
evicting
I
would
say
that
eventually,
it
should
become
instead
of
evicting
updated
in
place,
but
if
not,
there
might
be
a
new
mode
for
it.
I
don't
know
where
that
will
land
this
feature.
This
proposal
is
still
a
work
in
progress
and
once
it's
approved
we'll
work
on
implementing
it
and
then
a
full
land
it'll
take
some
time
for
that.
A
D
In
your
use
case,
you
have
this
very
important
job
that
you
could
not
kill,
but
if
there
are
not
resources
on
the
nodes,
can
you
also
evict
other
parts
to
other
nodes,
because
I
think
there
are
also
other
initiatives
to
make
the
scheduler
more
dynamic
to
spread
of
workloads
even
and
how
does
it
work
together
with
the
proposal
that
you
had
now?
Yes,.
A
In
the
new
proposal
we
are
considering
that
so
the
scheduler
looks
at
any
parts
that
are
of
lower
priority
than
the
current
part
and
it
fixed
them
as
part
of
giving
it
more
resources,
that's
natural.
If
you
were
to
schedule
a
new
part,
it
would
do
that
and
if
a
new
part
with
the
higher
priority
comes
in
it
does
the
eviction
anyways.
That
feature
came
in
recently,
I
believe
I,
don't
know,
maybe
in
the
last
year
or
so
so
that
feature
is
there.
It's
gonna
leverage
that.
A
A
A
We
we
don't
have
support
for
extended
resources
that
mainly
vp8
is
not
supported
today,
but
that's
something
that's
been
asked
before.
If
there
is
a
strong
desire
for
doing
this
for
extended
resources,
the
support
will
eventually
come
in.
The
scope
of
this
cap
is
kept
for
the
CPU
and
memory
at
this
point
and
potentially
ephemeral
disk
space.
A
Yes,
yes
in
the
new
cap,
we
are
gonna,
be
auto
scaling
down
the
auto
scaling
down
of
memory
is
a
little
bit
of
a
challenge,
because
you
can't
really
take
away
memory
if
the
pages
are
in
use,
so
we're
still
working
through
how
we
could
really
do
that.
I
think
the
solution
there
is
to
set
your
limit
at
slightly
below,
what's
being
currently
used
and
try
to
force
a
reclaim.
That
is
what
it
would
do
at
this
point,
but
CPU
we
can
take
away,
see
giving
and
taking
away.