►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How To Handle Node Shutdown In Kubernetes - Xing Yang & Ashutosh Kumar, VMware
Speakers: Ashutosh Kumar, Xing Yang
Shutting down of a node is an inevitable event and it can be graceful or non graceful in a Kubernetes cluster. A node shutdown can be graceful only if it can be detected by the Kubelet ahead of the actual shutdown. A node shutdown may not be detected by the Kubelet due to a variety of reasons causing the shutdown to be non graceful. In the talk, Xing and Ashutosh will explain the graceful shutdown concepts and its impact on the running workloads including the systemd inhibitor locks mechanism and configuration settings. In Kubernetes v1.24, alpha support for handling non graceful shutdown is introduced which enables replacement pods for StatefulSets to be created successfully on a different running node which otherwise would be stuck. The talk will explain how to use the non graceful shutdown feature using taints and the future roadmap around making the feature more automated.
A
A
A
A
But
no
shutdown
could
happen
for
many
reasons.
It
could
be
that
you
need
to
apply
a
security
patch.
You
need
to
do
your
kernel
upgrade
you
need
to
reboot
a
node
or
it
could
be
due
to
the
preemption
of
the
VM
instances,
or
it
could
be
that
there's
a
harbor
failure
or
some
software
problem
that
causes
that
to
happen.
A
A
Without
this
feature,
user
have
to
make
sure
they
manually
join
the
notes
before
shutting
down
the
node.
However,
no
shutdown
Could
Happen
unexpectedly,
in
that
case
the
parts
could
be
evicted
unsafely
if
the
node
is
not
drained
and
your
application
would
see
errors
and
your
workloads
may
not
function
properly.
A
A
There
are
two
config
parameters
in
couplet
that
you
need
to
set
for
this
feature
to
work.
The
first
one
is
the
Sharon
grace
period
That's,
the
total
duration
for
both
the
regular
and
the
critical
parts
to
be
terminated,
and
the
second
parameter
is
the
shutdown
grace
period
critical
parts.
That's
the
time
needed
for
the
critical
parts
to
be
terminated
and
that's
always
smaller
than
the
total
duration
and
the
graceful
notion
feature.
It
works
in
two
phases:
the
notes
the
parts
are
terminating
two
faces,
first,
the
regulars
and
then
the
critical
Parts.
A
But
if
you
want
to
have
more
granular
control,
then
there
is
another
feature
called
a
pod
priority
based
grease
for
no
shutdown.
That's
an
alpha
feature
introduced
in
1.23
release,
so
you
can
configure
your
Port
priority
classes
into
a
multiple
classes
types
and
then
the
parts
will
be
shut
down
in
multiple
phases,
depending
on
how
you
define
your
priority
classes.
B
B
Zing
just
talked
about
graceful
shutdown,
and
there
are
scenarios
when
the
shutdown
Is
Not,
So
Graceful.
It
could
be
due
to
some
configuration
error
or
maybe
due
to
a
hardware
failure
or
if
a
shutdown
is
not
detected
by
cubelet,
and
this
kind
of
shutdown
can
be
termed
as
known
graceful
one
and
it
can
be
problem
for
stateful
workloads.
B
I've
already
talked
about
system
D,
inhibitor
log.
If
the
shutdown
does
not
trigger
inhibitor
inhibitor
lock,
then
it
will
not
be
ejected
by
cubelet
or
if
you
set
shut
down
grace
period
or
shut
down
grace
period,
critical
paths
incorrectly.
This
can
also
let
lead
to
non-graceful
Saturn
and
as
a
side
effect
of
this.
What
happens
is
the
prod
moves
to
a
terminating
State,
especially
the
parts
that
are
using
volumes?
B
So
here
is
the
experiment
that
I
did
like
some
time
back.
I
used
vsphere
UI
to
test
that
so
I
deployed
this
stateful
set
and
I
triggered
a
set
down
from
the
vsphere
UI
button
and
that,
as
a
matter
of
fact,
is
a
non-graceful
set
down
and
what
I
observed
is
the
Pod
went
into
terminating
state
after
five
minutes
and
also
what
I
observed
is
that
part
is
stuck
in
terminating
State,
even
after
six
minutes,
I
code,
six
minutes
here
it
is
an
important
number
I'll
come
to
that
later.
B
B
There
has
also
been
talks
about
node
and
control
plane,
partitioning
in
case
node
shutdowns,
but
it
does
not
include
in
Gold,
because
when
a
node
is
not
ready,
it
could
be
due
to
variety
of
reasons.
One
could
be
due
to
a
split
of
the
network
and
we
really
don't
know
if
the
node
is
not
ready
because
the
network
is
and
Rachel
or
the
node
actually
went
into
a
non-graceful
set
down.
B
Let
us
take
a
look
at
how
the
gracefuls
are
down.
Non-Gracefuls
are
Done
Works,
so
we
use
the
native
taint
apis
of
kubernetes.
In
this
case,
and
as
of
now
to
utilize
this
feature,
one
will
have
to
do
manual
steps
and
what
happens
is
once
you
figure
out
that
a
node
has
went
into
a
shutdown
non-gracefully.
B
You
apply
this
well-known
taint
that
is
node.cubernetes
dot,
IO
out
of
service,
and
once
you
do
that
there
is
two
steps
that
is
happening
behind
the
scenes.
What
happens
is
the
Pod
GC
controller,
delete
support
that
do
not
have
a
matching
generation
and
it
deletes
it
forcefully
and
after
that
attached
t-test
controller
is
going
to
quickly
do
a
force
volume
detach
operation
so
that
the
newer
part
that
might
get
spawned
on
in
your
node,
the
volume
gets
attached
successfully.
B
B
You
have
to
set
this
flag
true
in
Cube
controller
manager,
and
once
you
do
that
you
just
have
to
utilize
the
taint
and
if
you,
if
you
know
that
a
node
has
went
into
shutdown
non-gracefully,
you
can
just
use
the
taint
command
that
you
can
see
on
the
screen.
It
is
a
well-known
taint,
node.cubernetes,
dot,
IO
out
of
service
state.
B
B
Now
this
is
one
more
experiment
that
I
did
with
this
feature.
I
enabled
the
non-graceful
set
down
feature.
I
took
a
kubernetes
version
that
has
this
feature
this
time.
I
created
a
stateful
set,
did
the
same
set
down
via
the
vsphere
UI
and
observe
that
after
five
minutes,
the
part
change
to
terminating
state
and
I
used
the
cube,
CTL
10
command
to
obtain
the
node.
B
Because
now
I
know
this
is
the
pod
which
went
into
a
non-graceful
shutdown
and
the
Pod
immediately
failed
over
to
a
different,
healthy
node,
and
it
didn't
hit
had
to
wait
like
six
minutes
for
the
path
to
come
to
running
state
and
just
take
caution
here
whenever
you
are
trying
to
utilize.
This
feature
just
make
sure
that
your
node
has
really
went
into
a
non-graceful
set
down,
and
you
are
tainting
that
particular
note.
B
B
B
B
B
B
B
B
B
B
And
we
can
see
an
error
is
reported
from
a
task
controller
that
is
multi-attacher
and
let's
do
right.
So
this
is
the
problem
that
we
are
into
now.
So
what
I'll
do
now
is
I'll,
wait
for
like
six
minutes
and
if
we
do
so,
this
part
should
come
to
running
state,
because
this
is
the
default
timeout
period
for
this
volume
detached
to
happen,
and
once
that
happens,
this
font
will
be
able
to
successfully
attach
the
volume.
B
B
So,
but
we
had
to
wait
like
for
a
considerable
amount
of
time,
and
this
is
unfavorable
now
I'm
going
to
repeat
the
same
experiment,
but
this
time
I'm
going
to
utilize
the
non-graceful
set
on
feature
and
realize
how
it
helps
here.
So,
but
before
that,
let
me
just
start
the
cubelet
back
onto
that
note.
B
B
B
B
B
B
And
a
new
part
is
got
scheduled
onto
a
different
node,
that
is
the
node
ending
with
6
KPS,
and
it
is
in
container
creating
State.
Now,
let's
check
once
more
awesome
now
this
part
came
to
running
state,
so
we
can
see
that
the
Pod
came
back
online
fairly
quickly
and
we
didn't
have
to
wait
like
much
longer,
so
that
is
it
I
wanted
to
show
in
this
demo.
Thank
you
thanks.
I
would
like
to
invite
same
zinc
to
continue
further.
C
A
A
A
A
But
the
problem,
of
course,
is
the
Cs
driver
needs
to
have
this
knowledge.
We
are
not
sure
if
that's
possible,
it's
possible
for
every
C-section
driver,
but
we
will
be
looking
into
this
approach
again
in
our
next
step
and
the
second
alternative
that
we
evaluated
is
note
fencing
in
this
approach.
There
will
be
a
controller
that
monitors
the
status
of
the
notes
and,
if
the
note
Went
to
went
into
not
not
ready
status,
it's
going
to
create
this.
The
note
fan
crd
and
work
on
node
fencing
this
meth.
A
A
There
is
a
CSI
Force
detached
proposal
apparently
proposed
some
new
capabilities
in
CSI
controller
and
the
node,
and
and
also
there
is
this
Portman
project.
It
also
watches
the
status
of
the
notes
and
if
it
is
not
ready
or
it's
going
to
check
from
the
storage
side
to
see
if
Universal
IO
and
if
there's
no
IO,
then
it's
going
to
forcefully
delete
the
detach
the
volumes
and
clean
up
the
parts.
A
A
A
A
A
A
E
Great
and
question
I'm
very
aware
of
this.
This
situation,
in
my
experience
with
rwo
volumes,
it
never
gets
out
of
terminate
I
mean
how
I
know
you
said
after
six
minutes.
Well,
this
six
minutes
another
six
minutes
is
there
a
condition
which
it
would
never
get
out
of
terminate
for
rwo
volumes?
Well,.
B
That
is
far
if
the
Pod
is
managed
by
deployment
controller,
because
in
case
of
deployment
you
know
you
can
have
the
part
come
up
again
depending
on
Lake
policies
and
your
deployment,
but
in
case
of
stateful
set.
If
your
part
is
into
terminating
state,
you
know
the
New
York
pod
won't
come
up
until
it
like
terminates
so
yeah
yeah.
Oh
you
have
to
you,
know
forcefully
delete
it
and
delete
the
volume
attachments.
Yeah
thanks.
A
B
C
G
A
Understand
the
question:
yeah
I,
don't
think
that's
going
to
solve
this
problem
because
you
still
need
to
be
able
to
kind
of
delete
the
original
pod.
But
if
the
origin
node
is
not
there
anymore,
you
can't
even
delete
that
part.
The
cubelet
is
not
there
to
delete
that
right,
so
you
still
need
to
have
a
way
to
delete
your
pod.
A
D
D
B
If,
if
you
have
it's
a
Down,
if,
during,
if
during
rebalancing,
you
have
to
figure
out
what
kind
of
shutdown
it
is
and
if
it
does
not
trigger
if
it
trigger,
if
it
is
not
a
graceful
shutdown,
it
will
help.
But
right
now
you
have
to
manually
taint
the
notes
and
you
have
to
actually
know
what
are
those
notes
that
are
going
out
of
the
cluster
and
then
you
can
utilize.
This
feature.
C
A
B
C
B
C
F
B
Don't
think
so
because
you
do
this
steps
when
you
know
that
you
have
a
non-graceful
shutdown
and
usually
those
kind
of
shutdowns
will
not
be
under
your
control
like
at
most
times
so
you
know
once
you
know
that
this
node
is
into
a
sat
down,
then
you
follow
these
steps
and
the
grace
if
you
are
talking
about
draining
what
I
was
mentioning
on
the
slide.
Is
you
know
when,
even
if
it
was
a
graceful
shutdown,
you
know
there?
B
There
are
some
shutdown
you
know
for
which
we
can
take
precautionary
steps,
but
we
couldn't
because
the
graceful
sit
down
feature
was
not
there
and
you
had
to
manually
go
and
drain
the
nodes
and
do
all
this
stuff.
But
now
because
we
have
this
graceful
sit
down
feature
you
really
do
not
require
to.
You
know:
do
all
this
stuff
and
yeah.
B
Yeah
I
I
I'm
not
sure
about
how
it
will
count
the
part
disruption
policies,
but
yeah
I
need
to
check
yeah,
but
in
general
the
idea
is:
you'll
have
Parts
marked
critical
parts
or
regular
pots
according
to
the
policies,
and
you
basically
configure
parameters
on
cubelet.
For
example,
you
have
30
seconds.