►
Description
Highly Available Services During Maintenance Events - Maisem Ali & Eric Tune, Google
Maintenance events occur and require taking down nodes for various reasons. Eric and Maisem talk about the best practices and lessons learned trying to minimize downtime during routine maintenance events.
They show how to use StatefulSets and PodDisruptionBudgets to achieve highly available services. They go on to explain what the best practices for performing node maintenance are using scenarios like failed pod evictions, non-responsive kubelets and network bisections.
About Maisem Ali
Maisem has been working on Kubernetes for the last two years. He has heavily contributed to upgradability and upgrade testing between Kubernetes versions, and primarily focuses on the Google Kubernetes Engine control plane
About Eric Tune
Eric has worked on Kubernetes since before the first public release. He has contributed to Security, Tenancy, Application Controllers, Charts, Jobs, documentation, and more. Before Kubernetes, he worked on Google's Borg system, on datacenter-scale efficiency and performance measurement, and on web search.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi
I'm
Messam
I've
been
working
on
gk
kubernetes
for
the
past
two
years
now
and
today
we're
gonna
talk
about
each
a
services
during
maintenance.
So
what
that
means
is
so
I
don't
know
if
you've
heard
about
all
the
cool
things
that
Clayton
and
Kelsey
have
been
talking
about.
Infrastructure
should
be
boring.
So
why
should
know
what
upgrades
be
any
boring
or
I
should
note?
It's
not
be
boring,
so
that's,
let's
make
him
boring
stock.
Rather
so
maintenance
happens.
We
all
know
that
you
know
it
just
does.
A
There
are
two
primary
things
that
we
see
in
maintenance,
involuntary
disruptions
and
wall,
introduce
options,
involuntary
are
unavoidable
and
under
the
de
label,
like
disk
failures,
kernel
panics,
all
of
those
things
happen
network
by
sections
happen,
racks
get
on
fire,
cooling
turns
off
fun,
stuff
happens.
All
the
time
wallet
interceptions
are
more
interesting.
They
have
so.
If
you
have
a
security
patch,
then
you
need
to
do
you
have
node
upgrades.
A
You
have
kubernetes
upgrades
there,
multiple
types
of
things
that
you
need
to
do
to
make
sure
that
you
are
not
compromised
or
you're,
not
running
a
really
old
version
and
you're
not
out
of
support.
So
how
do
we
do
all
of
that?
Talk
about
that,
so
the
primary
concern,
where
maintenance
is
that
it's
fundamentally
disruptive
a
you
need
to
replace
a
node
to
change
the
kernel.
For
example,
you
need
or
something
equivalent
they
need
to
be
either
rebooted
or
replaced,
and
when
you
do
that,
you
fundamentally
take
out
capacity.
A
You
need
to
your
pods
are
shuffled
in.
How
do
you
prevent
against
making
like
all
those
like
when
were
close
are
shuffled?
You
need
to
make
sure
that
your
when
your
pod
is
going
down.
You
need
to
be
sure
that
you
fulfill
any
incoming
requests,
any
requests
that
are
already
in
flight.
It
could
take
an
hour
for
you
to
fulfill
that
request,
so
you
should
be
very
careful
when
taking
down
those
spots
like
you
want
to
be
sure
that
you
have
enough
capacity,
you
don't
have
downtime
and
you
are
not,
and
the
fun
part
begins.
A
I,
don't
know
if
you
saw
the
Nordstrom
truck,
they
talked
about
how
you
could
end
up
in
a
split
brain
mode
and
we're
at
CD.
You
have
two
different
sets
of
at
CDs
talking
about
doing
different
things
and
they
have
different
information,
who's
right,
who's
wrong.
How
do
you
you
don't
want
to
be
in
that
say
that
state
is
very
messy,
so.
A
So
what
are
your
options
for
maintenance?
You
can
accept
downtime,
that
is
horrible.
You
don't
want
to
say
you
don't
wanna
tell
you,
users,
hey
we're
doing
a
maintenance
event
and
we
want
you
to
come
back
after
an
hour.
That's
not
okay
and
it
could
also
lead
to
under
your
application
or
if
you
have
a
service,
that's
like
a
memory
cache
and
you
don't
flush
it
properly.
A
Well,
you're
gonna
have
data
loss.
You
could
be
in
where
you
were
to
say
for
a
long
time,
and
getting
out
of
that
is
gonna
be
costly.
Another
approach
that
you
can
take
is
you
can
have
a
completely
replicated
cluster,
which
is
a
snap
to
failover
and
as
soon
as
something
goes
wrong.
You
need
to
move
on
to
that,
but
that
takes
human
effort
that
takes
time
and
it's
not
a
straightforward,
Federation
or
multi
cluster
will
make
it
better,
but
it's
still
fundamentally
hard
at
Google.
A
A
However,
it
does
obviously
come
with
extra
maintenance
cost
extra
developer
cost,
because
you
need
to
think
about
all
the
different
failure
scenarios.
What
happens
if
you're
fulfilling
your
request
and
you
crashed
and
all
that,
and
just
like
writing
a
distributed
application?
You
should
also
realize
that
you're
running
on
distributed
infrastructure.
That
means
the
same
principles
that
apply
here.
Applications
also
apply
to
the
infrastructure
that
things
will
go
down.
Things
will.
A
So
if
you
go
with
a
brooch
three,
you
can
build
more
automation.
You
can
have
things
automatically
move
across
you
nodes
or
across
resources,
and
the
great
thing
about
that
is
this
kubernetes
is
built
for
that.
That's
why
you
should
it's
great
that
you're
here
and
let's
talk
about
how
we
can
make
this
better,
so.
A
B
Build
maintenance,
tolerance,
applications,
the
kubernetes
you
a
number
of
different
tools
and
our
underlying
philosophy
is
of
so
there's
two
different
boxes
of
tools.
One
is
for
application
owners
and
one
is
for
cluster
owners
and
the
reason
we
think
about
splitting
the
roles
in
the
cluster
into
two
parts
is
to
allow
two
things.
B
One
is
if
you
have
many
different
teams
to
point
microservices
into
a
single
cluster:
it's
not
not
always
practical
for
the
it's
not
always
practical
for
the
cluster
operator
to
understand
all
of
them,
and
if
you
want
to
automate
cluster
operations
such
as
gke
is
increasingly
doing,
then
you
need
to
have
a
clear
contract
between
cluster
operators
and
application
owners.
Application
owners
understand
what
the
availability
model
quorum.
Whatever
the
requirements
are,
how
long
it
takes
things
to
drain,
whereas
the
cluster
operators
needs
to
understand
these
things
in
the
abstract.
B
So
the
application
owners
tools
are
graceful
termination
which
I
want
to
encourage
everyone
to
understand
and
use
if
you
haven't
used
it
before
and
a
graceful
termination
says
when
you
says
to
the
person
who
is
doing
maintenance,
be
patient
with
map,
here's
how
patient
you
have
to
be.
Let
me
know
when
you're
gonna,
when
you're
gonna
turn
me
off.
It
gives
you
a
chance
to
let
request
complete
and
it
gives
you
time
to
flush
or
commit
state.
You
know
to
a
local,
disc
or
the
remote
store.
B
Now
the
pro
move
is
to
use
pod
disruption,
budgets,
which
we'll
also
talk
about
in
demo
here
today
and
pod
disruption
budgets.
Allow
you
to
say
not
only
be
patient,
but
be
careful.
Don't
take
too
many
in
my
instances
down
at
once.
This
prevents
loss
of
quorum
which
causes
unavailability
and
it
can
prevent
losing
all
your
replicas
of
a
state
store
before
they
can
flush
state
which
could
cause
under
ability
for
lack
of
a
better
word.
B
So
graceful
termination
I'll
go
through
the
timeline
of
what
happens.
So
if
the
user
controller
deletes
a
pod
by
the
REST
API
anytime,
you
want
to
like
scale
down,
you
say:
delete
something
you're
rolling
out
a
new
thing.
That's
all
doing
a
pod
delete
because
pods
don't
really
get
modified.
They
just
get
deleted.
It
doesn't
delete
from
the
REST
API
when
you
delete
it.
It
just
gets
a
deletion
time
stamp
set
in
the
metadata
header.
So
then,
a
bunch
of
things
in
kubernetes
are
gonna
notice
that
it's
pending
to
delete
Kubla
is
gonna.
B
Send
a
signal
under
pre.
Stop
handlers.
If
you
have
any
for
your
pod,
the
endpoints
didn't
get
removed
from
the
service
and
then
you're
gonna
stop
load-balancing
to
that
instance,
but
it's
not
going
to
get
it
killed.
Yet
the
positive
gate
reporter
is
not
ready.
So
you
can
see
that
in
your
you
eyes,
but
the
couplets
gonna
wait
for
a
certain
amount
of
time.
The
default
is
30
seconds,
but
I
would
encourage
you
if
you
have
stateful
applications
to
set
a
longer
pod,
spec
termination
grace
period
seconds.
B
You
can
set
that
as
long
as
you
want
the
couplet.
Well,
after
that
time
period
is
exceeded,
it's
gonna
send
us
a
kill
because
it'll
assume
well,
maybe
if
it's
waited
longer
than
it
said
it
needed
that
it
must
be
wedged
or
if
the
application
exits
sooner,
then
you
know
the
couplets
got
immediately
gonna
clean
it
up.
B
So
when
application
is
either
reach
to
the
end
of
its
period
or
it's
exited
already,
the
Akula
is
going
to
clean
up
all
its
resources
and
then
which
really
make
sure
it's
completely
dead
and
the
last
thing
it's
going
to
do
is
delete
it
from
the
rest
api
so
like.
How
should
you,
as
an
application
owner,
understand
this
so
you're
gonna
want
one
option
is
to
handle
a
sig
term
or
you
can
use
a
pre
stop
Handler.
B
We're
gonna
want
to
do
that
and
then
you're
gonna
want
to
finish
outstanding
request
and
there's
two
ways
you
can
do
that
one
is
you
don't
do
anything
and
you
let
kubernetes
service
automatically
remove
you
from
the
endpoints
so
that
new
requests
don't
come
to
you
and
you
just
keep
handling
existing
requests,
as
you
normally
would.
So,
just
no
action
there's
a
little
bit
of
delay.
So
if
you
want
to
like
be
faster
more
agile,
then
you
can
actually
start
refusing
new
connections
immediately
with
something
like
HTTP
to
go
away.
B
But
that's
like
that's
for
you
don't
have
to
you.
Don't
have
to
do
that.
That's
so
the
way
that
you
so
I
would
definitely
encourage
everyone
to
look
into
a
graceful
termination.
If
you're
running
stateful
services
on
kubernetes-
and
you
want
to
be
able
to
run
node
upgrades
across
those
like
some
people,
I
talked
to
today-
are
running
at
CD
on
kubernetes,
and
then
they
have
all
their
service.
Depending
on
that,
you
really
want
to
make
sure
you
don't
lose
quorum
for
at
CD,
while
you're
updating
the
nodes.
B
So
for
that
I
encourage
you
to
create
a
pod
disruption.
Budget
object
for
your
EDD
cluster
so
asset.
So
this
is
a
long
name,
I'd
like
to
say
PDB,
because
whoever
thought
of
pod
disruption
budgets
didn't
have
RSI
yes,
so
it's
a
set
of
replicated
pods,
which
are
selected
by
a
label.
Selector
has
a
budget
which
is
how
many
can
be
down
at
once,
new
to
disruptions
so
stateful
sets
you
can
use
the
withing
use
the
deployments
they
don't
have
to
be
stateful.
B
B
So
the
way
that
you
say
that
you
want
to
delete
something
carefully
and
nicely
is
by
using
the
eviction
sub
resource.
We
actually
don't
have
a
great
command
to
do
that
directly,
but
what
it
does
it
says
if
there's
enough
of
these
pods
in
this
group,
then
delete
it
with
grace
period.
Otherwise,
it's
gonna
send
you
back
a
429
that
says:
try
again
later
so
normally
you
won't
need
to
do
this
manually,
but
you
can
figure
out
if
you
want.
B
B
B
Men
available,
saying,
like
I
need
at
least
two
out
of
three
to
have
quorum
max
unavailable
is
saying,
like
I
can
handle
10%
being
down
at
any
given
time,
because,
even
though
I'm
scaling
up
and
down
due
to
load,
I
still
have
a
10%
buffer
and
I
reserved
that
additional
10%
for
the
possibility
of
losing
capacity,
and
do
a
note
due
to
a
note
upgrade
so
Massa
was
gonna,
talk
more
about
the
draining
process
and
then
do
a
demo.
Thanks.
A
Thank
you
Eric,
so
we're
gonna
talk
about
draining
a
node
one
on
one,
it's
essentially
a
rapper.
This
is
essentially
how
q
pedal
drain
works
and
the
first
thing
that
you
need
to
do
before
you
drain
a
note
is
coordinate.
You
really
don't
want
anything
being
scheduled
on
top
of
it
when
it's
when
you're
doing
a
practice
rain.
If
you
get
more
pods
on
it,
then
it
means
that
is
bad
because
you
just
started
rain
anyways.
A
So
another
point
to
make
on
that
is:
what
happens?
The
reason
you
want
to
violate
the
pdb
after
one
hour
is
the
if
your
cubelet
on
that
node
is
dead,
if
it's
not
respecting
anything,
if
it's
not
there,
if
it's
not
present,
and
it's
not
taking
any
actions
that
you
asked
you
to
take,
then
you're
essentially
running
a
black
box
that
you
know
is
not
performing
so
in
that
scenario
we
do
want
to
move
forward
at
some
point.
A
We
want
to
wait
for
all
the
parts
to
terminate
by
the
grace
period.
So
let's
say:
if
you
have
a
pod
that
takes
30
minutes,
you
determined
that's
fine,
but
if
it
takes
six
hours
your
cloud
provider
might
might
not
be
fine
with
that.
So
on
GK
we
cap
this
at
one
hour.
If
you
have
any
concerns,
please
let
us
know
and
we
will
retweet
that,
and
so
it's
per
not
burnout.
A
A
So
yeah,
oh
and
you
should
because
kubernetes
is
a
distributed
model
and
there's
so
many
actress
playing
on
the
same
node
or
the
same
objects.
You
should
be
very
careful
when
you,
after
you've
drained
it
drained
it
out.
You
should
go
back
and
check
that
it's
actually
properly
data
trained.
If
not,
you
should
repeat
everything
again.
Otherwise
you
don't
know.
What's
going
so
with
that,
I'm
gonna
show
how
that
actually
looks
like.
A
Okay,
so
that's
the
controller
which
starts
doing
the
eviction.
First
is
going
to
set
it
to
on
schedule
and
then
it's
gonna
start
training
start
evicting
pods.
So
that's
evicting
acting
over
there,
as
you
can
see,
we
don't
have
any
disruptions
left.
We
have
no
way
we
can
move
over
right
now.
What
will
happen
is
that
the
API
server
will
reject
a
request.
It'll
say
that
no
I'm
not
going
to
allow
you
to
move
forward
at
that
point.
A
The
cloud
provider
is
supposed
to
back
off
and
wait
for
it
to
or
it
can
keep
you
trying
after
every
few
seconds
or
whatever,
until
all
the
pods
are
replicated.
So
one
thing
to
note
over
here
is
in
a
deployment
your
web
server
yep,
a
new
pod
for
the
web
server
will
show
up,
can
show
up
even
prior
to
your
victory,
pod
terminating.
That
is
e,
because
that
does
not
happen
in
a
safe
whole
set.
A
What
happens
in
a
stateful
set
is
that
until
this
replica
goes
down
completely,
the
other
one
won't
show
up
and
what's
not
shown
over
here
is
that
it
will
come
up
with
the
same
identity.
So
if
it
was
Cassandra
zero,
it
will
show
up
it's
Cassandra
zero
again
and
now,
if
you
do
in
effect,
it'll
succeed
and
it'll.
Get
reschedule,
and
one
thing
to
note
is
at
this
point:
it
is
still
not
safe
to
delete
the
note
object
from
kubernetes.
A
You
should
never
delete
the
node
object
and
that
the
node
controller
delete
it
at
least
on
GC.
That's
what
happens
I'm,
not
sure
if
other
cloud
providers
use
something
similar.
The
reason
is
you
actually
want
to
make
sure
that
the
node
has
left.
The
network
is
no
longer
active
before
you
delete
it,
so
you
didn't
like
you
should
check
that
against
a
cloud
provider
if
they
know
it
still
exists.
A
So
it
is
a
cassandra
disdainful
set
on
this
demo.
It's
a
it
has
five
replicas.
It
has
a
PD
reset
for
max
and
available
one,
so
you
can
I
can
only
take
it
on
order
play
at
a
time.
It
has
a
simple
replication
strategy
across
the
in
Cassandra
it
set
to
three
replicas.
That
means
each
data
is
replicated
thrice.
A
The
HTTP
server
is
just
deployment
which
uses
go
sequel
and
talks
to
persona
directly.
It
also
has
five
replicas
and
Matson
will
go
set
to
two.
It
essentially
means
that
I
can
say:
I
can
reduce
capacity
of
proof
like
60%
or
say
down
to
60%
yeah,
and
we
do
quorum
rights
and
single
reads
that
I
just
want
to
get
the
whatever
the
latest
value
was
and
I
want
to
go,
trim
right,
and
this
is
a
list.
It's
not
a
get
in
the
sense
that
it's
gonna
be
listening.
All
resources.
A
A
You're
gonna
see
that
the
pods
are
getting
shuffled.
Cassandra
just
got
really
scheduled.
The
web
server
has
not
been
rescheduled
yet
and
once
it
no,
it
comes
back
up
in
the
second.
No,
it
would
go
offline
or
start
and
that's
scheduling
and
you
can
see.
Cassandra
too
was
rescheduled
and
all
this
is
happening
and
you
have
static
ups
and
everything
works.
So
with
that
I
guess,
do
you
have
yeah?
You
can
watch
the
inter
I
think
it's
boring
essentially.
B
A
A
A
Not
necessarily
so,
if
you
take
down
so
the
web
server,
can
you
can
only
take
down
one
at
a
time?
Oh
sorry,
two
at
a
time
that
means
at
worse,
even
if
a
disaster
happens,
you're
down
to
40%
capacity
on
the
cassandra
the
pdb
sets
you,
you
can
only
take
down
one
at
a
time.
That
means
if
a
disaster
happens
you
still
above
quorum.
You
still
like
work.
B
B
C
B
A
B
D
E
So
one
issue
we
ran
into
when
using
PDP's
was
that
the
readiness
check
wasn't
actually
correct.
So
in
the
case
where
you
might
be
in
Cassandra,
even
at
City
right,
your
second
part
could
be
ready,
but
it
may
not
have
to
run
quorum
yet
so,
if
you're
that
and
then
then
it
will
allow
the
PDP
will
actually
satisfy.
Allow
a
third
note
to
be
taken
down.
E
So
the
issue
here
is:
if
you
rewrite
everybody
in
the
shack
to
consider
quorum,
then
you
have
a
scale
of
issue
where,
when
you
cut
when
you're
three
extenders
coming
up
there,
first
Cassandra
doesn't
have
quorum
and
you
can't
scale
up
again.
So
how
do
you
like
balance
this
back?
It's
like
a
very
small
timeframe,
but
when
we
have
network
partitions,
for
example,
you
are
no
to
report
ready,
but
they
don't
have
quorum
and
then
you
will.
Your
PD
view
will
allow
you
to
lose
quorum
on
a
subsequent
termination.
I.
E
Sure
so,
like
they're
90
solution,
we
do
site.
We
just
simply
made
the
readiness
check.
We
pour
quorum
instead
of
no
server
health,
but
then,
if
the
default
steeples,
our
rollout
scheme
is
that
yours,
Cassandra
0,
must
be
ready
before
I
will
scale
Cassandra
one.
So
you
won't
it
would
it's
actually
block
or
well
so.
B
I'd
say:
Cassandra
is
one
of
the
trickier
applications
to
make
work
with
stateful
sets.
I
would
probably
write
an
operator
for
Cassandra
that
and
possibly
have
like
a
site
car
that
ran
with
the
Cassandra
that,
like
could
detect
from
the
identity
or
communication
with
the
operator,
what
state
it
was
in
and
if
that's
something
how
many
people
would
like
the
project
to
write
a
really
great
Cassandra
operator,
one
two,
three
four,
five,
six,
seven,
eight
nine
10
ish
people
would
would
use
one
if
we
wrote
one.
Alright,
that's
good
feedback
Thanks!