►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Safely Upgrading Kubernetes Clusters - Chao Xu & Yu Liao, Google
You need to upgrade your Kubernetes clusters to pick up security patches and new features. Are you intimidated? What could go wrong? There are many pitfalls, including but not limited to: API deprecation, mismatching addon versions, deprecated control-plane extensions, incorrect upgrade order, and upgrades in HA setup, etc. The other side of the coin is also worth exploring, if an upgrade goes wrong, how to safely downgrade. This talk gives you a deep dive on the pitfalls, suggests rule-of-thumb, and reviews community efforts on safer upgrades. Attendees will walk away with more confidence in upgrading their Kubernetes clusters.
To learn more click here: https://sched.co/FuK8
B
A
A
So,
there's
an
oh
saying
that
the
only
certainties
in
life
are
death
and
taxes,
fortunately
for
kubernetes
cluster
operators.
There's
another
certainty
which
is
upgrades
like
every
other
software
upgrading
kubernetes
clusters
is
necessary
to
pick
up
the
latest
security
patch,
the
bug
fixes
or
to
use
any
new
features.
A
However,
it
is
very
hard
to
get
the
upgrade
correct
if
you're
operating
your
own
cluster
on
Prem,
you
don't
have
many
chances
to
practice.
Kubernetes
upgrades
because
after
all,
kubernetes
only
release
once
every
quarter
and
even
for
cloud
providers
like
Google,
although
we
do
how
we
do
upgrades
very
often
we
face
other
challenges.
For
example,
we
don't
control
the
workloads
running
on
the
nodes,
so
we
cannot
validate
kubernetes
the
new
version
of
kubernetes
with
every
client's
workloads
before
hand.
So
it's
very
difficult
for
us.
A
Okay
to
get
upgrades
100%
safe
for
everyone,
so
I
have
been
managing
g
k.
G:K
ease
upgrades
for
the
past
half
a
year,
I
think
it's
worth
sharing
my
experience
with
the
community.
I
hope
from
this
talk.
Cluster
admins
can
learn
what
are
the
current
best
practices
of
upgrades
and
some
some
known
issues
and
how
to
work
around
them
and
two
contributors
I
hope
you
can
learn
from
this
talk.
What
are
the?
A
A
I
plan
to
structure
the
talk
in
three
sections:
first,
I'll
talk
about
the
state
of
the
art.
What
are
the
current
diversity
practices?
Then
I'll
share
some
knowing
issues
and
how
to
work
around
them
and
at
last
I'll
talk
about
some
unforeseen
problems
and
how
to
tackle
with
them.
Let's
begin
with
the
first
section.
A
A
Yeah,
that's
the
that's
great,
in
my
opinion.
Backup
is
the
most
important
thing
to
do
as
the
first
step
of
upgrade.
You
can
achieve
this
easily
with
the
sed
control
command.
According
to
our
our
experience,
this
command
will
only
will
take
at
most
10
minutes
to
run
and
in
most
cases
it's
a
matter
of
seconds.
So
it's
a
very
cheap
insurance
to
buy
before
you
do
the
upgrade
that
that
way,
you
will
at
least
have
a
point
to
row
back
to.
If
your
upgrades
fails.
A
A
A
You're
right,
the
answer
is
no
all
right.
Let's,
let's
talk
offline,
if
you
think
it
is,
but
this
is
a
policy
decision
of
kubernetes
many
backward
compatibility.
Guarantees
of
kubernetes
only
supports
only
sub
only
guarantees
the
backward
compatibility
for
one
minor
version
difference.
So
if
you
want,
if
your
cluster
is
down
1.9-
and
you
want
to
upgrade
to
one
12
I
suggested
to
take
three
small
steps.
A
Instead
of
a
big
jump
and
in
later
in
the
talk,
I
will
actually
mention
some
of
the
issues
you
will
meet
if
you
jumped
minor
versions
and
last
but
not
least
before
you
do
that
wait.
Please
look
the
read
the
release
notes
carefully,
especially.
We
should
pay
attention
to
the
following
three
sections:
the
knowing
issues
the
actually
required
and
the
deprecations
and
removals.
They
are
very
important
because
if
there
is
any
backward
incompatibilities,
the
developers
will
highlight
them
in
these
three
sections.
A
Now,
let's
begin
our
upgrade,
there
are
many
tools
that
can
make
upgrades
easier.
All
these
popular
cluster
lifecycle
management
tools,
they
all
automated
the
upgrade
process
to
some
extent
like
the
cops,
the
cube,
ADM,
the
cube
spray
and
from
another
talk,
I
heard,
the
clutter.
Api
is
the
future.
It
will
be
the
unified
channel
to
orchestrate
upgrades.
A
A
A
The
notes
are
allowed
the
to
lag
behind
the
master
by
two
minor
versions,
but
not
the
other
way
around,
there's
no
guarantee,
if,
if
your
master
is
lagging
behind
it,
a
note
so
always
upgrade
a
note.
Sorry
always
upgrade
master
before
your
upgrade
note,
and
also
when
you're
actually
doing
the
master
upgrade.
It's
suggested
to
upgrade
to
your
cube,
API
server.
First,
because
your
clients
like
controller
managers,
they
might
depend
on
the
new
features
and
you
only
the
newer
version
of
cube.
A
Api
server
provide
those
features,
so
you
upgrade
your
cube,
API
server,
and
then
you
upgrade
to
your
clients,
like
the
controller
manager,
the
scheduler
and
add-ons
and
in
Cuba
ATMs
in
Cuba
diem
setup.
The
last
step
of
upgrading
master
is
setting
up
the
our
back
rules
for
the
node
bootstrap
tokens.
This
this
step
is
optional,
it's
required
only
if
you
are
using
the
feature,
but
this
is
a
very
good
example
of
how
small
features
miscellaneous
features
can
can
get
in
the
way
of
a
successful
upgrade.
A
Then
we
calculate
our
notes.
Usually
this
is
done
in
a
fashion
of
rolling
upgrade.
We
were
upgrade
one
node
at
a
time
and
to
do
that,
we
first
used
the
cube
control
joint
command,
which
will
remove
the
workloads
from
the
node,
and
it
will
mark
the
note
as
unschedulable,
to
prevent
any
further
assignments.
And
if
you
use
this
switch,
the
cube
control
Jane
was
the
port
disruption
budget
feature.
Then
you
can
minimize
that
disruptions
caused
the
two
your
services.
A
It's
worth
mentioning
that
the
in-place
cubelet
upgrade
is
not
supported
here
by
in
place,
I
mean
upgrading
your
cubelet,
while
they're
still
pods
or
containers
running
on
your
on
your
note.
It
is
not
supported
because
cubelet
has
States
and
it
will
recorded
the
states
on
a
node
and
there's
no
guarantee
that
the
newer
version
cubelet
will
treat
those
states
in
a
backward
compatible
way.
So,
for
example,
I
think
I've
seen
one
ten
cubelets
changed
where,
where
it
stores
the
logs
for
the
pots.
A
A
A
A
So
in
this
graph,
starting
from
the
left
top
corner,
a
happy
user
is
trying
to
create
a
batch
v1
job,
so
it
will
send.
He
will
send
the
content
to
the
battery
one
job
and
point
of
api
server.
Api
server
will
be
serialized,
the
content.
Do
the
validations
apply
the
defaults,
and
then
it
will
convert
the
object
to
the
storage
version,
which
is
extensions,
v1
beta
1,
and
then
it
will
store
the
the
version,
the
object
SCD.
A
So
the
content
in
SED
is
actually
version
versioned
and
it
will
be
extensions
v,
1
beta
1
by
default,
and
when
the
user
is
trying
to
read
the
data,
the
reverse
of
the
process
is
applied.
The
content
is
read
from
the
SED.
An
api
server
needs
to
deserialize
the
extensions
v1
beta
one
job
and
then
converted
to
whatever
version
the
user
user
is
requesting.
A
This
is
the
normal
process.
However,
if
you
create
a
job
today
and
you
forget
to-
and
you
didn't-
you
didn't
to
read
or
write
to
it
for
the
part
for
the
next
year,
then
this
job
will
stay
in
the
extensions
v1
beta
1.
You
know
in
your
STD,
however,
one
year
later,
extensions
v1
beta
one
might
get
removed
from
the
codebase.
A
Your
API
server
will
not
understand
extensions,
V,
1
beta
1
anymore.
So
at
that
time,
if
you
try
to
read
the
job
from
a
CD
again,
API
server
will
fail
to
read
it.
It
will
give
you
a
obscure
error
message
claiming
that
there's
no
job
registered
at
the
extensions
view
and
beta1
at
that
time.
You
cannot
read
the
data,
it
cannot
update
it.
You
cannot
even
delete
it
because
many
delete
options
in
Cuban
artists
will
require
read
from
sed
to
do
some
validation.
So
at
this
point
this
job,
yes,
is
completely
dead.
A
So
this
is
a
known
issue
in
the
upstream.
We
actually
have
taken
actions.
So
in
this,
in
this
issue
of
5
to
185,
we
have
declared
a
freeze
of
the
removal
of
API
versions
before
we
can
solve
this
this
problem,
so
the
message
to
the
contributors
is
that
when
you
are
doing
code
reviews,
we
have
to
reject
any
poor
request.
That's
trying
to
remove
a
deprecated
API
version.
They
need
to
stay
there
for
now
as
to
the
root
cause
of
the
issue,
we
are
working
on
a
proper
fix,
which
is
the
storage
migration
system.
A
A
Also
the
message
to
the
cluster
admins
is
that
you
don't
need
to
over
concern
with
this
issue
because
we
have
free.
We
have
freeze
the
removal
of
API
versions
since
1/6.
So
if
your
cluster
is
newer
than
1/6,
then
you
are
safe
from
this
issue
and
also
please
in
the
future.
Please
read
the
release
notes
because
once
the
storage
microd
migration
system
is
ready,
we'll
start
removing
API
versions
again
and
if
there's
any
action
required
on
your
site,
we
were
highlighted
in
the
release,
notes.
A
So,
if
you
have
any
custom,
automation
depend
depending
on
extensions,
v1,
beta1
workload,
the
api's
and
you
upgrade
your
cluster
to
115.
You
will
suddenly
see
all
those
automation
stops
working.
That
is
because
the
extensions
v1
best
beta
one
endpoints
are
scheduled
to
be
removed
from
API
server.
So
your
client
cannot
visit
these
endpoints
anymore.
A
This
is
a
chance
to
work
on
and
a
third
issue.
I
want
to
share
that's
after
the
upgrade
it
might
break
your
policy
dissing
your
policies,
so
our
use
admission
web
hooks
as
an
example
for
those
of
you
who
are
not
familiar.
What
admission
web
hooks
are.
They
are
just
an
extension
points
to
API
server.
You
can
instruct
the
API
server
to
fold
the
requests
of
any
specific
resources
to
your
web
book
and
in
your
web
book
you
can
do
some
custom
validation
to
enforce
policies,
so
in
this
example,
set
up
the
web
hook.
A
It
is
instructing
the
API
server
to
Ford
any
request
to
batch.
We
want
jobs
forwarded
the
requests
to
the
your
web
hook
and
the
web
hook
can
restrict
can
restrict
what
images
the
jobs
are
using
so
far
so
good.
However,
once
you
upgrade
your
cluster,
there
might
be
a
new
version
of
the
job
like
batch,
v2
or
even
trickier.
The
jobs
might
be
available
through
different
API
group.
Like
awesome
batch
v1.
A
So,
to
avoid
this
kind
of
issues
again
for
cluster
admins,
we
need
to
read
the
release,
notes
and
see.
What
are
the
new
API
is
that's
becoming
available
and
we
need
to
upgrade
our
web
hooks
and
the
web
hook
configurations
to
solve
such
issues
to
include
the
new
versions
to
prevent
any
security,
holes
and
I.
Think
this
issue
is
not
just
it's
not
limited
to
what
box
any
rules
by
rule-based
the
validation
in
the
system
suffer
from
the
same
problem.
A
A
So
in
this
example
setup
there
are
three
masters
forming
a
each
a
cluster.
They
can
all
independently
serve
their
clients,
and
here
is
the
real
issue
that
happened.
Maybe
one
year
ago
after
we
do,
when
we
do
the
rolling
upgrade
and
one
of
the
master
has
finished
the
operate,
while
the
other
two
are
still
using
the
old
of
the
coatis.
A
So
in
the
new
version
of
kubernetes,
the
port
has
a
new
fielding,
inspect
called
the
Toleration
x'
and
there
are
clients
actively
writing
to
this
field
to
a
non
default
value
harbor
because
there
are
still
two
masters
of
the
old
version
and
clients
are
reading
and
writing
through
those
old
masters.
The
and
those
old
masters
doesn't
even
know
about
the
existence
of
the
old
field.
So
once
the
right
happens,
occur
happens
in
the
old
master,
the
field
will
be
lost
and
this
will
lead
this
work.
A
So
so
this
is
a
this
is
an
issue,
but
you
don't
need
to
over
concern
with
it,
because
we
have
been
living
with
this
long-standing
issue
for
a
while,
and
we
haven't
seen
any
production,
serious
production
issue
with
it,
but
to
fail
on
the
safe
side.
Plus.
That
means
you
should
we
should.
It
will
be
safe,
it
will
be
safer
if
we
can,
if
we
restricted
the
use
of
the
new
AP
is
before
the
rolling
upgrade
has
completed,
and
two
contributors
I
think
this
the
root
cause
of
this
issue.
A
A
A
But
what
are
you
expecting
here?
It's
called
Oren
foreseeable
issues.
I
cannot
foresee
them
right,
that's
by
definition,
uncertainty.
Uncertainties
are
just
part
of
our
life.
We
have
to
live
with
it.
Ji-Hye
Union
comes
on
being
hot
right,
but
there's
still
some
measures
we
can
take
to
make
upgrade
safer,
so
I
think
the
first
thing
we
need
to
do
more
is
testing.
This
is
a
I
think
this
is
a
common-sense
but
I.
Think
it's
worth
emphasizing
here
for
kubernetes
upgrades.
The
open
source
testing
coverage
is
actually
very
limited.
A
We
only
tested
some
api's
in
some
very
specific
configurations
and
in
production
we
did
see
some
problems.
That
will
happen
in
very
specific
conditions.
So,
for
example,
we
have
seen
issues
that
will
only
occur
if
we
are
upgrading
from
a
very
specific
patch
version
to
the
new
version,
but
not
from
other
patch
versions.
We
have
also
seen
bugs
that
will
only
review
if
we
have
if
the
user
has
set
up
an
API
objects
with
a
very
specific
value.
So
all
these
issues
will
not
be
catched
will
not
be
caught
by
the
stand.
A
A
So
if
you
can
afford
the
resource,
it's
suggested
to
always
have
a
staging
environment,
stand
by
and
tested
your
upgrades
with
the
real
workloads,
and
then
you
can
proceed
well
with
the
production
environment,
and
even
if
we
do
all
the
preparation
up
rates,
upgrades
can
still
go
wrong
and
at
that
time
we
will
really
hope.
There's
a
boat
button
there,
so
we
can
press
it
and
undo
the
upgrade
for
kubernetes
such
an
abort
button
will
be
a
safe
done
great,
but
unfortunately,
downgrade
is
not
generally
safe.
A
A
First
issue
is
that
errands
might
crash
after
you
do
the
downgrade,
because
today
add-ons
might
use
the
latest
features.
So
if
you
downgrade
your
API
server
your
control
plane,
those
add-ons
will
not
work
anymore
and
if
the
atom
is
critical
enough,
they
will
do
a
bring
your
cluster
down
with
it.
So
I
think
for
contributors.
We
should
ban
this
kind
of
operation.
When
we
do
code
reviews
for
add-ons,
we
need
to
ban
them
from
using
the
latest
features.
This
is
not
safe
and
a
second
issue
I
think
it's
obvious.
A
If
user
have
already
created,
have
you
have
already
used
the
new
API
S
or
the
new
API
fields?
They
have
already
created
objects,
then,
once
you
done
great,
the
API
world
will
not
be
there
anymore,
but
the
data
was
still
lurking
the
sed
and
they
are
a
time
bomb
that
will
detonate
once
you
upgrade
again
so
for
this
kind
of
issue,
I
have
a
brainstorm
idea.
I
think
it's
worth
exploring.
We
can
do
Multi
multi
phase
upgrade.
A
For
example,
we
can
upgrade
the
clusters
to
use
the
new
kubernetes
binaries,
but
we
can
disable
the
new
API
at
first
and
only
after
we
are
satisfied
with
the
quality
of
the
new
binaries.
Then
we
were
enabled
a
new
API.
This
should
eliminate
should
reduce
the
chance
of
this
kind
of
data
loss.
Yes,
you
have
from
happening
as
I
said,
this
is
just
an
incomplete
list
and
downgrade
is
sung
on
the
top
of
my
wish
list
for
2019.
A
B
I
feel
comfortable
to
use
English
so
a
little
background
about
our
setup
as
as
we
use
criminalities
EWS
and
we
start
use
kubernetes
1.8
hosted
and
for
tab
ec2
instances
and
we
were
trying
to
upgrade
the
community's
version
from
one
point:
8
2,
2,
1,
1
10
by
manner
version.
But
we
the
reason
we
won't
do
that
is.
We
will
use
an
v
type
on
AWS
which
give
us
more
benefits
but
like
when
we're
trying
to
do
that
in
one
shot.
I
failed
and
we
don't
know
why.
It's
like
some
weird
message.
A
So
yes,
the
is
the
error
happening
when
you
upgrade
from
180
to
one
I
know
one
ninety
110,
but
I
think
this
underlying
issues
needs
to
be
analyzed
a
case
by
case
a
Google.
We
also
do
upgrades.
We
also
operated
the
backend,
the
resources
and
the
communities,
but
we
usually
do
them
at
a
different
pace
and
we
have.
We
will
have
a
staging
cluster
that
will
test
all
those
issues
together.
A
So
the
reason
we
do
them
at
different
paces
because
we
want
to
separate
causes
right,
I,
don't
know
if
you
in
your
case,
you
can
upgrade
to
the
backend
the
resource
first
and
test
it
and
then
upgrade
the
kubernetes.
Then
you
can
you
can
at
least
identify
which
one
causes
the
problem
or
if
it's
all,
your
fields
suggest
a
compatibility
issue,
but
in
general
I
think
this
kind
of
issues
is
difficult
and
we
need
to
analyze
some
case-by-case.
A
Let
me
try
to
repeat
a
question:
is
it?
Is
there
any
benefit
of
doing
in
place,
cubelet
upgrade,
then
ok,
so
I
think
one
is
one.
Is
the
resource
concern
right?
Maybe
not
not!
Every
cluster
is
flexible.
You
can.
Maybe
you
cannot.
When
you
do
the
cube
control
Jane,
you
do
need
to
find
a
new
home
for
all
the
pods
right.
So
if
you
don't
have
extra
resource
when
you
once
you
join
the
pods,
it
might
take
a
longer
time
to
schedule
them
and
also
I.
A
Think
if
you
don't
schedule
them
carefully,
there
will
be
actually
cost
the
cube.
Control
Jane
might
cause
all
the
poles
to
be
moved
in
multiple
times
and
that
will
cause
disruption
to
services.
So
if
we
can
do
increase
upgrade
that
will
be
the
best
right.
We
can't
the
worker
there
will
be
no
downtime
to
the
no
disruption
to
the
services.
D
You
suggested
always
testing
so
setting
up
some
extra
resources
to
validate
that
your
real
workloads
can
work
before
moving
your
cluster
at
that
point.
What's
the
technical
rationale
for
supporting
downgrade
in
my
engineering
experience
downgrade
is
extremely
difficult
to
support
and
it
feels
like
that
takes
resources
from
stabilizing
the
actual
upgrade
path.
A
A
Because
we
haven't
done
any
dunk
rating
production,
so
it's
hard
to
say,
but
in
general
I
think
engineers
are
feared
about
doing
downgrade,
because
we
we
don't
in
the
in
the
I,
think
in
the
kubernetes
open
source.
We
do
have
some
downgrade
tests,
but
they
are
like
the
upgrade
tests
that
they
only
tested
very
certain
cases,
I
think
to
find
the
order
done
great
blockers.
We
will
need
the
most
thorough
test
first
and
then
we
can
identify
the
problems.
I
think
that
will
be
the
first
step.
D
It
was
partly
a
question
of
why
even
choose
to
support
downgrade
in
the
community.
So
from
this
sig
release
perspective,
we've
very
often
had
the
downgrade
test
failing
and
it's
been
hard
to
find
somebody
to
work
on
fixing
them,
and
everybody
says
we
don't
care
about
downgrade.
Only
Google
cares
about
downgrade,
but
then
you
say
that
you
haven't
really
so
I
wonder
if
we're
spending
engineering
resources
trying
to
support
something
needlessly.
A
I
think
dunk
rate
is
just
a
safety
net.
It
will
give
all
the
customers
and
all
the
users
more
confidence
about
upgrade
right.
Hopefully
we
will
never
need
to
use
it
if
we
have
storoe
testing
of
the
upgrades
and
suppose
all
the
production
workload
production
environment
is
exactly
the
same
as
our
stage
in
the
environment.
Then
I
agree,
we
don't.
We
will
never
need
to
dunk
wait,
but
usually
the
reality
is
is
far
from
the
our
imagination
from
the
ideal
world
right,
so
I
think
it's
still
valuable
too.
A
E
Hi
you
mentioned,
you
talked
about
backing
up
your
ad
CD
cluster
and
I
was
wondering
if
you
can
briefly
talk
about
like
in
case
of
the
upgrade
failed
when
you
restore
your
ed
CD
cluster.
Like
a
brief
like
how
is
the
process
done
like?
Do
you
actually
perform
the
restore
on
the
same
master
node
that
that
you
fail
that
or
to
create
a
new
node
and
restore
the
data
to
the
new
new
master?
No
or
I
I'm,
not
yeah,.
F
Hi
I'm
Jo
Betts
I
work
on
that
CD,
so
this
is
actually
a
path
that
we
want
to
improve
over
time
right
now,
it's
a
little
bit
risky
to
just
restore
the
sed
state
under
kubernetes.
If
you
haven't
performed
any
scheduling,
tasks
and
the
cluster
is
very
stable,
he
didn't
theory
may
work,
but
I
would
definitely
consider
this
a
bit
of
an
unsolved
problem.
F
So
I
expect
to
see
work
over
this
in
time,
but
right
now
it
might
be
better
to
go.
Get
the
objects
out
of
sed
that
you
need,
if
you
run
into
a
problem,
but
the
best.
The
best
bet,
if
you're
going
to
do
this,
is
to
try
and
mostly
stop
operating
on
your
cluster
until
you're
done
with
your
upgrades,
so
that
if
you
have
to
restore
from
the
backup
that
deltas
as
minimal
as
possible,.