►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Storage Version Migrator: Never Worry About Stale API Objects Again - Chao Xu, Google
Have you ever had zombie Kubernetes API objects that your API server refuses to get, update or even delete? This is probably because the objects persisted in etcd were encoded in an obsolete version. This talk presents the Kubernetes storage version migrator, which solves this problem once and for all. After you enable this alpha feature, the migrator automatically makes sure that all your API objects stored in etcd are encoded in the proper version at all times. In the talk, Chao, a main contributor behind the design and the implementation of the storage version migrator, will share how the migrator manages not only the Kubernetes resources like Pods, but also your custom resources. You will also learn the caveats when using the migrator, e.g., if you are managing an HA cluster. Chao will also share the road map to graduate the storage migrator.
https://sched.co/NroQ
A
I'm,
a
software
engineer
and
I've
been
working
on
kubernetes
for
open
for
years.
This
is
my
second
Q
control
again.
Thank
you
for
coming
to
my
talk,
I'm
going
to
talk
about
storage
migrator
today
before
we
go
into
today's
agenda.
I
want
to
first
share
with
you
a
story
of
mine.
It
was
the
first
year
when
I
joined,
Google
I
was
freshly
from
graduated
from
graduate
school
and
it
was
my
first
her
uncle
shift
and
it's
late.
In
the
night
my
pager
suddenly
ran
customers.
Cluster
was
crushed.
B
A
I
opened
my
laptop
I
logged
into
my
account
logged
into
the
API
service
log
I,
easily
fund
culprit
in
the
log.
The
culprit
to
the
error
message
is:
there's
no
job.
Api,
there's
no
job
kind
registered
in
extensions,
B,
1,
beta,
1,
I,
agree.
I.
Think
this.
This
error
message
is
a
little
baked,
but
it
gives
me
some
clue
so
I
looked
into
the
code,
a
little
more
I
found
out
that
Jobs
is
not
available
in
this
API
version.
Anymore.
A
Gulps
has
graduated,
so
I
think
my
next
step
is
either
try
to
fix
the
objects
or
try
to
delete
the
object.
I
got
into
contact
with
the
customer.
The
customer
told
me
those
jobs
are
just
deprecated
objects,
I
have
their
permission.
We
they
delete
all
those
jobs.
I
was
very
happy.
I
thought,
that's
it
all
I
need
to
do
is
just
run
a
cue
card
elite
command
to
delete
all
those
doors.
B
A
Are
the
culprits
of
much
looping
API
servers,
but
when
I
try
to
do
the
cube,
color
developed
I
found
out
that
the
API
server
cannot
handle
the
delete
operation
either.
The
reason
is,
the
jobs
is
not,
but
HSL
just
doesn't
to
recognize
the
jobs
because
they
are
in
and
stale
version.
The
API
server
cannot
interpret
the
object
anymore,
so
I
cannot
do
any
operation
on
them,
but
my
last
results
I
really
want
to
use.
It
had
to
directly
talk
to
SED.
A
This
really
wasn't
to
the
situation.
I
want
to
be
in
right.
It's
my
first
year
at
Google
I,
don't
want
to
yo
SSE
DFP
API,
because
I've
never
touched
that
those
API,
oh
I,
know
it
is
kubernetes
api.
Also,
I
don't
want
to
accidentally
delete
a
customer's
other
data,
so
I
was
really
nervous.
But
what
can
I
do
that's
my
only
option.
Api
server
does
not
cannot
do
anything
at
this
point.
A
I
had
to
spend
like
half
an
hour,
search
as
the
sed
control
the
sed
control
their
binder
document
and
try
to
assemble
the
command
to
delete
those
stale
jobs
and
I
had
to
experiment
it
on
my
local
environment
to
make
sure
those
command
actually
works
after
half
an
hour,
it
I
got
the
job
done.
I
am
smart,
but
that's
really,
that's
really
not
a
situation.
I
want
to
be,
and
I
don't
want
any
of
you
to
be
in
this
kind
of
situation
either.
A
A
This
is
necessary
for
me
to
talk
about
storage,
migrated
and
I
just
like
to
talk
about
API
machinery,
stuff,
it's
fun,
then,
after
the
background
I'll
I'll
talk
about
how
come
we
will
ever
have
stale
objects
in
a
CD
and
how
come
I
went
I.
How
come
that
bug
happened
to
me
at
last.
I'll
talk
about
a
solution
that
storage
migrated.
A
So
let's
begin
was
the
background:
how
kubernetes
those
objects
in
a
CD.
This
will
be
a
little
long.
Actually
did
so.
If
you
go
to,
if
you
have
an
running
API
server,
and
if
you
go
to
the
root
path
of
the
API
server,
it
will
return
you
the
discovery
document
which
will
tell
you
all
the
API
versions,
all
the
API
is
are
supported
and,
as
you
can
notice
for
a
specific
API
groups
like
batch,
usually
there
will
be
multiple
API
versions
available
and
those
API
versions
adjusted
are
the
evolution
paths
of
the
API.
A
A
A
A
So
what
I
mean
is
if
you
have,
if
you
bid
it
to
all
these
three
different
URL
paths,
although
they
have
different
versions
in
the
URL,
but
because
their
name,
space
or
the
namespace
here
is
default
and
all
the
object
name
here
is
PI.
They
were
actually
they're
actually
referring
to
the
same
object
in
the
backend,
which
is
the
job
in
the
name,
space
support
and
in
there,
with
the
name
pipe.
So
those
versions
are
just
different.
You
can,
you
can
think
the
thing
tomorrow.
Think
of
them
as
a
elías.
A
A
So
now,
let's
look
at
these
api
path:
dynamically.
Let's
see
how
how
things
work
when
you're,
actually
when
the
api
server,
is
actually
taking
and
request.
So,
first,
let's
look
at
when
the
user
is
creating
an
app
object,
so
the
user
might
choose
to
post
to
their
object,
to
the
be
1
beta,
1
jobs,
URL
and
what
happens
is
the
the
user?
The
user
will
well
just
to
provide
a
a
job
object
at
this
URL
path,
and
this
object
in
the
request
must
be
encoded
in
the
b
1
beta
1
version.
A
A
However,
after
the
master
had
after
the
API,
server
has
processed
the
job
and
they
want
to
pay
for
it,
stores
it
into
the
backend.
The
sed.
The
API
server
will
actually
convert
to
the
object
from
B
1
beta
1
to
the
storage
version,
which
is
actually
the
badge
be
one
storage
version
the
little
committed
to
the
storage
version.
It
will
change
the
encoding
and
then
it
was
stow
it
into
the
back
end.
The
sed.
So
here
the
storage
version
is,
is
an
important
concept
we
will
use.
A
We
will
mention
a
few
times
later
in
the
talk
as
well.
The
storage
version-
it's
it's
so
literally,
it
means
literally
it's
the
version.
The
API
server
will
convert
to
the
object
two
before
the
API
server
can
stow
it
into
a
CD
and
for
butene
resources,
a
storage
version.
It's
already
it's
pre-configured,
it's
it's
coded.
It's
hard-coded,
that's
tighter
to
the
API
servers
version
the
for
example.
If
you
use
119,
where
we
are
we
now,
we
add
one
115
right.
A
A
Ok,
now,
let's
go
back
to
the
main
topic.
So
previously
we
talked
about
how
a
create
heart,
what
happens
when
you
create
an
object?
Now,
let's
look
at
what
happens
when
you're
reading
an
object,
so
I
use
the
user
can
choose
to
get
to
the
object
from
a
different
URL
path
at
a
different
version.
So
if
you
remember,
we
created
it
via
the
v1
beta
one
pass,
and
now
the
user
tries
to
get
it
from
the
V
to
alpha
one
pass.
A
Now
what
happens
is
the
API
server
will
read
the
data
from
the
SED
which
has
encoded
as
v1,
and
it
were
converted
to
the
version
that
user
requested,
so
it
took
here
that
the
read
packet
or
caste
is
hitting
HB
2
alpha
1,
the
API
server
will
convert
to
the
object
to
be
2,
alpha
1
and
then
give
it
to
the
user.
The
API
server
here
is
like
a
version
hub
of
all
the
versions
as
well
to
order
translation
for
users.
A
That's
because
we
want
to
make
sure
this,
the
data
for
the
server
and
the
client,
a
compatibility.
That's
for
that.
That's
the
best
to
enable
different
to
lifecycle
management
for
uphold
a
lifecycle
of
server
from
the
lifecycle
of
client.
Let's
look
at
an
example,
so,
let's
suppose
the
API
server
only
supports
one
version
at
a
time
and
the
CPI
server
is
active
version.
A
One
point
X
and
you
have
a
client
and
you
have
a
client
which
is
talking
to
the
batchview
and
beta
one
endpoint,
and
once
you
upgrade
your
IPS
over
to
the
next
two
version
and
suppose
that
only
supports
one
version
and
the
batch
API
has
graduated
from
you
and
beta
1
to
V
1.
Then,
suddenly,
your
clients
will
stop
to
working.
It
gets
all
the
fall
for
responses
right,
because
the
V
1
beta
1
isn't
available
anymore.
A
So
that's
why
we,
but,
but
but
yes,
but
as
things
work
today,
the
API
server
will
support
multiple
versions.
At
the
same
time,
then
you
can
upgrade
your
API
server
first
and
your
clients
will
still
keep
working
so
and
you
can
upgrade
your
client
afterwards
wait
before
you
do
next
upgrade
of
API.
So
that's
why
the
API
server
has
to
support
the
multiple
version.
It's
for
the
compatibility
between
the
server
and
their
clients,
a
second
Oh
y-yes.
Why
does
the
API
server
have
to
convert
objects
to
the
storage
version?
A
A
On
the
other
hand,
if
the,
if
things
are
stored
in
the
storage
version,
which
usually
is
the
the
most
recent
version
available
most
of
stable
version,
most
a
recent
stable
version
available,
then
even
if
even
if
the
previous,
the
B
1
beta
1
version-
is
deleted
because
the
objects
store
the
MCD
is
in
the
storage
version,
then
this
object
is
still
interpretable.
Of
course,
this
for
things
to
work.
This
relies
on
the
kubernetes
upstream
maintainers
to
pick
the
storage
version
carefully,
so
they
need
to
pick
the
most
stable
version.
A
A
So
previously
is
the
background
knowledge,
and
now,
let's
look
at
why
how
I
ended
up
in
that
story?
Why
would
there
ever
be
stale
objects
in
customers
clusters,
but
you
might
have
already
noticed,
there's
a
loophole
in
all
the
designs
I've
mentioned
before.
That
is,
if
you,
if,
if
as
a
user,
if
you
create
an
object
at
one
at
some
point,
let's
say
you
created
when
your
API
server
is
at
V
115
and
your
everything
is
working
as
designed.
A
The
object
is
converted
to
the
storage
version,
which
is
patch
v1,
and
that
is
storing
a
CD.
However,
suppose
you
let's
say
you
forgot
about
this
job,
you
never
you
never.
You
never
touched
the
job
anymore,
and
one
year
later,
your
API
server
has
been
upgraded
there
multiple
times
to
that's
a
$1.99.
It's
a
it's
an
imaginary
version
at
this
time,
the
bad
shot,
but
the
batch
API
can
already
be
at
v8.
No
one
understand
the
API
server
doesn't
understand
the
b1
anymore.
A
A
The
right,
yet
you
never
updated
the
thing
that
all
the
job-
yes,
they
just
become
stale,
and
at
this
point
you
never.
This
thing
is
not
interpretable
anymore.
This
is
actually
what
happened,
backing
the
story,
the
users,
the
clients
they
created,
those
jobs,
I.
Think
in
one
one
three.
So
at
that
at
that
time,
the
job,
the
storage
version
of
jobs,
extensions
beyond
beta-1
and
because
they
don't
they
actually
don't
need
those
jobs.
They
are
deprecated,
they
never.
A
A
A
Want
to
first
highlight
the
features
of
the
storage
migrated
from
the
users
perspective,
so
it's
some
very
nice
feature
of
the
storage
migrator.
First,
it's
deployable
by
the
cue
cuddle.
All
you
need
to
do
is
to
download
some
demo
files
and
deploy
it.
You
know
support
the
the
API
that
migrator
uses.
Yes,
kubernetes
style,
it's
very
its
component
to
that
generic
kakugane
are
these
api,
so
it's
the
learning
curve
will
be
very
low.
Also,
this
thing
is
fully
automated.
Once
you
deploy
it,
you
don't
need
to
worry
about.
Just
they
object
anymore.
A
Migrator
will
do
the
job
for
you
also
it's
resilient
to
failures
like
if
your
API
server
crashes,
if
the
migrator
self
crashes,
it
will
recover
from
the
fail
as
long
once
it
recovers
from
the
failure.
It
will
continue
the
migration
job
as
when
it
when
it's
as
before.
Also
it's
vendor
specific,
so
you
can
deploy
or
changing
any
cluster
and
environment.
You
like
let's
look
into
some
details,
so
this
is
how
you
can
deploy
the
storage
migrator
in
your
cluster.
A
And
let's
look
at
the
migration
API,
so
this
is
the.
This
is
a
simple
migration,
so
I
highlighted
the
important
of
field.
So,
let's
first
look
at
this
back
odorous
resource,
so
here
says
batch
jobs
v1.
What
it
means
is,
this
is
migration
request
for
batch
objects,
so
it
requests
the
migrator
to
migrate
all
the
jobs
in
your
sed
to
make
sure
they
are
in
the
latest,
a
storage
version
and
the
name
here.
It's
just
a
combination
of
the
resource
name
and
the
group
name.
A
The
next
important
bit
is
the
conditions,
so
this
is
also
a
regular
kubernetes
api.
It
says
so
here
the
condition
is
it's
running.
I
means
the
migration
is
still
ongoing
and
because
this
migration,
thanks
to
this
migration,
API
to
be
fully
kubernetes
component,
you
can
use
some
very
generic
Ricardo
command
to
talk
through
the
API,
the,
for
example,
why
you
want
to
upgrade
or
downgrade
your
API
server?
You
need
to
make
sure
all
the
migrations
have
completed,
and
you
can
do
this
with
a
one-liner.
A
A
So
basically,
the
migrator
is
to
loosely
cupholder
controllers
on
the
left
side
of
the
graph.
Is
the
migration
trigger
controller?
It's
just
that
it
will
just
affect
the
discovery
document
from
the
API
server
continuously,
actually
every
periodically
and
it
was
attacked
when
it
was
attacked
when
the
storage
version
has
changed.
So,
for
example,
if
your
API
server
has
upgraded
to
the
next
version,
the
storage
version
might
change.
A
What
hit
what
just
to
watch
us
for
all
the
migration
requests
so
once
it
attacked
once
the
trigger
controller
posted
the
migration
law
requests
the
migrator
controller,
it
will
actually
migrated
a
resource
for
you.
So,
for
example,
if
the
storage
version
of
the
job
has
changed
after
your
server
upgrade,
the
migration
trigger
controller
will
post
a
great
oops
and
the
migrator
controller.
It
notice
the
new
migration
request
and
actually
migrates
all
the
jobs
for
you
not
zooming
to
how
the
migrator
controller
actually
migrated
a
resource.
A
So
here
the
migrator
controller
is
on
the
left,
have
a
PR
survey
and
the
CDR
and
sed
on
the
right
so
to
actually
migrate
a
resource.
The
migrated
controller
will
first,
it
will
use
the
chunking
list
API
to
list
the
first
500
objects
of
the
resource.
So
in
our
example,
it
will
just
list
the
550
500
jobs,
and
then
it
will
you
get
the
name
or
the
name,
the
list
of
of
the
names
of
the
jobs.
A
Then
it
will
get
the
first
job
from
the
API
server
and
without
making
any
change
without
to
do
any
conversion
that
you
were
just
to
write
to
the
job
back
in
today.
Api
server,
it
actually
the
so
the
migrated
control
itself
is
actually
done.
It
doesn't
know
all
those
API
versions.
Oh
it
does
it's
just
to
read
the
object
and
then
write
it
back
to
the
API
server
and
the
API
server
once
set
to
receive
the
object.
Its
were
actually
converted
to
the
storage
version
and
the
writer
to
their
CD.
A
And
if
the
object
is
already
in
the
storage
version,
then
the
API
server
will
not
will
not
do
anything.
It
will
just
do
it
will
just
the
return
present
nowak.
So
this
is
the
actual
migration
how
the
migration
has
done.
It
actually
relies
on
the
API
server
to
do
the
conversion
and
the
migration
migrator
controller
will
do
this
thing
500
times,
and
here
it
finished
one
chunk
after
one
chunk
of
the
object
is
done.
It
will
record
the
progress
in
the
migration
goes
back,
that's
where
the
continue
to
continual
token
comes
from.
A
A
A
A
Currently
it's
still
alpha,
but
it
has
met
all
the
beta
requirements
we
listed
in
their
cap.
So
we
plan
to
graduate
it
to
beta
in
this
month
and
then
we
will
we
plan
to
make
it
GA
in
q3
of
1019,
and
so
at
this
point,
I
want
your
support.
I
hope
you
can
try
the
migrator
in
your
own
cluster
and
report
any
issues
you
found
so
that
we
can
make
it
to
stay.
B
A
The
interesting
thing
is
I,
don't
care
about
those
newly
created
their
jobs,
but
that's
because
after
you
start
your
migration
and
if
someone
comes
to
create
a
new
job,
those
new
jobs
are
there.
They
must
be
extorting
the
latest,
a
storage
work,
because
it's
the
latest
API
server,
handling
the
request.
But
all
we
actually
need
to
migrate
is
the
jobs
that's
created
in
was
the
previous
API
server,
so
I
don't
actually
care
about
those
things
and.
C
A
B
A
But
this
continua
token
it's
not
based
on
the
index
of
the
things.
Alright,
so
you're
saying
we
already
listed
500
things.
We
should
start
from
the
500
501
thing
and
because
it
deleted
something,
the
501
thing
has
actually
become
the
400
MB,
but
that's
not
how
the
chunking
API
works.
So
all
the
chunk,
the
chunking
API,
is
actually
based
on
the
the
name.
A
The
name
of
the
object
so
I
say
so
when
they
say
I
listed
the
first
500
thing
and
the
last
job
name
is
say
a
so
then
it
will
just
when
you,
when
you
send
the
continue
token
to
the
API
servers
or
just
to
continue
from
the
next
thing
in
the
alphabetical
order.
It's
not
so
based
on
the
index
of
the
things.
It's
not
so
the
remember.
A
A
A
D
A
D
Some
backup
and
restore
scenarios
where
you've
there
are
things
like
the
Bolero
project
that
backup
kubernetes
not
threw
in
a
full
at
CD
backup
but
through
API
calls
and
then
have
a
restore
process.
Now
it
could
be
that
since
the
recovery
would
be
writing,
they
they
trigger
an
auto
update
anyway.
But
I
was
just
wondering
about
politics.