►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes and Checkpoint Restore - Adrian Reber, Red Hat
Over 6 years ago a ticket (#3949) was opened asking for Pod migration in Kubernetes and until now there is no support in Kubernetes to migrate a container. Container migration is based on checkpointing and restoring containers and checkpointing and restoring containers is one the main reasons Checkpoint/Restore in User-Space (CRIU) exists. Although container migration is always viewed as an outlier or corner case of containers, because containers are supposed to be stateless, CRIU continues to get better at container migration and even if containers are supposed to be stateless, CRIU still sees growing interest in its container migration features and especially the integration in container runtimes. This talk wants to present the multiple use cases for checkpointing and restoring containers. The talk wants to give a technical background how CRIU is enabling container runtimes to checkpoint and restore containers and the plan how to integrate checkpoint and restore into Kubernetes.
A
My
name
is
adrian
rebo.
I
work
for
red
hat
since
2015
and
I'm
involved
in
process
migration,
which
is
the
basis
or
which
is
the
result
of
checkpoint
resource
restore
for
at
least
10
11
years
now,
I'm
involved
in
in
creo
in
some
way,
since
2012
crew
is
the
basis
as
the
tool
we
use
to
checkpoint
and
restore
our
processes
and
container.
Here
today
and
I'm
focusing
on
on
container
migration
around,
I
would
say,
2015
and
the
agenda
for
today
is.
I
first
want
to
give
an
introduction
about
a
few
definitions.
A
A
So,
first,
the
definition
of
container
life
migration,
because
the
work
to
get
checkpoint,
restore
working
with
container
is
always
also
the
possibility
to
migrate
containers
from
one
system
to
another.
One
and
and
and
basically
the
idea
behind
container
live.
Migration
is
transfer
a
running
container
from
one
system
to
another
system.
You
could
also
call
it
maybe
stateful
migration,
because
the
state
of
the
application
in
the
container
is
not
lost,
but
continues
to
be
there
and
a
very
on
a
from
a
very
high
level
view
container.
A
As
mentioned,
this
is
all
everything
I'm
talking
about
today
is
based
on
creo
checkpoint
in
restoring
user
space.
The
reason
for
the
name
is
at
the
time
creo
was
first
developed.
There
were
different
approaches
to
checkpointing
and
restoring
in
linux.
There
were
kernel-only
approaches,
there
were
mixed
kernel
and
user-based
approaches.
There
were
approaches
where
you
intercepted
system
calls
and
creo
took
another
approach
and
completely
did
it
from
user
space
using
existing
interfaces
to
collect
as
many
information
about
the
running
process
as
possible.
A
There
are
multiple
integrations
of
preview
in
container
engines
container
runtimes
and
want
to
present
some
of
those
here.
So
the
one
I
have
to
present
first
is
the
openvset
integration
because
they
invented
creo
and
they
had
some
mechanism
to
migrate
containers
before
crew,
but
I
think
they
were
not
able
to
to
upstream
it
to
the
linux
kernel,
so
they
worked
on
something
which
can
be
upstreamed
and
which
everyone
can
use,
and
so
they
invented
creo
and
they
are
now
using
it
to
migrate
containers
and
using
creo
sun.
Then
there's
borg
google's
container
engine.
A
It
also
uses
3u
to
migrate
container,
especially
for
long
running
jobs
which
take
a
couple
of
hours.
If
one
of
the
container
nodes
is
out
of
resources
or
is
soon
gonna
be
out
of
resources,
then
borg
can
move
the
container
from
one
system
to
another
without
stopping
it
and
without
losing
a
couple
hours
of
work
which
might
have
already
went
into
the
currently
running
job.
A
A
I
have
also
brought
to
to
cryo
and
I'm
working
on
this
now
for
about
for
about
a
year,
but
the
problem
is
I:
can
the
pull
request
cannot
be
merged
in
cryo
because
it
depends
on
changes
to
the
cri
api
and
this
requires
changes
to
kubernetes.
So
it's
it's
it's
a
difficult
situation,
because
you
cannot
implement
one
without
the
other,
and
so
you
have
to
start
at
some
point
and
then
you
can
implement
the
other
thing,
but
the
one
thing
doesn't
work
with
it.
A
Interestingly
enough,
there's
a
issue
of
an
issue
about
migration
open
in
in
kubernetes,
since
2015
there's
also
a
lot
of
discussion
going
on
there,
but
until
now
nothing
has
been
merged
into
kubernetes,
to
implement
migration
or
to
at
least
get
a
step
closer
to
to
implementing
migration
in
kubernetes.
So
now
about
a
couple
of
use
cases,
the
first
use
case
is
reboot
and
safe
state.
A
I
try
to
visualize
it
here,
so
we
have
a
container
running
on
a
host
and
the
host
is
memory
is
blue
and
the
container
memory
is
red.
So
to
to
see
the
differences
here,
so
we
we
want
to
reboot
a
host
because
we
have
to
upgrade
the
kernel,
but
we
do
not
want
to
lose
the
state
of
the
container
because
it
takes
forever
to
start.
So
what
do
we
do
with
checkpoint
and
restore?
We
checkpoint
a
container
write
it
to
the
local
disk.
A
Then
we
reboot
the
host,
so
the
original
host
with
its
memory
is
gone
and
then
once
the
container,
once
the
host
has
rebooted
into
new
new
kernel,
that's
why
we
have
a
different
color,
but
we
have
the
same
container
running
as
before.
It's
still
red
as
before.
I
have
prepared
a
demo
to
show
this
here.
So
let's
connect
to
a
rel
eight
system
using
podman.
A
So
let
me
start
a
container
podman
run,
and
so
this
is
a
wildfly
based
container.
It's
a
java
application
server
and
it
has
a
real,
simple
stateful
application
in
there,
which
returns
just
a
number
and
once
the
number
has
been
returned,
it
increases
it.
So,
the
next
time
I
get
a
number
or
increased
by
one.
So
it's
stateful,
and
now
I
can
talk
to
the
container
and
ask
for
the
value
so
give
me
a
zero
one
and
two
now
I
say
I
want
to
create
a
checkpoint
of
the
container.
A
So
I
say:
potman
container
check,
point
checkpoint,
the
last
container
and
export
it
to
the
file.
So
now
podman
will
write
the
checkpoint
file
to
the
disk,
and
then
I
can
reboot
the
host
and
later
I
can
restore
the
container
from
that
checkpoint
file.
The
checkpoint
file
does
not
only
include
the
checkpoint
of
the
processes
in
the
container,
but
it
also
includes
changes
to
the
file
system,
because
when
the
wi-fi
starts
it
creates
log
files
and
a
lot
of
files
to
to
to
to
manage
it
thing.
It's
things.
A
So
all
these
changes
to
the
file
system
are
also
part
of
the
checkpoint
archive.
So
now,
let's
reconnect
to
the
vm.
Let's
see,
there's
no
container
running
so,
let's
restore
the
container,
putman
container,
restore
and
I
say
import
and
I
give
it
the
guitar
file
from
before
and
now.
If
we
connect
to
the
container,
it
should
give
us
backup
three
instead
of
the
zero
as
it
would
when
starting
from
scratch
talk
to
the
container
and
we
get
a
three
and
a
four.
A
So
we
did
a
reboot
and
we
were
able
to
save
the
state
of
the
container
back
to
the
slides
and
next
use
case
is
what
I
call
a
quick
startup.
So
again
we
have
a
container
of
a
host
and
a
container
again
takes
a
long
time
to
initialize,
and
now
we
want
to
start
up
the
container
faster.
So
we
take
a
container
a
copy
of
the
container.
A
A
So
let's
say
I
create
another
checkpoint.
I
say:
apartment
container,
check,
checkpoint
last
container
and
now
I
say,
keep
it
running,
don't
stop
the
container
after
the
checkpointing
and
I
again
exported
the
file,
I'm
overwriting
the
existing
file.
So
we
don't
need
this
anymore.
So
now
again
the
same
thing:
it
will
create
a
checkpoint
and
the
file
system,
differences
or
everything
will
put
into
an
archive,
and
now
I
can
restore
it.
Let's
see
so
the
original
container
is
still
running.
A
I
can
still
talk
to
it
and
now
I
can
restore
a
copy
from
the
container.
So
when
I
say
portman
container
restore
import
again
the
file.
This
will
now
fail
because
portman
sees
that
the
id
and
the
name
is
already
in
use,
and
so
it
tells
me
I
cannot
restore
this
because
this
already
exists,
so
I
tell
portman
then
use
a
different
name
for
it.
So
let's
say
I
counter
one:
now
it
will
restore
the
container.
A
A
copy
of
the
container
has
counter
one
and
another
copy
as
counter
two
and
then
we
should
be
able
to
talk
all
containers
independently.
So
now
I
can
say:
potman
inspect
counter
one
to
get
the
ip
address
of
this
one.
So
it
says
again,
six
seven.
This
is
the
point
where
I
checkpointed
it
from
and
now
I
can
also
say,
inspect
counter
two
and
again,
six
seven
eight.
A
So
we
create
multiple
copies
of
an
existing
container
and
in
this
small
demo
case,
I
can
see
already
improvements
in
the
startup
time,
because
usually
my
wildfly
container
takes
like
eight
seconds
until
it
can
answer
the
first
request
and
it
takes
about
four
seconds
to
restore
from
from
a
checkpoint.
So
I
already
already
have
like
50
improvement
in
startup
time
re
starting
from
from
a
checkpoint,
so
the
next
use
cases
container
live
migration.
A
This
is
basically
a
combination
of
the
two
things
I've
shown
before
so
again,
I
have
a
container
running
on
a
source
system.
I
take
a
copy
of
the
container
and
write
it
to
disk,
and
then
I
transfer
it
to
a
destination
system
and
there
I
can
create
multiple
copies.
If
I
want
to
so,
let's
go
to
so
I
already
have
the
check
point.
Okay,
I
already
have
the
checkpoint,
so
let's
transfer
it
to
another
system
and
say
temp:
dump,
tar
and
transfer
it
to
another
vm.
A
Where
is
it
here,
and
so
now
it's
container
migrate
with
mid
middle
in
the
container
migration,
and
now
I'm
on
the
other
host
and
there's
no
container
hopefully
running
here
now
I
say:
potman
container
restore
import
again
from
the
file
I
just
transfer
to
the
host
and
it's
restoring
the
container
and
now
I
should
be
able
to
talk
to
it,
and
I
should
get
probably
again
a
six
right,
six,
seven,
eight!
So
and
now
we
have
migrated
the
container
from
one
host
to
another
host
using
portman's
feature
to
export
and
checkpoints.
A
So
I
have
one
more
use
case
here.
This
is
the
forensic
container
checkpointing,
and
this
is
also
the
the
use
case.
We
are
currently
working
on
our
kubernetes
enhancement
proposal
and
a
corresponding
pull
request
to
it.
So
the
idea
is,
we
want
to
introduce
checkpoint
restore
into
kubernetes,
but
it's
complicated,
as
I
said,
with
the
cri
api.
So
we
try
to
implement
a
really
simple
use
case
of
checkpoint,
restore
which
requires
minimal
changes
to
to
kubernetes,
and
most
of
the
changes
need
to
happen
in
the
container
engine.
A
So
what
we
want
to
do
is
we
want
to
checkpoint
a
container
which
we
think
something
is
not
right
with
the
container.
Something
seems
to
be
strange,
but
we
do
not
want
to
analyze
the
container
in
case
there
is
an
attacker
there
and
in
case
the
attacker
has
put
something
in
the
container
to
detect
analysis
of
the
container.
A
So
what
we
do
is
we
take
a
checkpoint
of
the
container
out
of
kubernetes
and
try,
and
then
we
can
transfer
the
container
in
a
sandbox
environment
where
we
can
start
it
without
kubernetes,
but
without
influencing
the
original
containers.
So
I
also
have
a
demo
for
this
here,
so
this
is
now
a
fedora
system
no
longer
the
system
from
before.
I
have
prepared
it
here
in
a
screen,
so
there's
kubernetes
running
with
cryo
in
combination,
so
let's
see
cry
control
ps.
I
know
this
find
out.
A
Dns
from
kubernetes
anyway,
so,
let's
start
a
pod
with
two
containers
so
part
with
two
containers
created
we
can
see
so
the
first
container
is
already
something
up
the
same
container
as
before.
Again
the
wildfly
based
container
and
there's
now
a
second
container,
it's
called
counter
it's
it
does
the
same,
but
it's
just
a
python
application.
A
This
time
we
want
to
checkpoint
the
python
applications.
So
so,
let's
see
we
first,
let's
talk
to
the
container
so
that
we
see
what
it
returns
to
us
returns
one
zero
two!
So
it's
it's
it's
the
same
as
before,
just
implemented
in
python,
and
now
we
want
to
check
on
the
container
without
stopping
it.
So
we
created
a
cubelet
interface
which
lets
us
checkpoint,
our
container,
so
it's
called
checkpoint.
Then
we
say
default,
namespace
podname
is
counters
and
container
name
is
counter.
A
So
we
now
say:
let's
checkpoint
it
now,
kubernetes
talk
to
cryo
cryo
talk
to
run
c
runs.
He
talked
to
screw
and
container
has
been
checkpointed.
If
we
access
the
container
again,
we
see
it's
still
running,
so
the
container
has
not
been
changed.
So
now.
Let's
stop
kubernetes
here
on
under
source,
and
I
will
now
my
windows
here
and
now
I
will
remove
to
con
all
the
containers
from
from
this
hose.
A
A
A
A
A
Creo
always
operates
on
a
process
tree.
So
we
point
clear
to
a
process
and
creo
will
checkpoint
this
process
and
all
child
processes.
It
will
collect
all
the
information
of
the
processes
and
write
it
to
disk
the
process.
This
will
be
paused
during
that
time
and
can
continue
to
run
after
the
checkpoint.
In
the
first
thing
quiu
does
and
where
it
got
it
got
its
name
from.
Is
it
collects
the
information
from
user
space
and
using
the
information
available
in
procped?
A
So
cryo
collects
all
the
information
from
profit
once
it
has
all
the
information
it
can
collect
from
the
outside
of
the
proset.
It
goes
into
the
process
to
collect
information
from
within
the
processes
address
namespace,
and
to
do
this,
crea
uses
something
called
a
parasite
code.
The
parasite
code
is
injected
into
the
process.
It
replaces
some
of
the
original
code
and
the
parasite
code
is
then
starting
again
and
it's
it's.
A
It's
basically
running
a
daemon
inside
of
the
process,
and
now
the
main
creo
binary
can
talk
to
the
daemon
running
inside
of
the
process,
as
as
the
parasite
code
and
collect
information
from
within
the
name
from
within
the
address
space
of
the
process.
So
we
can
see
inside
of
the
process.
Using
this
and
and
one
important
thing
creo
does
using
the
parasite
code
is,
for
example,
to
dump
all
the
memory
from
the
process
is
using
from
within
the
processes
address,
name
space
and
and
this
way
we
can
dump
the
memory
really
fast.
A
And
if
you
look
at
migration
times,
the
most
time
is
spent
in
in
transferring
the
data
over
the
network
from
one
host
to
another
host
so
dumping.
The
memory
from
the
process
to
disk
is
fast.
Also
reading
it
back
is
fast
and
compared
to
the
time
it
takes
to
transfer
the
process
from
one
system
to
another
over
the
network.
A
So
once
the
the
parasite
code
has
collected
all
the
information
and
and
written
to
disk,
the
parasite
code
is
removed.
Of
the
usage
this
is
called
or
creo
calls
is
curing
the
process.
The
process
will
never
know
that
it
was
running
under
the
control
of
of
crew
or
the
parasite
code.
It
will
just
continue
to
run.
A
A
We
collect
all
the
information
from
within
the
processes
address,
name
space
and
once
we
have
collected
all
the
information
we
need,
we
replace
the
parasite
code
with
original
code
and
the
checkpointing
is
then
finished
and
at
this
point,
after
all,
relevant
information
has
been
written
to
disk
and
the
target
process
can
be
killed
or
it
can
continue
to
run
in
the
demos
I've
showed
we.
We
have
seen
both
ways
either
the
process
keeps
on
running
or
not.
So
this
really
depends
on
the
use
case.
What
you
want
to
do
so
after
checkpointing.
A
The
second
last
step
is
now
restoring
the
process.
This
happens
by
reading
all
the
checkpoint
images
from
disk,
and
then
we
recreate
the
process
tree
for
each
pid
tid,
just
as
it
was
before
checkpointing
we
do
a
clone
for
each
or
clone
three,
depending
on
your
kernels
for
each
process
and
recreate
a
process
tree,
just
as
it
was
before
checkpointing
and
once
we
have
the
process
tree,
the
creo
moth
all
the
processes
into
the
original
processes
like
they
were
during
checkpointing
and
one
good
example,
I
think,
is
always
the
file
descriptors.
A
So
during
checkpointing
we
look
at
the
file
descriptors,
which
file
descriptor
points
to
which
file
and
what
position
in
the
in
the
file
do.
We
have
and
we
write
this
information
to
disk
and
during
restore,
we
re
open,
all
opened
file
with
the
same
file
descriptor,
and
then
we
position
the
file
script
at
the
correct
position,
and
once
the
process
continues
to
running
after
creo
has
done
all
its
work.
A
The
file
descriptor
will
be
at
exact
the
same
position
and
and
and
will
be
the
same
id
will
be
the
same
file
just
as
as
it
was
before,
so
the
code
which
will
access
the
file
descriptor,
will
just
get
the
same
file.
The
same
content,
same
position
and
creo
will
map
all
the
memory
pages
back
to
the
original
location.
A
A
What
I
did
in
my
demos,
just
if,
if
if
you
want
to
try
it
out
for
yourself
and
the
steps
are
listed
here
in
the
slides,
so
first
was
the
checkpointing,
then
the
restoring
and
restoring
with
a
new
name
and
creating
copies,
and
with
this
I'm
already
at
the
summary
of
my
talk,
so
creo
can
be
used
to
checkpoint
and
restore
containers
and
it's
integrated
in
different
container
engines.
Already
it's
used
in
production
and
the
use
cases
are
presented
where
you
can
reboot
into
a
new
kernel
without
losing
the
container
state.