►
From YouTube: Lessons Learned Migrating Kubernetes from Docker to containerd Runtime - Ana Calin, Paybase
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Lessons Learned Migrating Kubernetes from Docker to containerd Runtime - Ana Calin, Paybase
In 2016 Docker span out its core container runtime functionality into a standalone component, containerd. Using containerd developers can customize, extend and swap out functionality as needed, without unnecessary abstraction getting in their way. containerd provides all the core primitives needed to manage containers on Linux and Windows hosts allowing Docker and other leading container systems, including Kubernetes to use it as their core container runtime. containerd is fully OCI compliant. This talk covers lessons learned by migrating a Kubernetes platform from a docker runtime based OS to a containerd based OS. It includes observations around security, performance and usability, as well as the benefit of debugging using crictl - containerd’s cli, compared to the docker cli. Find out if there’s a real benefit in swapping and the kind of effort required to achieve complete migration.
https://sched.co/MPd8
A
A
A
Then,
a
couple
of
years
ago,
I
started
hearing
all
of
this
new
terminology
like
CRI
the
container
runtime
interface
or
OCI,
the
open
container
initiative
and
container
D
and
run
C
and
add
another
peps
technologies,
and
that's
when
a
massive
fear
of
missing
out
overcame
me.
I
didn't
want
to
be
left
behind,
but
I
didn't
have
time
to
think
about
it.
A
A
Your
workloads
from
kubernetes
cluster
running
on
docker
engine,
look
beneath
this
class,
the
running
on
container
D
so
for
the
last
well,
docker
brought
containers
to
mainstream
and
it
was
working
great
and
then-
and
this
is
what
everyone's
using
docker
and
that's
what
we've
been
using
for
the
last
five
years.
So
this
brings
me
to
today
and
I'm
gonna
talk
to
you
about
what
is
a
container
run
time,
because
the
whole
landscape
is
very,
very
confusing.
A
A
So
what
is
a
container
runtime?
According
for
definition,
I
found
a
container
runtime
is
responsible
for
all
the
parts
of
running
a
container
that
isn't
actually
running
the
program
itself.
Now,
please
raise
your
hand
if
you
understand
what
the
container
runtime
is
based
on
this
definition:
okay,
so
and
okay
you're
doing
than
me
when
I
first
read
this
I
didn't
quite
understand
it.
So
I
did
my
own
research.
A
So
the
way
I
used
to
think
about
containers
at
the
very
beginning
and
the
way
you
might
think
about
containers
as
well
is
a
bit
similar
to
how
we
used
to
think
of
in
the
VM
world.
So
you
have
the
kernel,
then
you
have
another
layer
which
is
the
time,
and
then
you
have
another
layer
which
is
the
container
itself
exactly
like
the
in
the
VMware
world.
Now
this
is
wrong
because
a
container
really
an
isolated
process
on
the
kernel.
A
What
gives
us
and
more
realistically
based
on
that
the
way
to
look
like
is
you
have
the
kernel
then?
So
you
have
a
client
that
talks
to
a
runtime
that
tells
the
kernel.
Okay,
give
me
all
of
these
processes
that
I
need
to
create
in
order
to
run
containers,
and
the
kernel
creates
those
for
those
processes
that
we
call
containers.
A
So
the
processes,
isolation
that
we
we
need
for
containers
is
given
by
a
few
features
in
the
Linux
kernel
and
I
wanted
to
very
quickly
recap
for
those
of
you
that
might
not
be
familiar.
So,
first
of
all,
we
have
the
concept
of
a
namespace,
not
open
apis
namespace,
a
Linux
kernel
namespace,
and
what
that
does.
A
Is
it
partitions,
different
kernel
resources
in
through
in
in
a
way
in
which
the
resources
in
the
same
namespace
she's
see
the
same
things,
but
that
is
related
from
each
other
to
namespace
level,
and
then
we
have
three
different
namespaces
that
are
relevant.
Here
we
have
the
PID,
which
is
the
process,
the
process
ID
namespace.
That
gives
us
the
numbering
of
processes
in
such
a
way
that
at
the
namespace
level
you
you
you
can
use
in
different
namespaces.
You
can
use
the
same
PID
without
being
aware
other
without
interacting
with
each
other.
A
Then
we
have
the
networking
namespace
and
what
this
does
is
it.
It
makes
sure
that
the
port's
that
you
are
using
in
containers
our
separate
and
don't
interfere
with
the
ports
that
are
already
open
on
the
kernel
itself.
Then
we
have
the
mountain
named
space
and
that
allows
us
to
mount
and
unmount
file
systems
on
containers
without
conflicting
with
the
host
file
system
and
then
we're
talking
about
cgroups
or
control
control
groups
and
they
allow
us
they
give
us
resource
limitations.
A
So
when
we
give
CPU
limitations
and
limitations
in
kubernetes,
that
translates
down
to
a
runtime
level
and
then
the
last
one
is
second
PPF.
I
hope
I
said
that
correctly.
But
what
this
does
is
it
allows
us
to
filter
what
kind
of
system
calls
certain
processes
can
and
cannot
do.
So.
That's
a
security
feature.
A
So
I,
like
this
picture
and
I,
think
that
you
could
say
that
this
might
be
a
good
analogy
of
how
runtimes
and
the
kernel
and
containers
sort
of
are
so
my
ok
port.
So
if
we
imagine
that
this
is
the
kernel
and
then
you
know,
the
wrong
x
gives
commands
to
the
kernel
and
then
each
one
of
these
it's
a
namespace
that
runs
the
containers
and
their
functionalities.
A
So,
let's
look
at
the
different
types
of
container
long
x,
so
we
have
low-level
runtimes
and
we
have
high-level
runtimes
and
then
we
have
docker,
which
is
which
does
low-level
high-level
and
additional
functionality
when
we
say
low-level
like
a
runtimes
women,
LX
c
1c
and
LMC
pfy
and
I'm
not
going
to
go
into
this.
But
all
you
need
to
know
about
them
in
this
context
is
that
they
are
the
the
runtimes
that
take
care
of
creating
them
those
namespaces,
and
that
process
has
isolation.
The
kernel
point
of
it.
A
Then
we
have
the
high
level
one
times
like
cryo
and
container
D,
and
there
are
a
few
others
now.
The
reason
why
we
have
high-level
as
well
is
because
just
creating
namespaces
and
isolating
processes-
that's
not
enough
to
run
a
container.
You
need
a
API
API,
you
need
additional
functionality,
you
need
the
ability
to
pull
and
push
images
and
unpack
them
and
then
pass
them
all
to
the
local
level
long
time.
So,
when
I'm
saying
we're,
gonna
talk
about
container
D
and
I'm
gonna
migrate
to
a
container
D
runtime.
A
Okay,
so
a
better
definition,
how
I
understand
container
long
times
in
a
very
simplified
right
is
a
container
one.
Time
is
responsible
for
setting
up
namespaces
and
C
groups
and
then
running
command
inside
those
namespaces,
plus
additional
things
and
I
feel
like
this
is
a
really
good
time
to
add
a
bit
more
terminology
so
that
we
can
use
it
so
the
CRI,
the
container
runtime
interface,
is
a
plug-in
interface,
which
enables
the
cubelet
to
use
a
wide
variety
of
container
runtimes
without
the
need
to
recompile.
A
The
OCI
is
the
open
container
initiative,
which
is
a
linux
foundation,
project
that
allows
us
to
design
open
source
standards
for
operating,
system-level
virtualization
and
for
Linux
containers,
and
then
we
have
the
CR
I/o,
which
is
contain
a
long
time
that
I'll
tell
you
about
in
a
bit.
So
let's
go
okay,
so
this
is
the
full
feature
set
of
docker
engine
and,
as
you
can
see,
it
does
a
lot
of
things
now.
Docker
was
the
first
open
source
container
run
time.
You
want
to
call
it
that
way.
A
That
was
taking
care
of
the
whole
lifecycle
of
building
containers,
building
images
and
pushing
them
pulling
them,
then
containing
then
creating
run-up
containers
out
of
them
and
so
on.
So
it
was
taking
care
of
the
whole
lifecycle
of
containers,
but
then,
some
time
ago,
I'm
not
a
historian,
so
I
don't
know
the
exact
time.
But
some
time
ago
docker
decided
to
split
its
monolith.
Daemon
functionality
into
different
features,
different
components,
and
it
looks
a
bit
like
this,
so
you
have
the
docker
CLI.
A
Then
you
have
the
docker
beam,
which
is
the
piece
of
functionality
that
does
all
of
the
additional
functionality
in
terms
of
building
images
and
packaging
bang
esophagram,
then
you
have
docker
container
d,
which
is
a
vanilla
version
of
created
by
docker
of
container
theme
that
we'll
talk
about
in
a
second
and
then
dr.
Wong
C,
which
is
the
low-level
that
creates
those
namespaces.
We've
talked
about.
A
Okay,
so
in
a
kubernetes
cluster,
when
when
we
want
to
run
kubernetes,
we
really
need
this,
so
we
need
the
low-level
functionality
and
we
need
the
high-level
functionality
and
the
rest
to
talk
to
kubernetes.
Now,
when
I
first
saw
this
and
was
thinking
about
this,
my
first
thought
was
wait.
So
you're
telling
me
that
I
don't
need
networking
to
run
in
a
kubernetes
cluster
or
I.
Don't
need
persistent
storage,
you
do,
but
that's
something
that
kubernetes
giving
it
gives
you.
So
you
have
the
services
resources
that
kubernetes
uses
to
take
care
of
networking.
A
You
have
the
storage
classes
and
the
persistent
volumes
that
you
use
for
persistent
storage.
So
purely
from
a
kubernetes
point
of
view,
you
don't
need
all
of
the
additional
functionality
of
docker
and
it's
really
not
fair
to
compare
docker
to
container
deep,
because
because
docker
has
container
D
and
relies
on
it.
So
it's
more
of
more
or
less
features.
A
In
terms
of
how
it
looks
in
terms
of
interaction
with
kubernetes,
a
docker
versus
container
team,
so
the
cubelet
talks
to
the
docker
shrimp,
which
is
the
sierra
implementation
for
docker,
and
the
Bocage
team
talks
to
the
docker
server
that
takes
care
of
building
images
and
so
on
and
docker
server
hands
over
commands,
container
d
and
container
didn't
talks
to
run
C
and
then
then
crates
containers.
Now
in
the
container
D
world,
it's
slightly
more
simplified.
A
So
in
terms
of
limitations
between
container
D
and
docker
docker
engine,
if
we
are
to
take
it
feature
by
feature,
it
sort
of
looks
like
container
D
is
losing
right.
Well,
it's
not
because
it's
meant
to
be
more
lightweight
and
because,
from
a
kubernetes
point
of
view,
you
still
get
to
do
all
of
the
same
things
that
you
don't
really
use.
You
know
kubernetes
cluster,
when
you,
when
you
use
docker
and
then
there's
another
limitation
more,
like
could
potentially
be
a
security
flaw
with
docker
engine.
A
A
Okay,
so
cryo
is
a
lightweight
see
a
long
time
that
was
made
specifically
for
kubernetes
at
the
high
level
runtime.
It
supports
the
management
of
OCI,
compatible
images,
and
it
supports
also
run
C
as
a
low-level
and
clear
containers
and
in
theory
it
supports
other
OCIE,
contain
compatible
low-level
runtimes,
I,
say
in
theory,
because
when
supporting
other
low-level
runtimes,
it
still
depends
on
some
of
the
features
of
Ramsey.
A
So
at
the
top
here
you
can
see
all
of
the
clients
that
can
be
used
with
or
by
container
d.
So
we
have
the
Q
blood
we
have
pouch
implemented
container.
The
docker
is
using
it.
You
have
the
docker
container,
the
client
and
the
container
the
CLI
lets
you
interact
directly
with
container
the
runtime.
Then
we
have
the
G
RPC
container,
the
API
that
allows
you
to
directly
communicate
with
the
container
B
without
using
client.
A
If
you
want
to
or
for
applicants
to
communicate
with
it,
then
there's
a
mate
matrix
API
and
the
kind
of
metrics
that
are
exposed.
Our
container
D
process-specific
might
matrix
as
well
as
container
specific
matrix,
which
are
very
useful
for
premedia,
and
a
really
really
neat
feature
is
the
runtime
manager
or
the
runtime
handler
that
in
version
1.2
will
let
you
switch
from
r
and
c
to
any
to
low-level
runtimes.
A
So
now,
let's
get
serious,
let's
talk
about
what
it
takes
to
migrate,
actually
quite
easy.
It
depends
on
your
setup,
so
AWS
eks
doesn't
currently
support
container
the
runtime
as
your
a
uses,
mobi,
which
is
a
version
of
flavor
of
docker
and
uses
that
as
a
container
runtime
and
with
aks
customer
can
deploy
their
own
kubernetes
clusters
with
a
different
runtime
using
the
AKS
engine
and
then
there's
GK
and
they
support
it
and
the
way
you'd
go
about
creating
a
cluster
with
container
at
the
wrong
time.
A
It
would
be
just
a
matter
of
selecting
the
cost
container
deep
image
of
your
cluster
or
just
creating
a
new
node
pool
in
an
existing
cluster
and
then
migrating
the
workloads.
Now
one
thing
that
I
want
to
be
clear
on
this
initially
before
I
started.
Looking
into
any
of
this
I
was
a
bit
worried
of
the
kind
of
effort
to
detect
migrate,
because
without
knowing
much
about
anything,
I
thought,
oh
well.
All
of
my
images
were
built
with
docker.
Does
that
mean
that
I
have
to
recreate?
All
of
my
images?
A
In
terms
of
performance,
I
am
a
systems
engineer
and
I
do
have
to
deploy
new
features
usually
yesterday
and
because
of
that,
I
really
care
about
performance.
So
container
beam
performs
better
in
terms
of
pod
startup
latency
compared
to
docker
engine
version.
18
there's
about
20
seconds
latency
that
container
be
container.
T
is
20
seconds
faster
than
on
docker,
and
this
is
something
that
I
haven't
created.
A
So
I
wanted
to
make
sure
that
that's
the
case,
so
I
sort
of
the
amount
so
I
spun
up
two
different
clusters:
GK
clusters,
one
running
docker
engine,
one
running
container,
D
on
the
exact
same
version
of
GK
and
then
I've
installed
with
helm.
The
nginx
ingress,
stable
home
child
and
I,
took
one
of
the
pods
and
I
looked
at
how
long
it
took
between
the
pod
being
initialized
and
the
port
thing
Marcus
ready.
So
you
can
see
that
this
is
on
the
docker
one
time,
and
you
can
see
here
the
timestamp.
A
The
time
it
was
initialized
and
then
you
can
see
the
time
its
type
in
and
and
yeah
it's
about
20
minutes
in
between
on
container
D
doing
the
same
thing
with
the
same
child.
Picking
the
exact
same
pod,
you
can
see
that
the
time
it
was
initialized
the
time
that
passed
it
in
when
it
was
initialized
and
when
the
pod
was
masters
ready.
It's
exactly
two
seconds
now.
A
Well,
I'm
also
an
engineer
with
PCI
DSS
compliance
in
my
mind
and
if
you
don't
know
what
this
idss
is
you're
very,
very
lucky,
because
it's
a
very
horrible
standard
that
the
payment
institutions
have
to
comply
with,
and
it
talks
about
a
lot
of
security
and
a
lot
of
really
ridiculous
requirements.
So
from
that
point
of
view,
container
VIII
has
a
smaller
attack
surface
than
docker,
simply
because
it
has
less
less
feature
when
container
be
graduated
from
CN
CF.
A
They
published
their
full
security
audit
and
it
looks
quite
good
and
in
the
event
that
my
auditor
comes
to
me
and
says,
can
you
prove
to
me
that
your
contain
a
long
time
is
secure
enough?
I
know
that
that's
not
gonna
happen,
but
I
can
say.
Yes,
I
can
here's
the
security
audit
and
what
I
mentioned
before
with
docker
an
attacker
with
access
to
my
nodes
can
build
images
locally
and
install
them
into
my
cluster.
A
A
Oh
container
B
is
more
more
flexible
and
much
more
lighter
than
docker,
and
that's
a
good
advantage
and
container
D
runs
on
top
of
low-level
runtimes
such
as
run
C,
and
you
can
plug
them
in
and
out,
and
it
performs
better
and
it's
more
secure
than
docker
it's
fairly
easy
to
migrate
and,
in
case
you're
wondering
what
step
we
are
with
on
migration,
we're
not
yet
in
production,
because
I'm
still
testing
and
yeah
that's.
My
summary
I
have
added
some
resources.
B
B
If
you
want
to
like
actually
come
up
to
the
front
like
that,
would
be
actually
a
really
helpful
and
I'll
just
give
you
the
mic
and
we
can
kind
of
go
from
there.
How's
that
working
yeah,
yeah,
just
maybe
like
kind
of
line
up
here,
is,
if
that's,
what
works.
Allen,
I'm
gonna.
Take
you
first,
if
you
don't
mind,
I.
C
A
C
A
A
So
one
of
them
is
in
version
1.1.
They
they
made
that
container
the
try
container
big
feature
into
a
plugin
so
that
reduced
latency
between
between
different
components
and
then
the
fact
that
docker
has
that
additional
feature
that
the
docker
server
component.
So
it's
a
bigger
chain
of
elements
I
could
go
through.
That's
one
of
them
in
terms
of
how
it
was
how
it
was
measured.
It
was,
it
was
an
average
over
a
hundred
five
pots
in
a
in
a
cluster
in
a
in
a
stable
state,
and
that
105
is
I.
Think
it's
the
maximum.
D
Hi
I'm
Jenna
Ellie,
and
thank
you
for
the
great
talk
and
introducing
me
with
the
opportunity.
I
have
a
question
so,
as
you
mentioned,
that
we
cannot
build
container
images
with
catenary
right
yeah,
so
it
means
that
we
cannot
use
any
CI
that
depends
on
building
and
pushing
images.
For
example,
we
talked
her
and
usually
in
enterprises,
they
have
CI
and
they
use
talking
in
docker
or
they
just
mount.
The
docker
proxy
Dakka
Dakka
socket
with
limited
access
for
a
fort
near
CI
a
moment.
So
it
is
not
possible
with
continually
right.
A
Is
it's
not
possible
with
container
people
who
actually
have
the
same
the
same
sort
of
implementation?
We
run
all
of
our
images
through
CI
before
that
get
pushed
into
a
Google
repository,
and
then
we
still
use
docker,
but
in
the
actual
Burnett
is
cluster.
So
we
don't
don't
run
the
CI
functionality
within
the
kubernetes
cluster.
That
would
would
probably
impact
our
applique
in
more
more
sense,
most
centrally
impact
our
application.
So
so
we
still
have
docker
in
our
you
know,
CI
process.