►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: containerd - Derek McGowan, Docker & Mike Brown, IBM
Come learn about the internals of containerd that make it extensible, efficient, and stable. We will discuss the gRPC APIs used to communicate with containerd, the smart client model, the pluggable runtime interface, and the provided components that enable pulling, unpacking, and running a container image. For container runtime providers, learn about the v2 runtime shim API and integrating your own runtime. For those with custom storage needs, learn about developing plugins to build your own snapshotter and content store. We will include examples from CRI, Docker, and Moby’s buildkit.
To learn more click here: https://sched.co/FuLU
A
Hello,
thank
you
for
coming.
This
is
the
container
D
deep
dive
session
I'm
in
which
McGowan
and
myself
will
try
to
elucidate.
You
know
how
container
D
works
a
little
bit
we'd
like
to
do
it
in
a
semi
relaxed
kind
of
mode.
We
know
it's
late
in
the
conference
and
we're
sure
that
you
guys
may
have
lots
of
questions.
So
we're
certainly
here
to
answer
any
questions.
Derek
we'll
go
through
all
the
decks
and
give
you
a
nice
nice
demo.
Derek
is
a
maintainer
from
docker
he's
been
keep
closer.
A
Is
that
louder?
Is
that
better?
Well,
I'm
sure
the
translators
can
probably
hear
me
better
too,
since
they
were
in
the
room
right
so
on
to
repeat
just
a
bit
introducing
Derek
McGowan
of
docker.
He
is
a
maintainer
on
container
D.
He
works
on
all
elements
of
it,
but
you
know
focuses
on
a
lot
of
the
file
system.
Components
he's
worked
on
docker
distribution,
the
docker
registry
been
around
for
a
long
time
and
he's
got
a
lot
to
share
okay
I'm
turning
it
over
to
Derek.
B
Essentially
it's
a
client-server
model.
We
have
a
smart
client.
The
client
implements
all
of
the
high-level
interfaces.
So
when
you
do,
for
example,
a
pole
operation,
the
pole
operation
is
actually
implemented
inside
the
client.
Any
sort
of
new
functionality
that
users
are
asking
for
most
of
those
functionalities
can
actually
be
implemented
in
the
client
rather
than
on
the
container
d
server.
B
The
interfaces
on
the
client
they
may
change
somewhat
rarely,
but
well.
What
we
really
focus
on
for
stability
with
container
D
is
our
G
RPC
interface,
so
the
G
RPC
interface
gives
you
access
to
all
the
low-level
components,
all
our
low-level
interfaces
and
we
consider
it
to
be
stable.
Anyone
dot,
X,
container,
D
client
should
be
able
to
use
any
any
server
so
sitting
underneath
inside
the
server
we
have.
B
We
have
our
service
layer
so
container
DS
service
interfaces.
These
are
what
provide
interfaces,
such
as
to
the
content
store
to
our
snapshot,
or
so
we
actually
have
a
service
layer
that
can
be
used
directly
by
plugins.
It
can
be
used
by
the
G
RPC
service.
It
provides
access
to
each
of
the
components,
and
then
we
have
a
metadata
store
that
sits
right
underneath
it.
So
our
metadata
store
is
what
will
actually
provide
all
the
namespace
scene
inside
of
container
D.
B
It's
what
adds
all
of
the
labels
and
it
is
actually
sitting
on
top
of
the
snapshot,
ER
and
content
store,
which
talked
to
the
which
actually
store
data
on
disk,
so
that
you
could
have
data,
that's
on
disk
for
one
user
and
another
user,
potentially
using
the
same
data
or
different
data
on
disk,
but
all
that's
going
to
be
managed
by
the
metadata
store,
so
think
of
the
metadata
stores.
It's
it's
just.
It's
just
a
bolt
database
in
container
D.
It
is
top
level
key
is
the
actual
namespace.
B
It
was
designed
this
way
in
order
to
support
multiple
clients.
So
if
your
the
CRI
plug-in
your
recent,
a
namespace
called
Katie's
I/o,
if
your
docker
you're
using
the
namespace
called
Moby
when
you're
using
CTR
the
namespaces
just
called
default,
but
you
can
set
your
namespace
to
anything
and
under
that
namespace
we.
B
B
B
Of
how
that
data
is
structured,
so
we
have
data
types
for
images.
We
have
data
types
for
containers,
content
which
think
of
content
is
just
a
blob
of
data
referenced
by
a
hash.
So,
if
you're
familiar
with
the
registry
API
at
all,
there's
like
a
blobs
endpoint,
which
just
is
referenced
by
hash,
OCI
images
descriptors,
they
always
point
to
a
specific
hash.
So
the
content
store
is
just
what
actually
stores
those
blobs
of
data.
So
in
container
D
we
decided
we
need
a.
B
We
need
a
way
to
track
all
of
this
data,
which
could
potentially
get
kind
of
large
a
problem
that
people
often
have
with
docker
is
after
they've,
been
using
docker
for
a
while.
There
just
starts
to
fill
up
and
we've
added
some
stuff
within
docker
like
pruning.
That
can
actually
help
reduce
some
of
that
usage.
But
in
the
end
it's
it's
kind
of
harder,
at
least
in
the
data
model
that
docker
has
to
clean
up
data
so
for
container
D.
B
We
decided
to
build
that
in
from
the
beginning,
by
having
actually
references
between
the
data
and
having
top-level
references
that
users
would
be
more
familiar
with,
such
as
images
and
containers.
So
when
you
go
to
pull
an
image,
it's
going
to
pull
down
all
the
content
and
then
it's
going
to
create
an
image.
That's
going
to
point
at
that
content,
so,
in
this
chart,
I
tried
to
demonstrate
what
what
your,
what
the
container
deed
meditative
store,
might
look
like.
B
If
you
had
two
images
and
you
had
a
running
container,
so
the
Redis
container
will
be
based
off
the
Redis
image
that
was
pulled
from
the
registry
and
all
these
arrows
in
between
in
container
D.
These
are
actually
just
labels,
but
to
the
garbage
collector,
they
look
like
references
so
that
we
know
what
what
can
actually
be
kept
and
what
can
be
removed.
B
So
if
we
were
to
come
in
and
now
say,
delete
the
Redis
container,
even
with
a
or
delete
the
Redis
image,
even
with
the
Redis
container
running,
the
immediate
result
would
just
be
this.
The
image
is
gone.
There's
no
longer
a
link
between
that
image
and
the
content.
That
is,
reference,
that's
being
referenced
by
that
image,
the
garbage
collector
is
going
to
come
along.
B
It's
going
to
see
that
there's
content
here
that
is
no
longer
referenced
by
a
root
and
it's
going
to
go
ahead
and
and
delete,
delete
that
content
and,
as
you
can
see
when,
as
soon
as
that
content
gets
deleted,
the
links
go
away.
What
I
said
before
how
each
of
the
each
of
the
pieces
of
data
in
container
D
can
have
labels
and
those
links
are
just
part
of
those
labels,
so
once
the
content
is
gone
or
all
the
links,
so
the
end
result
is
just
going
to
look
like
this.
B
B
B
So
let
me
switch
over
to
talk
to
talk
about
plugins
for
the
snapshot
er.
If
anyone
has
any
questions
as
Mike
was
saying,
we
can
answer
some
of
them
along
the
way,
it's
kind
of
a
kind
of
a
big
room.
But
if
you
have
questions
a
lot
about,
some
of
the
data
model
go
ahead
and
ask
them
so
there's
not
too
many
at
the
end.
B
Okay,
so
for
the,
what
a
snapshot
are
looks
like
the
snap
shutter
is
a
fairly
simple
interface.
If
you're
familiar
with
kind
of
the
graph
drivers
that
we
have
in
docker,
there's
there's
a
lot
more
operations,
you'd
expect.
Whereas
for
the
snapshot
er
there's
no
data
operations,
there's
no
mounting
operations,
we
have
immutability
for
snapshots,
as
in
once
you
create
a
new
snapshot
and
you
commit
it.
That
snapshot
is
immutable.
There's
no
way
through
the
API
to
mutate
that
that
snapshot
anymore
snapshots
themselves
can
be
labeled,
as
you
saw
in
the
previous
diagram.
B
B
B
So
in
1.2
we
added
a
feature
called
a
proxy
snapshot
err.
So
today
we
have
three
proc
or
C.
We
have
three
snapshot
eyes
that
are
built-in,
maybe
for
three
or
four
snap
shutters
built
in
the
main
ones,
being
overlay,
FS
and
then
and
yes,
the
most
popular
one
and
the
native
driver,
which
is
just
a
flat
which
is
every
layer,
is
just
the
full
flat
structure,
kind
of
like
the
naive
driver
and
docker.
B
So
what
a
proxy
snapchatter
allows
is
you
can
actually
use
in
a
configuration
you
can
configure
a
UNIX
socket
to
actually
represent
a
snapshot
ur
and
in
that
unit
socket
serve
a
gr,
PC
API
of
our
snapshot
or
interface.
So,
if
you
have
this
implemented,
we
have
AG
RPC
interface,
set
mirrors
this
and
can
actually
be
made
available
through
G
RPC.
B
So
to
configure
it
today,
you
would
update
the
Etsy
container
D
config
tamil
there's
a
line
in
there
called
proxy
plugins,
let's
type
snapshot,
and
you
just
give
it
a
UNIX
socket.
So
this
is
an
example
of
how
you
would
create
one.
So
the
second
line
in
the
main
function,
a
custom
snapshot
er.
All
that
is,
is
an
implementation
of
the
previous
interface
that
I
showed
register
that
with
a
G,
RPC
service,
start
listening
and
then
container
D
to
make
use
of
it.
B
Now,
one
of
the
one
of
the
reasons
why
this
is
somewhat
practical
with
snapshots
and
the
way
we
designed
it
is
the
same
since,
since
there
is
no
mounting-
and
there
is
no
data
operations,
the
API
is
pretty
light
that
what
actually
gets
requested
of
the
snapshot,
err
and
what
gets
returned
by
the
snapshot
err
is
is
mostly
metadata
in
nature.
There's
not
a
lot
of
transactional
operations
that
need
to
happen
between
different
runs
of
the
snapshot,
or
so
it
can
act
fairly.
Stateless.
B
B
So
the
main
feature
here
stabilization,
but
we
also
added
a
stats
API
here,
whereas
previously
and
container
t10
and
1.1
to
get
stats.
You
would
have
to
actually
talk
to
see
groups
on
the
hosts
which,
as
you
can
imagine,
is
problematic
when
you're
running
inside
of
a
VM,
you
can't
make
use
of
those
stats.
So
this
allows
any
runtime
to
to
fully
implement
this
interface
and
be
much
more
much
more
useful
to
a
broader
range
of
runtimes.
B
B
The
other
way
to
kind
of
extend
container
D
today
is
in
the
client,
so
the
client
extensibility
is
mostly
through
our
go
interfaces,
so
we
chose
to
take
an
approach
in
our
go
client,
where
pretty
much
anything
can
be
swapped
out.
So
I
gave
an
example
of
when
you
actually
estancia
taken
a
nerdy
client.
We
have
this
service
object,
which
is
responsible
for
actually
connecting
to
these
services.
That
would
be
served
by
the
G
RBC
API
and
when
you'e
Stan
she
ate
that
service,
you
can
overwrite
any
of
the
individual
services
with
any
implementation.
B
So
the
arguments
to
all
of
these
are
mostly
just
the
interfaces
that
container
D
itself
has
defined
and
uses
all
over
the
place,
such
as
with
snapchatters.
This
snapchatter
interface
is
that
same
interface,
I
showed
before
you
could
actually
add
your
own
custom
snap
shutters
directly
on
the
client.
If
you
wanted.
C
B
This
is
used
by
the
CRI
plug-in
as
well,
so
the
CRI
plug-in,
even
though
it's
running
inside
the
daemon,
it
actually
uses
the
container
D
client
and
it
justice
Stan
sheets
all
of
these
services
with
a
local
service.
So
it
can
be
run
both
externally
or
internally,
without
having
any
changes
in
the
way
it's
designed.
I
also
wanted
to
highlight
it's.
B
It's
not
absolutely
obvious
at
first,
what
the
what
a
resolver
means,
but
the
with
resolver
is
an
option
that
you
can
give
on
any
push
and
pull
operation.
Think
of
the
resolver
as
a
way
to
take
a
name
and
resolve
it
to
a
method
for
getting
bytes
or
getting
the
blobs
for
for
that
image.
So
in
a
pull
flow,
that
would
look
like
a
process
of
resolving
a
name
to
a
manifest
and
then
just
having
a
way
to
pull
down
the
blobs
further
for
that
manifest.
B
So
it's
it's
somewhat
OCI
specific
in
that
it
assumes
that
you
have
OCI,
manifests
and
and
tar
blobs,
but
using
this
functionality
you
could
actually
completely
overwrite
how
you
do
distribution
in
container
d.
You
can
have
it
something.
That's
completely
customized.
If
you
don't
want
to
use
the
the
to
find
docker
base
implementation.
That's
now
the
OCI
distribution
spec,
so
pull
flow
ends
up.
Looking
like
this
in
container
D,
this
is
completely
orchestrated
by
the
client.
When
a
pull
operation
runs
that
can
be
broken
down
into
a
fetch
operation.
B
The
fetch
gets
all
the
content
from
the
remote.
It
puts
it
inside
the
content
store.
Once
all
that
content
is
fetched,
the
pol
will
create
an
image
that
points
to
the
manifests.
After
that
image
is
created.
That's
when
we
actually
do
the
unpack,
which
will
read
the
content
that
has
been
pulled
and
verified
inside
of
the
container
D
content
store,
and
it
will
unpack
it
into
a
snapshot
er.
B
So
the
push
flow
is
very
simplified
as
well
in
container
D,
since
we
have
the
content
container
D
defines
no
method
of
creation.
Of
that
content,
we
defer
that
to
tools
like
don't
get
so
from
container
DS
perspective,
that
has
images
in
s
content
and
when
you
go
to
do
a
push
operation,
it
just
takes
that
content
and
it
pushes
it
directly
to
the
remote
in
a
docker
flow.
There
would
be
an
extra
step
here
that
would
be
reading
from
a
graph
driver
in
order
to
to
recreate
the
content,
recreate
all
the
manifests.
B
The
container
remember
earlier
when
I
was
talking
about
it,
snapshot
are
not
having
the
responsibility
to
mounts.
We
move
that
responsibility
to
the
running
container
step
so
by
having
the
running
containers
actually
deal
with
the
mounts
we
no
longer
have
to
worry
about.
Inside
the
snapshot,
ur
various
issues
around
mount
namespacing
and
cleaning
up
some
of
those
mounts
and
how
many
times
those
snapshots
had
actually
been
mounted
and
Mike.
You
want
to
talk
about.
D
Sungeum
thanks
Mac
hi
Dirk,
you
have
done
a
great
job.
I
have
a
little
confusion
about
the
GC
connect
to
the
copy's
connector
yeah.
If
we
delete
an
image,
then
the
word
image
information
help
it
elated,
even
though
the
the
container
is
running
so
I
have
Cochin.
Is
that
if
the
upper
Sitton
want
to
know
the
containers
image
information,
how
do
we
do
that?
Oh.
B
B
If
the
client
can
implement
this
in
different
ways,
it
could
also
set
a
label
on
the
container
that
says
what
the
image
name
is,
but
at
least
from
dock
or
from
the
data
model
that
we
have
in
here,
or
we're
not
opinionated
about
whether
or
not
whether
or
not
that
container
should
be
linked
to
that
name
after
it
is
created,
at
least
from
from
here.
We
don't
have
a
bunch
of
checks
that
would
go
in
and
try
to
kind
of
define
these
policies.
So
this
policy
start
up
to
see.
A
A
A
E
E
Proportion
and
a
I
already
put
an
image
to
this
note,
but
henan
be
supposed
to
not
have
access
to
this
image,
but
he
can
guess
and
try-
and
maybe
he
is
successful-
run
a
Latinas
image,
because
the
image
already
on
a
note-
and
he
he
just
get
the
process-
is
there
any
kind
of
this
roadmap
to
prohibit.
This
kind
of
situation
happened
so.
B
There's
there's
two
features
that
we
have
and
community
to
both
support
that
and
disallow
it.
So
the
default
behavior
is
that
when,
when
you
have
content,
that's
pulled
between
two
namespaces
when
it
goes
to
create
the
content
in
the
second
namespace
it
it
sees
that
that
content
is
there,
it'll
actually
be
able
to
copy
that,
but
we
do
have
a
feature
in
container
D,
where
you
can
disallow
that.
So
it's
it's
like
a
strict
mode
where
every
single
namespace
is
responsible
for
repo
lling,
that
content
verifying
it
and
after
it
has
verified
it.
B
It
goes
to
commit
that
content
on
the
on
the
back
end.
It
will
share
the
content,
but
it
first
has
to
go
through
the
verification
step
before
it
will
add
it
into
the
new
namespace.
So
that's
that's
similar
to
how
the
registry
works
today,
whereas
the
registry,
you
can't
just
tell
it
you're,
you're,
creating
content,
you
are
you
already
have
it.
B
You
actually
have
to
prove
that
you
have
it,
so
we
both
have
a
a
mode
where
you
have
to
prove
it,
but
by
default
we
don't
have
that,
since
we
don't
really,
we
haven't
really
defined.
How
the
security
model
on
top
of
container
D
should
work.
It's
really
up
to
the
clients
to
define
but
yeah
that
that
is
a.
That
is
something
that
we
thought
about
and
made
sure
that
at
least
from
Canadia
spective.
C
B
A
A
To
the
intro
section,
you
probably
understand
that
we
we
do
some
namespace
isolation
around
the
pod,
it's
basically
we
it's
applied
namespace
for
networking
and
user
namespaces
things
like
that.
We
also
have
C
groups
that
get
attached
permissions
and
they're
all
shared
across
the
across
the
entire
pod.
Alright,
this
represents
all
the
resources
that
are
going
to
use
for
your
pot
and
it
represents
the
the
container
that
you're
going
to
generate
you'll
you'll
note
here
that
we're
using
CI,
for
you
know
all
networking.
A
A
So
this
is
a
quick
flow,
we're
just
we're
just
going
to
create
apply.
It's
pretty
you'll
notice,
it's
very
similar
to
creating
a
detainer,
because
the
pod
is
actually
in
our
implementation
actually
includes
creating
a
container
called
the
pause
container.
We
pull
that
image
down,
it's
really
small,
we
run
it,
we
get
it
get
it
going
and
we
attach
a
network
namespace
to
it.
A
You
hear
what
we
were
showing
is
actually
now
running
the
container
in
the
pod
kulit
request
to
launch
the
container
we
go,
get
the
image
create
a
snapshot
using
the
tools
you
talked
about
before
we
set
up
our
container.
We
store
our
metadata
that
best
sort
of
thing
and
then,
when
we
start
a
task
and
you
don't
running.
C
A
A
A
B
Kind
of
a
an
example
proxy
plugin,
so
this
is
similar
to
what
I
showed
earlier,
but
I
kind
of
want
to
show
it
working.
So
we
have
a
simple
app
here:
I'm
just
going
to
run
a
proxy
plugin
which
uses
the
native
snapshot
ur.
So
it's
basically
gonna
take
the
native
snapshot
ur,
which
obviously
implements
the
snapshot
or
interface.
It's
going
to
register
it
as
a
gr,
PC
plugin.
B
B
C
E
B
D
B
B
Want
to
show
real,
quick
we're
at
the
end
here.
Let
me
go
ahead
and
show
how
the
taken
view
these
with
CTR.
So
we
have
a
tree
command
under
CTR.
That's
pretty
useful!
It
actually
shows
the
relationships
between
the
different
snapshot
airs.
So
this
one
here
is
actually
going
to
be
the
the
leaf.
So
this
would
be
the
parent
most
layer.
B
B
We
also
have
a
neat
command,
never
said
snapshots,
don't
mount,
but
CTR
will
output
a
mount
for
you.
So
this
is.
This
is
a
mount
command
you
could
just
give
to
Linux
and
it
will
run
it
and
mount
it
wherever
you
want,
but,
as
you
can
see,
since
it's
the
native
snapshot,
ur
a
mount
is
just
a
bind
and
it's
binding
to
the
directory
7,
which
has
the
full
root
filesystem
for
that
image.
So
let
me
just
go
ahead
and
clean
that
up
and
show
you
what
happens
when
we
remove
this.
B
B
This
is
actually
all
the
content
that
got
removed
from
disk.
So
this
scheduling
of
the
content
cleanup.
This
is
actually
what
removes
data
from
disk,
since
it's
pretty
small,
it
was
didn't.
Take
that
long.
Only
six
milliseconds
of
the
four
point-
nine
milliseconds
is
the
amount
of
time
that
we
actually
had
the
metadata
store
locked.
So
we
made
sure
to
put
that
as.