►
Description
Embedding the Containerd Runtime for Fun and Profit [I] - Phil Estes, IBM
The containerd project, one of the youngest in CNCF, is purpose-built to be an embeddable container runtime expected for use within higher layer container systems like the Docker engine and the Kubernetes orchestrator. Of course, the intent is that it will be used and embedded within a variety of software systems and has been designed for easy consumption via a gRPC API and client library.
In this talk we'll walk through a straightforward example of building up a container "client" written in Go, using today's containerd client library and API. Similar to how the Kubernetes CRI uses the containerd endpoints or how the Docker engine's libcontainerd operates, our small client will have access to all the same capabilities of container lifecycle management and registry interactions provided by containerd.
To finish our tour of building a fully functioning containerd client, we will pair our new sample application with LinuxKit and the Moby tool project. Using these tools, we'll build a simple virtual machine that embeds containerd and our sample client to test interesting aspects of containerd's capabilities in our own customized Linux OS image.
About Phil Estes
Phil is a Senior Technical Staff Member in the office of the CTO of IBM Cloud. Phil is a core contributor and maintainer on the Docker engine project where he has contributed key features like user namespace support and multi-platform image capabilities. Phil is also a founding maintainer of the containerd project, and participates in the Open Container Initiative (OCI) as a contributor to the development of runc/libcontainer.
Phil guides both IBM product teams and IBM's customers in applying container technology and concepts to their own cloud native efforts. Phil speaks regularly at industry conferences and meetups and enjoys helping customers and developers alike understand this fast growing ecosystem. Phil is a member of the Docker Captains program and maintains an active blog on container topics at https://integratedcode.us. You can find him on Twitter @estesp.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
So
that's
gonna,
be
the
focus
if
you
again,
if
you
want
to
hear
more
about
architecture
and
design,
Stephen
Day
who's
here
and
myself
have
given
two
or
three
talks
over
the
last
few
months.
At
least
the
Moby
summit
talk
is
online,
so
you
can
find
that
and
kind
of
get
a
deeper
dive
into
architecture,
but
again
I'm
one
of
the
maintainer
--zz.
We
have
a
couple
more
maintain
errs
here.
We've
got
maintain
errs
from
the
CRI
container
deep
project
here.
A
So
if
everything
goes
horribly
bad
up
here,
we
can
just
chat
around
about
container
D
together,
but
hopefully
you'll
get
something
out
of
kind
of
how
you
can
actually
use
container
D
in
your
project
for
your
use
case.
So
why
did
container
D
come
about
Michelle
covered
a
very
brief
highlight
on
that
earlier
in
the
keynotes.
A
Again
it
came
out
of
this
desire
across
the
ecosystem.
For
this,
this
boring,
stable
infrastructure
idea.
The
docker
engine
obviously
grew
with
a
lot
of
features
and
capabilities
that
wasn't
as
interesting
to
the
entire
ecosystem,
and
so
you
saw
lib
container,
which
was
originally
part
of
the
docker
engine,
got
spun
out.
That's
what
created
the
OCI
run
c
and
and
that
work
and
then
container
d
was
a
a
lightweight
management
layer
above
that
that
that
became
as
of
last
December,
a
much
more
ambitious
project.
A
So
again
that
the
intent
was
here
like
Craig
son
and
his
tweet,
let's
create
some
boring
core
infrastructure
that
everyone
can
use
usable
by
any
higher
layer,
container
native
orchestration
piece
and
then
more
recently,
along
with
that
year
ago,
promise
was
that
the
governance
model
would
move
away
from
btfl,
and
so,
as
of
a
few
weeks
ago,
the
mobi
technical
steering
committee
has
been
formed,
I'm
part
of
that.
If
you
want
again
more
information
on
that
I'm
in
the
middle
of
writing,
a
blog
post
about
that
that
should
be
out
soon.
A
But
again,
these
are
all
the
pieces,
boring,
cork
and
container
infrastructure
governance.
That's
open
contributed
to
CN
CF,
since
that's
really
why
we
have
container
D
today
who's
using
it
today,
there
are
a
few
kind
of
core
use
cases
already
using
it
actively.
We
we've
talked
to
a
lot
of
people
who
are
kind
of
earlier
earlier
on
which
I'm
not
going
to
talk
about
here,
but
hopefully
we'll
see
this
list
grow.
A
Obviously
a
very
important
use
case
is
CRI
container
D
implementing
that
container
runtime
interface,
so
kubernetes
can
use
container
D
as
its
couplet
runtime
Linux
get
the
project
out
of
the
the
mobi
project
for
a
minimal,
secure,
OS
image.
Assembler
uses
container
D
as
its
core
runtime
and
again
higher
layers
can
use
that.
A
So
that's
really
where
you
see
container
D
use
today,
as
I
said,
I'm
not
going
to
try
and
unpack
this
whole
architecture.
There's
a
couple
things
that
are
important
to
to
say
here:
one
is
the
that
the
container
D
architecture
really
applies
a
lot
of
the
learnings
from
the
history
of
the
docker
engine,
all
the
OCI
work.
So
a
lot
of
the
the
the
concepts
used
in
container
D
are
either
hey.
This
didn't
work
so
well
and
the
early
versions
of
the
docker
engine.
Let's
try
and
do
that
better.
A
So
the
snapshot
er
model
is
very
powerful.
The
graph
drivers
were
very
tangled
in
with
the
rest
of
the
engine.
The
snapshot
or
design
is
much
clearer
again.
If
you
want
more
on
that,
we
can
provide
more
resources
and
then
using
no
good
technology
again
using
the
OCI
runtime,
underneath
using
gr,
pc
exporting
metrics
with
Prometheus
and
not
opinionated
toward
a
specific
platform,
so
docker
engine
can
use
it
and
that
it's
opinionated
kind
of
capabilities
on
top
kubernetes
can
use
it,
but
again
I'm
gonna,
you
know
skip
over.
A
You
know
a
lot
of
the
detail
here
then
we're
going
to
jump
into
actually
using
the
API,
which
will
bring
out
some
more
of
these
components
in
a
more
you
know,
hands-on
fashion,
the
API
I
think
has
been
kind
of
one
of
the
shining
examples
of
what
container
D
has
done
to
apply
some
of
the
learnings
of
of
the
earlier
runtimes
one
was
to
have
an
API
that
was
simple
and
clear.
Had
stability
guarantees
again
using
G,
RPC
and
protobuf,
you
know
versioning.
We
can
provide
that
now.
So
container
d1o
is
out.
A
So
now
the
API
is
versioned.
You
can
guarantee
to
use
that
API.
We
have
CI
tests
that
make
sure
it
doesn't
break
and
it's
also
the
API
uses
a
past
your
model.
So
you
know
we're
not
trying
to
abstract
the
whole
OCI
json
configuration
so
that
if
it
changes,
then
we
have
to
change
so
it
gives
our
api
more
of
you
know
just
that
lightweight
metadata
and
passing
through
the
OCI
config
to
the
OCI
runtime,
and
then,
of
course,
we
have
the
go
doc
published
which
gives
you.
A
A
So
we
are
doing
fairly
well
on
time.
What
I'm
going
to
do
is
switch
to
an
editor
that
has
a
skeleton
of
what
I
want
to
do.
You
can
go
to
this
github
repository
example
CTR
under
my
github
ID.
You
can
see
the
finished
example
there,
but
the
part
we're
going
to
code
is
just
quite
simple:
we're
going
to
connect
to
the
daemon,
we're
going
to
pull
an
image,
create
a
container
and
start
the
task
I'm
also
going
to
set
a
namespace
again.
A
A
So
let's
try
this
out
so
here
I've
is
the
func
gonna
be
okay
for
folks
in
the
back?
Alright,
if
not,
there
are
some
seats
further
forward.
What
I'm
going
to
do,
first
of
all,
is
we're.
Gonna
have
to
connect
to
the
UNIX
socket
where
container
D
is
listening,
so
I'm
just
going
to
set
that
as
a
a
console
run
container
d
container
D
sock
is
going
to
be
my
default.
Address.
I'm
gonna
set
a
a
default
image
that
I'm
going
to
pull
and
run
again.
A
Container
d
is
not
opinionated,
so
it's
not
going
to
in
docker
hub
or
the
tag
so
I
have
to
provide
all
of
that
so
I'm
going
to
talk
to
dr.
hub
one
of
the
library
image
images
I'm
going
to
run
the
Redis
Alpine
tag,
so
those
are
just
some
defaults.
That'll
help
me
out
to
connect
to
the
daemon
very
simply
I
provided
the
address.
So
that's
the
address
I'm
going
to
connect
to
that's
going
to
return
to
me
a
client.
A
A
A
A
A
Will
do
contacts
background
and,
let's
just
call
it
example,
CTR,
and
that
will
return
the
context
that
I'm
now
going
to
use
for
all
the
rest
of
the
API
calls.
So
at
this
point,
I
want
to
pull
this
image.
That
I've
already
talked
about.
One
of
the
things
I'll
do
with
my
client
is
I'm,
first
going
to
say,
get
image
with
this
context:
the
the
name
of
the
image,
the
reference
to
it,
because
it's
possible
in
this
namespace
I've
already
pulled
the
image.
A
A
A
If
I
get
an
error
at
this
point,
we're
probably
enough
to
just
get
out
of
here,
because
that
means
I
couldn't
even
get
the
image,
couldn't
pull
image
and
the
error.
So
at
this
point,
I've
connected
to
my
client
I've
set
my
namespace
I
pulled
an
image.
The
next
thing
I
want
to
do
is
create
a
container,
and
this
again
is
fairly
straightforward
context.
A
Kind
of
a
container
ID
I'll
call
it
example
CTR.
Now
the
interesting
thing
is
I
am
a
set
of.
If
you
look
in
the
go,
dock
you'll
find
a
bunch
of
kind
of
with
helpers
container
D
width
and
one
of
the
things
we're
going
to
want
is
a
snapshot
I'm,
sorry
with
new
snapshot.
So
that
means
that
now
my
image
is
unpacked
in
the
snapshot
driver,
in
this
case
the
overlay.
So,
if
use
docker
graph
driver
the
overlay
driver,
that's
your
default
in
container
D.
A
A
The
next
thing
I
need
to
do
so
if
you've,
if
you've
looked
at,
run
C
or
the
OCR
runtime
spec,
you
know
that
to
run
a
container,
I
need
two
pieces
of
information.
I've
now
provided
kind
of
this
root.
Filesystem
with
a
snapshot
or
I,
also
need
a
OCR
runtime
config.
That's
the
JSON
file
that
has
you
know
what
what
program
is
gonna
be
run.
A
What
namespaces
Linux
kernel,
namespaces
I'm
gonna,
isolate
in
all
those
settings,
are
within
the
OCI
config,
that's
bound
up
in
the
image,
so
there's
a
nice
helper
that
I
can
use
to
I'm.
Sorry
I
actually
need
the
OCI
package
here
because
we're
going
to
basically
say
OCI
with
image
config
and
not
make
any
modifications
to
it,
and
so
we'll
just
be
this
Redis
Alpine
image
already
says
you
know
it's
configuration
is
in
in
its
image.
A
A
Yep,
okay:
well,
that
was
a
little
more
fun.
What
in
the
world
just
happened,
there
I
think
I
did
a
yeah
there
we
go
so
we
have
a
container.
That
was
a
little
more
exciting,
but
at
this
point
we
do.
Obviously
we
don't
have
a
task.
We
don't
have
a
runnings
there.
This
is
just
the
the
metadata
is
now
set
up.
A
A
Cio
is
contain
it's
not
about
the
the
person
in
your
company,
the
chief
information
officer,
it's
about
container
DS
I/o,
and
so,
when
I
create
this
task,
I'm,
basically
saying
I,
don't
care
about
standard
in
standard
error
standard
out
just
get
rid
of
it
for
me,
pipe
it
to
Devon
ole.
This
will
create
a
task
object.
A
A
A
A
And
actually,
we
should
really
delete
the
tasks
and
we
should
really
delete
the
container
at
this
point.
That
makes
it
easier.
We
can
obviously
go
this
one
I
always
forget
with
snapshot
cleanup.
Yes,
so
that
means
that
that
root
file
system,
I
assembled,
will
actually
get
cleaned
up
and
deleted.
So
I
don't
have
to
use
the
snapshot
tooling
to
go.
Do
that
and
I
really
should
have
done
this
as
well
here
just
to
make
it
cleaner?
A
Alright,
so
in
you
know,
sixty
lines
of
code
we've
connected
to
container
D
we've
set
up
a
namespace,
pulled
an
image,
create
a
container
created
a
task
and
started
it.
Who
believes
this
will
actually
run
any
votes
all
right
all
right,
so
I
called
this
file.
Skeleton
go
I've
already
vendored
in
everything
into
into
this
project.
Again,
you
can
see
this
on
github
under
examples.
Ctr
dot
go
I
didn't
want
to
check
in
the
blank
skeleton,
so
I
just
created
that
for
this
alright,
that's
built,
you
can
see.
I
already
have
container
D
running
here.
A
A
If
you
wonder
why
I
didn't
have
to
say
anything
about
the
namespace
I've
already
put
that
in
a
variable
in
my
environment,
so
container
D
underscore
namespace,
I
didn't
have
to
type
I
could
have
type
namespace
example
CTR
and
seen
the
same
container.
It's
running
the
Redis
Alpine
image.
If
we
look
at
tasks,
we
should
actually
see
that
there's
one
running
if
we
PS
on
this
system,
you'll
see
that
Redis
server
is
actually
running.
Ok,
so
we
wrote
a
container
D,
simple
client.
It
actually
worked.
It
started
a
container.
A
Hopefully
you
see
again
just
the
simplicity
of
the
API
and
how
simple
it
was
to
assemble
that
quickly,
I'm
going
to
stop
the
task
with
the
seat.
Again,
CTR
is
a
simple
command-line
tool.
It's
not
really
something
we
officially
see
as
part
of
the
container
D
promises
about
stability,
but
obviously
for
administrative
tasks.
It's
quite
easy
to
use
so
again,
I
can
stop
my
Retta
server,
so
I
can
kill
my
read
a
server
task
and
I
can
remove
that
container
and
you
can
see
that
it's
gone.
A
If
you
know
that
I've
worked
on
user
namespaces
and
various
container
runtime
contacts
you're
not
going
to
get
out
of
here
without
a
user
name,
space
example,
so
I've
added
that
too,
to
this
to
this
example,
I've
also
added
a
with
helper
in
my
own
code.
So
again
we
use
some
with
helpers
that
container
D
provided
us
now
I'm
going
to
add
my
own
with
helper
to
do
a
bind
mount
for
my
host
and
then
we
can
also
do
things
like
an
optional
custom
command
to
override
the
image
config.
A
So
you
know
if
you
do
docker
run
and
some
image
and
a
command
it
runs
that
command
instead
of
the
default,
went
in
the
image.
So
we've
added
that
to
this
example.
So,
let's
go
back
to
our
editor.
Let's
look
at
advanced,
I'll,
try
and
skip
over
the.
This
is
the
ugliness
that
gets
produced
when
you
don't
use
a
flagged
parsing
library,
I
thought
it
would
be
easy
and
then
it
got
really
ugly,
so
don't
follow.
A
This
example
use
a
nice
to
e
library
for
that
what
I've
done
is
I've
basically
said
when
you
call
my
example:
CTR
now
you're
going
to
provide
a
username
I'm
going
to
look
up
to
see
if
that
username
has
any
sub?
U
IDs
and
G
IDs.
So
if
you
know
anything
about
user
name
spaces
that'll
give
me
some
ranges
that
I
can
then
that
I
can
then
apply
to
the
Container
config
and
then
obviously
I've.
Let
you
specify
an
image
and
a
command.
A
You
can
see
here
I'm
using
a
package
that
I
just
copied
over
from
the
mobi
project,
that
I
wrote
when
I
originally
did
user
name
spaces,
so
the
ID
tools,
new
ID
mappings
just
does
that
look
into
sub
UID
and
sub
GID
and
gives
me
those
mappings.
A
bunch
of
this
is
the
same.
Obviously
connecting
setting
my
namespace
I've
created
a
small
object.
So
I
can
have
this
run
container
function
and
not
have
to
pass
around
all
the
client
and
context.
But
you
can
see
the
same.
A
Things
are
happening,
getting
an
image,
pulling
an
image,
creating
a
container,
creating
a
task.
Creating
a
task
got
a
little
more
interesting
because
now
I
actually
want
to
modify
the
OCI
config
to
add
in
the
user
name,
space
configuration
actually
in
this
case
sorry,
the
task
I
need
to
tell
run
see
by
the
way,
don't
don't
make
the
iOS
owned
by
root,
or
else
you
know
my
user
name.
A
Space
process
won't
be
able
to
read
that
we'll
look
at
new
container,
which
actually
does
the
other
thing
I
was
about
to
to
mention,
but
once
we've
done
that
all
we're
doing
is
creating
the
task
and
checking
for
exits
at
us,
because
now,
if
you
give
me
a
custom
command,
I
want
to
know
that
it
actually
ran
and
give
you
the
output.
So
again,
this
client
is
is
a
little
more
involved.
A
A
If
so,
let
me
grab
those
maps
and
pass
it
in
to
the
OCI
helper,
with
user
name
space,
and
so
again,
at
this
point,
I'm
modifying
that
images
default,
OCI,
config
to
add
the
user
name,
space
options,
and
then
again
we
passed
that
spec
on
and
actually
create
the
container
the
same
way
we
did
before
alright,
so
hopefully
that
wasn't
too
too
much
of
a
whirlwind
tour
through
that
again.
This
is
all
in
that
github
project.
So
you
can
look
at
this
at
your
own
leisure
I've
already
built
this
example.
A
Again,
I
need
to
sudo
that
estes
P
is
my
local
user.
I
know,
I
have
UID
maps
already
on
this
system,
for
that
I
can
provide,
let's
just
run
Alpine
latest
and
we
can
do
actually.
Let's
run
it
like
this
first
because
I
mean
it,
we
you
can.
Actually
we
can
verify
that
it's
running
in
a
user
name
space.
So
let's
try
this
out.
A
Yes,
you'll
find
out
in
a
minute
why
well
I
had
to
do
this,
but
this
container
D
path
was
modified
because
in
a
minute
I'm
going
to
run
this
inside
Linux
kit.
Hopefully
we're
going
to
have
time
to
wrap
that
up
and
I
had
to
run
a
separate
container
D
because
we're
working
on
a
bug
with
username
spaces
inside
Linux
kit,
so
I
had
to
point
it
at
a
different.
So
let's
try
this
again
alright.
So
now
we
have
a
running
container.
If
I
look
at
the
task,
there's
it's
pid'.
A
A
A
A
It
has
a
couple
more
tools.
You
could
build
out
your
client
a
little
more
with
stop
delete.
So
again,
you
don't
have
to
use
the
CTR
to
low
for
that,
but
here
I've
added
with
mounts
and
the
way
I've
done
this
just
for
demo
purposes-
has
actually
looked
for
a
file
in
the
local
directory
called
mounts.
I
expect
the
inside
it
I'll
have
a
target,
the
type
of
mount
like
mind
and
the
destination,
and
then
I'll
basically
modify
the
OCI
spec.
A
So
you
can
see
at
the
end
here
I'm
appending
a
new
mount
to
the
specs
mount.
So
this
is
my
own
with
mounts
helper
that
I've
created
so
now
my
client
has
the
ability
to
do
mounts
and,
let's
just
actually
I
think
I
already
had
a
mounts
file
here
in
the
local
directory
I'm
gonna
mount
Etsy
host
bind
mount
that
lets.
A
Let's
just
mount
that
over
home
in
Alpine,
so
you
can
make
sure
that
it
works,
and
so
at
this
point,
if
I
do
LS
home
with
this
mounts
file,
nope
yeah,
so
estes
p
was
actually
mounted
in
there.
Let's
do
home,
SSP
and
there's
a
couple
files
that
are
readable
to
the
user
namespace.
So
again,
it's
not
showing
a
ton
of
content
in
there,
but
our
filename
fred
was
located
in
that
folder.
A
A
And
now,
if
I
run
multiple
containers
with
different
mappings
I
can't
you
know
actually
come
up
with
a
with
privileges
and
and
permissions
that
make
sense
and
that's
pretty
frustrating
to
a
lot
of
user
name
space
interested
parties
who
want
to
use
it
in
a
broader
way,
and
so
what
I'd
like
to
kind
of
finish
with
is
that
Linna
skit
is
an
interesting
way
to
try
out
new
features
in
the
Linux
kernel.
That's
one
use
case
of
many,
and
so
there
is
a
shift
FS
patch
set
that
is
attempting
to
solve
this.
A
This
problem
of
mounts
inside
a
user
name
space,
and
so
what
I've
done
is
I've
already
I've
already
created
a
Linux
kit
assembly.
Again,
if
you
look
in
my
github
project,
I
don't
have
time
at
this
point
to
to
go
through
that.
You'll
see,
you'll,
see
the
Linux
kid
directory
and
you'll
see
a
yamo
file
in
there
called
shift
FS
and
that's
what
that's
what's
being
used
to
build
this
Linux
image,
there's
also
scripts
to
build
that
there.
A
Here
you
can
see
I'm
running
Linux
kit,
as
I
said
I'm
already
running
my
my
own
container
container
d
here
that
example
CTR
is
using
so
if
I
run
example
CTR
2.
Here,
if
you
look
in
that
yamo
file,
I
created
two
users
Bob
and
Alice,
they
have
different.
They
have
different
UID
mappings.
So
if
I
run
a
container
for
Bob.
A
A
You
can
see
that's
running
with
UID
200,000
now
what,
if
I
mount
the
same
volume
into
each
of
their
containers
or
bind
mount
a
director
from
the
host
they're
not
going
to
have
permissions
to
access
that
I
was
going
to
show
both
modes
where
I
couldn't
have
access
and
where
I
could,
but
we're
running
out
of
time
pretty
quickly.
So
let
me
kill
these
these
tasks.
A
And
just
remove
the
containers
just
to
make
the
output
clear
when
we
start
the
other
ones,
linux
kid
isn't
being
friendly
to
my
output
here.
But
that's
fine.
So
at
this
point,
I
should
have
no
containers
running
as
I
said.
The
example
ctr2
looks
for
a
mounts
file,
so
if
I
do
mounts,
shift
FS
and
copy
that
here
to
where
I'm
going
to
run
it,
and
we
look
at
this.
A
A
We
have
both
of
them
running
I've
got
a
simple
script,
so
I
don't
have
to
type
the
whole
command,
but
if
I
go
if
I
exact
into
those
containers,
so
I'm
going
to
go
into
I
think
this
is
Bob's.
The
first
one
I
started
and
I
look
at
home.
It's
empty
right
now,
but
if
I
touch
a
file,
I
can
actually
I
can
do
that
and
it
actually
looks
like
it's
owned
by
root
here,
we'll
go.
A
A
A
Let's
do
it
again
and
if
we
go
back
to
our
external
volume,
we
can
see
that
that
was
created
with
two
hundred
thousand
two
hundred
thousand
so
again,
I've
had
to
rush
through
this
part,
because
we're
basically
out
of
time
but
again
shift
FS
is
giving
us
a
way
to
do
that
without
being
blocked
by
the
permissions
shift.
Fs
needs
more
work
upstream,
but
Linda's
kit
gives
us
an
easy
way
to
play
with
those
features
and
with
my
container
D
client
I
can
actually
try
out
those
features.
A
So
a
few
final
slides
there
are
some
resources
here:
I'll
post
these
slides
and
as
soon
as
we're
out
of
this
talk,
if
you
want
to
see
more
interesting
uses
of
the
API
beyond
kind
of
what
I've
just
showed
you
look
at
CRI
container
D,
you
can
look
at
my
bucket
bench
project,
which
has
the
container
D
driver
using
the
same
API.
You
can
look
at
Lib
container
D
in
the
mobi
project.
These
are
all
users
of
the
API.