►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Deploying Unikernels in Production with Kubernetes - Alexander Jung, Lancaster University
In data center and cloud deployments, the virtual machine is the basic unit of isolation and Kubernetes has emerged as the de-facto standard for orchestration. This points to the container-in-a-vm model: the VM for isolation, and the container to seamlessly integrate with the orchestrator. But is this optimal for deployments where containers are not a requirements? Is it possible to use VMs for isolation, use Kubernetes and all of its wide array of features for orchestration, but without having to rely on containers? In this talk we will present KubeKraft, a novel runtime along with modifications to containerd that allow for running VMs with Kubernetes *natively*, without any container overhead. We will present KubeKraft's architecture, its integration with Kubernetes and will show performance results; for instance, when using a Unikraft NGINX unikernel (less than 2MBs in size) KubeKraft results in 2x the throughput of an official Docker NGINX image (42M image size).
A
Hi
everyone,
my
name,
is
alexander
young
and
I'm
the
cpo
at
unicraft,
and
I'm
also
a
phd
student
at
lancaster
university,
focusing
primarily
on
lightweight
virtualization.
I'm
very
honored
today
to
be
able
to
give
you
a
presentation
today
at
kubecon
on
some
of
the
work
we've
been
cooking
up
at
unicroft.
This
has
been
an
ongoing
effort
with
the
university
of
lancaster
university
of
polytechnic
bulcarest
and
nec
labs
europe,
and
today's
talk
it's
all
about
how
we
can
introduce
unikernels
into
the
ecosystem.
A
But,
let's
start
with
our
premise:
achieving
higher
cluster
utilization,
whilst
maintaining
performance
and
security
is
fundamental
to
service
operators
such
as
infrastructures
or
service
providers
or
devops
engineers
who
wish
to
decrease
operational
expenditure.
We're
actually
seeing
a
lot
of
great
work
in
this
space
recently
on
how
to
reduce
the
cost
of
services
running
in
the
cloud.
A
However,
virtualization
strategies,
such
as
with
containers,
have
gained
immense
popularity
thanks
to
orchestration
frameworks
such
as
kubernetes,
thanks
to
their
ease
of
use,
containers
and
kubernetes
have
been
adopted
as
the
de
facto
industry
standard
for
deployments
of
diverse
workloads
on
heterogeneous
infrastructure
deployments
vary
from
big
data
web
services
and
edge
computing.
Some
really
awesome
stuff
and
the
mix
is
enabled
in
vast
advancements
in
scheduling
strategies,
deployment,
architectures
and
workload
monitoring.
A
But
let's
take
a
look
at
what
a
typical
deployment
looks
like
with
kubernetes
and
dive
into
the
full
stack
to
see.
What's
going
on,
you
can
see
here
in
this
figure
that
what
may
resemble
a
typical
worker
node
in
a
cluster.
The
node
is
provisioned
in
a
cluster,
typically
as
a
virtual
machine
itself
running
on
top
of
a
hypervisor.
A
A
But
what's
the
problem?
Well,
there
are
actually
four
degrees
of
virtualization
and
indirection,
there's,
of
course,
the
hypervisor,
which
is
hosting
the
virtual
machine.
It's
keeping
the
vm
safe
from
other
vms
in
what
is
probably
the
most
secure
way
possible.
The
os
itself
is
managing
a
number
of
internal
system
processes,
along
with
the
container
runtime.
A
The
container
runtime
is
managing
the
lifecycle
and
managing
the
namespaces
and
isolation
of
all
the
services
and
all
the
pods,
and
in
some
cases,
the
container
parts
introduce
their
own
level
of
virtualization
separating
internal
services
to
achieve
a
functionality
of
the
application,
to
quote
anil
merhada
petty.
Are
we
really
doomed
to
adding
new
layers
of
indirection
and
abstraction
every
few
years,
leaving
future
generations
of
programmers
to
become
virtual
archaeologists,
as
they
dig
through
hundreds
of
layers
of
software
emulation
to
debug?
Even
the
simplest
of
applications?
A
Not
only
is
the
amount
of
indirection
between
the
different
levels
of
virtualizations
difficult
to
debug,
especially
for
mission,
critical
applications
or
performance
sensitive
applications,
but
it
adds
performance
penalties
every
step
of
the
way
it's
making
things
slower.
It's
making
things
slower
and
it's
adding
bloat
to
the
system,
meaning
that
we
have
to
scale
horizontally
to
new
workers
when
the
resources
of
the
machine
are
no
longer
adequate
for
the
service.
A
Another
thing
to
mention
is
that
the
system
pods
are
mingling
with
actual
service
pods
they're
only
separated
by
name
spacing.
There
have
been
some
work
to
make
the
separation
stronger
but
and
I'll
get
on
to
this
in
a
second,
but
it's
very
soft,
meaning
that
we're
relying
on
the
operating
system
to
ensure
this.
For
us,
we
rely
on
the
operating
system
to
be
secure
and
hardened
okay.
What
can
we
do
to
try
and
address
this
bloat?
A
Over
the
last
few
years,
we've
been
seeing
some
great
work
pop
up
to
creating
a
full
stack
operating
system
tailored
for
the
kubernetes
experience.
They
have
a
single
goal:
tailor
the
os
for
kubernetes
only
include
its
libraries,
services
and
daemons,
and
what
have
yous
to
make
kubernetes
run
and
they're
pretty
good
at
what
they
do.
We've
got
k3os
and
rhinzos,
which
creates
separate
daemon
processes
for
system
containers
and
service
containers.
We
also
have
some
tailoring
of
the
linux
kernel
from
the
folks
at
docker
who
build
it
only
only
the
necessary
kernel
components.
A
So
the
question
I'm
really
getting
at
is
whether
we
can
reduce
the
degrees
of
unnecessary
components
even
further
by
looking
directly
at
linux.
Linux
kit
does
this
to
some
extent.
I
actually
really
like
the
way
they
approach
the
problem.
It's
a
nice
tool
and
you
should
check
it
out.
If
you
can,
it
allows
for
very
bespoke,
builds
of
linux
kernel,
it's
really
cool
too,
but
it
suffers
from
the
same
premise
and
that's
the
linux
kernel
is
inherently
monolithic
and
it's
totally
understandable.
It
was
built
organically.
A
Over
the
last
30
years
it's
been
growing
and
growing
and
suiting
every
need
that's
been
thrown
at
it
to
increase
performance
or
adjust
to
new
use
cases
for
code
reuse.
It
has
very
strong
interdependencies
with
other
sub
components,
but
if
you
build
the
linux
kernel
from
scratch,
you'll
just
know
how
many
configuration
options
I'm
talking
about
thousands,
many
thousands,
but
turning
one
of
these
off
doesn't
necessarily
mean
the
whole.
Linux
is
excluded
from
one
another.
A
A
We
tried
to
quantify
this
api
complexity
in
the
linux
kernel
and
analyze
the
dependencies
between
the
main
components.
We
use
a
program
called
c-scope
to
extract
all
the
functions
from
their
sources
of
the
kernel
components
and
then
for
each
cool
check
to
see
if
there's
a
function
in
defined
in
the
same
component
or
in
a
different
one.
In
the
latter
case,
we
recorded
the
dependency
in
this
graph.
You
can
see
all
the
dependencies
in
between
different
kernel
components.
A
So
back
to
our
premise,
the
challenge
operators
face
with
containers
is
found
with
the
increased
desire
to
utilize
the
public
cloud
through
ease
of
use,
despite
its
negative
effect
on
cost
performance
and
security.
Actually,
there's
a
very
interesting
article
by
sarah
wang
and
martin
casado
on
the
paradox
of
the
cloud
which
goes
into
more
of
this
problem.
In
this
article,
they
essentially
discuss
the
ease
of
use
and
to
get
developers,
operators,
companies,
everyone
to
use
the
public
cloud
and
attempt
to
centralize
and
achieve
economies
of
scale,
but
it
isn't
quite
scaling
as
expected.
A
It's
very
easy
to
get
started
with
aws
google
compute
microsoft.
Azure
is
actually
quite
cheap
for
a
small
project,
but
as
soon
as
your
services
start
to
scale
so
does
the
cost
of
the
bill.
Naturally,
but
it's
not
linear,
it's
not
linear,
because
it's
not
enough
physical
infrastructure
specialized
or
ready
for
your
exact
use
case.
A
This
is
where
I
get
to
to
introduce
you
to
the
concept
of
a
unikernel,
I'm
very
excited
about
this
technology.
It's
been
on
the
radar
for
a
few
years
now,
but
largely
poop
of
concepts,
experimentation,
research
projects
etc,
but
unit
kernels,
in
contrast
to
containers
and
the
full
stack
full
os
stack,
are
an
increasingly
relevant
model
for
deploying
services,
as
they
serve
for
an
uncompromising
approach
for
gains
and
performance
by
tailoring
the
os
specializing.
The
application's
needs
and
stripping
away
unnecessary
functionality
right
to
the
core.
A
The
focus
has
been
largely
targeted
toward
memory
safety
through
single
address
space,
single
application
performance
through
a
specialized
language,
runtime
or
customization
of
os
primitives,
such
as
the
memory
allocator
and
scheduler,
reducing
latency
and
increasing
bandwidth
by
specializing.
The
networking
stack,
however,
until
now
these
efforts
have
fallen
short
when
considering
large-scale
heterogeneous
deployments,
with
overheads
experienced
in
real-world
context,
such
as
those
experienced
with
services
deployed
by
kubernetes.
A
Okay.
First,
let's
cover
some
basics
of
how
a
unikernel
is
built.
We
start
with
the
full
stack
application,
and
you
can
see
here
that
we
have
your
application
and
its
runtime
configuration
right
at
the
top
of
the
stack.
It
runs
as,
for
instance,
a
linux
user
space
program.
A
typical
go-to
example
is
say:
nginx
nginx
itself
requires
a
number
of
third-party
libraries
to
work.
For
example,
openssl
nginx
is
also
usually
dynamically
compiled
and
then
so
shares
libraries
that
are
installed
on
the
operating
system.
A
A
If
we
look
at
the
operating
system
and
its
requirements
its
needs,
and
it
needs
a
lot
of
functionality
itself
to
run,
it
runs
demons
services
like
ssh
and
also
relies
on
the
kernel
for
multitasking.
The
kernel
is
also
compiled
and
has
diverse
sorry
drivers
which
are
tailored
to
the
platform.
It
runs
on
like
kvm,
zen
or
vmware,
and
lastly,
it
has
hardware
specific
code,
machine
code
which
interfaces
with
the
processor,
such
as
with
intel
x86.
A
The
idea
of
the
uni
kernel
and
with
unicraft
in
particular,
is
that
we
libra
size
all
of
these
components
and
create
a
library
operating
system,
all
traditional
components
of
the
monolithic
operating
system.
The
kernel
and
all
third-party
libraries
are
made
available
in
a
pick
and
choose
fashion.
You
select
them
based
on
your
application's
needs.
A
You
don't
need
to
schedule,
because
your
application
is
run
to
completion,
no
problem,
but
not
only
that
we
can
pick
different
libraries
with
the
same
interface
which
interact
in
the
same
way
but
offer
different
functionality
I'll
cover
more
of
this
later,
the
unicroft
is
built
on
this
system
and
this
library
operating
system.
So
you
start
with
your
application.
Your
configuration
and
then
through
a
number
of
steps
such
as
retrieving
relevant
libraries,
preparation,
compilation
and
linking
you
get
a
final
binary,
a
full
virtual
machine
image.
A
It
only
has
the
necessary
components
for
the
application
to
run
and
nothing
more.
It's
tailored
exactly
as
your
application
needs
it
so
uni
kernels.
They
are
compile
time
specialization
strategy.
You
can
actually
very
easily
integrate
them
into
a
ci
cd
pipeline
and
generate
the
artifact
ready
for
deploying
deployment
and
to
the
relevant
platform
and
hardware
architecture.
A
They
are
a
form
of
lightweight
virtual
machine.
They
just
boot
like
any
other
os,
but
unlike
traditional
operating
systems,
they
boot
much
faster
I'll
cover
how
much
faster
a
little
bit
later.
The
application
and
the
kernel
share
the
same
address
space.
So
there
is
no
separation
between
the
two
there's,
no
user
space,
there's
no
kernel
space.
This
means
that
the
cost
of
cisco's
are
much
cheaper.
They're
actually
just
function
calls
now.
This
means
we
can
also
do
optimizations
on
their
call
making
them
in
line
when
possible
through
compile
time
optimization
strategies.
A
The
unikernel
only
has
the
necessary
functionalities
for
the
application
to
run
nothing
more,
which
means
that
there
are
no
daemons,
no
system,
libraries,
background
tasks
and
really,
you
know
no
ssh,
for
example,
oops
uni,
kernels
target
a
specific
platform
and
hardware
combination,
they're
tailored
for
not
only
the
application
but
for
their
runtime
environment
with
unicroft
nginx
has
only
the
necessary
dependencies
dependencies
for
it
to
run
networking
ipc
file,
system,
memory,
management
and
scheduling.
We
don't
need,
for
instance,
usb
drivers.
A
We
don't
need
iscsi,
we
don't
need
it
and
we
don't
and
if
we
know
the
file
system
that
we're
working
with,
we
only
need
to
compile
in
those
particular
drivers,
such
as
ext4,
9pfs,
etc.
So,
okay,
let's
check
out
some
numbers.
We
compared
the
ports
of
nginx
and
compared
the
performance
of
nginx
against
different
solutions,
including
a
full
vm
running
linux,
nginx
running,
of
course,
inside
and
other
unique
kernel
projects
and,
of
course,
docker.
A
A
We
also
measured
the
boot
time
of
unicraft.
Here
we
tried
different
virtual
machine
monitors
more
on
how
we
can
use
these
later
later
with
kubernetes,
but
we
can
also
achieve
boot
times
into
the
running
main
thread
program
in
as
little
as
3.1
milliseconds
and
just
for
fun.
I
ran
some
experiments
on
my
raspberry
pi
and
just
wanted
to
quickly
show
you
that
it
also
runs
directly
on
hardware.
A
There
was
an
order
of
magnitude
difference
in
boot
time
between
raspbian
os,
the
default
standard
os
that
ships
with
raspberry
pi's
and
the
unicropped
unicorn
image
unicraft's
build
system
can
tailor
it
to
different
platforms
and
architectures,
and
here
I
just
had
to
rebuild
the
nginx
application,
targeting
a
different
platform
and
hardware
architecture,
and
when
it
comes
to
the
separation
between
kernel,
space
and
user,
space
cisco's
represent
a
huge
bottleneck.
It's
understandable.
A
There
are
a
lot
of
checks
which
occur
to
ensure
that
the
user
space
application
does
not
make
incorrect
or
potentially
malicious
calls
to
the
kernel
which
can
disrupt
the
runtime
of
other
services.
If
our,
if
our
vm
is
running
in
the
same
address
space,
which
it
is
because
we
know
the
exact
exactly
what
we
want,
we
don't
need
this
mitigation
here.
A
We
demonstrated
that
a
function
call
is
two
orders
of
magnitude
faster
in
terms
of
cpu
cycles
at
unicraft,
I'm
going
to
dive
into
the
specifics
of
how
we
get
to
these
images
in
a
few
slides
yet.
But
I
want
to
show
you
that
when
it
comes
to
packing
images
in
an
oci
registry
ready
to
be
retrieved
by
the
relevant
worker
node,
we
can
often
try
to
minimize
the
file
system
as
much
as
possible
right.
There
are
some
classic
techniques
like
with
the
go
program.
A
You
often
create
the
final
static,
binary
image
and
you
place
it
in
a
scratch
and
it's
the
last
step
in
the
build
step,
and
you
just
hook
it
as
the
main
entry
program.
In
a
way
this
is
almost
exactly
what
we're
doing
with
uni
kernels.
Just
with
the
whole
os
and
kernel
to
give
you
an
idea,
however,
of
the
difference
between
grabbing
the
official
docker
image
for
nginx
is
4.2
megabytes
in
size,
but
our
complete
uni
kernel
image
for
nginx
of
the
same
version
is
1.3
megabytes
in
size.
A
A
The
vm
is
really
just
the
standard
units
unit
which
large
infrastructure,
as
a
service
providers,
offer
so
hardened.
It's
way
more
hardened
and
it's
more
hardened
than
the
os
that
they
actually
provide
to
you
and
the
provision
to
you.
Finally,
I'd
like
to
mention
that
at
unicraft
we
have
some
ongoing
work,
which
is
introducing
in
unikernel
memory
protection
using
hardware
acceleration
to
separate,
for
instance,
different
library
components
such
as
the
network
stack
and
the
scheduler
or
application
from
each
other,
for
instance,
by
using
intel
npk.
A
Okay,
we
have
uni
kernels
and
they
seem
to
provide
a
battery
of
different
benefits.
A
stepping
stone,
however,
towards
the
ultimate
use
of
lightweight
virtualization
via
uni
kernels,
requires
integration
into
an
orchestration
framework
which
maintains
this
ability
to
dynamically
and
quickly
provision
new
services
to
schedule
and
reschedule
services
based
on
workload
known
as
auto
scaling
and
collecting
co-locating
relevant
jobs
and
required
resources
such
as
static
volumes.
A
Kubernetes
provides
all
of
these
requirements
and,
as
such,
acts
as
an
ideal
tool
to
be
able
to
test
large
scale
deployments
for
uni
kernels
the
same
or
varying
comparable
workloads.
Our
main
goal
is
to
allow
for
the
seamless
use
of
unikernels
through
the
orchestration
platform
kubernetes
at
a
high
level.
We
envision
dynamic
workloads
scheduled
in
the
same
way
as
their
traditional
container-based
counterparts.
A
The
use
of
kubernetes
itself
is
fueled
by
its
widespread
adoption,
as
an
industry
leading
platform
for
the
scheduling
and
orchestration
of
services
and
two
make
it
easy
as
possible
to
achieve
the
first
goal.
We
want
to
make
everything
as
pluginizable
as
possible,
make
the
experience
seamless
attempting
to
make
little
to
no
modifications
to
existing
ecosystem
internals,
which
would
require
operators
to
make
non-trivial
require
modifications
to
the
host.
A
A
In
the
same
with
the
same
assumptions,
a
monolithic
os
running
multiple
services,
including
your
application,
actually,
one
of
the
way,
some
of
the
ways
that
we
interface
with
configuring,
these
vm
based
containers
through
kubernetes,
was
via
ssh,
and
we,
as
you
saw
before.
We
can't
do
this
with
uni
kernels,
because
it
boots,
and
then
it
starts
the
application
and
does
nothing
more
okay.
Let
in
the
next
few
slides
I'd
like
to
explore
with
you
the
necessary
steps
to
make
our
integration
with
uni
kernels
into
kubernetes
possible.
A
We
start
by
studying
how
kubernetes
as
a
whole
operates
and
how
kubernetes
talks
to
a
container
runtime
engine
via
something
called
the
container
runtime
interface,
the
cri,
which
is
implemented
using
a
tool,
for
instance
like
container
d,
and
how
the
container
runtime
engine
speaks
with
a
oci
or
open
container
initiative.
Runtime
image
or
a
specification
compliant
program
such
as
one
c
here,
is
a
general
overview
of
a
multi-node
kubernetes
cluster
with
a
controller
node.
The
api
server
speaks
with
all
worker
nodes
by
a
protobuf,
but
on
the
host.
A
A
A
We
introduce
a
new
plugin
to
the
container
d
and
for
a
program
which
matches
at
the
runtime
type
we've
seen
this
here
and
sets
the
path
of
the
binary
of
the
union
kernel
runtime
manager.
So
our
first
contribution
is
a
new
tool
run.
You
to
run
unicorns
to
be
oci
compliant.
This
program
must
essentially
have
the
following
sub-commands
and
arguments
available
to
it.
That's
create
delete,
kill
spec,
which
I'll
cover
in
the
next
few
slides
start
and
a
method
to
query
the
state.
A
Okay,
so
that's
all
well
and
good,
but
we're
still
dealing
with
oci
formatted
images.
This
means
we
must
disguise
unit
kernels
in
oci,
compatible
images.
The
ecosystem,
aka
container
d,
relies
on
the
use
of
remotely
accessible
registries
which
understand
the
contents
of
container
images,
standardize
again
through
the
oci
image
specification.
A
The
distribution
of
such
images
makes
the
instantiation
of
services
easier
as
their
content
are
remotely
accessible
and
can
be
transported
when
necessary.
Uni
kernels
are
typically
binary
format.
A
single
virtual
machine
image,
which
can
be
launched
via
virtual
machine,
monitor
such
as
camu
typical
standardization
for
virtual
machine
images,
are
iso
qca2
and
raw
format
and
are
not
compatible
with
existing
container-based
infrastructure
to
package
a
uni
kernel
into
oci
clothing.
We
must
follow
the
specification
dictated
by
oci
to
do
so.
A
A
These
layers
are
simply
file
systems
which
are
extracted
over
one
one
another
and
to
create
a
complete
file
system.
So
we
go
back
to
the
life
cycle
of
our
oci
image
and
once
it's
retrieved,
this
is
part
of
the
spec
command.
Before
we
unpack
it,
we
unpack
it
and
prepare
it
on
the
host
environment.
In
all
scenarios,
each
layer
is
extracted
atop
one
another
to
create
a
final
file
system
of
the
container.
In
our
case,
we
simply
need
to
extract
the
binary
image
of
the
unikernel.
A
A
So
here
we
have
a
manifest.yaml
file.
It
contains
a
specification
that
will
deploy
two
kubernetes
and
you
can
see
here
that
it
requires
the
run
u
class
and
pulls
the
relevant
image
from
our
registry
containing
the
packaged
uni
kernel.
So
when
we
run
it
it
launches
via
kubernetes,
you
just
do
apply.
We
can
see
that
we
created
a
new
pod
and
then
it
is
running
now.
If
we
inspect
this
pod,
we
can
just
follow
along
the
logs
you'll
see
it
boots
and
it's
now
running
unicraft.
A
This
is
running
the
nginx
instance
and
it's
been
allocated
an
ip
address
and
when
we
curl
it,
we
of
course
get
our
unicroft
image
with
the
payload
saying
powered
by
unicroft.
I've
just
opened
up
the
kubernetes
dashboard,
and
you
can
see
here
everything
is
sort
of
running
and
it's
appearing
as
usual
in
the
same
place
that
you
would
expect
cool
so
for
future
work.
There's
still
lots
to
do
and
we're
looking
forward
to
getting
this
release
very
soon.
A
So
there's
a
lot
of
information
about
how
the
unikernel
operates
before
it's
even
instantiated
that
we
can
make
use
of
as
we
boot
them
in
different
places.
And
so,
if
we
have
a
particular
file
system
baked
in,
for
instance,
we
can
use
the
different
metadata
there
to
locate
it
in
a
in
a
specific
place.
That
is
a
able
to
facilitate
the
runtime,
we're
also
starting
to
experiment,
with
new
scheduling
techniques
for
kubernetes
to
be
able
to
co-locate
unit
kernels
unit.
A
Another
thing
that
we
are
working
on
is
expanding
our
oci
registry,
with
matrices
of
builds
with
the
same
application,
but
for
different
applications
and
hardware
targets.
You
already
see
this
in
major
registries
for
oci
containers.
Docker
hub,
for
instance,
allows
you
to
download
a
different
architecture
depending
on
the
oci
or
depending
on
the
you
know,
the
host
platform-
that's
requesting
it,
so
you
can
get
an
arm
image
or
you
can
get
a
x86
image
for
instance,
and
so
we
want
to
tailor
that
experience,
but
to
a
lot
more
of
a
diverse
workload.