►
From YouTube: Container Isolation via Virtualization: Don't Forget to Shr... Dan Williams & Hsuan-Chi (Austin) Kuo
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Container Isolation via Virtualization: Don't Forget to Shrink the Guest - Dan Williams, IBM & Hsuan-Chi (Austin) Kuo, UIUC
Virtualization must be lightweight to be useful for improving the isolation of container runtimes (e.g., Kata containers): adding traditional (heavyweight) virtualization layers to container primitives would, for example, result in unacceptable boot time and performance for important use cases like serverless computing. Fortunately, the community has made great strides towards lightweight virtualization with new VM monitors (e.g., AWS Firecracker) and associated tooling (e.g., Weaveworks Ignite). However, there has been relatively little attention paid to the guest kernel itself, which remains unnecessarily bloated, affecting both performance and security. We will make the case for guest kernel specialization via kernel configuration and highlight key challenges in applying these techniques in a sandboxed container context.
https://sched.co/ZelG
A
A
But
of
course
you
know
the
the
vms
that
that
we're
seeing
happen
in
the
context
of
containers
are
not
like
the
virtual
machines
of
old.
In
fact,
the
virtual
machines
are
becoming
much
much
more
lightweight,
and
you
know
if
you
look
at
the
aws
firecracker,
they
don't
call
them
just
you
know
virtual
machines.
They
call
them
micro
vms,
to
kind
of
capture
that
lightweight
essence
and
so
to
dig
a
little
bit
deeper
into.
Why
and
how
vms
are
becoming
lightweight?
A
I
have
a
little
picture
here
on
the
slide,
which
will
which
represents
a
kind
of
standard
virtualization
stack.
So
at
the
bottom
you
have
a
host
kernel
or
a
hypervisor,
so
this
would
be
something
like
linux,
with
kvm.
On
top
of
that,
you'd
have
a
monitor
like
the
this
is
sort
of
like
the
thing
that
does
the
virtualization
and
uses
the
the
host
kernel
and
hypervisors.
A
The
important
thing
here
is
that
there's
a
very
low
level
of
abstraction
between
the
virtual
machine
and
the
host
software
below,
and
this
is
where
the
isolation
and
the
security
from
virtual
machines
is
coming
from
inside
the
virtual
machine.
You
typically
have
a
guest
kernel,
so
normally
linux
and
your
application
finally
runs
on
top
of
that.
So,
given
that
huge
stack
that
we
just
we
just
showed
here,
there's
there's
no
wonder
that
that
vms
have
this
reputation
for
being
very
heavyweight
now.
A
So
the
first
way
to
get
vms
to
be
more
lightweight,
is
to
look
at
the
monitor
piece
and
try
to
make
that
thinner.
So,
and
this
is
what
is
is
done
by
aws
firecracker,
for
example,
they
take
that
piece.
The
qmu,
for
example,
and
cut
out
things
like
pci
and
reduce
complexity,
implement
it
in
a
very
lightweight
way
and
they
end
up
with
much
better
performance.
A
So
I
have
a
graph
here
that
that
copied
from
a
paper
on
firecracker
from
nsdi
2020,
and
what
you
see
is
that
firecracker
can
boot
basically
twice
as
fast
as
qmu,
and
this
type
of
this
type
of
performance
improvement
is
not
is
not.
This
type
of
you
know.
Observation
is
not
new.
As
far
back
as
2017,
there
was
a
paper
in
in
20
in
sosp.
A
So
if
you
see,
even
though
we
have
a
thin
process
a
thin
monitor
now
we
still
have
a
pretty
heavyweight
guest
there
and
there
has
been
some
work
to
make
the
guest
lighter
both
in
user
space
for
the
guests,
so
going
from,
let's
say
an
ubuntu
container
to
an
alpine
lace
container
and
also
through
kernel
configuration
for
that
guest
kernel.
Things
like
tinyx,
which
was
from
that
same
ssp
paper
that
we
mentioned.
But
the
question
that
we
asked
here
is
in
this
context:
how
thin
can
you
go?
A
So
a
unikernel
is
an
application
that
is
linked
only
with
those
library
os
components
that
it
needs
to
run
directly
on
the
virtual
hardware
abstraction.
So
you
can
think
of
this.
As
a
virtual
machine
with
no
guest
os
live
linux
inside
and
just
the
application
and
whatever
library
components,
it
needs
to
run
the
original
uni
kernels
were
language,
specific,
so
mirage
os,
for
example,
used
ocamel.
A
But
since
then,
there's
been
a
lot
more
kind
of
legacy,
oriented
unicorns
things
like
rump
run,
which
is
based
on
that
bsd,
as
well
as
hermatox
and
osv,
which
are
which
claim
binary
compatibility
with
linux,
and
so
these
properties
that
uni
kernels
have
so
things
like
small
kernel,
size,
fast
boot
time,
performance
security.
These
all
seem
like
things
that
we
want
and
those
are
great
properties.
A
However,
there's
a
big
downside
to
the
uni
kernels,
which
is
the
lack
of
full
linux
support,
so,
for
instance,
hermitage
is
only
supporting
97
system
calls.
Osb
has
a
laundry
list
of
kind
of
criteria
that
your
application
has
to
obey
in
order
to
be
able
to
be
run
on
it,
and
this
leads
these
these
unicorn
communities
to
have
to
kind
of
maintain
and
curate
applications
to
run
on
their
platforms.
A
B
Hello,
so
the
idea
is
lupine
is
that
we
take
an
application
container
and
we
specialize
the
kernel
just
for
the
application.
So
we
apply
unicorn-like
technique
to
specialize
the
linux
source
and
get
our
new
kernel,
that's
called
lupine
and
for
the
fix
that
is
running
along
with
the
kernel.
We
take
the
application
container
image
and
then
directly
translate
that
into
a
disk
that
can
be
used
in
the
virtual
machine.
B
So
unique
kernels
are
all
about
specialization,
as
definition
unit
kernel
includes
only
what
is
needed,
and
surprisingly
linux
is
very
configurable.
You
get
more
than
6000
configuration
options
from
the
k-config,
and
the
options
include
everything
you
need,
including
drivers,
file
systems,
network
protocols
and
power
features.
B
So
we
specialize
linux
through
configuration
and
our
starting
point
is
firecracker
macro
vm
configuration
which
takes
roughly
about
five
percent
of
the
configuration
space,
but
lupine
removes
more
options.
So
first
we
classify
283
options
that
is
essential
for
the
vm
to
work,
and
we
call
these
options.
Lupine
base
and
remaining
options
are
either
application,
specific
or
potentially
unnecessary.
B
So
those
remaining
options
are
application,
specific
and
potentially
unnecessary,
because
some
of
them
are
modern
processing
and
some
of
them
are
about
hardware
management
and
the
reason
I
said
it's
potentially
unnecessary
is
that
we
only
we.
We
can
only
answer
this
question
until
we
answer
these
two
us
two
questions.
B
The
first
question
is:
do
we
need
to
support
for
multiple
trust
domains
since
it's
a
lightweight
vm
and
there's
only
one
container
and
application
are
going
to
run
on
rpm,
so
options
relating
to
isolating
and
accounting
for
processes
might
not
be
necessary,
such
as
c
groups
and
name
spaces
and
other
kernel.
Security
features,
and
the
other
question
is:
do
we
need
to
support
for
general
hardware,
since
it's
a
virtual
machine
and
virtual
machines
are,
by
definition,
going
to
run
a
hypervisor?
B
B
So
for
application
specific
options,
I
can
give
you
some
examples.
The
first
obvious
one
is
system
codes,
so
some
system
codes
that
you
can
enable
or
disable
and
make
it
optional
in
your
kernel.
But,
however,
if
your
application
happens
to
use
those
system
codes,
you
need
to
make
sure
they
are
in
your
kernel.
So,
for
example,
nginx
server
means
eventfd,
because
it's
an
event-driven
framework
and
I
use
e-paw
and
event
of
the
system
calls,
and
there
are
other
features
like
kernel
services.
B
B
B
B
B
We
see
that
the
configuration
is
very,
very
effective
because
lupine
is
already
is
only
four
megabytes
and
it's
only
27
of
the
microvia
in
size
and
lupine
general.
The
one
that
supports
20
applications
is
only
slightly
larger
than
the
loop
pi
and
it's
still
much
smaller
than
micro
vn
and
when
comparing
to
uni
kernel
systems,
lupine
general
is
still
smaller
than
some
unique
kernels,
including
hermitage
and
osb,
and
for
bhutan.
B
B
B
So
that's
why
we
escape
the
evaluation
and
so
for
application
performance.
We
measure
radius
and
nginx,
and
the
throughput
is
normalized
to
micro
vm,
and
we
see
that
lupine
outperforms
micro
vm
by
up
to
29
percent
and
surprisingly
lupine
general,
is
just
robot
general,
just
as
good
as
lupine
and
some
and
for
the
other
unicorn
systems.
A
B
And
here's
some
related
work
that
people
trying
to
leverage
linux
to
achieve
some
unique
kernel
benefits,
including
live
vm,
x,
containers
and
ukl,
and
there
are
some
studies
about
how
you
can
how
much
security
benefits
you
can
get
by
configuring,
your
kernel.
So,
for
example,
you
see
that
oh
85
percent
of
the
attack
surface
can
be
reduced
via
configuration.
A
Okay,
great
thanks
austin,
so
just
to
kind
of
give
a
bit
of
a
summary
about
some
of
the
some
of
the
key
takeaways
that
we
get
from
that
evaluation
that
that
austin
showed
us.
The
first
one
is,
that
is
that
specialization
is
really
important.
So
if
you
remember
what
we
were
talking
about,
which
was
that
we
want
to
have
this,
this
guest
vm,
as
well
as
the
monitor,
be
as
lightweight
as
possible.
So
we
can
get
these
properties
that
are
so
important
for
container-like
workloads.
A
Specialization
is
really
key,
so
what
we
found
was
that
we
could
get
73
smaller
image,
size,
59,
faster
boot
time,
28,
lower
memory
footprint
and
33
higher
throughput
than
the
state-of-the-art
micro
vm.
So
what
this
means
is
that,
even
though,
in
the
context
of
of
aws
firecracker,
the
kernel
has
already
been
slimmed
down
to
something
that's
fairly
small,
even
with
that
there's
a
lot
of
performance
that
is
left
on
the
table
that
that
we
could
possibly
take
through
specialization.
A
The
other
point
that
I
wanted
to
make
was
that,
even
though
specialization
is
important,
specialization
per
individual
applications,
which
is
what
we
were
showing
in
that
evaluation,
what
austin
was
showing
us
specialization
per
that
each
application
may
not
be
quite
so
important,
and
this
is
actually
really
good
news.
What
this
means
is
that
this
process
of
trying
to
figure
out
what
exactly
the
configuration,
the
most
lightweight
configuration
for
each
application
is,
may
not
be
necessary.
A
Today
is
how
do
we
get
these
special
specialization
benefits
into
the
community?
So
we
know
what
they
are
now:
we've
measured
them
we've
seen
them
and
we
want
to
be
able
to
get
those
into
the
micro
vm
community
so
that
our
vms
are
even
less
like
that
turtle
that
we
saw
at
the
start
and
and
more
like.
I
don't
know
the
the
future.
I
guess
so,
but
unfortunately
there
are
a
number
of
challenges
that
we
do
not
yet
know
how
to
solve.
A
So,
the
first
one
is
how
how
we
know
whether
or
not
this
lupine
general
configuration
is
general
enough
for
the,
for
you
know
a
wide
variety
of
workloads,
so
I
mentioned
that
it
was
a
great
thing
that
we
didn't
have
to
do
the
per
application
specialization.
But
there's
always
this
this
question
of.
If
we
give
you
a
specialized
configuration,
will
it
work
for
your
workload
and
we
don't
know
how
to
guarantee
that.
A
So
you
know
any
kind
of
ideas
and
and
things
things
like
that
from
the
community
would
be
very
happy
to
have
either
you
know
this
could
go
in
a
few
different
directions.
It
could
go
towards
trying
to
discover
exactly
what
requirements
from
the
from
the
guest
kernels
are
needed
versus
some
fallback
measure
mechanisms.
What
do
you
do
if
you
do
over
specialize?
I
think
there's
a
lot
of
opportunity
there
for
for
future
work.
A
Like
features
may
be
pushing
towards
a
more
general
configuration,
whereas
the
specialization
that
we
were
advocating
for
is
pushing
towards
less
of
a
system-like
configuration
and
more
of
a
you
know,
specialized
application
like
configuration.
So
I
think
that
there's
this
kind
of
tension
here
that
that
is
is
not
not
super.
A
A
So
I
just
wanted
to
share
a
couple
of
ideas
that
we've
had
related
to
how
how
you
could
go
about
trying
to
do
this.
So
I
think
one
thing
is,
if
you
think
about
perhaps
somehow
tying
this
into
like
a
ci
cd
environment
or
something
like
that,
it
might
be
possible
to
automatically
try
different
kernel
configurations
in
your
micro,
vm
and-
and
just
you
know
see
if
your
test
cases
are
are
actually
happening.
A
I
don't
know
of
people
who
are
actually
doing
this
in
practice,
but
I
think,
as
as
we
look
at
specialization
more
especially
for
micro
vms,
I
think
it's
going
to
become
more
important.
The
other
idea
that
could
happen
is
having
some
kind
of
like
reactionary
approach.
So
it's
possible
that
at
run
time
you
could,
you
could
actually
switch
out
to
a
more
more
general
version
of
the
kernel.
A
The
implications
on
performance
on
security
are
a
little
bit
questionable
there,
but,
but
that's
that
that's
another
case
and
then
finally,
a
third,
a
third
idea
here
is
that
you
could
potentially
take
something
that
you
already
have
about
the
application.
So,
for
example,
if
you
have
a
setcomp
profile
which
says
you
know,
basically
my
application
is
no
longer
going
to
use
these
system
calls.
You
could
use
that
also
as
a
way
to
guide
your
specialization
of
the
kernel
to
some
extent.
A
B
B
So,
for
example,
your
application
does
not
need
security
features
like
kpti,
but
if
you
use
those
kind
of
tracing
approach
you
get
like
whatever
that
is
used,
but
not
necessary,
but
not
essentially
necessary.
So,
yes,
you
can
do
that,
but
for
the
purpose
of
lupine,
we
want
to
specialize
the
kernel
to
that
extreme.
So
that's
why
we
choose
this
manual
way.
A
Yeah
yeah,
it's
a
good
point,
because
so
some
of
the
things
when
you're
specializing
are
are
you
know
just
functionality.
Things
like
you
know,
inclusion
or
not.
Inclusion
of
a
system
called
implementation
is,
is
a
very
kind
of
straightforward
thing
that
may
or
may
not
work.
You
know
inclusion
or
not,
inclusion
of
the
the
pocketfest
or
the
cfs.
For
example.
I
mean
these
things,
are,
you
know
they're
very
binary?
They
either
work
or
they
don't.
A
But
there
are
some
other
things
that
austin
was
alluding
to
here,
which
are
a
little
bit
more
complex
or
more
difficult
to
think
about.
So
it
might
be
that
you
want
a
particular
file
system
that
has
certain
properties,
but
it's
under
the
same
file
system
api
as
some
other
file
system.
So
if
you
go
from
ext4
to
gxt2,
esd2
is
potentially
a
simpler
file
system,
but
you
might
be
losing
a
lot
of
the
benefits
from
the
xd4
without
even
realizing
it.
A
So
these
are
some
of
the
other
kind
of
nuances
that
go
on
into
some
of
these
decisions,
and
especially
like
you
know,
you
know
this
ext
example
might
be
a
little
bit
contrived,
but
especially
when
you
start
thinking
about
security
and
security
options
that
you
may
want
in
the
kernel
that
are,
you
know,
enabled
or
disabled,
but
you
can't
necessarily
your
application
works
either
way.
It's
just.
Maybe
it's
working
in
a
less
secure
environment.
B
A
Yeah
so
there's
another
question
on
here
about
using
bpf
trace
to
record
the
system,
calls
and
the
open
files.
So
I
I
don't
know
I
don't
know
about
you
austin,
but
I
haven't,
I
haven't,
haven't
looked
at
bf
bpf
trace
at
all.
I
think
that
in
in
general
I
think
that
austin
had
made
you
know
kind
of
a
comment
about
tracing
like
approaches
and
the
fact
that
you
know
you're
always
gonna
get
into
this.
A
This
situation,
where
you're
not
sure
if
the
testing
that
you're
tracing
is
the
same
as
what
you
actually
need,
but
it
it.
You
know,
I
think,
the
better
the
tracing
framework,
the
better
the
more
information
you
can
get-
and
you
know
if
bpf
trace
is
a-
is
a
way
that
we
can
have
more
robust
tracing
or
more
informative
tracing
in
some
ways.
That
could
be
very
interesting.
A
Let's
see
so
there's
this
other
one
down
here.
Do
you
think
it's
possible
to
automate
the
process
of
creating
a
kernel
for
a
specific
container?
Besides
system
call
inclusion?
Surely
there
are
other
type
of
configurations
that
could
be
required
by
apps
yeah,
so
I
mean,
like
you
know.
I
think
that
we're
you
know
most
of
the
questions
are
around
the
same
sort
of
basic
idea,
which
is
like
you
know.
A
I
think
that
what
we've
got
so
far
is
we
know
that
that
we
have
some
evidence
now
that
that
that
getting
the
getting
this
kind
of
specialized
kernel
or
specialized
micro,
vm
image
is,
is
going
to
be
valuable
in
some
way.
But
you
know
the
questions
are
absolutely
right.
The
the
big
challenge
here
is
is
how
how
can
you
get
these
things
automatically?
A
I
think
one
other
point
to
make,
though,
that
to
drive
home
a
little
bit,
is
you
know
the
degree
of
specialization
that
really
seems
to
matter,
at
least
from
our
our
evaluation?
That
we've
done
in
this
work
is
is.
Is
you
don't
need
specialization
that
is
so
fine-grained
that
you
know
you
need
to
know
every
single
little
system
call
and
and
do
those
types
of
things,
this
type
of
specialization
that
we're
talking
about?
A
If
you
remember
the
the
slide
that
that
austin
showed
the
lupine
general,
which
was
this
configuration
which
had
basically,
it
was
the
union
of
all
these
different,
smaller
configurations
that
we
had
together.
So
in
some
sense
it
was
a
much
more
general
purpose
specialized
kernel,
it
was
still
very
specialized,
but
it's
more
general
than
individual
ones,
and
that
that
particular
one
showed
us
that
that
maybe
fine
grain
specialization
isn't
really
as
important
as
we
may
have
originally
thought.
A
So
I
think,
when
we
think
about
some
of
these
tools
to
automate
how
to
get
the
specialization,
I
think
that
it's
important
to
to
remember
that
that
we
don't
need
to
go.
You
know
we
don't
need
to
go
too
crazy
with
with
how
specialized
we're
looking
if
it's
a
pretty
coarse,
grained
hammer.
It's
probably
it's
probably
good
enough
yep,
all
right!
A
A
I
think
we're
at
the
end
of
the
questions,
so
thank
you
all
for
for
coming
to
the
to
the
session
and
we're
gonna
be
hanging
out
on
the
the
runtime
slack
yeah
and
we'll
be
there
for
for
a
little
while.
So
if
you
have
further
questions-
or
you
want
further
discussions,
we're
more
than
happy
to
to
hear
your
thoughts.
So
thank
you
and
have
a
nice
day.