►
From YouTube: In a Container, Nobody Hears Your Screams: Next Generation Process Isolation - Andrew Martin
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
In a Container, Nobody Hears Your Screams: Next Generation Process Isolation - Andrew Martin, ControlPlane
Is it ever safe to run untrusted code in containers? Should process isolation keep workloads in, or attackers out? What would it take to run a malware test lab in Kubernetes? With fast startup times and consistent execution environments containers beat traditionally slow, monolithic VMs -- but with the advancement of micro VMs the boundaries have become blurred. It is increasingly difficult to know which isolation technology to choose for our next application. Can we run different workloads in different “container” types -- on the same cluster? In this talk we: - examine the history of trying to safely run unsafe processes - compare and contrast the emerging generation of process isolation and security techniques - rationalise the design decisions that drive each project - demo how to break in, out, and learn about what workloads are best suited to run in each technology
https://sched.co/Zerb
A
I
am
andy,
I
love
to
break
stuff
and
then
put
it
back
together
again
once
I
know
what
it
is
made
of,
and
I'm
also
very
proud
to
be
an
official
trainer
and
occasional
course,
author
for
sans
watch
out
for
the
upcoming
sec,
584,
hashicorp,
docker,
o'reilly
and,
of
course,
control
planes.
Detailed
and
extensive
low-level
cloud-native
security
courses
and,
as
you
may
recognize
from
the
title,
this
talk
contains
aliens,
I'm
a
founder
at
control,
plane
secops
for
kubernetes,
focusing
on
regulated
industries,
including
financial
services.
A
A
However,
like
all
software,
it
has
had
exploitable
vulnerabilities
and
advanced
sandboxing
and
isolation
techniques
have
developed
to
mitigate
the
risk
of
unknown
existing
or
future
vulnerabilities,
while
these
techniques
may
also
protect
against
kubernetes
misconfigurations,
robust
software
and
infrastructure
delivery
pipelines
are
a
less
complex
way
of
managing
that
risk
for
high
sensitivity,
infrastructure,
workloads
or
data.
A
new
generation
of
sandboxing
technologies
has
emerged
sometimes
referred
to
as
micro
vms.
A
The
linux
kernel
is
written
in
c,
a
language
with
security,
bugs
that
have
proven,
notoriously
difficult
to
entirely
eradicate
virtualization
can
be
thought
of
as
a
way
to
abstract
a
process,
as
far
from
the
linux
system
call
interface
as
possible
to
reduce
the
chance
of
exploitable
vulnerabilities
using
an
assortment
of
lsms
and
kernel
modules,
containers
and
capabilities,
dedicated
software
and
hardware,
assisted
virtualization,
use
of
a
use
of
a
type
safe
language
for
the
virtual
machine.
Security
such
as
golang
or
rust
reduces
the
likelihood
of
bugs
much
more
difficult
to
avoid
in
software
programmed
in
c.
A
A
It
may
also
be
the
algorithms
within
including
quant
trading,
algorithms,
machine
learning
models
or
an
organization's
secret
source,
so
an
untrusted
workload
is
software
that
cannot
be
certified
as
safe
to
run.
That
is
a
human
has
not
cryptographically
signed
it
with
a
gpg
or
similar
trusted
thing.
A
Untrusted
workloads
may
also
include
software
with
published
zero
days
or
cves.
If
no
patch
is
available
and
the
workload
is
business,
critical
isolating
it
further
may
decrease
the
potential
impact
of
the
vulnerability
if
exploited
a
container
image.
Build
can
be
considered
untrusted
if
its
desired
contents
are
unknown,
as
it
may
contain
malicious
code
to
scan
network
endpoints
exfiltrate
code
in
the
container
or
attack
the
host
container
images
and
code
from
unauthenticated
external
sources
may
also
be
considered
untrusted
for
risk-averse
organizations.
A
A
sensitive
workload
is
one
whose
data
or
code
is
too
important
to
permit
unauthorized.
Access
to
this
may
include
fraud,
detection
systems,
pricing
engines
or
high
frequency
trading,
algorithms
on
a
kubernetes
cluster,
a
single
workload
or
an
entire
tenant
may
be
considered.
Untrusted
mitigation
with
kubernetes
native
solutions,
such
as
admission,
control
and
network
policy,
may
be
sufficient
in
some
cases,
but
for
advanced
isolation
from
unknown
vulnerabilities
in
the
linux
kernel
and
the
run
c.
Demon,
sensitive
or
untrusted
workloads
may
be
isolated
further
with
next
generation.
Sandboxing.
A
Interfacing
with
the
kernel
and
creating
that
isolated
containerized
process,
a
major
difference
between
a
container
and
a
vm
is
the
containers,
do
not
use
hardware,
assisted,
virtualization
and
so
are
more
widely
compatible.
But,
of
course,
containers
do
not
really
exist.
They
are
merely
a
user
space
fiction.
A
A
A
A
So
what
is
wrong
with
containers?
Containers
are
not
inherently
insecure,
but
for
some
workloads
they
leak.
The
host's
abstractions
too
much
underlying
concerns
about
a
root
owned.
Daemon
can
be
assuaged
by
running
rootless
containers
in
unprivileged
username
space
mode,
but
this
introduces
another
risk.
User
name
spaces
have
historically
been
a
rich
source
of
vulnerabilities.
A
Linux
was
originally
written
with
a
lot
of
code,
assuming
that
it
was
being
run
by
root
in
the
host
namespace,
and
some
of
these
assumptions
have
changed
with
the
introduction
of
username
spaces.
The
last
kernel
namespace
to
be
completed,
so
the
answer
to
whether
it
is
riskier
to
run
a
root
owned,
daemon
or
user
namespaces
isn't
entirely
clear.
A
A
A
This
allows
the
route
in
user
namespace,
which
is
unprivileged
in
the
host
namespace
to
create
new
containers.
To
achieve
this,
additional
work
must
be
done.
Network
connections
into
the
host
network
namespace
can
only
be
created
by
the
host's
routes
so
for
rootless
containers,
an
unprivileged
slurp
for
net
ns
guarded
by
setcomp
into
kernel,
is
used
to
create
a
virtual
network
device.
A
Capnet
bind
service
is
required
by
the
kernel
to
bind
to
ports
below
1024,
which
is
historically
considered
a
privileged
boundary
and
ping
is
not
supported
for
users
with
high
uids
if
the
id
is
not
in
proxis
net
ipv4
ping
group
range,
but
this
can
be
changed
by
the
host
root
user
host.
Networking
is
not
permitted
as
it
breaks
network
isolation
and
cgroup's
v2
function,
but
only
when
running
under
system
d
c
group
v1
is
not
supported
by
either
rootless
implementation.
A
So
docker
and
moby
and
podman
share
much
of
the
same
code
for
rootlessness
and
have
been
developed
in
parallel.
They
share
similar
performance
and
features.
Although
docker
has
an
established
networking
model
that
doesn't
support
host
networking,
whereas
podman
reuses
kubernetes
cni
interface,
while
rootless
containers
protect
the
host
from
the
container,
some
abstractions
may
still
leak
from
the
host,
although
they
become
much
less
dangerous.
For
example,
proc
host
devices,
the
kernel
interface,
so
is
this
exposure
to
c-based
system
calls
in
the
linux
kernel
from
a
rootless
container,
runtime
bad?
A
Well,
the
kernel
of
linux
powers,
the
internet
and
the
world,
and
has
done
so
for
decades,
but
its
lack
of
memory
management
leads
to
the
same
critical
bugs
over
and
over
again
and
when
the
kernel
opens.
Ssl
and
other
critical
software
are
written
in
c.
We
just
want
to
move
everything
as
far
away
from
trusted
kernel
space
as
possible.
A
The
host
machine
is
split
into
smaller,
isolated
compute
units
traditionally
referred
to
as
guests.
These
guests
interact
with
a
virtualized
interface
to
the
host's
cpu
and
devices,
and
the
interface
intercepts
system
calls
to
handle
them
itself
by
proxying
to
the
host's
kernel
or
using
its
own
code
to
handle
the
request.
A
Full
virtualization,
for
example,
vmware
emulates
hardware
and
boots
a
full
kernel
inside
the
guest
operating
system
level,
virtualization,
for
example,
a
container
emulates,
the
host's
kernel
using
namespaces
c
groups,
capabilities
and
setcomp,
and
so
can
start
a
containerized
process
directly
on
the
host
kernel
processes
in
containers
share
many
of
the
kernel
pathways
and
security
mechanisms
that
processes
in
vms
also
execute
to
boot.
A
kernel.
A
It
should
be
noted
that,
despite
many
decades
of
effort
in
practice,
no
virtual
machine
is
completely
equivalent
to
its
real
machine
counterpart
inside
each
guest.
Virtual
machine
is
an
environment
in
which
processes
or
workloads
can
run.
The
virtual
machine
itself
is
owned
by
a
privileged
parent
process
that
manages
its
setup
and
interaction
with
the
host.
A
A
Linux
has
a
built-in
virtual
machine
manager
called
kvm
that
allows
a
host
kernel
to
run
virtual
machines
along
with
qmu,
which
emulates
physical
devices
and
provides
memory
management
to
the
guest
and
can
also
run
by
itself.
If
necessary,
an
operating
system
can
run
fully
emulated
by
the
guest,
os
and
qmu.
A
This
emulation
narrows
the
interface
between
the
vm
and
the
host
kernel
and
reduces
the
amount
of
kernel
code.
The
process
inside
the
vm
can
reach
directly.
This
provides
a
greater
level
of
isolation
from
unknown
kernel.
Vulnerabilities,
different
technologies.
Take
different
approaches
to
moving
away
from
linux
system
call
interaction
for
the
guest
linux
containers
are
the
most
lightweight
form
of
isolation
as
they
allow
workloads
to
use
kernel
apis
directly.
A
A
Finally,
it
also
implements
its
own
user
space.
Networking
stack,
conversely
firecracker,
while
also
using
kvm
instead
of
implementing
the
heavyweight
qmu
process
to
emulate
devices.
As
a
traditional
linux,
virtual
machine
might
do
it
starts
a
stripped-down
device
emulator.
Instead,
this
reduces
the
host
attack
surface
and
removes
unnecessary
code
requiring
only
36
system
calls
for
itself
to
function.
A
Visor,
firecracker
and
cata
containers
all
take
different
approaches
to
virtual
machine
isolation
and
aim
to
challenge
the
perception
of
slow
startup
time
and
high
memory.
Overhead
each
system
is
a
combination
of
virtual
machine
and
container
technologies,
some
vmm
processes,
a
linux
kernel
within
the
virtual
machine,
and
once
the
kernel
has
booted
a
linux
user
space
in
which
to
run
the
process
and
some
combination
of
based
isolation,
that
is
container
style,
name,
spaces,
c
groups
and
set
comp
either
within
the
vm
around
the
vmm
or
some
combination
thereof.
A
A
These
file
systems
have
historically
leaked
the
container
abstraction
by
sharing
information
from
the
host,
such
as
memory
devices,
processes,
etc.
Sentry
prevents
the
application
interacting
directly
with
the
host
kernel
and
setcomp
is
used
around.
The
g
visor
kernel
to
limit
system
calls
and
prevent
escalation
in
case
of
tenants
breaking
into
sentry
and
attacking
the
host
kernel.
A
A
A
side
process
to
the
sentry
called
gopher
handles
io
eg
disks
devices,
which
is
historically
a
common
attack
vector
for
vms
separating
this
process,
provides
resistance
to
compromise.
If
sentry
has
an
exploitable
bug,
it
can't
be
used
to
attack
the
host's
devices
because
they
are
all
proxied
through
gopher.
A
A
Sentry's
user
space
os
kernel
implements
all
the
kernel
functionality
needed
by
the
untrusted
application.
Although
it
does
make
some
host
system
calls
to
support
its
operation,
it
will
not
allow
the
application
to
directly
control
the
system
calls
that
it
makes
to
the
underlying
linux.
Kernel
application
system
calls
are
redirected
to
sentry
by
a
platform
syscall
switcher
that
intercepts
the
application
when
it
tries
to
make
system
calls.
A
A
The
p
tray
system
call
provides
a
mechanism
for
a
parent
process
to
observe
and
modify
another
processor's
p
trace.
Cism,
u
forces
the
traced
process
to
stop
an
entry
to
this
next
syscall
and
gvisor
is
able
to
respond
to
it
or
proxy
the
request
to
the
host
kernel
going
via
gopher.
If
io
is
required,.
A
Firecracker
is
a
vmm
that
boots
a
dedicated
virtual
machine
for
its
guest
using
kvm,
but
instead
of
using
kvm's
traditional
device.
Emulation
pairing
with
qmu
firecracker
implements
its
own
memory
management
and
device
emulation.
It
has
no
bios.
Instead
implementing
linux
boot
protocol,
no
pci
support
and
stripped
down
simple
virtualized
devices
with
a
single
network
interface,
a
block.
I
o
device,
a
timer
clock,
serial
console
and
a
keyboard
device
that
only
simulates
control
or
delete
in
order
to
reset
the
vm
firecracker
itself
will
be
compatible
with
kubernetes
and
oci.
A
A
A
It
was
created
from
for
aws's
lambda
service
forked
from
google's
chrome
os
vm
cross.
Vm
firecracker
is
a
statically
linked,
rust,
binary
that
doesn't
pollute
the
host
file
system
and
is
compatible
with
catacontainers
ignite
firecube
firecracker
container
d.
It
provides
soft
allocation
for
more
aggressive
bin
packing
and
a
greater
resource
utilization.
A
Cat
containers
are
lightweight
vms,
containing
a
container
engine,
highly
optimized
for
running
containers.
They
are
the
oldest
and
most
mature
of
the
recent
sandboxes
and
were
originally
called
clear.
Containers
compatibility
is
wide
with
support
for
most
container
orchestrators
grown
from
a
combination
of
intel,
clear
containers
and
hyper
dot.
Sh
run
v
cat
containers
wraps
containers
with
a
dedicated
kvm
virtual
machine
and
device
emulation
from
a
plugable
back
end
qmu,
qmu,
qmulite,
nemu,
a
custom
strip
down
qmu
or
firecracker.
A
A
A
An
honorable
mention
many
new
vmm
technologies
have
some
rustling
components,
so
is
rust
any
good,
it's
similar
to
golang,
in
that
it
is
memory
safe.
It's
memory
model,
vertio
et
cetera,
but
it
is
built
atop,
a
memory
ownership
model
which
avoids
whole
classes
of
bugs,
including
use
after
free
double
free
and
dangling
pointer
issues.
A
A
A
A
Docker
is
able
to
run
any
oci
compliance.
Runtime
kubernetes
requires
a
runtime
to
also
be
cri
compliance,
while
kubernetes
does
not
yet
distinguish
between
types
of
sandboxes.
We
can
still
set
node
affinity
and
toleration
so
that
we
schedule
our
pods
onto
nodes
that
have
the
relevant
sandbox
technology
installed.
A
What
are
the
risks?
Well,
the
degree
of
access
and
privilege
that
a
guest
process
has
to
host
features
or
virtualized
versions
of
them
impacts
the
attack
surface
available
to
an
attacker
in
control
of
the
guest
process.
So
this
new
tranche
of
sandbox
technologies
is
under
active
development.
Its
code
and,
like
all
new
code,
is
at
risk
of
exploitable
bugs.
This
is
a
fact
of
software
and
is
infinitely
better
than
no
new
software
at
all.
A
Potentially,
these
sandboxes
are
not
yet
a
target
for
attackers.
The
level
of
innovation
and
baseline
knowledge
to
contribute
means
the
barrier
to
entry
is
set
high
from
an
administrator's
perspective.
Modifying
or
debugging
applications
within
the
sandbox
becomes
slightly
more
difficult,
similar
to
the
difference
between
bare
metal
and
containerized
processes.
A
Narrowing
the
interface
between
a
sandbox
process
and
the
host
is
a
risk-based
decision.
There
are
some
trade-offs.
Debugging
becomes
much
harder
and
traditional
tracing
tools
may
not
have
great
compatibility.
There
is
a
performance
impact,
but
this
may
be
negligible
for
some
workloads
and
benchmarking
is
strongly
encouraged.
A
So
application
workloads
should
be
categorized
by
risk.
Does
this
application
access
a
high
value
asset?
Is
this
application
able
to
receive
untrusted
traffic
or
have
there
been
vulnerabilities
or
bugs
in
this
application
before?
If
the
answer
to
any
of
those
is
yes,
you
may
want
to
consider
a
next
generation
sandboxing
technology.
A
So
unless
you
have
specific
problems,
containers
are
probably
just
fine
for
higher
sensitivity,
workloads
and
data.
Perhaps
we
want
more
isolation.
There
is
an
affinity
between
some
of
these
technologies
and
their
supporting
cloud
provider,
which
makes
it
a
lot
easier
to
use.
The
supported,
runtime
and
rustvmm
means
that
there
will
be
many
more
hypervisor
based
container
runtimes
in
the
future.
Thanks
for
listening
have
a
wonderful
day.