►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Secure Pods - Tim Allclair, Google (Advanced Skill Level)
What is a "secure pod"? What does it mean for a Kubernetes workload to have strong isolation? With the announcement of Kata Containers and the overflowing multitenancy deep-dive at the last Kubecon, it's clear that these topics are building momentum. This talk will cover the current state of container isolation and why there is a need for technologies like hypervisor-based containers in order to provide stronger security boundaries. It will also include a discussion of how these technologies fit into Kubernetes and a roadmap for secure pods. This is a rapidly evolving area, and Tim anticipates that a proposal for secure pods will be finalized by May. This talk will be shaped by the status or outcome of that proposal.
"About Tim
Tim Allclair joined the Kubernetes project with Google just after the 1.0 launch in 2015. He is a sig-auth lead, an active sig-node contributor, and a maintainer of features including advanced audit logging, AppArmor support, PodSecurityPolicy and cAdvisor. He is a member of the Kubernetes Product Security Team (responsible for responding to vulnerabilities in Kubernetes), and a member of the Kata Containers Architecture Committee. His most recent charter is providing stronger container isolation, a topic in which he led discussions at the sig-node & sig-multitenancy deep-dives at Kubecon North America 2017."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hopefully,
you're
all
still
awake
on
the
last
day
of
the
conference,
so
welcome
to
secure
pods,
sandboxing
workloads
and
kubernetes.
My
name
is
Tim
all
Claire
I'm,
a
software
engineer
at
Google
I
work
on
the
kubernetes
engine
team,
doing
various
security
things
I'm,
also
a
co-chair
of
sig
off
and
a
member
of
the
kubernetes
product
security
team.
A
So
when
I,
when
I
first
submitted
the
CFP
first,
the
only
time
I
did
it
was
the
last
day
the
CF
keys
were
due
and
we
had
just
started
to
dig
into
secure
pods.
We
knew
we
wanted
to
build
on
a
lot
of
the
great
work
that
was
coming
out
of
the
cotta
container
and
frock
D
and
cryo
communities
around
kind
of
building
these
sand
boxes,
sandbox
runtimes.
A
We
knew
we
wanted
to
kind
of
integrate
that
deeper
into
kubernetes
and
formalize
the
concept
of
a
secure
pod.
But
we
didn't
really
know
exactly
what
a
secure
pod
was
so
five
months
later
I'm
here
we
have
a
much
better
idea
of
what
your
secure
pod
means
and
now
we're
calling
them
sandbox
pods
to
disambiguate.
But
the
title
of
the
talk
is
still
secure
pods.
So
let's
run
with
that
for
a
minute
and
talk
about
what,
if
secure
pod
is
for
people
just
getting
here,
there's
still
some
seats
up
front.
A
So
just
because
a
is
isolated
from
B
doesn't
necessarily
mean
that
B
is
isolated
from
a
so
in
the
context
of
pods
and
kubernetes,
we
can
talk
about
threats
from
the
outside,
so
threats
from
the
the
internet,
big
scary,
place
and
all
the
attackers
that
want
to
get
at
whatever
is
running
in
our
pods
in
our
cluster,
and
this
is
a
really
big
rich
topic
area,
a
lot
of
different
things
to
help
prevent
attackers
from
the
outside
getting
into
our
cluster.
But
this
isn't
what
I'm
going
to
focus
on
today.
A
A
First
reason
to
care
is
if
you're
running
untrusted
code,
if
you
are
running
a
multi-tenant
SAS,
for
instance,
where
your
users
submit
binaries
or
they
submit
container
images
or
anything
like
that,
you
can't
trust
the
code.
You
don't
know
what
your
users
are,
giving
you,
maybe
you're
running
a
third-party
service.
A
A
So
what
can
we
do
about
this?
How
do
we
deal
with
internal
threats?
It's
actually
much
harder,
I
think
to
deal
with
these
internal
threats.
There's
much
much
larger
attack
surfaces.
Not
only
do
we
have
to
worry
about
some
of
these
same
external
attacks,
but
we
also
have
to
worry
about
attacks
on
the
the
kernel,
the
syscall
interface,
that's
exposed
to
containers.
We
have
to
worry
about
the
various
storage
interfaces.
A
There's
the
daemons
that
provide
different
services
to
your
to
your
cluster
and
then
we'll
touch
on
hardware
as
well
in
the
wake
of
Spectre
and
meltdown.
But
let's
start
with
the
kernel,
so
I'm
sure
you've
all
heard
the
term
container
escape.
This
is
most
commonly
talked
about
in
the
context
of
the
kernel,
I
think
looking
at
attacks
via
sis
calls
and
there's
actually
a
lot
of
nice
things
we
can
do
about
this.
Oh
wait,
I'm
supposed
to
say,
let
me
engage
in
a
little
bit
of
good
old-fashioned
fear-mongering.
A
In
2017
there
were
169
code
execution,
CVEs
published
in
the
linux
kernel.
The
good
news
is
most
of
these
are
in
drivers
in
drivers
that
are
primarily
used
in
mobile
devices,
so
keep
your
phone's
up-to-date,
but
unless
you're
running
kubernetes
on
your
phone,
I
bet
there's
at
least
one
person
at
this
conference.
Doing
that
you
don't
need
to
worry
about
this,
and
we
actually
have
a
lot
of
good
tools
to
deal
with
the
kernel
attack
surface
today.
A
Seccomp
is
a
way
of
filtering
out.
Sis
calls
and
reducing
the
attack.
Surface
docker
has
some
really
nice
defaults,
a
comp
profile
that
we
turn
off
in
kubernetes
by
default.
So
you
probably
want
to
turn
it
back
on,
and
this
is
the
annotation
that
you
can
use
to
do
that.
We're
trying
to
make
that
the
default,
but
security
defaults
are
hard.
A
parmer
is
another
another
tool.
Selinux
is
sort
of
the
the
counterpart
if
you're
running
a
selinux
system.
A
A
But
what
happens
when
these
features
aren't
enough?
Dirty
cow
I've,
already
heard
mentioned
a
few
times
at
this
conference.
This
was
a
vulnerability
that
came
out
two
years
ago.
It's
a
race
condition
in
the
copy-on-write
memory,
handling
of
the
linux
kernel
and
there's
a
cool
proof
of
concept
escaped
and
demonstrated
by
the
via
the
virtual
dynamically
shared
object
that
is
injected
into
every
process
in
linux
and
I'll
link
to
that
at
the
end.
A
At
Google
we
have
this
principle
security
principle
that
between
any
untrusted
code
and
user
data,
there
have
to
be
at
least
two
distinct
security
boundaries.
So
that
means
two
independent
security
mechanisms
need
to
fail
in
order
to
for
that
untrusted
code
to
get
out
that
user
data
in
the
context
of
the
kernel
attack
surface.
Since
the
kernel
is
a
single
kind
of
thing
in
memory,
we
basically
need
a
separate
kernel
for
the
sandbox
pod.
A
Otherwise,
a
single
bug
in
the
the
kernel
like
dirty
cow,
could
lead
to
an
escape
from
the
sandbox
and
that
that
two
security
boundaries
is
an
important
concept
and
we're
going
to
come
back
to
that
throughout
the
talk,
so
one
implementation
of
sandbox
is
is
kata
containers.
There
have
been
a
couple
of
talks,
maybe
you've
been
to
at
the
conference.
It
was
announced
in
December,
cube
con
north
america
and
it's
a
combination
of
Hyper's
run
v
technology
and
intel's,
clear
containers
coming
together
to
make
lightweight
VMS
that
run
containers
in
the
context
of
kubernetes.
A
Maybe
you
heard
the
announcement
about
gee
visor
two
days
ago.
This
is
a
new
open
source
project
from
Google,
and
this
is
actually
the
sandbox
that
we
use
at
Google
in
a
lot
of
places
in
the
production
environment.
This
is
a
little
different
from
a
traditional
VM.
You
have
a
an
emulated
kernel,
it's
basically
something
that
implements
the
Linux
Cisco
interface
running
in
user
space
and
then
this
platform
thing
that
just
bounces
syscalls
into
the
emulated
kernel
and
I
just
wanted
to
highlight
that
sandboxes
are
at
the
pod
level,
we're
still
discussing
the
API.
A
Actually
I
should
mention.
This
whole
feature
is
still
in
the
design
phase,
and
but
we
want
to
keep
it
simple.
So
this
is
one
possible
implement
what
the
API
might
look
like.
So
what
happens
if
we
have
dirty
cow
and
we're
running
in
a
sandbox?
If
you
are
running
kata
containers
and
your
guest
kernel
is
vulnerable,
then
you
can
escape
the
container
but
you're
still
trapped
in
the
VM.
You
need
a
hypervisor
exploit.
You
know
some
bugging
KVM,
for
instance,
to
actually
get
out
of
there.
A
A
Obviously,
if
you're
sandboxed,
you
shouldn't
be
able
to
do
that
now.
Also,
if
you're
running
in
a
sandbox,
you
don't
have
the
same
visibility
from
the
hosts,
as
you
would
into
a
native
container
process.
This
isn't
because
the
sandbox
is
isolated
from
the
host.
It's
not,
but
when
you're
running
a
native
container
processes
in
the
container,
just
like
processes
on
the
host-
and
it's
really
easy
to
kind
of
you
know
introspect
in
those
processes,
with
a
lot
of
the
cool
monitoring
tools
that
you
might
be
familiar
with.
A
A
You
may
have
heard
about
the
CVE
we
announced
in
kubernetes
a
couple
months
ago.
This
was
a
vulnerability
in
the
sim
link
handling
in
volumes,
so
normally,
when
a
container
reads
a
symlink
in
a
volume
that
sim
link
is
evaluated
in
the
containers
mount
namespace.
This
means,
if
that
sim
link
points
to
Etsy
password,
for
instance,
when
the
container
reads
that
it's
just
going
to
see
the
Etsy
password
in
the
container
filesystem,
which
hopefully
isn't
too
sensitive.
A
So
that's
bad.
How
does
sandboxes
help
us
here?
Well,
one
approach
to
sandboxes
volumes
in
sandboxes
is
to
mount
the
volume
on
the
host
and
then
expose
it
into
the
sandbox,
so,
for
instance,
in
cata
containers,
one
approach
is
to
use
the
9p
protocol
over
a
ver.
Do
interface
to
effectively
bind
mount
the
volume
into
the
sandbox.
A
The
problem
is
we're
left
with
the
same
vulnerability
here
when
the
host
goes
to
mount
that
volume
on
its
filesystem.
It
follows
the
sim
link.
We
end
up
mapping
that
sensitive
data
into
the
sandbox,
and
we
see
that
we
only
have
one
single
point
of
failure
here
that
leads
to
an
exploit.
So
what
can
we
do
about
this?
This
is
still
design
phase
needs
more
kind
of
research,
but
there's
a
couple
ideas
what
we
can
do
about
it.
The
first
principle
is
that
read-only
read-only
storage
should
be
exposed
via
read-only
protocols.
A
We
do
a
similar
thing
with
block
volumes
and
mount
block
volumes
directly
into
the
sandbox
and
then
and
not
onto
the
host,
so
that
again
they're
opaque
to
the
host,
and
then
we
have
this
whole
like
networked
file
system
stuff,
that's
probably
the
most
significant
portion
of
the
volumes,
and
this
is
kind
of
a
hard
problem.
One
possible
solution
is
we
run
the
client
software
for
those
network
file
systems
like
NFS
cluster
staff.
A
We
run
those
in
the
sandbox
and
then
a
vulnerability
in
that
client
would
just
lead
to
an
escape
within
the
sandbox
and
not
into
the
hosts
the
problem
there
is.
We
then
need
to
also
mount
the
credentials
for
that
remote
storage
into
the
sandbox
and
we're
effectively
moving
that
security
boundary
into
the
the
remote
storage,
authentication
and
authorization
layer.
So
it's
a
bit
of
a
trade
off
and
we
need
to
do
more
research
in
this
area.
A
We
have
various
ports
on
the
node.
This
read-only
port,
that's
used
in
most
clusters,
is
unauthenticated
and
leaks,
a
bunch
of
data
metadata
about
the
workloads
or
trying
to
fix
that
metadata
service.
That's
exposed
by
a
lot
of
cloud
providers
is
a
common
attack
vector
if
you're
mounting
service
account
tokens
into
your
sandbox,
then
we
have
to
worry
about
attacks
via
the
API
server.
Any
authenticate,
unauthenticated
services
within
the
cluster
and
also
DNS
can
can
leak
some
information.
A
So
what
can
we
do
about
this
here?
We're
going
to
look
at
a
different
feature
other
than
sandboxes
there's,
actually
a
lot
of
options
in
this
space.
A
lot
of
the
service
meshes
can
help
us
here,
and
a
network
policy
can
also
help
I'm
not
going
to
go
too
deep
into
this,
but
this
is
just
an
example
of
a
network
policy
that
says,
allow
all
traffic
coming
out
of
my
sandbox
pod,
except
traffic
to
private
IP
addresses,
and
this
will
vary
a
little
bit
depending
on
your
network
topology.
A
A
This
is
sort
of
a
simplified
diagram
of
the
log
flow
on
the
node,
so
the
logs
start
from
the
container
they're
written
to
standard
out
or
standard
air
they
go.
That
is
a
pipe
to
the
runtime,
typically,
which
adds
a
little
bit
of
metadata
to
those
logs
like
the
timestamp
and
which
stream
it's
coming
from
then
they're
written
to
the
log
file
and
that
log
file
is
read
by
the
cubelet
to
serve
logs
requests
and
eventually
gets
back
to
the
user.
A
But
that
means
that
fluent
D
is
actually
looking
at
that
untrusted
data
and
parsing
that
so
what
happens?
If
there's
a
vulnerability
in
fluent
e
triggered
by
that
untrusted
log
data
that
leads
to
code
execution
again,
we
have-
and
we
see,
there's
a
single
point
of
failure
here
and
I
was
kind
of
looking
back
through
the
history
of
fluent
evil.
Nor
abilities
and
I
didn't
see
anything
like
this
as
having
happened
in
the
past,
but
that
doesn't
mean
it
can't
happen
in
the
future.
A
Om
and
so
what
can
we
do
about
this?
One
option
is
just
to
sandbox
all
the
things
sandbox
flew
in
D,
so
now,
if
you
actually
do
manage
to
trigger
a
vulnerability
and
flew
in
D
you're
still
trapped
within
flew
in
DS
sandbox,
and
maybe
we
can
use
native
processes
here
or
sorry
native
containers,
because
fluent
D
is
our
first
security
boundary
and
so
a
single
security
boundary
might
be
acceptable
there
and
finally,
I
want
to
talk
about
attacks
via
hardware
in
the
wake
of
Spectre
and
meltdown.
A
Sandboxes
is
an
upcoming
feature
that
can
help
prevent
a
lot
of
those
internal
threat,
vectors
leveraging,
hypervisor
isolation
and
then
we're
looking
at
deeper
integration
into
the
kubernetes
system
to
mitigate
a
lot
of
threats.
Via
these
other
attack,
vectors
I
talked
about
the
kernel
or
I.
Guess
that's.
You
know.
The
hypervisor
isolation
gives
us
kernel
isolation,
but
then
there's
the
storage
interfaces,
the
logging
interfaces
and
the
network
interfaces.
A
So
when
can
you
use
sandboxes
now
we
actually
have
experimental
support
today,
both
cryo
and
container
D
support,
choosing
between
different
OCI
runtimes,
based
on
an
annotation
on
the
container
and
this
plugs
into
the
CRI.
So
you
can
annotate
your
pod
and
take
advantage
of
using
kata
and
G
visor.
If
you
have
your
cluster
set
up
the
right
way,
this
is
only
protection
at
the
kernel
interface
again,
I
was
really
hoping
to
get
an
alpha
out
in
111.
A
It
looks
like
that's
probably
not
going
to
happen
so
I
expect
to
see
an
alpha
in
around
112
with
an
actual
kubernetes
api,
better
implementation
via
kata
and
G
visor,
and
also
some
improved
resource
management
that
I
didn't
touch
on
today
and
then,
as
we
go
to
beta,
actually
looking
at
hardening
a
lot
of
those
other
interfaces
as
well.
If
this
sounds
cool
to
you,
you
can
get
involved.
A
This
first
link
is
to
a
document
on
sandboxes
that
goes
into
a
lot
of
the
challenges
with
building
it
and
there's
a
lot
of
design
discussion
happening
in
the
comment
section.
We're
also
talking
about
the
design
in
the
signal
community,
and
this
second
link
is
to
where
you
can
find
out
more
about
sig
node
and
our
weekly
meetings
and
the
different
ways
to
engage
there.
If
you
want
to
contribute
code,
of
course,
we're
welcoming
contributions
in
sig
note,
but
also
in
cotta
containers
in
G
visor.
These
are
all
open
source
projects.
A
B
A
Yeah,
so
the
question
is
about
user
name,
spaces,
user
name.
Spaces
is
a
good
solution
for
running
as
non-root.
It's
still
relying
on
kernel
isolation,
so
still
that
single
security
boundary
we
there
are
a
few
proposals
open
for
how
to
tie
user
name
spaces
into
kubernetes.
One
of
the
biggest
challenges
is
around
handling
permissions
of
volumes.
A
So
if
you
run,
if
your
pod
starts
up
and
mounts
a
volume
and
write
some
data
to
it
and
then
your
pod
gets
killed
and
restarted
elsewhere,
maybe
it
has
a
different
name:
space
mapping,
you
end
up
with
different
user
IDs,
and
it
can
no
longer
read
that
data
from
the
volume.
So
we
still
need
to
figure
out
how
to
deal
with
that
problem,
but
definitely
something
we
want
to
add
any.
C
And
my
question
is:
would
the
legacy
be
I
must
be
able
to
run
in
sandbox
without
huge
notification
or
to
expect
legacy
to
be
rewritten
to
run
on
sandbox
sorry,
I
didn't
catch
the
first
part
of
the
question
the
legacy
VMs.
Would
they
be
able
to
run
in
sandbox
legacy
virtual
machines,
Oh
legacy,
verge
yeah.
A
Sorry,
so
the
question
is
about
legacy
VMs
and
non
containerized
workloads.
It's
definitely
a
possibility.
Cotta
containers
is
running
a
VM
environment.
It
is
a
stripped-down
VM.
So
it's
not.
It
might
not
have
the
full
set
of
hardware
interfaces
that
your
legacy
workload
needs.
It's
also
a
stripped
down
guest
operating
system,
so
we
would
need
a
way
of
specifying
the
the
full
VM
image
to
the
sandbox.