►
From YouTube: Securing Untrusted Workloads with Kata Containers on Kubernetes - David Angot & Alex Price
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Securing Untrusted Workloads with Kata Containers on Kubernetes - David Angot & Alex Price, Atlassian
Whilst containers have allowed for quick and easy deployment and execution of workloads, they come with their drawbacks in terms of security and isolation. This is evident when running untrusted workloads, where isolation and separation of customer workloads is paramount in a multi-tenanted environment.
With years of experience running the Bitbucket Pipelines infrastructure, Atlassian engineers David Angot and Alex Price will explore the challenges faced, such as kernel vulnerabilities, providing access to Docker in Docker (DinD) and “privileged” containers when managing a platform that executes untrusted code.
With security in mind, they will explore Kata Containers, a runtime for Containerd and CRI-O that provisions Kubernetes pods as Virtual Machines, each with their own kernel and resources and most importantly isolation.
A
Hi
everyone,
thanks
for
coming
to
our
talk,
I'm,
David
and
I'm
here
with
Alex.
We
both
from
the
Cuban
esteem
at
lassen
here
in
Sydney
and
today,
we're
here
to
talk
about
securing
untrusted
were
close.
We've
cut
our
containers,
so
just
get
a
quick
hands
up
so
who
heard
of
Kara
containers
in
the
room?
Oh
cool,
okay,
that's
good!
A
So
I'll
just
give
a
little
brief
overview
of
kubernetes
Atlassian's.
So
we've
been
running
in
Prague
for
three
years
now
and
we
didn't
have
many
tools
at
the
time
to
deploy
accumulates
clusters.
We
really
talk,
we
will
say:
we've
done
it
the
hard
way,
there's
a
tutorial
communities
the
hard
way.
Well,
we've
pretty
much
done
that
we
have
a
dedicated
team
to
run
those
clusters.
A
50
clusters
on
growing
was
that
correct.
That
was
yeah
well,
which
one
she
multi-tenant
our
clusters
as
much
as
possible.
But
sometimes
it's
not
always
possible
or
different
reasons,
security,
networking
or
things
like
that,
but
most
of
the
time
we
remove
to
tenant
our
cluster
zhim
and
then
which
once
you
separate
concerns
like,
for
instance,
we
run
cumulus
clusters
for
job
based
workload
and
in
some
cases
for
stay
lists.
A
One
of
the
the
tools
that
Alison
offering
that
runs
on
top
of
I
cubanelles
clusters
is
bitbucket
pipeline.
So,
if
you
don't
know
bitbucket
pipeline,
it
allows
you
to
deploy
and
build
your
code
directly
from
where
it
is
hosting
on
bitbucket.
So
within
a
few
clicks
you
can
actually
start
building
your
code
and
and
deploy
as
well,
so
that
runs
on
top
of
kubernetes
and
each
whenever
you
do
a
did
push.
A
For
instance,
each
jobs
were
billed
will
be
a
cube
or
any
spot
in
its
own
namespace,
and
that
runs
on
a
pool
of
co-locating
notes.
So
it's
not
really
hard
to
understand.
Why
are
we
talking
about?
You
know
hard
multi
penalty.
Here
everything
has
to
be
isolated
from
each
other.
Every
container
like
need
not
to
know
about
the
other,
so
we
shall
have
a
a
job
from
a
customer.
A
it
needs
to
be
completely
isolated
from
another
job
from
customer
B.
A
A
So
if
I
have
a
look
at
how
the
traditional
containers
and
when
I
say
traditional
containers,
you
probably
understand
better
if
I
say
docker
containers,
we
don't
run
docker
demon
anymore
in
our
clusters,
but
it
is
effectively
a
docker
container.
So
the
isolation
works
with
kernel
namespaces
and
that's
basically
going
to
control
what
the
process
are
going
to
be
able
to
to
see
or
not
see.
So
here,
I
have
like
three
containers
process,
a-b
and
fee
with
their
own
kernel,
namespaces
and
then
a
half
see
groups.
A
A
So
when
you
talk
about
securing
those
those
process
that
the
worst
that
could
happen
is
having
a
container
break
out,
it's
pretty
rare,
but
it
could
potentially
happen
so
this
fission
mechanism,
you
can
or
things
you
can
do
to
secure
those
does
contain
a
bit
further
and
then
it
was
just
to
talk
about
during
the
best
practices
what
you
can
do
for
for
security,
and
then
we
do
most
of
that.
If
not
everything
network
policies,
for
instance,
is
a
good
one.
A
Where
you're
gonna
have
your
container
a
running
from
a
customer
not
going
to
be
able
to
talk
to
you
container,
be
the
rules
are
pretty
simple
in
those
clusters,
for
instance,
where
you're
only
going
to
be
able
to
talk
to
the
internet
you're
going
to
be
able
to
push
pull
some
code,
do
things
that
are
be
never
going
to
be
able
to
talk
to
you.
Another
container
running
that
cluster
or
the
underlying
host
said.
A
It's
a
good
thing
for
obvious
reasons
right
so,
as
I
said,
the
worst
thing
that
could
happen
is
really
a
container
breakout,
and
then
we
have
some
interesting
requirements
in
that
cluster
and
in
one
of
those
requirement,
is
docking.
Docker
developers,
love,
docker
and
dagger.
So
do
you
start
a
container
for
them
with
a
build
and
they
want
to
be
able
to
build
their
code
in
that
container,
but
they
also
want
to
build
their
docker
images
and
then
also
want
to
push
their
double
images
so
like
that.
A
Build
actually
has
a
something
they
can
start
using
straightaway.
So,
in
order
to
do
that,
we
have
to
run
docker
and
docker
I
create
a
docker
container
and
then
you're
gonna
run
docker
container
in
Dhaka,
so
that
requires
what
root
uses,
privilege
mode
extra
capabilities.
It's
a
bit
of
a
nightmare
for
at
you,
bananas.
That
means
to
secure
it
so
really
challenging
security.
Think
that
doctor
of
Dhaka
requirement,
another
thing
is
C.
B's
CV
is
a
really
this
big
scary
thing
that
you
don't
know.
A
What's
gonna
hit
you
and
most
of
the
time
when
they
are
announced,
you
actually
have
a
fix
already
and
then
you
can
push
that
fix.
But
it
is
the
big
scary
thing
here,
where
you
running
infrastructure,
but
really
untrusted
workload
and
and
well
you're
trying
to
protect
yourself
against
unknown
CVS.
So
this
is
where
we
really
require
the
extra
level
of
security
where
we
think
well,
the
container
isolation
might
not
be
just
enough
and
then
we
need
another
level.
A
So
this
is
where
we
started
with
Keira
containers,
and
that
comes
straight
from
their
website
where
it
says
the
speed
of
containers
and
the
security
of
VMs.
Well,
it
almost
sounds
too
good
to
be
true.
Well,
you
usually
spinning
up
a
VM.
Will
take
you
30
seconds
a
minute
starting.
You
contain.
It
takes
service
sub-second
like
if
you
have
the
image
the
image
stunning
in
containers,
almost
instant
well
kara
can
provide
stuff
and
it
does
up
with
running
what
they
call
micro
virtual
machine.
A
So
if
I
have
a
look
at
the
isolation
aware,
I
have
the
same
traditional
container
isolation
with
my
three
same
process,
a
B
and
C
that
sharing
that
Linux
kernel
in
the
cutter
container
world
I
have
a
micro,
virtual
machine
wrapping
those
containers.
So
each
of
my
process,
a
B
and
C
which
different
containers
will
have
their
own
Linux
kernel
a
B
and
C,
and
then
this
is
really
where
that
extra
level
of
isolation
comes
from.
Where
you
really
have
the
hardware
virtualization
on
the
underlying
host,
you
have
like
the
normal
like
in
its
kernel.
A
So,
in
order
to
start
a
car
a
container,
you
have
to
tell
Cuban
Aires
how
to
start
the
container,
so
this
is
done
with
an
annotation
and
in
a
more
recent
version
of
kubernetes,
a
runtime
class
car
can
run
on
the
same
machine
as
a
normal,
traditional
container,
which
is
a
run
sea
container.
Basically
so
in
run
container
the
cryo
or
even
docker,
alongside
Keira
containers
that
runs
gonna,
run
inside
a
virtual
machine,
color
containers
for
different
hypervisor.
A
So
this
a
Big
Bertha
firecracker,
which
has
been
released
recently
by
AWS,
but
we
run
the
normal
QM.
You
manage
k,
vm
based
virtual
machine.
So
really
all
you
need
to
do
is
drop
the
packages
for
four
color
containers
and
an
annotate
or
use
every
one-time
class
for
your
pod
and
then
automatically
cubelet
will
just
start
your
containers
in
that
micro
virtual
machine.
A
A
B
David
so
as
we
were
transitioning
to
Carter
containers,
one
of
the
first
hurdles
that
we
encountered
was
this
performance
inside
the
virtual
machine
by
default,
Carter
uses
a
system
called
nine
PFS
to
share
the
route
FS
of
the
container
config
Maps
secrets
volumes
into
the
VM.
Any
changes
made
to
inside
the
VM
are
propagated
back
to
the
host
side
and
vice
versa.
This
allows
the
qubit
to
manage
things
like
secrets
and
config
maps
from
the
host
side
and
then
have
those
changes
replicated
back
into
the
VM.
B
A
benefit
of
using
an
old
established
protocol
like
9p
FS
is
that
it
is
really
reliable
and
it
has
really
good
compatibility
with
certain
file
systems.
At
one
point,
we
were
able
to
run
an
overlay
file
system
on
top
of
9p
inside
that
VM,
but
well
we
soon
got
reports
that
builds
are
running
really
really
slowly
in
Carter
containers,
verse,
natively
in
run
C.
In
some
cases
these
builds
were
taking
much
longer
and
were
performing
worse
than
on
the
developers
laptop.
B
So
we
started
doing
some
performance
benchmarks
of
the
file
system
used
by
the
containers
inside
the
VM,
and
we
quickly
discovered
why
it
was
subpar.
This
is
a
sequential
right
of
a
of
an
eight
gigabyte
file
to
the
root
of
us
and
it
performed
pretty
poorly
at
eighty
eight
megabytes,
a
second
for
some
perspective,
consumer
hard
drives
operate
at
around
150
megabytes,
a
second.
When
writing.
If
our
goal
was
to
provide
a
system
from
the
early
2000s,
then
we
definitely
achieved
that
and
here's
the
same
performance
on
my
macbook,
which
is
considerably
better.
B
So
it
was
really
obvious
now
why
developers
were
getting
better
performance
on
their
laptops.
So,
with
some
research
we
discovered
that
cada
has
support
for
plugging
block
devices
into
the
VM.
This
is
done
by
leveraging
device,
mapper
and
lugging
built-in
to
contain
addy
that's
fairly
new
rather
than
provisioning.
The
docker
images
lays
on
the
host
with
overlay
FS
and
using
that
for
the
containers
read
FS.
The
container
D
plug-in
will
create
thin
provision,
copy-on-write
block
devices
for
the
docker
image
layers
with
each
block
device.
B
They're
then
plugged
into
the
virtual
machine
and
mounted
into
the
correct
location
for
the
containers
route
FS
by
using
block
devices.
We
avoid
the
encoding
and
decoding
overhead
of
9pf
s,
while
also
maintaining
that
compatibility.
That
being
said,
we
still
have
to
rely
on
9
p
FS
for
some
sharing
of
directives
and
pass,
but
we've
ensured
that,
though
these
are
not
in
the
critical
path
of
builds.
For
example,
the
container
secrets
and
comping
Maps
still
use
9
p
FS,
but
these
are
small
files
where
the
penalty
of
9
p,
FS
isn't
as
significant.
B
Finally,
we've
ensured
that
device
mapper
blocks
are
provisioned
on
top
of
an
array
of
nvme
SSDs
and
help
boost
performance,
and
so
our
sequential
performance
now
looks
like
this,
which
is
pretty
good,
considering
that
it's
inside
a
virtual
machine,
it's
running
on
a
multi-tenant
anode
and
we've
found
that
out
device.
Mapper
setup
has
been
able
to
sustain
250,000
a
ops
without
issue
which
is
really
ideal
for
a
platform
that
relies
on
running
a
large
number
of
builds
on
a
single
host.
B
So
the
next
challenge
that
we
ran
into
was
related
to
one
of
our
requirements.
Moving
Takada
that
was
being
able
to
run
privileged
containers
in
a
secure
fashion,
as
David
mentioned
earlier.
This
requirement
would
allow
us
to
run
docker
and
docker
inside
the
VM,
giving
customers
access
to
not
only
building
images
but
running
those
images
within
their
build.
B
B
This
will
create
the
container
with
elevated
privileges,
all
of
the
capabilities
that
you
could
ever
want
and
access
to
all
host
devices,
and
this
is
what
we
require
to
run
docker
in
docker,
letting
people
build
images
in
run
containers
within
their
pod,
so
by
default,
if
you
don't
have
privileged
setup,
containers
do
not
have
access
to
devices
on
the
host,
but
one
of
the
things
that
privilege
does
grant.
You
is
read/write
access
to
all
of
the
host
devices.
B
If
you
ever
needed
to
manage
a
disk
on
the
host
or
a
USB
device
from
a
container,
then
giving
privileges-
probably
the
easiest
way
to
achieve
this.
But
this
ease
of
use
can
have
some
disastrous
effects
if,
if
not
understood
properly
or
done
in
a
multi-tenant,
tentative
environment,
so
having
access
to
those
devices
in
the
host
minute,
you
can
not
only
read
and
modify
the
disk
file
systems,
but
also
file
systems
of
other
containers
on
the
host
itself.
These
capabilities
in
the
wrong
hands
could
result
in
data
loss
secrets,
exfiltrated
and
customer
data
stolen.
B
So
what
does
this?
Let
us
do
well
inside
our
privilege,
container
or
as
brooders
inside
the
VM.
We
can
then
mount
and
explore
any
file
system
mounted
into
the
virtual
machine.
In
this
case
there
was
another
container
running
on
the
host
file
system,
which
was
dam
25.
This
particular
container
was
an
odious
Xpress
up.
I've
then
managed
to
mount
dam
25
to
a
path
inside
the
vehm,
with
the
Linux
mount
command
once
mounted,
we
can
list
any
files
in
the
container,
including
readwrite.
B
As
a
remedy
for
this,
we've
contributed
a
flag
to
the
CRI
component
within
container
D,
to
allow
us
to
overload
the
default
privilege
behavior
with
this
flag.
We
can
disable
adding
of
those
host
devices
for
certain
runtimes
in
container
D
and
that's
not
mounting
them
into
the
card,
a
virtual
machine.
B
So
overall,
this
was
a
really
good
learning
experience
for
us,
and
some
of
our
key
takeaways
are
that
we
shouldn't
assume
isolation
with
anything,
especially
virtual
machines.
Performing
an
audit
of
the
resources
being
exposed
in
shared
Takada
helps
uncover
this
problem.
We
did
this
from
both
the
host
side
and
also
from
within
the
VAM,
as
it
might
not
be
evident
on
the
host
side
that
something
is
being
shared
into
the
van.
Finally,
we
are
interested
in
improving
the
improving
the
state
of
privilege
in
communities.
B
So
that
leads
me
to
the
final
challenge
that
we've
encountered.
Monitoring
of
containers
running
in
Carter,
so
I'm
wondering
who
containers
the
regular
containers
is
fairly
straightforward
tasks
when
using
run
C.
There
are
few
components,
but
it
essentially
boils
down
to
the
qubit
exposing
an
endpoint
with
metrics
of
all
the
containers
on
the
host.
The
qubit
then
gets
these
metrics
from
C
advisor,
which
in
turn
scrapes
the
C
groups
for
the
plods
and
containers
on
the
host
side
of
the
actual
metric
value.
B
So
when
we
look
at
that
same
metrics
pipeline
in
the
context
of
Carter,
we
run
into
a
few
problems.
Initially
things
look
pretty
good.
Q,
Buddha's
I
would
have
query
C
advisor,
which
is
in
turn,
able
to
scrape
the
C
groups
from
metrics.
However,
the
metrics
return
from
C
groups.
We
see
individual
container
metrics.
Do
the
container
process
now
running
in
a
completely
separate
virtual
machine
and
not
on
the
host
itself.
B
B
So
with
that,
we
looked
for
an
alternative
method
for
metrics
collection
will
have
defined
in
humanities
114.
There
is
now
a
new
metrics
endpoint
being
exposed
on
the
cubelet.
The
main
benefit
we
say
for
this
endpoint
is.
It
is
now
runtime
aware
it
will
delegate
its
metrics
collection
off
to
the
runtime,
so
whether
you
were
running
Carter
or
run
C
or
some
other
runtime,
the
Rob,
the
runtime
itself
is
responsible
for
collecting
metrics
and
shipping
them
back
to
the
Cuba
for
unseen.
Nothing
much
has
changed.
B
It'll
still
look
at
the
C
groups
on
the
host
for
resource
consumption,
and
so
when
we
go
to
get
metrics
for
our
Carter
pods,
the
runtime
knows
it
needs
to
take
a
few
extra
hops
or
steps
to
collect
the
metrics
for
its
containers.
It
will
query
the
C
groups
from
within
the
virtual
machine,
rather
than
the
C
groups
on
the
host.
B
And
finally,
if
you
have
mixed
workloads
running
on
the
host,
the
endpoint
works
seamlessly
with
both
run
C
and
card
a
container
and
whatever
other
runtime
you
want
to
run
so
what's
next
for
us,
with
our
journey
in
Bukhara
still
got
a
few
issues
to
work
through
and
a
few
things
that
we're
looking
to
utilize
in
the
future.
First
is
auto
scaling,
I'm,
not
sure
if
David
mentioned
it,
but
we
require
large
metal
instances
to
run
these
Carter
pods
for
a
while.
B
Now
we've
been
used
to
the
start
time
of
virtual
instances
in
AWS,
which
can
be
up
and
running
in
a
matter
of
seconds
but
with
metal
instances
start
times
can
be
up
to
10
minutes
for
a
whole
node
to
come
up
with
this,
we
need
to
be
ensure
that
we
have
enough
capacity
to
absorb
spikes
and
load,
but
also
be
really
mindful
of
the
cost
of
running
these
large
instances.
I,
don't
think
we're
looking
forward
to
is
annotations
that
Carta
provides
for
us.
B
So
this
lets,
you
add
a
notation
to
your
pods
back
and
then
configure
things
that
would
normally
be
statically.
Configured
we've
cut
off
so
things
like
our
the
CPU
counts
that
are
exposed
to
the
bam,
but
we're
also
like
different
kernel
versions.
So
we
could
have
applied
to
different
pods
on
the
same
host
running
to
different
kernel
versions.
If
we
wanted
to
test
a
new
kernel.
B
Finally,
we're
looking
forward
to
using
firecracker
as
our
virtual
machine
monitor
firecracker
is
really
appealing
to
us,
with
its
focus
on
security
performance
and
the
workload
density,
but
unfortunately
it
does
not
fully
support
all
the
features
that
we
require
just
yet
and
that's
what
we
had
today.
Thank
you.
I.
A
C
Hello
thanks
a
lot
for
great
presentation,
a
question
about
your
runtime
engine.
So,
basically,
have
you
considered
utterly
opposite
direction
instead
of
having
huge
machine
with
high
tendency
and
like
a
lot
of
pot
on
that
small
machines,
one
container
per
machine,
there
are
a
few
project
that
can
implement
that
thanks.
There's.
A
A
really
good
question:
yes,
one
of
the
key
features
like
the
key
feature
is
that
the
start
time
we're
having
having
those
nodes
already
there
and
being
able
to
pull
more?
They
already
have
the
agents
images
pulled
already
so
that
the
start
time
is
almost
instant
with
Cuban
Aires
and
having
warm
nodes.
A
If
you
will
I
think
that
that
was
our
agreed
to
trade
off,
we
might
have
reconsidered
up
with
new
things
like
Fargate,
for
instance,
where
we
eventually
we
will
be
able
to
run
a
container
in
in
its
own
virtual
machine
or
cube
fit,
but
they're
not
there.
Yet.
So
that's
why
we
had
to
go
the
car
out
to
run
those
privileged
containers
in
their
own
kernel.
So
did
it's
not
there
yet?
But
yes,
absolutely
in
the
future,
I
think.
With
the
start,
I'm
improving,
we
will
be
able
to.