►
From YouTube: Layers of Isolation in Kubernetes - Tim Allclair, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Layers of Isolation in Kubernetes - Tim Allclair, Google
How much isolation can you reasonably expect between two applications in the same cluster? Should every application have its own namespace? Every service? Between containers, pods, nodes, namespaces, and even clusters, it can be hard to know how to architect a secure system, and what layers of isolation can be depended on. In this talk we will start at the bottom and build up. You will learn which resources are isolated between two containers in the same pod, and which are not. From there we will explore what changes as the workloads are increasingly separated. You will see examples of real-world attacks, and how these attacks are mitigated at different layers of the stack. By the end, you will have a better understanding of how workloads can and should be separated for your own threat models.
To learn more click here: https://sched.co/FuLz
A
Welcome
to
layers
of
isolation
in
kubernetes,
my
name
is
Tim
all
Claire
I'm,
a
software
engineer
working
on
Google,
concur,
burn,
Eddy's
engine
I'm,
also
a
co-chair
of
kubernetes,
sig
auth
and
a
member
of
the
product
security
team.
So
today,
today,
I'm
going
to
tell
you
about
isolation
in
kubernetes,
but
before
we
start
with
that,
let's
make
sure
we're
all
on
the
same
page
for
what
isolation
means:
a
model
of
isolation
and
security
that
I
really
like
is
the
CIA
model.
This
stands
for
confidentiality,
integrity
and
availability.
A
Confidentiality
means
that
isolated
process
cannot
read
information
or
State
or
data
from
outside
of
its
isolation.
Boundary
integrity
means
that
the
isolated
process
cannot
alter
the
state
or
the
data
of
things
outside
of
its
isolation.
Boundary
and
availability
means
that
the
isolated
process
cannot
compromise
the
ability
of
things
outside
of
its
isolation
boundary
to
do
their
job
affecting
services
and
whatnot,
and
before
we
go
on
there's
two
key
properties
of
isolation.
I
want
to
point
out.
The
first
is
that
this
is
really
a
multi-dimensional
problem.
A
It's
not
enough
to,
for
instance,
isolate
just
the
CPU
and
memory
resources
to
prevent
denial
of
service,
compromising
availability,
type
attacks,
there's
a
lot
of
other
ways
that
you
can
cause
a
denial
of
service
and
so
to
have
a
secure
isolation
boundary.
This
really
requires
a
holistic
approach,
because
attackers
will
find
the
weakest
link.
The
other
property
I
want
to
mention
is
that
this
is
a
directional
thing.
As
an
example,
a
process
in
a
container
has
a
good
isolation,
boundary
around
it
and
it's
not
able
to
access
things
on
the
node.
A
A
So
these
are
the
six
layers
I'm
going
to
talk
about
today
and
along
with
the
theme
of
directionality,
we
see
that
the
first
four
layers
are
within
the
cluster,
so
we're
talking
about
something
that's
already
running
in
the
cluster
and
what
is
isolated
within
the
cluster
from
that.
So
why
might
you
care
about
this?
The
first
reason
is
defense-in-depth.
If
you
have
a
service,
that's
exposed
externally,
maybe
there's
a
remote
code,
Vollmer
ability,
remote
code,
execution,
vulnerability
in
that
service
and
an
attacker
manages
to
get
access
to
running
within
your
container.
A
If
you
have
good
isolation,
then
you
can
prevent
that
attacker
from
being
able
to
further
escalate
privileges
and
access
data
within
the
cluster
that
they
shouldn't
have
access
to.
The
second
reason
you
might
care
about
this
is
if
you're
running
any
sort
of
multi-tenant
environment,
maybe
you're
a
SAS
provider,
and
you
have
different
customers
running
on
your
cluster
and
you
don't
want
them
to
be
able
to
affect
each
other.
You
want
good
isolation
between
those
customers.
A
A
Sorry
yeah,
so
they've
done
a
lot
of
work.
We've
adopted
those
in
kubernetes
and
short
of
known
vulnerabilities
that
it's
very
hard
to
escalate
privileges
out
of
a
container
if
you're
using
best
practices.
So
what
are
some
of
those
features
that
enable
container
isolation
in
kubernetes?
We
isolate
CPU
and
memory
resources
and
disk
as
well,
by
setting
requests
and
limits
on
containers.
Cpu
and
memory
is
enforced
by
the
kernel
through
C
groups.
Setting
limits
that
on
what
the
container
can
use
and
disk
is
sort
of
a
softer
enforcement.
That's
done
by
the
cubelet.
A
Kernel
resources
are
isolated
through
things
like
namespaces,
so,
for
instance,
containers
run
in
separate
mount
namespaces
have
separate
file
systems,
and
this
means
that
a
container
one
container
writing
a
file
to
its
root
filesystem.
That
file
isn't
going
to
be
accessible
in
the
other
container
unless
of
course,
you're
writing
it
to
a
shared
volume.
Similarly,
with
process
IDs,
they
can't
read
each
other's
processes
and
there's
a
number
of
other
kernel.
Security
features
that
we
enable
to
reduce
the
attack
service
and
prevent
known
privilege.
A
Escalations
capabilities
is
a
sort
of
coarse
grained
permissions
in
the
Linux
kernel
that
let
us
prevent
certain
classes
of
syscalls
Linux
security
modules
such
as
a
farmer
and
SELinux,
can
further
scope
down
what
a
process
is
able
to
do
and
on
a
parmer
enabled
nodes.
We
enable
the
docker
default
app
armour
profile,
which
prevents
access
to
Sisyphus
and
other
pseudo
file
systems
that
can
be
used
maliciously,
and
then
we
have
some
best
practices.
A
You
can
able
enable
set
comp
through
an
annotation
on
the
pod,
and
this
will
further
reduce
the
number
of
the
syscalls
that
the
container
is
able
to
make
blocking
some
dangerous
syscalls.
But
of
course,
the
best
thing
that
you
can
do
is
to
just
run
as
non-root
if
you're
able
to
do
so.
This
is
by
far
the
the
best
security
guarantee
you
have
in
a
container.
So
what
isn't
isolated
at
the
container
level?
First
of
all,
we
haven't
done
anything
with
network
we'll
get
to
that
in
a
minute
when
we
talk
about
pods.
A
Well,
we've
isolated
some
hardware
resources.
This
isn't
perfect,
sandbox
and
there's
still
some
things
like
disk
contention
or
I,
ops
and
bandwidth
that
aren't
properly
aren't
fully
accounted
for
at
the
container
level.
This
is
something
that
we're
constantly
working
to
improve,
and
so
you
can
look
forward
to
improvements,
and
you
know
becoming
years
in
these
areas,
but
we're
not
quite
up
to
par
with
VM
level
isolation.
A
Here,
another
one
is
kernel
resource
exhaustion,
although
we
have
a
big
namespace,
it's
still
possible
to
run
out
of
process
IDs
through
things
like
fork,
bombs,
we'll
see
an
example
of
that
in
a
second
and
lastly,
identity
is
typically
bound
to
pods.
It
is
possible
to
have
separate
identities
per
container,
but
it's
not
typically
done
so.
A
This
isn't
particularly
sensitive
data,
but
this
is
an
example
of
some
data
about
the
node
that
is
accessible
to
within
the
container.
This
is
the
uptime
of
the
node,
not
the
uptime
of
the
container,
so
let's
go
ahead
and
shutdown
that
node.
Fortunately,
this
operation
is
not
permitted
by
default.
This
is
because
it
requires
the
system
boot
capability,
that's
dropped
by
default
and
docker
there's
actually
a
few
other
mechanisms
as
well
that
prevent
us
from
doing
this,
but
we
have
other
tools
in
our
toolbox.
A
Remember
security
requires
a
holistic
approach,
so
this
you
might
recognize
as
a
fork
bomb.
This
will
spawn
a
process
that
continuously
spawns
two
copies
of
itself
in
doing
so,
it'll
eventually
exhaust
the
kids
that
are
available,
so
new
processes
can't
be
run,
it'll,
cause
CPU
contention
and
eventually
eat
up
a
lot
of
memory
on
the
system
as
well,
and
some
of
these
things
aren't
fully
accounted
for
with
the
C
groups,
so
even
setting
CPU
and
limits
won't
help
you,
fortunately,
as
of
kubernetes
110,
we
have
an
alpha
feature
called
support,
pod
pids
limit.
A
A
So
with
that,
let's
move
on
to
pods
how
much
isolation
is
there
between
two
pods
in
the
same
namespace
on
the
same
node?
Well,
first
of
all
processes
aren't
really
running
in
pods
they're
running
in
containers,
so
we
get
all
of
the
isolation
that
we
already
talked
about
with
containers
and
then
we're
going
to
layer
on
a
bit
more.
A
First
of
all,
we
get
we
finally
get
name
network
isolation,
so
we
have
a
separate
network
namespace
at
the
pod,
and
that
comes
with
a
virtual
Ethernet,
add
device,
that's
associated
with
that
pod,
its
own
IP
address,
and
this
lets
us
apply
network
policy
as
well
for
good
network
isolation.
We'll
see
an
example
in
a
minute
of
why
network
policy
isn't
important.
A
Service
counts
are
typically
associated
with
a
pod,
so
we
now
have
the
ability
to
have
two
separate
pods
with
different
identities.
Running
and
a
lot
of
kubernetes
policies
are
applied
at
the
pod
level.
Pod
security
policy
network
policy
we
just
mentioned
and
scheduling
policy
is
a
work
in
progress
for
constraining
the
way
pods
are
scheduled,
and
then
volumes
are
another
resource
to
mention
that
are
bound
to
the
pod
emptied
ear
volumes,
for
instance,
have
their
lifecycle
tied
to
the
pod.
A
So
when
the
pod
is
terminated,
that
data
is
deleted,
so
what
isn't
isolated
at
the
pod
level?
Well,
as
with
containers,
we
haven't
really
added
a
lot
of
hardware
or
resource
isolation.
We
at
least
have
a
separate
network
namespace.
So
we
have
the
building
blocks
to
add
in
bandwidth
isolation
here,
bandwidth
limiting,
but
we
haven't
implemented
that
yet
kernel
resource
exhaustion
is
still
an
issue.
A
We
saw
how
the
kit
pod
Pibbs
limit
feature
can
be
used
to
prevent
pit
exhaustion,
but
file
descriptors
is
still
an
issue
and
it's
important
to
remember
that
this
is
still
only
a
single
security
boundary.
So,
even
though
we're
talking
second
layer
in
that
initial
diagram,
we're
still
relying
on
the
kernel
to
enforce
all
of
these
things.
For
us
that
means
that
a
single
vulnerability
that
is
exposed,
despite
our
attack
surface
reduction,
could
potentially
be
used
to
escape
out
of
a
container
and
out
of
the
pot
as
well.
A
So
let's
look
at
an
example
of
why
Network
policy
is
valuable
again
I'm,
going
to
run
an
alpine
container
to
get
access
to
my
cluster
Network
and
I'll
go
ahead
and
grab
the
end
map
utility
that
lets
me
scan.
The
network
I
could
scan
every
port
on
every
IP
address
within
the
the
private
cluster
IP
range,
but
this
would
take
a
very
long
time.
So,
let's
not
do
that
instead,
I'll
just
grab
the
IP
address
of
the
first
node.
The
JSON
path
is
just
a
trick
to
grab
the
the
IP
address
I'm.
A
Looking
for
you
could
also
do
it
manually
and
I
can
do
a
similar
thing
to
get
the
the
pod
site
arranged.
So
all
of
the
IP
addresses
that
can
be
used
for
pods.
Now
that
I
have
those
I
can
scan
just
the
the
node
IP
address,
and
we
see
that
this
is
a
kubernetes
master
node.
Taking
a
closer
look
at
some
of
those
ports
as
it's
a
master,
we
see
the
kubernetes
api
port
is
there.
A
We
also
see
that
etsy
D
is
actually
exposed
on
the
internal
network.
Now,
fortunately,
in
this
cluster,
this
is
an
authenticated,
@cd,
endpoint
and
so
I
don't
actually
have
access
to
it.
But
it's
important
to
make
sure
that
@
çd
is
in
fact
authenticated
on
your
network
because
having
access
to
etsy
D
is
equivalent
to
having
root
access
in
the
cluster.
A
Now,
let's
jump
forward
to
port
10
255,
this
is
the
cubelets
unauthenticated
read-only
port,
it's
read-only,
so
I
can't
do
too
much
with
it,
but
I
can
get
some
interesting
metadata
about
workloads
running
in
the
cluster.
Let's
take
a
closer
look
at
that
there's
a
pods
endpoint,
that's
exposed
and
if
I
look
at
that,
endpoint
I
actually
get
a
list
of
every
pod.
Spec
running
on
that
node
I
can
query
every
node
in
the
cluster
and
now
I
know
every
workload
that's
running
in
the
cluster.
A
Fortunately,
the
modern
kubernetes
dashboard
is
used
as
authentication
and
authorization
to
make
sure
that
without
proper
credentials,
I
can't
do
anything
too
harmful
to
the
cluster.
That
wasn't
always
the
case,
and
there
have
been
a
few
high
profile
breaches
caused
by
dashboards
that
were
exposed
publicly
by
accident,
so
make
sure
you're
running
an
up-to-date
dashboard
and
have
proper
controls
on
that.
A
So
all
of
these
things
could
be
blocked
with
a
good
Network
policy.
For
instance,
if
I
have
a
workload
that
doesn't
need
to
talk
to
nodes
or
need
access
to
the
dashboard.
I
can
easily
block
that
with
network
policy
as
just
a
couple
examples.
So,
let's
move
on
to
talking
about
kubernetes
namespaces.
This
is
different
from
the
Linux
namespaces.
We
talked
about
at
the
container
level,
so
with
namespaces
I
get
stronger
identity,
isolation.
We
already
mentioned
that
we
have.
A
We
can
have
different
identities
at
the
pod
level,
but
by
default,
all
pods
in
the
namespace
run
with
the
same
service
account
with
different
namespaces.
It's
actually
not
possible
to
run
those
pods
with
the
same
service
account
and
along
with
that,
I
have
stronger
authorization,
controls
and
we'll
look
at
an
example
of
that
in
a
second
and
then.
A
Lastly,
a
bunch
of
kubernetes
resources
are
bound
to
namespaces
and
having
pods
in
separate
namespaces
limits,
their
ability
to
access
some
of
these
resources
without
without
granting
access
explicitly
through
the
control
plane
through
our
back
or
something
like
that.
But
it's
important
to
note
that
namespace
isolation
and
named
kubernetes
namespaces
is
purely
a
control,
plane
concept.
A
So
I
mentioned
that
namespaces
are
sort
of
a
building
block
of
the
fundamental
building
block
for
authorization.
It's
actually
possible
to
limit
access
to
specific
resources
in
a
namespace
through
our
back,
but
doing
so
has
some
nuances
that
can
be
problematic.
So,
let's
look
at
why
here's
a
user
that
is
forbidden
from
accessing
secrets
in
the
default
namespace.
However,
they
do
have
the
ability
to
create
pods.
A
I
have
the
permissions
and,
at
this
point,
there's
very
many
different
ways
that
I
can
exfiltrate
this
data
out
of
this
pod,
just
to
keep
things
simple,
I'll
assume
that
I
have
exec
access
and
can
copy
the
data
out
and
now
I
have
access
to
the
secret
despite
being
forbidden
from
accessing
secrets.
So
what
can
you
do
about
this?
A
It's
important
to
kind
of
understand
the
nuances
of
the
all
the
different
permissions
that
you're
granting,
but
to
make
it
a
little
easier.
We
have
a
few
built-in
roles
in
kubernetes,
the
viewer
and
the
roll
which
went
down
dat.
The
namespace
level
will
appropriately
limit
to
what
is
actually
granted
by
that
role.
A
So,
let's
move
on
to
talking
about
nodes
now
and
we're
gonna
shift
gears
a
bit
here.
So
far,
we've
been
talking
about
isolating
processes
running
within
a
container,
but
now
at
the
node
level,
I'm
actually
is
going
to
assume
that
an
attacker
has
already
managed
to
escape
the
container.
So
maybe
it
was
running
as
a
privileged
container
and
there's
a
number
of
ways
to
get
out
from
there
or
maybe
maybe
there
was
a
kernel
vulnerability.
They
were
able
to
exploit.
A
So,
in
addition
to
that,
we
finally
have
hardware
level
resource
isolation,
or
at
least
as
good
as
our
VM
infrastructure,
that
we're
running
on
so
things
like
dis
contention
and
bandwidth
generally
are
accounted
for
much
better
at
the
node
and
what
isn't
isolated
at
nodes?
Well,
we've
actually
taken
a
step
back
in
network
isolation.
At
the
pod
level,
we
were
able
to
use
network
policy
to
firewall
off
the
network
from
the
pod,
but
pod
security
or
sorry
network
policy
applies
to
pods
and
not
to
nodes.
A
Additionally,
even
if
we
firewalled
off
the
network
nodes
have
access
to
a
lot
of
metadata
about
the
cluster.
The
cubelet
can
list
all
pods
running
in
the
network
or
running
in
the
cluster,
and
the
the
cubelet
needs
access
to
a
number
of
other
things
like
pod
credentials
and
secrets.
Also,
in
addition
to
the
cubelets
resources,
we
also
have
access
to
every
credential
used
by
every
workload
on
that
node
and
all
of
the
permissions
associated
with
those
credentials.
A
So
let's
look
at
an
example
of
that.
Here's,
a
simple
setup
I,
have
two
nodes:
two
pods
two
secrets
to
make
it
easy
to
remember:
pod
one
mounts
secret
one
and
runs
on
node
one
same
with
pod
two,
so
let's
SSH
into
node
one
and
see
what
we
can
do
using
the
the
cubelet
config
and
the
cubelet
credentials.
I
can
use
cube
cuttle
as
I
normally
would,
but
acting
as
the
nude,
the
cubelet
identity.
A
A
Similarly
pod
two
mounts
secret
two
runs
on
node
two
remember
I'm
still
running
as
node
one
node
one
doesn't
need
access
to
secret
two
assuming
there
aren't
any
other
pods
running
on
node,
one
that
mount
that
secret.
So
we
see
that
if
I
try
and
get
secret,
actually
I'm
forbidden
from
doing
so,
and
the
reason
this
works
is
because
of
the
node
authorizer
and
node
restriction
plugins.
A
These
plugins
maintain
a
graph
of
all
of
the
resources
that
are
referenced
by
every
workload
running
on
a
node
and
restrict
access
to
to
the
resources
that
the
node
doesn't
actually
need.
Access
to.
This
was
introduced
in
kubernetes,
1
7
and
we've
been
making
improvements
to
it
since
then
further
scoping
down
what
the
node
can
do
and
ways
that
it
can
steer
pods
to
itself
in
order
to
gain
access
to
things
that
those
pods
need
now
I
want
to.
A
The
goal
of
sandbox
isolation
is
to
bring
the
second
security
boundary
and
a
lot
of
the
things
we
just
talked
about
around
node
isolation
and
bring
them
to
the
pod
level.
So
in
some
ways
this
is
actually
even
better
than
note
isolation
right,
because
we
talked
about
how
at
the
node
level,
we
took
a
step
back
in
the
network,
isolation
by
applying
a
sandbox
to
the
pod.
A
We
get
a
second
security
boundary
at
the
pod
and
can
still
maintain
that
network
isolation
and
the
way
we
are
doing
this
is
with
technologies
like
G,
Weiser
and
kata
containers
that
bring
kind
of
VM
strength,
isolation
and
apply
it
at
the
pod
level.
So
this
isn't
increasing
sanba
or
sorry
container
isolation,
but
it's
applying
a
new
sandbox
to
the
pod.
A
I'm
gonna
do
it
on
time.
Alright.
So
let's
talk
about
clusters
now
so
up
until
now,
we've
been
talking
about
things
running
within
the
cluster
and
what
what
is
isolated
once
you
already
have
access
within
the
cluster
at
the
cluster
level,
we
now
have
an
exterior
boundary.
So
the
first
thing
we
add
in
cluster
isolation
is
this
network
perimeter
it's
now
possible
to
firewall
off
the
cluster
in
a
much
more
fine-grained
way
or
even
to
air-gap
the
cluster
completely
in
extreme
scenarios.
A
We
also
have
a
separate
control
plane.
This
means
possible
to
have
a
completely
separate
identities
and
access
controls
for
each
cluster
and
a
separate
data
store
means
that
a
compromise
of
the
cluster
will
still
be
limited
to
the
scope
of
that
cluster.
So
now
we
have
full-on
kubernetes
security,
totally
separate
clusters.
Are
we
done
spoilers
I
already
said
we
have,
but
one
more
layer
to
talk
about,
so
what
isn't
isolated
at
the
cluster
level?
Well,
this
at
this
point.
It
really
depends
on
your
setup.
Where
are
you
running
your
cluster?
A
If
you
are
running
in
google
kubernetes
engine,
your
security
model
at
this
point
is
going
to
look
very
different
than
if
you're
running
on
bare
metal
in
your
own
data
center.
So
the
questions
you
need
to
ask
are
what
other
resources
are
exposed
on
the
network,
what
other
services
are
exposed
if
you're
running
in
the
cloud?
How
are
acts?
How
is
access
to
those
services
authorized
and
do
nodes
in
your
cluster,
for
instance,
have
access
to
those
cloud
services,
and
so
this
brings
us
to
the
the
final
layer
here.
A
A
But
it's
just
kind
of
comes
back
to
these
two
questions
of
understanding.
What's
what
else
is
exposed
on
your
network
in
the
data
center?
Do
you
have
other
workloads
running
outside
of
kubernetes
that
might
be
might
be
running
in
an
unauthenticated
way
that
attacker
could
exploit
from
a
cluster,
and
we've
actually
seen
we've
seen
examples
of
this.
A
So,
to
recap:
we
started
out
at
the
container
level
talking
about
some
resource
isolation
that
we
get
and
a
number
of
kernel
security
features
that
hard
in
that
container
isolation
boundary
at
the
pod
level.
We
add
in
network
isolation
and
a
little
bit
more
resource
isolation
with
kubernetes
namespace.
A
We
get
stronger
control,
plane,
isolation
and
then
at
the
node
level
we
finally
add
in
a
second
security
boundary,
much
stronger
resource
isolation
and
better
data
isolation,
as
well
with
the
cluster
level,
we
have
stronger
network
isolation,
still
separate
control
planes
and
separate
authorization
mechanisms,
mean
better
control,
plane,
isolation
and
data
access.
That
comes
with
that
and
then
finally,
at
the
infrastructure
level.
This
is
really
as
good
as
it
gets.
A
So
in
conclusion,
which
layer
should
you
design
your
application,
for
which
layer
should
you
plan
on
having
isolation
at
I'm,
not
suggesting
that
every
workload
should
be
in
its
own
cluster
in
its
own
data
center?
That
would
be
absurd,
very
expensive
and
hard
to
design
and
defeat
the
purpose
of
kubernetes
completely.