►
From YouTube: How Symlinks Pwned Kubernetes (And How We Fixed It) - Michelle Au, Google & Jan Šafránek, Red Hat
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How Symlinks Pwned Kubernetes (And How We Fixed It) - Michelle Au, Google & Jan Šafránek, Red Hat
Ever wonder how Kubernetes deals with security vulnerabilities? This talk illustrates the process by walking through the discovery, patching, and disclosure of CVE-2017-1002101. In Nov 2017, we received a report about how misusing the volume subpath feature could result in access to host files. A team was assembled to investigate the vulnerability, develop a patch, and release it to all supported versions of Kubernetes -- ALL in secret. As we walk through the story from discovery to disclosure, we will also deep dive into the technical details of how this feature allowed a container to escape to the host filesystem, and how it was fixed. You will walk away with techniques for secure file handling in multi-tenant environments, best practices for restricting volume access in your Kubernetes clusters, and an understanding of how a large open source project manages security issues.
To Learn More: https://sched.co/GrZc
A
A
Well,
take
you
through
the
whole
process
from
when
the
issue
was
first
reported
to
when
the
fix
was
publicly
released
along
the
way
we
will
deep
dive
into
the
technical
details
of
the
vulnerability
and
the
fix
suggest
some
best
practices
that
you
can
follow
in
your
kubernetes
clusters
and
also
give
a
sneak
peek
at
future.
Directions
were
exploring
in
kubernetes
to
make
it
more
secure.
A
Well,
in
late
November
of
2017,
a
kubernetes
issue
was
opened
with
the
title.
Pop
security
policy
can
be
sidestepped
with
innocent
and
Peter
and
subpath.
The
issue
then
goes
on
to
describe
the
exploit
in
full
detail
with
the
pod
examples
and
output.
I
want
to
emphasize
here
that
this
is
not
the
proper
channel
to
report
security
issues
in
kubernetes.
Github
issues
are
public
and
don't
allow
time
for
us
to
privately
develop
and
release
a
fix
before
others
can
start
exploiting
the
vulnerability.
A
So
if
you
ever
think
you
have
found
a
security
issue
in
kubernetes,
please
follow
our
documented
process
on
kubernetes
I/o.
It
starts
with
sending
an
email
to
security
at
kubernetes
IO,
with
the
details
of
the
issue
and
a
member
of
the
product
security
team
will
evaluate
the
impact,
respond
back
to
you
and
then
coordinate
the
development
of
the
fix
and
its
release.
A
So
now
I'm
going
to
go
into
the
technical
details
of
this
particular
vulnerability
that
we
worked
on
to
understand
the
vulnerability.
Some
basic
background.
Knowledge
on
volumes
is
needed
so
to
illustrate
this
I
have
an
example
pots
back
here,
and
this
pot
is
using
a
volume
when
this
pod
gets
scheduled
to
a
node.
A
What
we
do
is
we
create
a
directory
on
the
host
file
system
underneath
the
pod
volumes
root
path,
as
shown
here
and
once
all
the
volumes
have
been
set
up
and
mounted
on
the
host
system,
then
it's
time
to
start
the
containers
to
start
the
container
cubelet
needs
to
tell
the
container
run
time
in.
In
this
case
it
could
be
docker
and
needs
to.
You
need
to
tell
docker
where,
on
the
host
file
system,
this
volume
is
located
at
and
where,
in
the
container
it
should
be
mounted
to.
A
You
can
also
see
in
this
prospect
that
there
is
a
sub
path
of
data.
One
specified
what
this
sub
path
lets
you
do
is,
instead
of
mounting
the
root
directory
of
the
volume
into
the
container
it
lets
you
mount
a
subdirectory
of
the
volume
into
the
container
and
to
handle
this.
What
we
do
in
kubernetes
is
instead
of
telling
docker
the
root
directory
of
the
volume
we
append
the
sub
path
to
that
root
directory
and
give
that
combined
path
to
doctor.
A
So
if
we
take
this
specific
example,
we'll
take
the
root
directory
at
the
volume
from
before
append
the
sub
path
of
data
1
to
it,
give
this
entire
path
to
doctor
and
then
docker
will
go
ahead.
Take
that
path
and
bind
mount
it
into
the
container
at
the
containers.
Mount
path
of
slash
mount,
slash
data.
A
B
A
A
Symbolic
Lake
gray,
yes,
someone
paid
attention
to
the
title.
This
talk
great,
so,
if
you
take,
if
you
pay
careful
attention
here,
you'll
notice
that
this
data,
one
which
is
specified
in
the
sub
path,
is
a
file
that's
created
by
the
user
inside
their
volume
and
because
it's
created
by
the
user.
That
means
this
file
can
be
anything
including
a
symlink.
A
So
if
we
pretend
let's
say
this
sim
link
points
to
something
on
the
root
file
system
of
the
host,
then
when
we
end
up
giving
this
path
to
docker,
docker
will
go
and
bind
mouth
whatever.
That's
in
link
points
to
into
the
container,
instead
of
the
your
actual
volume
that
you
specified
so
now,
yon
is
going
to
go,
demonstrate
exactly
how
you
can
create
such
a
sin.
Link
thanks.
B
So
I
hope
it's
readable.
I
have
kubernetes
1.9
dot,
oh,
that
is
still
vulnerable
and
I
have
a
port.
That's
just
very
innocent
pot
that
has
one
empty
air
volume
called
my
volume
as
on
initial
examples
and
I
have
an
init
on
container
that
creates
a
simmering
inside
the
volume.
This
tubing
is
called
data
one
and
it
points
to
the
root
of
the
container
file
system.
B
B
It
there
in
container
starts,
it
creates
a
symlink
and
the
real
container
starts
and
if
I
look
inside
the
container
I
have
the
my
volume
there
and
it
contains
what
I
created
it
contains
just
the
same
link
to
root
of
the
container,
and
there
is
nothing
wrong
with
the
ceiling,
but
the
magic
happens.
If
I
look
at
the
sub
path
data
one
it
contains
root
of
some
file
system,
but
it's
not.
The
host
host
is
much
smaller
and
this
is
sorry.
B
It's
not
the
root
of
the
container
and
the
sub
path
contains
root
of
the
host,
so
I
can
look
around
and
I
can
see,
for
example,
docker
socket,
so
I
can
kill
or
start
any
containers
on.
The
note
and
the
whole
note
is
compromised.
I
can
do
whatever
I
can
look
at
water,
Lape,
cubelet
pots,
I,
don't
run
so
many
pots,
but
I
can
see
all
volumes
mounted
on
the
node
I
can
see
all
the
secrets
mounted
on
the
node.
B
B
B
B
The
first
and
most
obvious
solution
was
to
check
ceilings
before
we
give
them
to
contain
runtime.
So
if
user
wants
volume,
my
volume
and
sub
path
data
1,
we
compute
the
whole
path
of
the
volume
that
user
once
we
receive
all
symlinks
and
if
see,
if
we
see
that
the
ceiling
points
somewhere
outside
of
the
volume.
That
means
that
the
user
is
trying
to
chip
very
few
such
volume
and
we
won't
start
the
pot
if
the
sub
path
is
not
something
at
all,
which
is
what
symlink.
B
Instead
in
that
we
get
the
same
path
as
the
result.
We
can
see
the
sub
path
points
inside
the
volume
and
it's
safe
to
use.
So
we
allow
that
and
if
the
stop
bath
is
symlink
but
the
ceiling
points
inside
the
volume
we
want
to
allow
that
to
because
people
already
use
that
and
it's
pretty
useful
feature
of
kubernetes.
So
we
want
to
allow
this
use
case.
So
when
we
evaluate
this,
we
either
allow
or
deny
the
pot
and
give
the
final
destination
of
the
symlink
to
contain
the
runtime.
This
is
obviously
wrong
because.
B
User
can
modesty
or
modify
in
the
directories
that
we
are
checking.
So
if
we
check
that
whatever
user
wants
to
use,
the
sub
half
is
safe
to
use
it's
inside
the
volume.
The
act
of
the
check
user
can
remove,
for
example,
in
this
case
directory
C
and
replace
it
with
a
symlink
to
a
bad
place
to
the
objector
host,
and
we
give
this
a
stir
to
be
/c
to
contain
runtime.
But
at
this
time
see
is
a
symlink
to
the
host
and
again
user
can
get
access
to
some
bad
places.
We
don't
want
to.
B
We
don't
want
them
to
use,
so
we
were
thinking.
We
need
to
freeze
the
filesystem
somehow,
while
we
are
checking
the
siblings
and
when
we
are
giving
the
final
destination
to
continue
runtime
on
Windows,
the
situation
is
pretty
straightforward
because
they
have
create
five
Cisco,
which
not
only
creates
file,
but
with
the
right
parameters.
It
opens
them
and
it
can
open
them
exclusively,
so
they
can't
be
replaced
with
symlinks.
So
on
Windows
we
just
lock
each
component
of
the
path.
B
We
check
it's,
not
a
symlink
and
when
everything
is
alright,
it
looks
valid
and
it's
inside
the
volume
we
give
it
to
container
runtime
and
container
runtime
stars
container
after
the
container
runs.
We
unlock
the
path,
that's
easy
on
Linux,
we
don't
have
such
luxury.
We
can't
freeze
file
system
while
we
can,
but
it
would
stop
the
application,
that's
running
and
uses
the
file
system,
and
we
don't
want
to
do
that.
So
if
we
were
thinking
how
how
to
make
things
safer
and
the
solution
was
to
use,
bind
mounts,
you
don't
want.
B
If
you
don't
know
what
the
bank
mount
is,
it
takes
piece
of
filesystem
hierarchy
for
example
directory
and
makes
it
available
somewhere
else.
So
in
this
example,
we
bind
mount
the
volume
and
it's
a
path.
My
volume
/,
a
/
b
/
see
that's
the
step
pass
that
user
wants
to
make
available
to
his
container
and
we
bind
mount
to
a
safe
place
that
it's
outside
of
users.
Control
user
can
modify
my
volume
and
its
content,
but
user
can't
modify
this
safe
place
and
benefit
of
this
bind
mount.
B
Is
that
after
the
vine
mount
this,
my
balloon?
Abc
shows
the
same
content
like
the
safe
place,
but
use,
and
if
user
changes
anything
in
ABC,
for
example,
create
some
bed
symlinks,
it
doesn't
influence
the
safe
place.
The
safe
place
is
still
safe
and
it's
still
shows
the
original
content
of
the
volume
so
another
very
naive
and
very
wrong
solution.
B
We
check
the
same
lines
as
before
in
validate
that
the
symlinks
are
quoting
inside
the
volume
and
user
is
not
trying
to
cheat.
We
bind
mount
the
final
destination
of
any
sim
link
that
you
user
used
to
a
safe
place.
In
this
case.
It's
something
var
lip
cubed
pots
volume
that
path
and
something
unique,
and
we
give
this
safe
place
to
contain
a
run
time
instead
of
content
of
the
volume.
So
this
way
we
have
a
safe
way
how
to
give
that
path,
volume
to
contain
runtime,
but
everything
before
the
bank
mount
is
still
unsafe.
B
While
we
are
resolving
siblings,
while
we
are
checking
the
destination,
users
still
can't
modify
content
of
his
volume
or
her
volume
and
do
bad
things
there
for
a
long
time.
We
didn't
know
how
to
fix
it
properly,
but
then,
if
somebody
suggested
a
smart
trick
to
open
file
descriptor
of
the
sub
path
and
bind
file
descriptors
instead
of
directories
or
files,
we
start
the
same
like
before.
We
dissolve
the
same
link,
a.
B
And
what
this
gives
us
is
that
at
some
point
in
time,
not
so
long
ago,
user
wanted
to
use
directory
a
slash
b
/c
as
the
sub
as
the
suppose
volume.
It
doesn't
need
to
the
the
data.
One
sibling
can
be
now
completely
different,
but
at
some
point
in
the
history
it
was
the
volume
that
user
wanted
and
now
the
goal
is
to
open
the
last
component
of
the
path
that
means
C
and
gets
its
file
descriptor.
B
We
can't
just
do
open
ABC
because
it
would
resolve
all
the
same
links
in
the
path
and
user
could
replace
already
a
or
b
or
c
if
siblings,
and
but
we
must
do
it
smarter
and
we
can
open
the
last
directory.
That's
not
an
recent
role.
My
volume
user
can
see
content
of
the
directory
inside
the
airport's,
but
they
can't
replace
my
volume
itself
if
a
symlink
and
then
we
safely
dive
in
into
a
into
B
into
C.
B
We
don't
follow
any
same
links
and
we
get
safe
file
descriptor
of
the
volume
that
user
once
wanted.
That
was
once
reachable
as
a
slash
b
/c.
It
doesn't
need
to
be
reachable
as
h.
/
b
s
/
c
now,
because
user
can
rename
things
and
move
them,
but
the
point
is
user
can
move
them
only
inside
the
pot?
They
can't
escape
the
container
right
now.
B
So
now
we
have
the
file
descriptor
of
the
thing
that
user
who
want
it
to
use,
and
how
do
we
feel
every
bind
mount
file
descriptors,
that
not
possible
you
can
bind
mount
fires
or
directories,
but
not
file
descriptors.
The
Lance
doesn't
make
sense,
but
there
is
there's
the
trick.
If
you
look
at
the
proc
file
system
of
cubelet
broke
is
basically
virtual
file
system.
That
shows
two
numbers
and
statistics
of
all
processes.
On
linux.
B
There
is
a
virtual
directory
called
FD
that
contains
all
of
open
file
descriptors
and
if
you
look
at
item
item
name,
13
is
a
sim
link
to
the
the
location
of
the
file
as
open
and
the
sibling
is
not
tied
to
the
path
it's
tied
to
I
know.
So,
whenever
user
moves
A
or
B
or
C
inside
the
container,
the
sealing
follows
to
the
nudist
destination
and
we
might
buy
Mount
this
magic
symlink
instead
of
ABC
directly,
and
what
can
cattle
does
he?
B
B
It
took
us
quite
a
long
time
to
implement
this
solution.
This
was
just
the
high-level
idea.
There
were
many
hurdles
to
climb
right.
End-To-End
test
fix
the
issue,
also
on
Windows
and
so
on,
and
how
could
we
do
that?
Actually
in
secret
in
kubernetes?
If
we
develop
the
fix
in
kubernetes
/
kubernetes
repository,
we
could
create
PR
there
and
develop
the
fix
there,
but
the
development
would
be
open
and
somebody
smart
could
see
hey
this
is
security.
Related
I
can
I
can
I
can
misuse
it.
B
So
there
is
a
kubernetes
/,
kubernetes
security
repository
on
github.
You
can
look
at
it
and
you
won't
find
anything
there.
You
get
four
or
four,
it's
private.
Only
the
security
product
security
team
can
access
it
and
they
give
access
to
the
repository
only
to
selected
people
who
need
to
fix
security
issues.
So
in
this
case
it
was
me
it
was
Michelle
and
Andy
from
Microsoft.
Nobody
else
could
know
about
the
you
and
we
immediate
immediately
lost
access
to
the
repository
when
the
fix
was
released.
B
B
The
only
difference
to
me
was
that
locks
from
the
CI
who
are
in
private
buckets
accessible
only
to
Google
employees,
so
I'm
from
Red,
Hat
and
I'm
in
Europe,
and
if
CI
geo
failed
and
I
needed
to
wait
for
Michele
to
wake
up
and
tell
me
what's
wrong
and
we
needed
to
be
careful
where
we
push
things,
because
if
we
pushed
our
code
to
our
public
Forks
on
github
again,
it
would
make
the
issue
publicly
available
and
all
the
development
is
coordinated
by
product
security
team.
So
as
a
developer,
I
just
follow
orders
and
I.
B
Don't
need
to
worry
about
the
security
aspect
of
the
thing.
I
have
a
safe
place
to
develop
the
issue
and
do
some
tests.
When
we
had
the
fix
it
was
time
to
start
thinking
about
releasing
it
and
again,
it
was
coordinated
by
product
security
team
me
as
the
developer.
I
just
follow
orders
and
the
products,
even
with
security
team,
talked
to
all
the
branch
managers
when
it's
a
good
time
to
release
the
fix
and
also
the
security
issue
could
be
announced
to
private
distributors
list,
which
is
a
list
of
third-party
kubernetes
distributors.
B
These
distributors
and
vendors
they
get
the
fix
and
the
issue
before
it's
publicly
released
under
Margot,
so
they
can
prepare
the
fix,
they
can
patch
their
products
and
they
can
publish
the
fix
after
the
are
Margaux
is
lifted.
So
all
the
products
earth
are
safe,
then
the
issue
is
public,
so
the
public
disclosure
happened
four
months
after
the
initial
discovery
on
github.
So
it
took
us
four
months
to
prepare
everything.
B
When
this
was
done,
then
all
the
people
who
were
involved
in
the
security
fix
that
means
developers,
testers
release
team,
told
the
security
they
met
together
and
created
post-motor
document.
This
post
mortem
is
public.
You
can
read
it
and
it
contains
like
what
went
wrong,
but
then
right
what
can
be
improved
in
the
future.
B
Yeah
and
that's
about
it
like
at
this
point,
we
lost
our
access
to
security
repository,
so
we
don't
know,
I
can't
even
check
the
history
of
the
pull
request
and
I'm
handing
over
back
to
Michel
Google,
to
tell
you
something
about
how
to
protect
yourself
from
this
issue.
If
something
like
that
is
found
in
the
future,.
A
First
I
recommend
that
you
don't
run
your
containers
as
the
root
user
whenever
possible
for
this
particular
vulnerability.
If
a
container
was
exploited
and
got
access
to
the
hosts
file
system,
then
the
damage
it
could
have
done
would
be
limited
to
only
those
files
that
the
user
has
permissions
to
access.
A
A
If
you
still
have
to
use
hosts
path
volumes,
then
you
should
at
least
set
both
the
allowed
host
path.
Prefixes
and
the
read-only
setting
it's
very
important
to
note
that
the
allowed
host
path
prefix
setting
in
pod
security
policy
is
only
effective
if
you
also
require
that
your
containers
mount
them
as
read-only
as
well.
A
A
Some
of
this
work
is
based
off
of
a
core
principle
that
we
follow
at
Google,
which
is
that
any
untrusted
code
must
be
protected
by
at
least
two
security
boundaries.
The
basic
idea
here
is
that
if
any
one
of
the
layers
security
layers
gets
compromised,
then
there
are
still
other
layers
that
can
provide
isolation.
A
So
how
can
we
apply
this
principle
to
protect
us
from
the
subpath
loner
ability
going
back
the
exploit
involved,
making
the
host
system
follow
siblings
that
were
created
by
the
user,
so
if,
instead,
we
have
a
sandbox
environment
that
can
provide
our
additional
security
boundary
and
we
and
we
moved
that
processing
of
the
untrusted
sub
paths
into
the
sandbox,
then
any
send
links
that
we
would
end
up.
Resolving
could
only
resolve
two
paths
contained
inside
that
sandbox
preventing
access
to
the
hosts
file
system.
A
So
this
is
an
idea
that
we
are
currently
investigating
today,
so
this
doesn't
exist
yet
in
kubernetes,
but
expect
to
see
some
movement
over
the
next
year
or
so
right.
So
that
is
our
full
story.
I
know
that
was
a
lot
of
information
that
we
packed
into
here.
So,
to
summarize,
the
key
points:
first,
if
you
ever
think
you
found
the
security
issue
in
kubernetes,
please
responsibly
disclose
the
issue
by
following
our
documented
process.
A
Next
be
extra
cautious
when
handling
untrusted
paths
in
your
own
programs.
We
detailed
today
two
common
vulnerabilities,
a
sibling
race
and
a
time
of
checked
a
time
of
use
race
and,
lastly,
asset
restrictive
policies
in
your
kubernetes
clusters
by
default
and
design.
Your
applications
to
provide
multiple
security
boundaries.
A
A
So,
if
anything
we
talked
about
today,
interest
you
check
out
these
links.
First
is
the
link
to
the
product
security
team,
which
coordinates
and
handles
all
the
security
issues
in
kubernetes.
They
are
looking
to
four
new
members,
so
you
can
check
out
their
page
on
the
process
on
how
to
join.
Second
is
the
secure
container
isolation
project?
This
is
the
project
that
is
looking
into
providing
sandbox
environments
and
there's
actually
a
talk
later
today
in
this
very
room
on
this
project.
A
B
A
B
When
we
had
the
fix
the
product
security
team
has
some
rules.
When
a
security
issue
can
be
made,
public
can
be
published
like
never
disclosed
and
issues
on
Fridays
and
yeah.
They
just
took
a
random
date
when
all
the
branch
managers
had
I
don't
want
to
say
that
time
they
had
time
all
the
day,
but
they
were.
There
was
no
release
in
the
progress
process
and
there
was
enough
time
to
do
let
the
kubernetes
vendors
know
before
the
issue
is
published,
so
they
can
fix
their.
All.
B
A
A
A
Okay,
the
question
was
so
we
said
it
took
about
four
months
to
develop
the
fix.
How
much
time
was
spent
actually
developing
and
testing
to
fix
versus
the
release
process.
I
would
probably
say,
like
three
and
a
half
months
were
spent
fixing
and
testing
the
fix.
We
went
through
probably
four
or
five
different
iterations
of
the
fix
until
we
arrived
at
one
that
we
were
confident
that
would
actually
patch
the
issue.
Oh.
A
B
A
I
think
this
was
one
of
the
first
vulnerabilities
where
we
really
tested
the
CI
infrastructure
for
the
this
private
repo,
so
I
think
actually
the
first,
like
two
months
of
it,
was
the
test
and
for
team
trying
to
like
get
everything
the
CI
to
work
so,
but
definitely
in
the
post-mortem
document,
which
is
public,
there's
a
whole
list
of
action
items
and
stuff
on
how
to
improve
the
process.
So
you
can
check
that
out
if
you're
interested
I.