►
From YouTube: Isolate the Users! Supporting User Namespaces in K8s for Increased Security - Mauricio Vásquez
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Isolate the Users! Supporting User Namespaces in K8s for Increased Security - Mauricio Vásquez, Kinvolk
Running a process as root inside containers is a security risk: if such a process is able to break out of the container into the host, it can cause considerable damage as it will be running as a privileged user there. The good news is that Linux has a solution for this problem: user namespaces isolate user and group IDs, so a process running as root in a container runs as non-root in the host. The bad news is that Kubernetes doesn’t yet support user namespaces. So, we created a Kubernetes Enhancement Proposal (KEP-127) with a plan to bring this support to a future release. We also implemented a prototype of this idea in Kubernetes and containerd. In this talk, I’ll introduce user namespaces and how they can increase the security of a Kubernetes cluster. I’ll explain how we are working with the community to bring this support to Kubernetes, the challenges we are facing, in particular with volumes, and how different approaches like shiftfs and idmapped mounts are trying to fix them.
A
Hello,
everybody
welcome
to
this
presentation
about
supporting
user
namespaces
in
kubernetes
and
mauricio.
I
work
as
a
software
engineer
for
kinfolk,
and
this
is
my
social
data.
Just
in
case
you
want
to
reach
out
in
this
presentation.
I
will
be
explaining
you:
what
is
the
motivation
to
implement
support
for
username
spaces
in
kubernetes?
That
most
situation
is
the
risk
of
running
containers
as
well.
A
I
will
also
present
a
demo
about
the
approval
of
concept
that
we
have
implemented
and,
finally,
I
will
explain
what
are
the
next
steps
that
we
have
to
do
in
order
to
have
this
support
implemented
in
coordinates?
So,
let's
get
started
with
the
problem.
The
main
motivation
of
implementing
username
spaces
in
kubernetes
is
that
running
containers
as
root
is
very
dangerous,
and
when
we
say
the
a
container
is
running
normal,
what
it
means
is
that
a
process
inside
the
container
is
running
as
root.
A
A
These
are
some
examples
of
some
vulnerabilities
that
have
been
found
in
the
last
year
that
could
have
been
mitigated
by
user
name
spaces.
The
last
one
is
a
special
critical
one,
because
in
this
case,
an
attacker
is
able
to
overwrite
the
run
c
binary
in
the
host
by
just
using
a
special
container
image.
So
after
that
happens,
the
attacker
will
have
full
control
over
the
house
machine.
A
As
I
told
you,
these
are
just
some
examples
of
vulnerabilities
that
had
been
found
had
been
found
in
the
past
and
we
changed.
This
is
very
probably
the
in
the
future.
There
are
going
to
be
more
vulnerabilities,
and
also
this
this.
This
is
possible
that
there
are
some
current
vulnerabilities
are
being
exploited
by
the
attackers.
A
Let's
see
some
of
the
mitigations
that
we
could
use
in
order
to
lower
the
risk
of
running
route
containers,
but
actually
the
first
question
that
we
have
to
do
is:
do
we
really
need
to
run
the
containers
as
well
in
many
of
the
cases?
The
answer
to
that
question
is
not
many
of
the
containers
that
are
running
iranians
rule
today
could
be
running
without
road
without
any
issue.
A
A
We
could
use
the
policies
called
security
policies
and
also
a
mechanism
like
the
open
policy
agent
to
control,
who
is
able
to
run
containers
as
roots
other
mitigations
that
we
that
we
could
do
is
to
limit
the
set
of
capabilities
that
we
give
to
the
pod
just
to
give
the
capabilities
that
the
pod
needs
and
not
give
any
extra
capability
to
the
port.
Also,
another
possibility
is
to
use
orthogonal
security
features
that
are
supported
by
linux
and
also
by
coordinators
and
yeah.
A
The
relationship
between
the
ids
inside
and
outside
the
name
space
is
con
is
controlled
by
an
id
mapping.
So
what
we
usually
have
is
all
we
have
in
the
initial
host
on
the
host
username
space.
We
have
all
the
ids
available,
and
then
we
assign
a
portion
of
the
range
to
a
pod.
For
instance,
in
this
case,
we
are
saying
that
the
uid0
on
the
pod
is
going
to
be
100
000
on
the
horse,
and
we
are
going
to
map
64
key
ids.
A
We
can
have
multiple
ports
for,
for
instance,
in
this
case
we
have
a
second
part.
The
only
difference
here
is
that
we
are
mapping
to
the
id
200
000
on
the
host.
This
is
interesting
to
notice.
In
this
example,
the
both
parts
have
a
different
uid
on
the
horse.
So
this
is
what
we
call
non-overlapping
ranges
on
the
hose,
and
this
is
important
from
a
security
point
of
view,
because
we
are
able
to
insulate
the
pots.
It
means
that
we
could
mitigate
some
of
the
attacks
that
one
part
could
perform
against
other
paths.
A
A
A
process
is
running
in
in
order
to
understand
if
the
process
is
able
to
perform
the
privilege
operation
or
not,
and
there
is
a
relationship
between
non-username
spaces
on
username
spaces,
user
non-username
spaces,
like
their
network,
namespace,
pid,
namespace
and
so
on,
are
always
owned
by
a
username
space
and
a
process
can
a
process
that
has
a
capability
on
an
username
space
can
only
use
that
capability
on
resources
that
are
owned
by
that
specific
user
name
space.
I
know
that
this
is
not
easy
to
understand
from
from
that
definition.
A
A
The
the
initial
username
is
based
on
the
host
and
we
also
have
a
network
name
space
on
the
host.
In
this
case,
the
network
name
space
of
the
host
is
on
by
the
initial
user
name
space.
In
this
example,
we
also
have
a
port.
This
part
is
running
with
a
different
username
space.
This
username
space
is
owned
by
the
username
space
of
the
host
and
also
the
network.
Namespace
of
the
pod
is
owned
by
the
username
space
of
the
path.
Let's
suppose
that
we
have
a
process
that
is
running
inside
the
pot.
A
What
it
means
is
that
the
process
is
running
inside
that
username
space
and
inside
that
network
namespace
and
the
process
wants
to
perform
a
privileged
operation
on
a
resource
that
is
a
controlled
by
the
network
namespace
of
the
pod,
for
instance.
Let's
suppose
that
the
process
wants
to
bind
to
up
to
the
480.
That
requires
a
capability.
A
A
On
the
other
hand,
let's
suppose
that
in
this
case,
the
process
wants
to
open
the
same
port
by
this
sign
of
on
the
network
name:
space.
Okay,
the
network
name
space
is
on
by
the
initial
username
space.
The
capability
check
this
time
is
performing
against
the
initial
username
space,
but
this
time
that
process
doesn't
have
the
capability
in
the
user
name
space.
Hence
the
process
is
not
able
to
perform
that
operation.
A
Okay,
let
me
talk
about
the
username
spaces
supporting
coordinates.
This
is
the
number
of
the
cape
that
we
have
been
working
on
before
going
into
details.
Let
me
show
you
a
bit
of
history,
about
the
username
spaces
coordinates
and
so
on.
So
the
name
were
introduced
in
linux
almost
20
years
ago
and
10
years
after
in
2013,
the
username
spaces
were
implemented
implemented
in
linux.
A
There
was
also
a
pr
with
that
implementation
for
for
coordinates,
unfortunately,
that
pr
was
closed,
so
this
support
was
not
merged
into
kubernetes,
and
then
last
year
in
2020
we
opened
a
new
proposal
for
implementing
the
support
in
coordinates
and
also
in
21
this
year.
The
support
for
id
mapping
months
was
merged
in
lenos.
I
will
talk
a
little
more
about
why
this
is
important
for
this
work
that
we
are
doing.
A
You
might
be
wondering
if
username
spaces
are
that
old.
They
were
introduced
to
linus
almost
eight
years
ago
and
many
of
the
tools
there
are
already
supporting
that,
for
instance,
the
different
container
and
science,
the
oci
specification
and
so
on.
So
why
coordinates
is
not
supporting
users
in
space
as
well.
This
is
not
the
easy
to
do,
and
there
are
some
challenges
that
we
have
to
overcome
in
order
to
provide
support
for
that.
A
The
the
one
of
the
challenges
are
the
duplicates
and
options.
So
when
we
use
username
spaces
in
some
cases,
we
have
to
create
a
copy
of
the
snapshot
and
we
have
to
show
to
face
downward
shape
of
that
snapshot
in
order
to
create
the
container.
So
what
it
means
is
that
when
we
create
a
container
that
is
used
in
username
spaces,
we
need
more
time
to
copy
there's
an
option
to
change
the
open
shape
of
dab
is
not
sure.
A
Another
challenge
that
we
have
is
the
support
for
volumes.
The
problem
is
that
if
we
have
multiple
containers,
multiple
parts
that
are
using
a
different
id
mapping,
those
spots
are
not
able
to
share
files
using
volume,
because
the
host
id
of
those
parts
is
different.
So
there
could
be
some
issues
while
accessing
those
files
in
a
shared
storage
and
other
of
the
challenges
that
we
have
is
to
support
high
ids.
A
A
There
are
some
solutions.
I
would
say
that
this
is
in
some
cases
more
can
work
around
a
solution,
but
this
is
the
best
we
can
do
right
now.
The
main
solution,
the
main
work
around
let's
say,
is
to
run
multiple
containers
with
the
same
id
mapping.
By
doing
the
way
we
solve
the
problem
of
the
volumes,
we
also
solve
the
problem
of
the
duplication
of
the
snapshots
and
so
on.
The
key
point
here
is:
what
is
the
granularity?
How
many
posts
do
we
want
to
share
that
shares
the
same
id
mapping?
A
A
A
A
So
the
idea
is
not
to
implement
only
one
of
them.
The
idea
is
to
to
see
what
of
them
make
sense
and
which
one
of
them
we
should
implement.
So,
let's
make
a
comparison
of
the
different
modes.
So
in
the
left
we
have
the
different
modes
I
just
presented
to
you
and
on
the
top
we
have
the
different
challenges
that
we
have
and
also
the
path
to
both
isolation
offered
by
each
mode.
A
So,
in
the
case
of
the
cluster
mode
for
the
duplicate
is
natural
issue,
it
provides
a
good
solution
because
we
have
all
the
posts
sharing
this
energy
mapping
on
the
on
the
cluster.
So
basically
we
are
able
to
to
reuse
the
same
snaps
for
all
the
containers
in
the
cluster.
In
the
case
of
the
name,
space,
a
and
service
account
modes,
the
this
is
a
this
offer
more
or
less
good
solution.
Let's
say
amazing
solution,
because
in
this
case
we
are
only
able
to
reuse
the
snapshot
in
the
same
namespace.
A
So
we
we
have
a
container
that
is
running
in
two
different
namespaces
or
two
different
service
account.
Then
we
will
have
to
create
different
copies
of
the
snapshot
for
the
for
those
containers
for
the
pod.
This
is
really
bad.
In
this
case,
all
the
pods
are
using
a
different
id
mapping.
We
have
to
create
a
different
snapshot
for
each
both
regarding
the
support
for
volumes
for
the
cluster
mode.
A
For
the
name
space,
the
support
is
more
or
less
medium,
because
in
this
case
we
are
able
to
share
the
volume
across
the
ports
inside
the
same
name,
space
also
in
service
account,
but
we
are
not
able
to
share
that
volume
between
parts
that
are
running
in
different
name
spaces
and
finally,
for
the
both
case.
This
is
again
very
bad,
because
we
could
only
share
a
volume
with
a
single
path.
A
Regarding
the
support
for
high
uids,
the
cluster
offers
a
very
good
solution
for
that,
because
in
this
case
we
could
assign
a
pretty
big
range
of
uids
to
the
cluster
and
danish
board.
We're
in
the
cluster
we'll
be
able
to
use
all
those
ids
for
this.
For
the
name
space
on
service
account
modes.
This
is
bad
because
in
this
case
we
have
to
coordinate
the
allocation
in
the
different
nodes
of
the
cluster,
and
we
also
have
to
warranty
that
the
probability
of
a
collision
is
very
low.
A
What
it
means
is
that
we
only
cannot
can
assign
a
low
number
of
ids
to
each
name,
space
or
turish
service
account
for
the
both
case.
The
solution
in
this
case
is
good,
because
we
we
don't
have
to
coordinate
the
location
between
the
different
nodes.
So
in
this
case,
the
only
thing
that
we
have
to
guarantee
is
that
the
pot
in
each
node
doesn't
get
an
overlapping
mapping.
A
So
in
this
case
we
are
able
to
give
to
each
path,
let's
say
a
pretty
big
range
regarding
the
path
to
pole,
isolation
in
the
cluster
case.
This
is
very
bad.
Of
course
we
are
sharing
the
same
id
mapping,
so
basically,
there
is
no
insulation
between
the
parts.
In
this
case,
for
the
name
space
and
the
service
account
case,
this
is
more
or
less
medium.
A
We
have
some
insulation
between
pots
running
on
different
name
spaces
or
different
service
accounts,
but
we
don't
have
isolation
for
parts
running
on
the
same
name,
space
of
service
account
for
the
both
case.
It
offers
the
develop
isolation,
because
in
this
case
we
have
a
different
id
map
pin
for
each
port.
So
basically
the
ports
are
easily
are
fully
isolated.
A
Okay,
let
me
show
you
a
demo
about
a
proof
of
concept
that
we
implemented
with
the
support
in
coordinates
in
this
demonstration.
I
will
show
how
we
can
protect
the
host
against
one
of
the
vulnerabilities.
I
showed
you
before
by
using
username
spaces,
I
will
use
just
the
run
c
burnell
ability.
So
what
I
have
here
is
a
run
c
that
has
that
vulnerability.
This
is
a
very
old
version
that
was
released
before
that
vulnerability
was
fixed.
A
A
A
Let's
wait
a
little
bit
until
that
part
is
running,
so
this
is
already
running.
Let's
look
at
the
locks
of
that
ball,
so
we
can
see
the
yeah
in
this
case.
This
plot
says
that
the
ransom
binary
was
updated.
Let's
check
if
that
actually
happened.
So,
yes,
in
this
case,
we
can
see
that
the
exploit
in
the
container
was
able
to
override
the
run
c
binary,
and
in
this
case
it
just
add
a
string
to
the
binary.
Let
me
reinstall
the
original
run
c
binary.
A
The
strain
is
not
present
anymore
and
let
me
show
you
how
we
can
mitigate
that
vulnerability
by
using
username
space.
So
this
is
another
path
template.
This
is
almost
the
same
template
I
have
before.
The
only
modification
is
that,
in
this
case,
unenabled
the
usage
of
user
name
spaces
by
using
the
cluster
mode
I
display
before
so.
Let's
create
this
path.
A
Okay,
so
the
pot
is
done,
let's
get
the
logs
of
the
polls,
so
in
this
case
we
we
see
that
there
was
an
error.
What
it
means
is
that
this
ploy
was
not
able
to
modify
the
run
c
binary
because
in
this
case
the
uid
the
display
was
using
on
the
house
was
not
the
owner
of
the
rancid
binary
and
yeah.
In
fact,
if
we
check
the
rancid
binary,
we
can
see
that
the
string
is
not
present
in
that
case
so
yeah.
A
A
Probably
the
first
thing
that
we
should
start
doing
is
to
consider
the
item
up
in
one
facial.
There
is
no
merge
in
the
linux
kernel,
so
this
is
a
feature
that
will
be
available
in
linux
5.12,
and
this
is
a
facial
that
provides
a
real
solution
for
the
duplicate
snapshots
and
the
volume
sharing
issues.
A
So
if
we
look
how
the
modes
comparison
again,
taking
into
consideration
the
item
up
and
mount
feature
where
we
can
see
that
now
the
duplicator
snapshots
and
the
support
for
volumes
in
all
the
solutions,
we
could
have
an
excellent
support
because
we
have
support
for
that
and
the
kernel.
So
this
is
something
that
we
don't
have
to
care
about.
A
A
We
don't
have
to
consider
anymore,
the
duplicated
snapchats
on
the
support
for
volumes
issues
and,
as
we
can
see
in
this
case,
the
pod
mode
looks
like
a
very
good
candidate
to
implement,
because
it
has
good
support
for
high
uits
and
also
he
offers
sln
a
path
to
path,
isolation,
okay,
so
just
before
finishing,
this
is
just
some
reference
material.
Just
in
case,
you
want
to
know
more
details,
some
documentation
about
username
spaces
capabilities
and
so
on.
In
the
linux
kernel,
two
blog
posts.