►
From YouTube: The Hitchhikers Guide to Container Security - Tunde Olu-Isa, Oteemo & Jed Salazar, Isovalent
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
The Hitchhikers Guide to Container Security - Tunde Olu-Isa, Oteemo & Jed Salazar, Isovalent
Speakers: Jed Salazar, Tunde Olu-Isa
You’ve seen the Kubernetes security announcements: CAP_NET_RAW rogue advertisements, runc breakout, hostNetwork hijack.. oh my! It seems that the best you can do is keep up with patching, but often these vulnerabilities take advantage of overly permissive Pods. In this talk, we’ll introduce the riskiest privileges that Pods can request, what allowing those privileges means for your cluster, and how to create security policy to protect your cluster and its users. If you’ve ever wondered, “what does enabling privileged actually mean?”, hitch a ride on a whirlwind and fun guide about the basics of Pod security and how you can easily configure security policy to keep Kubernetes safe. Any level of experience can benefit from learning about the riskiest privileges Pods can request and reducing the threats in your environment by running your workloads safely.
A
Hi
everyone
welcome
to
a
hitchhiker's
guide
to
container
security.
Today,
we're
going
to
be
talking
about
container
security
through
the
eyes
of
a
couple
of
characters
in
the
story
that
we
created
also
be
showing
a
couple
of
demos,
but
before
we
get
started
on
the
talk
just
wanted
to
give
a
brief
introduction
so
hi,
I'm
jed
salazar.
A
B
Hi
everyone,
I'm
tinder,
luisa.
I
enjoy
teaching
and
also
solving
complex
problems.
I
currently
work
with
a
company
called
ultimo,
where
it's
a
consulting
company
that
specializes
in
solving
cloud
native
problems
and
working
with
modern
technology.
We
do
this
within
enterprise,
I.t
companies
and
also
the
federal
government
space.
B
A
A
Let's
meet
our
node,
this
is
gnome
the
kubernetes
control
plane,
node
gnome
is
super
laid
back
and
will
run
any
containerized
workload
that
is
scheduled
to
them.
Also,
like
other
nodes,
they'll
run
workloads
with
any
settings.
The
containers
ask
even
if
they're,
risky
or
dangerous
gnome
always
runs
the
newest
version
of
kubernetes
and
they
take
pride
in
providing
a
safe
environment
for
their
hitchhiker
friends.
A
So
we've
talked
about
containers
a
couple
times,
but
we
haven't
touched
on
what
a
container
actually
is.
So
what
is
a
container
containers?
Are
linux
processes
that
run
in
the
context
of
a
namespace
namespaces,
provide
isolated
resources
for
containers
like
volumes
and
networking,
making
a
container
think
it's
the
only
container
on
the
node
due
to
this
isolation,
some
of
the
key
there's
a
lot
of
name
spaces,
but
some
of
the
key
namespaces
include
pid.
A
A
Not
shown
here,
but
also
part
of
containers
is
something
called
c
groups
and
c
groups
are
used
to
limit
the
cpu
and
memory
resources
that
a
container
can
consume
in
the
context
of
security.
This
is
nice
because
it
allows
us
to
not
have
one
what's
known
as
a
noisy
neighbor
container,
take
all
of
the
resources
from
the
node
from
other
containers.
A
So
paulie
asked
their
friend
gnome
the
note
if
they
could
help
them
get
started
hitchhiking
as
a
container
of
course,
that
node
stoked
to
help
their
friend
paulie
was
given
a
lot
of
choices
to
get
started
overwhelmed
with
the
choices
and
really
just
wanted
to
get
things
to
work.
Paulie
chose
a
privileged
setting.
A
A
A
So
I've
got
a
couple
of
manifests,
we're
gonna
be
using
both
in
this
demo,
so
the
first
one
is
paulie's
manifest
itself
and
we'll
actually
take
a
look
at
this
manifest
kind
of
jump
through
the
important
attributes
that
are
kind
of
specific
to
this
demo.
But
we
also
have
another
pod
manifesto
called
incognito
which
we'll
be
going
over
here
soon,
but
for
now
let's
take
a
look
at
poly.
A
So
let's
highlight
some
of
the
important
attributes
about
poly
that
are
actually
going
to
facilitate
us
to,
in
this
case
kind
of
jump
out
of
the
container
and
do
a
container
escape
the
way
that
I
like
to
think
about
a
container
escape
each
time
that
I
do
it
is.
You
know
the
the
chest
burster
scene
in
alien,
but
let's
take
a
look
all
right,
so
we
have
a
couple
of
things
under
spec
here
that
are
going
to
be
particularly
interesting.
A
So
we
have
host
pid
and
host
network.
So
if
you
remember
back
in
the
presentation
where
I
was
describing
name
spaces,
so,
for
example,
we
have
the
pid
name
space
and
the
network
name
space.
The
pid
name
space
basically
meant
that
you
know
in
linux,
every
process
gets
a
process
id
or
a
pid,
and
the
pit
namespace
makes
it
so
that
the
container
can
only
see
the
pids
within
its
own
namespace.
A
We
also
have
host
network,
and
in
this
case,
if
you
remember
a
network,
namespace
basically
creates
a
generally
a
network
interface
in
that
namespace
routing
tables
allows
you
to
send
traffic.
Basically,
we
are
removing
that
namespace
altogether
and
we're
using
the
host's
network
interface
directly
and
routing
tables
directly.
So
what
that
means
is
if
we
have
something
like
network
policy,
which
is
kind
of
like
a
firewall
for
your
pods,
we're
able
to
circumvent
that
network
policy
and
send
traffic
directly
out
as
the
node
so
moving
forward.
A
We
have
a
couple
of
other
things
I
wanted
to
discuss.
We
have
this
a
concept
called
volume
mounts
and
basically
what
this
means
is
inside
of
my
container,
I'm
going
to
actually
mount
a
slash,
kubernetes
directory,
and
if
I
jump
down
here,
we
can
see
that
the
thing
that
I'm
actually
going
to
mount
in
that
directory
is
from
the
host
itself.
So
we're
going
to
mount
the
etsy
kubernetes
directory
from
the
host
inside
of
the
container
at
slash
kubernetes.
Now
this
directory
at
sea
kubernetes
is
particularly
dangerous.
A
This
tends
to
contain
things
like
all
of
the
credentials.
That's
used
to
basically
make
up
the
cluster
as
well
as
some
pod
manifests
that
are
run
locally
by
the
cube,
which
we'll
take
a
look
at
here
soon
and
then.
Lastly,
to
kind
of
make
this
even
more
evil
and
powerful
is
oops,
we're
going
to
specify
that
we're
going
to
basically
have
paulie
land
on
a
control,
plane
node.
A
So
when
we
look
at
this
etsy
kubernetes
directory
we're
going
to
be
looking
at
that
from
a
control
plane
mode,
which
kind
of
gives
us
more.
You
know
additional
resources
to
take
advantage
of,
and,
of
course,
as
mentioned,
paulie
is
actually
going
to
be
running
with
a
security
context
of
privilege
equals
true.
So
we
kind
of
have
the
option
to
you
know
decide
whether
we
want
to
use
these
namespaces
that
we
defined
or
not
awesome.
A
A
A
A
A
This
right
here
is
basically
the
keys
to
the
kingdom,
so
this
is
basically
all
the
kind
of
authentication
for
the
cluster
itself.
That
kind
of
gives
you
kubernetes
admin.
So
at
this
point,
if
you
just
made,
you
know
these
configurations
into
your
pod
mounted
this
in
utilize
this
this
data
here
and
you
could
basically
do
whatever
you
wanted
with
the
kubernetes
cluster.
You
could
basically
impersonate
any
component.
You
could
basically
launch
any
pod
join
any
node.
A
You
know
the
the
possibilities
are
endless,
but
what
we
want
to
do
in
this
demo
is
we
actually
want
to
take
advantage
of
this
directory
here?
So
this
is
the
manifest
directory.
So
in
this
directory,
basically
there's
a
component
in
each
node
called
the
cubelet,
and
the
keyboard
will
actually
it's
responsible
for
running
pods
and
making
sure
that
those
pods
are
constantly
running.
A
So
if
the
pod
fails
for
whatever
reason,
the
qubit
will
basically
restart
it,
and
you
can
see
here
that
inside
of
this
directory,
we
have
all
of
the
kind
of
kubernetes,
you
know
control,
plane,
components
that
are
run
from
the
cubelet,
so
the
api,
server
controller
manager,
scheduler
etc.
So,
if
we
wanted
to,
what
we
could
do
here
is
because
we
have
kind
of,
like
you
know,
direct
write
access
into
this
directory.
We
could
basically
inject
our
own
api,
server
or
kind
of
tamper
with
the
controller
manager,
etc,
etc.
A
But
what
I
want
to
do
is
I
want
to
kind
of
showcase
this
kind
of
little
known,
kubernetes
bug,
and
what
I'm
going
to
do
is
I'm
basically
going
to
insert
my
basically
that
incognito
pod
and
have
the
cubelet
run
it
directly
now?
Most
specifically,
is
this
pod
actually
is
going
to
be
running
inside
of
a
kubernetes
namespace
that
doesn't
exist,
so
we
just
created
the
poly
name,
space.
A
We're
actually
going
to
be
just
running
this
and
and
pointing
to
a
name
space
that
doesn't
exist,
and
the
reason
that
this
is
important
is
because
basically
kubernetes
will
not
know
about
the
pod
as
a
result
of
this.
So
if
we
run
something
like
cube,
cuddle
get
pods
or
if
you
look
at
like
the
kubernetes
audit
logs
or
just
you
know,
ask
about
pods
in
general.
Basically,
it
will
basically
be
invisible
to
the
kubernetes
cluster.
A
A
So
there's
a
couple
of
things
here
that
are
that
are
interesting
about
this,
namely
this
name
space
that
doesn't
exist.
So
again,
this
is
a
pod
spec,
its
name
is
incognito,
and
basically
it's
also
going
to
be
running
as
privileged.
We
actually
picked
an
image
that
kind
of
gives
you
some
hacking
tools,
so
we
can
basically
take
advantage
of
that,
and
you
know
if
we
ever
needed
to.
We
could
basically
jump
into
this
pod
and
do
things
like
you
know,
send
network
connections.
A
You
know
just
basically
have
the
tools
that
we
need
to
do
to
be
able
to
kind
of
continually
own
the
system
great.
So
we
should
actually
see
this
pod
running
locally
if
we
ssh
into
the
node
and
the
way
that
we're
gonna
do.
That
is
because
this
is
a
kind
cluster,
so
kind
is
basically
running
kubernetes
via
docker
nodes.
Is
I'm
actually
just
going
to
exec
into
the
the
gnome,
the
node
directly,
which
in
this
case
is
kind
control
plane?
And
what
I'm
going
to
do?
A
Kubernetes
won't
actually
know
about
it,
so
we
have
this
running
locally
and
basically,
what
I
want
to
show
you
is
that
we
jump
out
of
the
control,
plane,
node
and
ask
kubernetes
about
this
pod
directly.
We're
not
actually
going
to
be
able
to
see
anything.
So
in
this
case,
what
I'm
going
to
do
is
I'm
going
to
say,
let's
get
pods
and
I
want
to
have
all
the
pause.
A
You
know
listening
and
you
can
see
here
that
we
we
have
poly
shown
so
paulie
is
running
in
in
in
their
name
space
and
we
actually
have
kind
of
all
the
you
know
cube
system,
pods
that
we
have
running
here.
But
what
we
don't
see
is
our
incognito
pod
and
what
this
means
is
basically
cubelet
is
going
to
be
running
our
incognito
pod,
secretly
and
invisibly
in
the
background,
and
if
whatever
reason
that
pod
stops
or
needs
to
be
restarted,
cubelet
will
dutifully
do
so.
So
it's
kind
of
a
high
availability.
A
You
know
command
and
control,
or
you
know
incognito
system
that
we
have
running
so
that
kind
of
shows
you
just
a
little
bit
about
what
you
can
actually
do
if
you
actually
pull
in
node
resources
into
a
pod.
So
some
of
these
configurations-
all
you
really
have
to
do-
is
ask
in
the
pod
spec
and
by
default,
you'll
kind
of
be
granted
these
privileges
cool
thanks
for
watching
and
let's
get
back
to
the
talk.
B
So
paulie
now
understands
the
implications
of
using
a
privileged
container
and
knows
the
ability
to
give
such
permissions
should
be
controlled.
So
what
paulie
does
is
poorly
employs?
Oprah
gatekeeper,
oprah
stands
for
open
policy.
Agent
and
oprah
gatekeeper
is
a
sub
project
of
the
opa
project
itself,
which
is
a
kubernetes
native
solution
that
provides
a
way
to
validate
what
goes
into
new.
B
B
So
what
gatekeeper
is
able
to
inform
polio
violations
of
rules
that
has
been
specified
via
its
auditing
feature
and
also
enforce
the
denial
of
malicious
configurations
to
noon
by
any
of
the
violators?
In
this
case,
as
you
can
see
from
the
image
in
this
case,
gatekeeper,
denies
this
specific,
malicious,
cryptominer
hi
curve
from
applying
dangerous
port
configurations
to
noon.
B
B
B
B
B
Great
that
is
applied
next,
we'll
apply
the
constraint
template
for
the
privilege.
Ball
remember.
This
is
the
logic
that
enforces
the
constraints
that
will
end
up
applying,
which
are
the
given
set
of
rules
or
requirements
that
must
be
met
by
configurations
being
applied
by
either
port
or
deployment
or
kubernetes
objects
in
general.
B
B
So
let's
go
ahead
and
apply
that
so
we
apply
the
deny
host
parts,
that's
applied
next,
let's
go
ahead
and
apply
the
let's
look
at
the
privileged
container
constraints
and
then
apply
that
too.
So,
as
you
can
see
here,
we
explain
we
explicitly
had
the
line
enforcement
action
deny,
but
this
is
actually
implied
by
default.
B
So
great
now
that
we've
gone
ahead
and
apply
that
the
next
thing
we're
going
to
do
is
we're
going
to
change
into
our
example
manifest
directory
and
we're
going
to
run.
Examples
show
examples
of
configurations
that
violate
our
rules
and
the
first
one
in
this
case
is
engine
x,
privilege,
and
you
can
see
here
we're
setting
the
security
contest
privilege.
It
was
true.
This
violates
our
rule,
so
let's
go
ahead
and
apply
that
and
see
what
happens.
B
So
we've
tried
to
apply
that
and
we
get
that
response
back
from
our
constraint,
template
logic
that
says
that
privilege
containers
is
not
allowed.
So
again,
it
returns
the
issue
of
why
it's
failing
to
us.
So
the
next
example
we
look
at
here
is
the
oh
spot
example
remember
this
was
this
was
exploited
in
the
last
demo.
We
showed
now
in
this
hospital
we're
trying
to
add
etsy
cumulatives,
so
let's
go
ahead
and
apply
the
configuration
and
see
if
this
is
admitted
into
our
kubernetes
poster
again
we
tried
this.
B
This
violates
our
rules
and
we
get
a
response
back
that
says
this
part
is
not
allowed,
and
so
what
we're
done
seeing
is
that
configurations
that
valid
our
rules
are
being
denied
entry,
and
this
concludes
our
demo.
Oprah
gatekeeper
is
a
great
solution
amongst
many
other
tools
within
a
cloud
native
ecosystem.