►
From YouTube: Kubernetes Exposed! Seven of Nine Hidden Secrets That Will Give You... Ian Coldwater & Brad Geesaman
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes Exposed! Seven of Nine Hidden Secrets That Will Give You Pause - Ian Coldwater, Twilio & Brad Geesaman, Aqua Security
Think you know Kubernetes? Think again. Kubernetes is full of uncommon knowledge and doesn’t always behave the way we assume, containing unexpected gotchas and surprising behaviors that’ll make you say, “how come nobody told me this earlier?” In this talk, Ian Coldwater and Brad Geesaman will shine a light on hidden secrets in Kubernetes, demonstrating scary science such as pods in non-existent namespaces, bypassing network policies via DNS, fun with capable sidecar containers, and one weird trick attackers don’t want you to know. Defenders hate it! Don’t build your next threat model before watching this! Attendees will learn how not to get caught off guard by learning what to watch out for and how to better secure their systems. You won’t believe what happens next.
A
A
B
B
B
If
this
sounds
familiar
to
you,
you're,
not
alone,
we've
been
working
with
kubernetes
for
a
while
now
and
we've
seen
a
lot
of
strange
behaviors
wild
surprises,
unexpected
gadgets
things
that
don't
always
work
the
way.
They
think
the
way
you
think
they
will
and
frankly
there
are
a
lot
of
things
that
you
might
not
be
aware
of
that
you
might
not
need
to
know.
B
So
here's
a
quick
one.
You
might
think
that
since
kubernetes
heavily
uses
pki
for
authentication
that
there
would
be
a
way
to
revoke
a
specific
certificate
if
one
of
the
certificate
pairs
ever
got
compromised,
that
would
be
sensible
right.
That
seems
like
that
would
make
sense,
but
that's
not
actually
how
that
works.
Ofc,
sp
stapling
is
not
supported.
This
is
a
long
time
issue
in
kubernetes,
like
no
literally
a
long
time
issue.
There
is
a
kubernetes
issue
from
2015
about
this
specifically
so.
B
Well,
some
some
managed
providers
will
help
you
with
this,
but
even
if
they
do
it
almost
is
always
going
to
need
some
down
time.
So
it's
an
important
thing
to
be
aware
of
and
to
be
prepared
for
here's
another
one.
Docker
has
a
solid
default
set
comp
profile
for
containers.
They
restrict
capabilities
in
a
really
smart
way,
great
to
just
purcell.
For
that
awesome
work
since
docker
builds
containers
and
kubernetes
orchestrates
containers.
You
might
think
that
kubernetes
would
inherit
those
capabilities
out
of
the
box
right.
B
That
would
be
sensible,
but
it's
not
actually
how
it
works.
Kubernetes
overrides
docker's
default
second
profiles,
and
it
makes
it
unconfined
what
this
means
for
people
who
are
not
linux
security
nerds
in
the
room
is
that
it
turns
them
off
all
of
those
capabilities
that
docker
very
sensibly
restricts.
Kubernetes
just
turns
around
and
puts
back
on.
B
Default
set
up
is
a
great
sounding
name.
Right:
we've
got
default
second,
by
default,
that's
great.
We've
been
waiting
for
that
forever.
Okay,
the
thing
is
that
it's
not
actually
on
by
default.
You
have
to
turn
it
on
and
not
only
that
you
have
to
turn
it
on
twice
the
cluster
administrator
or
your
cloud
provider
needs
to
turn
that
feature
flag
on,
since
it's
an
alpha
and
even
after
that
feature
flag's
been
turned
on.
There
is
a
cli
flag
that
needs
to
be
invoked
every
single
time.
B
Here's
another
thing
you
might
not
be
aware
of
you
might
think
if
you
delete
a
cluster
role
binding,
it's
gone
right
like
that
seems
like
that
would
be
sensible,
but
it
doesn't
always
work
that
way.
There's
one
special
exception
and
it
can
be
a
dangerous
one.
That
exception
is
system
masters,
which
is
hard
coded
in
kubernetes
to
be
cluster
admin
and
surprise.
Even
if
you
delete
it,
this
one
sticks
around
because
it's
hard-coded
you
can't
actually
really
get
rid
of
it.
B
C
So
I'm
running
a
kind
cluster,
so
that's
running
kube
adm
under
the
hood
and
it
puts
that
system
masters
certificate
pair
in
the
local
cube
config.
So
I'm
going
to
run
a
fancy
command
under
the
hood.
I
won't
bore
you
with
it,
but
I'm
just
going
to
extract
and
show
you
that
that
is
what
is
in
the
actual
certificate.
So
you
see
system
masters.
So
I
have
this
credential
here
locally.
B
C
C
So,
let's
look
at
the
cluster
roll
binding
named
cluster
admin,
it's
kind
of
confusing,
but
this
is
the
default
cluster
role
binding.
It
says
that
system
masters
the
group
is
bound
to
cluster
admin.
That's
where
the
explicit
definition
of
those
stars
happens,
so
we're
going
to
go
ahead
and
delete
that.
C
C
So
did
we
lose
that
permission?
Well,
let's
just
find
out.
Let's
go
ahead
and
recreate
that
exact
same
cluster
role,
mining
and
those
astute
viewers
out
there
will
know
that
you
shouldn't
be
able
to
give
yourself
permissions
that
you
don't
have
access
to,
but
you
can
there
I'm
cluster
admin
again
and
we're
going
to
show
that
just
to
prove
it,
there's
no
cards
up
the
sleeves
off.
Can
I
list
and
the
stars
are
back.
B
So
speaking
of
some
things
that
you
shouldn't
be
able
to
do,
but
you
can,
you
might
think
that
validating
web
hooks
are.
You
know
good
for
enforcing
policy
on
resources
as
they
hit
the
api
server
right,
but
they're
good
for
a
lot
more
than
that,
they're
useful
for
all
kinds
of
things.
For
example,
they
can
be
a
high
performance
data,
exfiltration
tool,
sometimes
avoiding
those
pesky
network,
egress
charges.
B
So
that's
how
admission
control
is
supposed
to
work.
We
can
use
the
way
that
those
things
work
to
make
them
useful
for
us
as
attackers
too.
So
here's
one
thing
we
could
do.
We
could
set
up
an
external
server,
maybe
one
in
a
different
cloud
provider,
and
we
can
set
it
up
to
listen
on
tls
and
log
what
gets
sent
to
it.
Then
we
could
set
up
a
validating,
webhook
configuration
that
points
to
it
when
resources,
like
secrets,
are
created.
C
Yeah,
you
could
embed
the
data
that
you
want
to
leak
or
exfiltrate
or
back
up
to
yourself
right
in
the
request
to
create
the
secret.
We
know
that
secrets
and
crew
config
maps
can
store
about
a
megabyte
each
so
and
the
control
plane
will
use
its
highly
efficient
go
routines
thanks
developers
who
wrote
those
to
send
that
data
to
the
listener,
the
control
plane
systems
in
managed.
Kubernetes
providers
don't
seem
to
filter
out
tcp
443
outbound
and
it
comes
from
a
vpc
that
in
some
cases,
might
not
be
one
that
you
get
billed
for
disclaimer.
C
So,
let's
exfiltrate
some
data,
I
have
very
important
documents
in
latin
about
a
megabyte
of
it,
3
400
lines
or
so
just
to
give
your
mind
a
mental
picture
of
how
much
that
is,
the
works
of
shakespeare
are
in
there
somewhere
and
we
have
an
extra
trading
web
hook
and
what
we've
configured
here.
If
you
look
at
the
very
bottom
line,
it
says
we're
going
to
send
this
to
a
external
server
named
hong
kong
sh
whenever
a
create
or
an
update
on
a
secret
happens,
so
a
copy
of
that
secret.
C
So
we're
going
to
apply
that
validating
web
hook
configuration
like
a
good
administrator
or
not
evil.
Okay,
we're
going
to
look
at
that
configuration
just
so
you
can
see
what
it
looks
like.
There's
some
ca
bundle
things
in
here.
This
is
basically
just
showing
you
that
we
have
every
time
a
secret
is
touched,
we're
going
to
send
it
to
that
url.
C
So
we're
going
to
do
is
we're
going
to
deploy
a
pod
inside
our
cluster
and
it's
job
I'll
show
you
in
a
second.
I
just
want
to
show
you.
The
deployment
right
here
at
the
bottom
is
to
do
a
loop
to
simulate
our
highly
efficient
exfiltration
system,
we're
going
to
do
a
cubecontrol,
create
secret
and
send
that
one
megabyte
with
every
one
of
those
requests.
C
C
So
let's
go
over
to
our
external
server
ssh
intuit
and
for
those
nginx
admins
out
there
you'll
recognize
the
access
logs,
we're
just
going
to
tail
those
and,
as
you
can
see,
we're
backing
up
at
high
speed
again,
I
don't
know
if
who
in
the
room
here
might
be
from
gcp.
I
don't
know
if
those
ips
look
familiar
to
you,
they
might.
B
C
C
Any
pod
that
needs
to
communicate
with
another
service
needs
dns,
so
most
admins
doing
the
right
thing.
Using
network
policy
still
have
to
poke
a
hole
in
for
dns
lookups
in
their
firewall
rules
and
that's
actually
all
an
attacker
needs
for
a
command
and
control
channel
if
they
compromise
a
running
container.
So
you
see
this
pod
in
the
red.
Without
any
egress
filtering,
it
can
directly
communicate
with
an
external
server.
It's
often
the
default,
although
not
the
most
secure
approach.
B
As
a
red,
teamer,
dns
is
really
useful
for
me.
Besides,
you
know
causing
outages.
If
I
have
data,
I
need
to
exfiltrate
a
lot
of
the
time,
I'll
need
to
figure
out
a
way
to
do
that,
and
sometimes
that
can
be
really
hard,
especially
if
egress
is
really
locked
down,
but
there's
often
at
least
one
exception.
C
B
C
C
So
on
my
attacking
server
out
in
the
internet
on
the
public
internet,
I'm
going
to
ssh
to
it
and
I'm
going
to
run
something
called
dns
cat
raise
your
hand
if
you've
ever
heard
of
dns
cat.
Oh
we've
got
some
dns
television
in
the
room
cool.
This
is
our
c2.
This
is
what's
going
to
capture
all
the
sessions
from
our
compromise
systems.
C2.
C
So
over
here,
switching
gears,
I'm
on
my
cluster,
that
I'm
going
to
compromise
now.
For
the
sake
of
this,
we're
going
to
presume
that
we've
gotten
access
to
this
specific
pod
once
it
runs
in
the
cluster,
so
our
world
view
from
an
attacker's
perspective
is
I
have
a
shell
on
this
pod?
What
things
can
I
connect
to
and
how.
C
What
we're
going
to
do
is
we're
going
to
use
various
methods
to
probe
and
test
that
we
have
egress
or
outbound
access
in
this
case
we're
going
to
use
standard
curl
my
favorite
hacking
tool.
I
know
you
have
those
as
well
curl
to
honk.sh
on
tcp
443
right.
This
succeeds.
This
is
there's
no
surprises
here.
Okay,
we're
just
validating.
We
have
tcp
outbound,
but
we
were
talking
about
dns
weren't.
C
We
so
in
this
same
pod
we're
going
to
switch
to
the
dns
cat
binary
that
we've
already
embedded
and
deployed
to
this
system
and
we're
going
to
reach
directly
out
from
this
pod
inside
this
cluster
directly
out
to
this
c2
server
using
udp,
53.,
okay.
So
it's
a
direct
connection,
there's
no
no
mallory
in
the
middle,
and
it's
establishes
a
session
just
like
that.
So
if
we
switch
back
over
now
we
go
back
over
to
our
server.
C
You
can
see
it
has
created
a
new
session.
So
if
I
run
sessions
you
see
the
number
one
is
added
there.
That's
a
new
session,
so
I'm
going
to
interact
with
it
using
dash.
I
it's
kind
of
like
a
tab
in
a
browser.
So
we're
switching
tabs
over
you
can
see
the
things
that
we
have
access
to.
Personally,
I,
like
rolling
shells,
I'm
gonna
run
a
shell
and
again
it
sort
of
spawns
another
new
tab
on
session
two,
so
we'll
control
z
out
of
that
and
interact
with
session
two
cool
now
entirely
via
dns
requests.
C
Let's,
let's
be
good
administrators
and
apply
some
network
policy.
We
have
two
policies
here.
I
know
they're
hard
to
understand.
That's
okay,
we'll
walk
through
this.
The
top
one
says
ingress
and
egress
deny
everything
in
the
default
name
space.
So
the
pod,
if
you
applied
just
that,
would
be
able
to
communicate
with
nothing
the
second
policy
right
below
it,
pokes
that
hole
for
udp
53
within
the
private
ip
space
of
this
cluster.
So
it
can
do
dns
within
the
cluster.
C
Lock
it
down
apply
security.
Okay,
good
control,
apply
security
done;
okay,
so
we're
gonna
go
back
to
that
same
pod.
Now
it
has
that
that
wrapping
blanket
of
a
nice
network
policy
only
allows
dns
to
the
cube
dns
pods
we're
going
to
try
those
connections
again.
So
we're
going
to
be
on
that
pod
we're
going
to
try
to
reach
out
and
it
blocks
right.
That's
expected
cool!
C
C
B
Okay,
so
if
we're
thinking
about
wild
things
that
we
could
potentially
do
with
kubernetes
networking,
we've
got
other
options.
It
isn't
just
dns.
Have
you
seen
cbe
2020
8554,
it's
okay!
If
you
don't
remember,
cve
numbers
are
not
terribly
human
friendly
in
this
cve
services
that
use
the
external
ip
configuration
can
disrupt
or
redirect
traffic
within
the
cluster
external
ip
services
are
pretty
uncommon,
but
they're
very
powerful,
and
they
can
be
very
useful.
For
example,
they
have
a
useful
ability
to
modify
ip
tables.
They.
C
Are
literally
redirection
as
a
service,
so
let
me
explain
you
see
this
pod,
yellow
or
sorry
the
service
yet
and
what
it
really
says
is
all
traffic
within
the
cluster
going
to
this
external
ip
is
redirected
to
the
nginx
pod
on
port
80.
I'll
say
that
again,
all
traffic
going
to
an
external
ip
is
going
to
this
pod
right
here
and
what
name
space
are
we
in
we're
in
default?
C
B
C
B
B
C
That
depends
if
the
configuration
is
set
to
fail
open
after
a
few
seconds
of
it
not
responding.
It
allows
the
request,
so
the
failure
policy
ignore
setting
is
pretty
typical
of
default
settings
for
web
hooks
because
failing
closed
might
cause
even
more
damage.
It
might
actually
cause
an
outage
or
block
and
upgrade
raise
your
hand
if
you've
ever
blocked
with
a
failing,
close
anyone.
Yes,
I
see
some
hands
fantastic,
so
you
know
the
pain.
B
C
So
we
have
a
default
cluster.
This
is
a
separate
cluster
entirely
from
the
previous
demos
and
we've
installed
caverno
from
the
defaults.
Okay,
just
control
apply
from
the
docs
straight
in
by
default.
Validating
web
hooks.
Don't
have
enforcement
policies
on
okay,
but
we
want
to
block
privileged
pods
in
this
specific
case.
So
what
we
have
to
do
is
submit
this
policy
and
change
the
word
audit
to
enforce,
so
I'm
going
to
go
ahead
and
apply
that
to
block
privilege
pods.
So
let's
do
that
again
we're
applying
security
good
one
shot
there
we
go.
C
So,
let's
test
that
we
always
need
to
validate
these
assumptions
right.
That's
what
this
whole
talk
is
about
validating
these
assumptions,
so
ian
and
duffy
brought
the
duffy
there.
You
are
shout
out
brought
this
to
my
attention
a
long
time
ago
and
it's
just
elegant
and
brilliant
that
you
can
run
a
privileged
pod
and
in
the
same
breath
basically
and
a
center
and
escape
to
the
to
the
node
if
you
can
run
a
privileged
pod.
So
this
is
a
one
shot
escape
it's
blocked.
It
did
its
job
hate.
C
Are
we
done?
I
don't
think
so:
let's
create
an
external
ip
service
with
those
pods
of
the
validating
web
hook
and
we're
going
to
get
them
in
the
top
line,
so
we're
going
to
get
the
caverno
service
ip
and
the
caverno
pod
replica
ip
we're
going
to
grab
those
we're
going
to
also
deploy
an
nginx
deployment
in
the
default
namespace.
So
I'm
just
a
developer
in
the
default
namespace
right.
C
So
I
don't
really
have
that
many
permissions,
but
maybe
I
can
create
pods
and
create
services
right,
so
I'm
going
to
create
nginx
and
have
it
listed
on
port
80..
That's
important
in
a
second
then
we're
going
to
create
a
service
with
the
pod
ip
of
caverno
and
the
service
ip
of
caberno
such
that
all
traffic
in
the
cluster.
Remember
all
traffic,
including
control,
plane
to
the
validating
webhook
traffic.
All
traffic
going
to
port
9443
gets
redirected
to
port
80
on
nginx.
C
C
Then
we
scale
engine
up
engine
x
up
to
three
and
then
we
immediately
scale
it
back
down
to
two
now.
Why
would
we
want
to
do
that
because
we're
forcing
code
proxy
to
reset
ip
tables
and
reshuffle
things
we're
causing
chaos
in
churn
in
the
networking
stack
to
try
to
increase
the
chances
that
we're
blocking
connections
that
may
have
already
been
passed
through
the
session
table?
C
So
let's
run
this
attack
and
it's
going
to
go
in
this
loop
and
so
we're
going
to
create
the
deployment
make
its
two
replicas.
Try
the
privileged
pod
three
back
to
two,
and
this
is
going
to
go
on
a
loop.
What
I
want
to
show
you
over
here
is
the
logs
from
the
validating
web
hook
pod.
So
it's
just
doing
its
job.
It's
blocking
the
privileged
pod
right,
we're
all
good
right.
B
B
If
you
would
like
to
hear
me
talk
more
about
race
conditions
and
hero,
brad
talk
more
about
race
conditions,
come
see
the
talk
in
this
exact
room
immediately
after
this
one
called
exploiting
a
slightly
peculiar
volume
configuration
where
we
with
our
hacker
crewmates
sig
hunk
exploit
a
race
condition
in
run
c.
What's
going
on
over
here,
brad.
C
C
C
C
B
So
yeah
now
that
you've
seen
what
can
be
done
with
these,
you
should
probably
block
or
restrict
external
ip
services
with
admission
control
policies,
because,
as
we've
seen,
external
ip
services
are
extremely
powerful
and
can
cause
a
lot
of
problems
on
purpose,
especially
for
multi-tenant
clusters.
Indeed,
as
we
said,
kubernetes
is
full
of
unexpected
surprises.
Things
might
not
quite
work.
The
way
you
think
they
do.
Things
might
actually
be
dangerous
in
the
way
that
they
differ
from
your
assumptions
so
check
your
assumptions.
B
It
is
possible
to
secure
kubernetes,
but
in
order
to
get
there,
you
really
need
to
know
how
it
works
and
not
just
how
you
think
it
works
know
how
it
really
works,
and
you
can
do
it.
We
believe
in
you,
here's
a
list
of
resources
and
links
for
further
reading
and
viewing
and
we
hope,
you've
had
as
much
fun
as
we
have
exploring
some
of
our
favorite
interesting
behaviors
and
maybe
a
jump
scare
or
two
in
the
fun
land
of
kubernetes.