►
From YouTube: The Devil in the Details: Kubernetes’ First Security Assessment - Jay Beale & Aaron Small
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
The Devil in the Details: Kubernetes’ First Security Assessment - Jay Beale, InGuardians & Aaron Small, Google
In October of last year, the Kubernetes project created a new Security Audit working group and began Kubernetes’ first comprehensive third-party security assessment. In the months that followed, we worked closely with Trail of Bits and Atredis Partners to assess and improve Kubernetes’ security posture. Through code review and penetration testing, we found and addressed 37 new vulnerabilities. With support from many Kubernetes contributors, the third party security firms and Kubernetes project produced a formal threat model covering eight critical components across six different trust zones. In this talk, we will share our findings, methodology, and vision for future security investments. We’ll discuss what the work uncovered, and what this means to Kubernetes security both now and for the future.
A
So
I
know
I'm
the
last
speaker
before
we
all
get
to
relax
and
have
a
couple
drinks.
I
will
try
and
keep
it
brief
at
least
not
go
over.
This
is
the
devil
in
the
details.
Kubernetes.
First
here
the
assessment,
my
name
is
Aaron
small
by
day
I'm,
a
product
manager
at
Google,
I
work
on
kubernetes
security
and
gke.
With
me,
I've
got
Jay
Beall
he's
the
CTO
of
n
Guardians.
They
do
a
bunch
of
kubernetes
penetration
testing
and
he's
also
a
seasoned
instructor
at
blackhat.
B
B
Craig
and
guram
works
for
Salesforce,
where
he
works,
on
securing
kubernetes
container
and
cloud
for
Heroku
Joel
Smith
of
Red
Hat
works
on
OpenShift
and
he's
also
a
member
of
the
product
security
committee
and
we'll
talk
about
them
both
a
little
bit
more
you're,
going
to
notice
that
there
are
little
logos
next
to
our
names,
Aaron
and
I
both
have
our
open
source
projects.
You
can
get
a
sticker
for
one
of
mine.
If
you
catch
me
afterwards,
Craig's
is
for
his
capture.
B
A
Okay,
so
we
have
like
30
minutes
to
talk
about
a
security
audit
that
took
us
a
year
to
plan
and
execute
and
yielded
241
pages
of
white
papers
and
reports.
We
won't
be
able
to
cover
everything
so
instead,
I
want
to
give
like
kind
of
an
overview
of
the
kubernetes
ecosystem
that
surrounded
the
audit.
Why
now
was
the
right
time
to
run
it
and
to
invest
all
this
time
and
money
and
energy
into
it?
B
A
Okay,
backstory
this
whole
journey
started
back
in
2018
the
CNC
F
reached
out
to
a
couple
of
their
projects.
It
was
a
core
DNS,
Prometheus
and
envoy,
and
asked
them
to
run
a
security
audit.
It
was
kind
of
a
trial
balloon
to
see
if
a
formal
audit
process
like
this
would
bring
value.
The
short
end
to
that
long
story
is
that
it
did.
They
got
good
findings,
they
got
good
bugs
and
overall,
it
improved
the
security
of
those
products.
At
the
end,
we'll
have
links
to
those
initial
reviews.
A
If
you
want
to
take
a
look,
so
we
left
at
the
opportunity
when
they
opened
it
up.
Two
graduated
products
and
the
steering
committee
said
to
the
community,
who
would
be
some
good
people
to
run
an
audit
like
this,
and
we
formed
the
working
group.
It's
gone
through
a
few
iterations
since
then,
but
by
and
large
the
people
you
just
saw
up
there
been
working
on
this
for
just
over
a
year.
A
So
I
said
why,
now?
Why
is
this
the
time
to
invest
in
an
audit
like
this
I
think
it's
important
to
acknowledge
that
kubernetes
has
become
business
critical?
It's
the
de
facto
way
to
run
containerized
workloads
and
containerize
workloads
aren't
just
for
our
side
projects
anymore.
The
world
runs
on
them.
This
is
a
survey
that
the
new
stack
ran
back
in
2017
and
even
at
first
glance.
It's
apparent
that
kubernetes
was
like
far
and
away
the
leader,
but
then
we
dug
in
a
little
bit.
It
turns
out.
It's
actually
a
check
all
that
apply.
A
So
many
of
the
other
responses
were
probably
also
running
kubernetes
and
if
you
take
an
even
deeper
you'll
notice
that
a
bunch
of
the
options
are
just
kubernetes
under
the
hood.
Also,
security
is
top
of
mind.
Is
the
number
one
challenge
reported
for
people
trying
to
adopt
containers
and
the
containerization
methodologies,
and
it's
an
entirely
different
security
model
than
these
more
traditional
companies
were
accustomed
to,
and
while
this
survey
was
taken
back
in
2017,
it's
not
like
interest
has
been
declining.
A
Also,
this
is
kubernetes
time
in
the
spotlight.
Not
only
is
it
business
Krita
goal
to
those
people
who
have
adopted
it,
but
it's
being
very
widely
adopted
and
it's
being
adopted
both
in
cloud
native
companies,
but
also
more
traditional
ones.
Some
of
those
companies
operate
in
very
security,
sensitive
industries.
We
have
banks,
we
have
financial
institutions,
we
have
e-commerce,
we
have
retail,
we
even
have
CERN
so
like
don't
let
CERN
get
hacked.
A
Finally,
this
isn't
theoretical
the
vulnerabilities
exist
and
they're
being
exploited
in
the
wild
I
know
a
couple
of
people
in
this
room
work
on
managing
and
reporting
those
vulnerabilities
every
day,
it's
very
real,
I
charge
and
headlines
aside.
What
this
means
to
me
is
that
our
market
is
calling
out
for
maturity
and
security
maturity
at
that
one
of
the
ways
that
we
can
demonstrate
this
maturity
is
to
run
these
independent
third
party
audits
in
the
more
traditional
software
acquisition
model.
A
A
So
when
we
approach
this
audit,
a
number
of
us
had
done
this
before
I've
run.
Lots
of
audits,
allocable
on
the
on
the
team,
had
run
audits,
but
this
was
different
than
anything.
Any
of
us
had
done,
and
that's
basically
for
two
reasons:
kubernetes
is
very,
very,
very
big
and
it's
open
source.
So,
instead
of
trying
to
run
our
old
play
books,
we
decided
to
build
some
fundamental
philosophy
around
how
we
wanted
to
approach
it.
First
kubernetes
is
built
in
the
open
and
we
wanted
to
audit
kubernetes
the
kubernetes
way.
A
So,
to
this
end,
the
RFP
that
we
published
to
get
proposals
from
potential
vendors
was
all
written
and
published
in
the
open,
I
think
more
interesting
than
that,
though,
the
selection
methodology
for
how
we
picked
each
vendor
and
what
the
criteria
was
for
a
selection
was
also
all
published
in
the
open.
I
want
a
big
thank
you
to
Craig
Ingram,
who
wrote
our
our
publication
around.
A
Why
we
picked
who
and
what
the
criteria
was
and
we're
doing
this,
because
we
hope
that
by
keeping
the
framework
open,
it
enables
repeatability
not
only
repeatability
for
his
subsequent
open
source,
kubernetes
audits,
but
any
open
source
project
can
take
a
look
at
our
methodology
and
either
you
just
use
it
or
they
can
improve
upon
it.
We
also
opted
for
as
much
transparency
as
possible,
of
course,
when
you're
handling
really
sensitive
information
like
undisclosed,
unreported
unpatched
vulnerabilities
and
something
as
big
as
kubernetes.
A
You
can't
be
very
carefree
about
it,
so
we
did
follow
like
industry
best
practices.
It
turns
out
that
attackers
are
notoriously
faster
at
building
exploits.
Then
defenders
are
rolling
out
patches,
so
we
gave
the
product
Security
Council
time
to
review
all
of
our
findings
and
potentially
embargo
them
before
we
published
all
of
them,
but
we
did
eventually
publish
all
of
the
findings
and
also
all
of
the
qualitative
analysis
of
kubernetes.
A
Also
in
the
vein
of
transparency,
the
vendors
who
are
working
on
it,
the
security
vendors
they
did
all
of
their
work
and
all
their
research
in
their
own
private,
github
repo
and
then,
when
we
finished
after
we
published
all
the
findings
after
we
published
the
white
papers,
they
actually
opened
that
repo
up
to
the
world,
so
you
can
read
there.
You
can
read
all
their
scripts.
You
can
read
all
of
their
incomplete
findings,
all
the
dead
ends
and
all
of
their.
A
Let's
call
them
enthusiastic
debates
about
whether
something
was
a
vulnerability
or
not
like
that
links
at
the
end,
I'd
highly
recommend
you
go
dig
in
frugal.
Alright,
we
were
working
with
the
CNC
F.
The
CNC
F
is
a
nonprofit.
We
wanted
to
be
good
custodians
of
their
resources,
one
of
the
ways
that
we
did
this
was
we
tried
to
focus
our
efforts
on
what
we
thought
were
the
most
security,
critical
components
of
kubernetes.
We.
A
To
like
peanut
butter,
our
efforts
over
the
1.4
million
lines
of
code,
that
is
kubernetes,
so
we
just
picked
the
key
areas
to
get
to
some
level
of
depth.
We
didn't
in
the
areas
that
we
did
look
at.
We
didn't
want
to
leave
anything
on
the
table.
That
was
obvious.
We
want
to
have
some
level
of
confidence
we
had
found.
What
was
there?
A
Finally,
we
wanted
to
be
future
focused
to
this
end.
We
knew
we
couldn't
cover
everything
and
we
knew
this
couldn't
be
the
last
audit,
so
we
wanted
to
invest
some
of
our
resources
into
creating
a
threat
model
and
an
attackers
guide.
We
want
this
effort
to
like
to
pave
the
way
for
future
research
by
independent
researchers
and
more
formal
processes.
A
So
from
that
philosophy
we
asked
for
these
four
deliverables.
I
know
it's
a
very
pretty
slide.
Thank
you,
these
four
core
deliverables
from
the
researchers.
We
wanted
the
traditional
assessment,
which
is
a
list
of
vulnerabilities
who
wanted
to
give
the
researchers
time
to
go,
find
these
good
vulnerabilities
learn
how
to
exploit
them
and
report
them
back
to
the
PSC,
but
we
also
wanted
to
create
some
documents
to
help
future
research,
and
these
were
the
attackers
guide,
which
is
it
covers
how
to
build
a
reference
cluster.
That's
a
reference
cluster.
That's
good
for
attacking
it'll.
A
Tell
you
what
the
networking
layer
might
be
might
look
like
in
that
cluster
and
it
discusses
what
the
available
attack
surface
is
on
all
of
the
critical
components:
the
threat
model.
Of
course,
we
talked
about
data
flow
between
these
components.
What
the
trust
relationships
are
between
them,
as
well
as
a
body
of
qualitative
findings
that
I
think
are
really
useful
for
attackers
and
defenders
and
then
there's
the
operator
white
paper.
The
operator
white
paper
is
effectively
what
you
need
to
know
about
defense
from
an
attackers
perspective.
A
B
Thanks
Aaron,
so,
let's
see
so
when
we're
when
we're
exploring
or
exploiting
a
cluster,
we're
thinking,
here's
how
we're
thinking
about
it.
We're
thinking
about
how
data
flows
through
the
cluster,
both
in
the
control
plane
and
even
through
the
workloads,
were
targeting
critical
components,
especially
anything
anything
that
could
grant
us
access
or
do
something
that
developers
or
operators
didn't
intend
or
hopefully
didn't
even
think
of
we're
watching
to
see
whether
network
isolation
is
either
hampering
us.
B
Well,
more
often
restricting
our
view
of
the
situation
of
the
cluster
we're
looking
to
evade
that
isolation,
more
otherwise,
try
to
break
it
so
part
of
how
we
do
that,
we'll
look
at
how
we
can
traverse
trust
boundaries
and
how
he
can
gain
lateral
levels
of
privilege.
They
might
give
us
better
visibility
around
that
network.
Isolation,
we'll
be
looking
at
how
we
can
escalate
privilege
we're
trying
to
find
any
opportunity
to
perform
a
monkey
in
the
middle
attack.
I
demoed,
one
of
these
in
a
recent
talk,
actually
we're
showing.
B
If
an
attacker
can
create
pods
in
a
namespace,
then
they
can
basically
monkey
in
the
middle
any
traffic.
That's
headed
to
a
service
in
that
namespace.
Let's
see
and
perhaps
most
crucially
we're
trying
to
get
or
modify
secrets
and
those
could
be
access
tokens,
they
could
also
be
passwords.
Tls
private
keys
configuration
lines.
Anything
the
defenders
don't
want
us
to
get
at.
So
the
other
interesting
thing
from
an
attackers
perspective
is
to
think
about
the
vantage
point
at
which
you're
attacking
a
cluster.
B
So
on
first
glance,
people
tend
to
think
about
attack
attacking
something
meaning
that
you're
hitting
it
across
the
internet.
As
an
attacker,
we
consider
those
situations
to
be
lucky
breaks.
Those
places
like
where
Tesla
happened
to
leave
there
happen
to
leave
their
dashboard
available
to
the
Internet.
Basically,
on
authenticated
as
I
recall,
those
are
that
that
happens,
but
it's
kind
of
rare
so
as
an
attacker
most
often
we're
attacking
kubernetes
clusters
from
inside
the
cluster.
So
how
do
we
get
inside
the
cluster?
Most
the
time
we're
working?
B
We
start
working
from
some
compromised
workload
on
the
cluster.
If
we're
really
lucky,
that's
some
control
plane
elements
if
we're
really
really
lucky
either
way.
There
just
happens
to
be
a
container
breakout
vulnerability
that
gets
us
execution
on
the
node.
We're
gonna
look
at
this
a
little
more
graphically
on
the
next
slide.
This
is
the
one
time
where
I
get
to
use
my
laser
pointer.
B
So
Erin,
can
you
just
put
your
eyeballs
right?
No,
no.
We
had
to
we've
had
to
argue
here
and
there
about
content.
So
you
know
give
me
the
laser.
So
this
diagram
actually
came
right.
It
came
right
out
of
the
threat
model
and
and
it's
a
and
it's
pretty
useful.
So
as
an
attacker
like
the
the
first
place,
is
I
might
be
coming
from
outside
the
cluster
and
you'll
note.
B
It
says
user
here
and
it's
his
external
attacker
here
and
for
me
those
are
the
same
thing:
I'm
a
I'm,
a
security
guy
so
and
I'm
a
defender,
I
think
every
user
is
an
attacker
and
when
I'm
attacker
I
well
yeah,
I'm
an
attacker.
So
of
course
I
think
everybody's,
an
attacker.
So
anyway,
I
can
be
what
I'm
going
from
the
outside
of
the
cluster
I.
Don't
see
a
lot
of
attack
surface
I
see
any
of
the
in
grasses
any
of
the
services
that
actually
land.
B
You
know
any
of
the
services
that
are
exposed
outside
that
land
on
pods
and
that's
what
am
I
said:
I'm,
sorry,
so
so
I've
got
so
so
that's
you
know!
That's
what
I'm,
seeing
from
outside
I
and
I,
need
that
and
that's
my
main
way
of
actually
getting
into
the
clusters
basically
can
I
find
a
vulnerable
workload.
I
have
an
intentionally
vulnerable
cluster
called
Busta
Kubb!
B
B
Often
I,
don't
see
much
right,
I
see
services
that
land
that
land
on
pods,
if
I'm,
lucky
I,
might
have
one
of
those
great
situations
where
I'm
attacking
something
in
the
cloud
and
the
Installer
is
set
up
the
API
server
so
that
it's
available
to
the
entire
Internet.
That
kind
of
mistake
is
rare,
but
I
might
find
it
finding
any
other
control
plane
component,
that's
accessible
from
the
Internet
at
large.
Good
luck,
so
I
don't
see
a
whole
lot
from
the
outside.
B
B
In
fact,
since
most
clusters
don't
have
egress
network
policies,
I
can
often
also
communicate
with
the
cubelet
on
each
of
the
pods,
so
I'm,
almost
always
able
to
see
the
cloud
provider
eight
and
honestly,
that's
where,
like
half
of
the
juice
comes
from,
including
this
metadata
API
that'll
give
me
cloud
credentials
that
let
me
interact
with
all
the
cloud.
Api's
and
honestly,
a
bunch
of
cluster
compromises
come
that
way,
but
I'm
on
a
tangent.
B
Let's
see
what
else
so
attacking
the
scheduler
and
the
controllers
is
available,
but,
to
be
honest,
most
of
my
access
to
those
is
coming
through
API
server
before
I
can
get
to
before.
I
can
get
to
those
ritu.
The
to
the
other
master
components
at
CD
again
is
that
similar
is
also
similarly
something
I,
basically
attack
from
starting
from
the
API
server.
So,
finally,
I
might
be
able
to
attack
the
container
runtime
when
it
come
and
and
when
I
get
lucky
there
I
really
like
it.
B
The
biggest
place
I
end
up
getting
an
ability
to
attack
container
runtime
if
I
don't
have
some
kind
of
way
of
breaking
out
of
the
container
is,
if
I
end
up
in
a
wonderful
container
and
wonderful
pod,
that
has
the
docker
socket
or
what
have
you
mount
it
in
and
before
you
think,
that's
unrealistic.
It's
it's
not
talk
to
me
after
so
ok,
so
Aaron
Aaron
actually
convinced
us
on
the
working
group
to
put
threat
modeling
on
this
effort
on
this
effort.
That
was
looking
for
bugs
and
vulnerabilities.
B
A
I
felt
so
strongly
about
this,
because
I
kept
talking
to
independent
researchers
in
the
security
community
and
they
kept
telling
me
they
wanted
to
hack
on
kubernetes,
but
they
didn't
really
know
where
to
get
started.
I
wanted
to
unlock
all
that
talent,
but
the
exists
out
there
and
one
of
the
ways
I
could
think
of
to
do.
That
was
to
hand
to
them
a
threat
model
or
like
a
roadmap
for
how
to
attack
kubernetes.
A
So
we
need
to
unlock
that
knowledge.
We
need
to
view
it
through
a
security
lens
and
then
we
need
to
draw
conclusions
from
it.
The
rapid
risk
assessment
is
just
a
really
good
at
getting
people
who
aren't
security
people
to
talk
about
security
stuff.
So
in
the
end
we
use
that
process
and
we
interviewed
a
bunch
of
people
across
a
bunch
of
different
SIG's
and
distilled
that
into
a
consumable
report.
Jay's
gonna
talk
about
some
one
stuff.
We
found
cool
thanks.
B
So
the
most
common
question
we've
gotten
when
we
talk
to
people
what
the
threat
model
is,
why
these
components
and
not
others,
well,
look
the
question
really
should
be
why
these
trust
zones-
and
some
of
you
are
probably
thinking
to
yourself-
well
wait
what's
a
trust
zone,
so
we
looked
at
the
architecture
of
a
kubernetes
cluster
and
broke
it
down
into
these
six
areas.
So
components
in
these
areas
and
I
really
should
be
using
our
pointer,
I
guess.
B
Components
in
these
areas
have
very
few
security
controls
between
them,
so
trust
zones
areas,
so
components
in
these
areas
have
very
few
security
controls
between
them
within
the
area
but
there,
but
each
of
them
are
responsible
for
enforcing
controls
on
communication
that
are
passing
into
and
out
of
their
area
or
trust
zone.
So,
for
example,
the
API
server
can't
blindly
trust
the
Internet,
but
the
worker
components
implicitly
trust
the
API
server
and
the
API
server
in
turn
fully
trusts
@cd
for
its
state
of
the
war.
It's
for
its
view
of
the
world.
B
So
we
looked
at
each
of
these
trust
zones
and
chose
the
components
in
them
that
had
the
biggest
security
impact
on
that
zone.
So
at
the
top
of
the
food
chain
you
have
Etsy
D
and
the
API
server,
and
now
these
are
the
these
are
the
crown
jewels
of
the
cluster.
If
you're
able
to
control
these
components,
you
have
full
control
over
the
cluster
itself
and
it's
just
game
over
easy.
B
In
the
controller
manager
zone
we
looked
at
coop,
scheduler,
coop
controller
manager
and
the
cloud
controller
manager
and
these
components
control
the
topology
of
the
cluster
and,
if
you
own
them
boom
privilege
escalation.
So
a
lot
of
the
kubernetes
security
controls
also
are
actually
really
node.
Centric,
for
example,
kubernetes
will
make
sure
that
the
secrets
that
are
the
secrets
that
secrets
are
available
only
to
the
nodes
that
have
pods
scheduled
on
them.
That
actually
need
those
secrets.
B
Well,
if
you
have
any
control
of
scheduling,
then
you
can
trivially
bypass
that
control
and
make
sure
that
the
node
that
you've
got
compromise
on
has
the
secrets
you
want.
We
looked
at
the
couplet
which
ties
kubernetes
to
the
container
runtime
and
des
coop
proxy,
which
handles
the
pods
networking
needs.
If
you
own
the
couplet
you
own
every
workload
secret
data
volume
on
the
node.
B
If
you
don't,
but
you
own,
coop
proxy,
you
get
to
control
where
all
the
network
traffic
goes
that's
on
its
way
to
or
from
pods
we
didn't
look
at
any
specific
container
runtimes.
Instead,
we
would
looked
at
the
container
runtime
interface.
We
had
to
make
sure
there
weren't
any
bad
behaviors
introduced
by
that
by
the
container
runtime
interfaces,
abstraction
layer
that
between
the
kulit
and
the
container
runtime.
So
next
we're
going
to
talk
a
little
bit
about
the
threat
models
results
so
we're
giving
you
the
highlights.
B
As
Erin
said,
this
effort
produced
hundreds
of
pages
of
really
really
useful
stuff
and
we
have
35
minutes
so
we're
gonna,
give
you
the
we're.
Gonna
give
you
the
highlights.
So
first,
there
are
at
least
a
couple
areas
where
a
user
can
configure
security
control.
Have
the
API
server
accept
that
security
control
and
then
have
that
configuration
achieve
no
result?
B
We'll
call
that
a
we'll
call
that
a
no
op
so
as
an
example,
if
you're
using
a
networking
provider
that
doesn't
support
egress,
doesn't
support
egress
policies,
you
can
put
egress
rules
in
place,
but
crib
annuities,
won't
warn
you
that
you
haven't
actually
got
any
firewall
rules
governing
outbound
traffic.
Now
you
find
out
only
if
you
either
do
your
own
test
or
if
you're,
like
most
people
you
find
out
when
attacker
shows
you
that
there
was
no
egress
policy
enforced
and
they
show
you
in
a
really
bad
way.
B
So
pod
security
policies
have
the
same
problem.
You
can
apply
a
policy,
but
if
you
don't
realize
that
you
have
to
actually
activate
the
pod
security
policy,
admission
controller
you're
left
thinking,
you
have
policy
enforcement
when
you
don't
and
that's
kind
of
it's
just
another.
It's
just
a
that's
another
case
where
you
think
you
but
control
you
don't
have
it.
It's
kind
of
sale,
fail,
silently
or
apparently
sale
filin.
B
B
Unfortunately,
the
one
thing
we
are
doing
consistently
is
setting
kubernetes
internal
SSH
connections
to
never
check
the
survey
to
check
the
server
certs.
Let's
see.
Third,
the
threat
model
found
out
that
we
weren't
doing
enough
to
stop
a
worker
node
compromise
from
leading
to
a
cluster
compromise.
We're
going
to
speak
to
this
a
little
bit
more
in
a
couple
slides
forth,
researchers
told
us
that
we
really
needed
to
provide
a
unified
audit
trail,
so
you
could
trace
an
attackers
path
through
the
system.
B
B
So
the
researchers
recommended
that
we
figure
out
how
to
do
least
privilege
here,
in
addition,
I'm
going
to
move
on
and
talk
about
the
sources
husband
findings,
but
you
should
know
that,
in
addition
to
the
final
model
report,
which
has
more
than
I'm
just
giving
you
these
highlights
all
the
raw
rapid
risk
assessment
Doc's
are
available
in
our
repo.
If
you
want
to
dig
a
little
deeper
so
now,
I
want
to
talk
with
the
source
assessment
in
this
2019
round
weighted
split
the
time
between
a
bunch
of
deliverables.
B
We've
still
got
a
heavy
focused
on
bug,
hunting.
They
found
37
vulnerabilities,
which
got
supported
in
through
the
reported
in
through
the
product
Security
Committee,
the
PSC,
so
Craig
opened
an
issue
for
each
Joel
coordinator
with
a
PSC
and
what
actions
we
could
take
in
it
like
embargoes
and
so
on.
On
the
next
slide,
we've
got
a
chart
with
the
with
those
same
with
some
of
the
highlights,
so
the
first
one
is
those
non
authenticated,
HTTP
connections
which
we've
already
talked
about
in
the
second
one.
We
got
dinged
on
not
supporting
certificate.
Revocation.
B
This
is
another
one.
That's
really
important!
You
have
a
breach
if
the
certificates
can't
be
revoked
and
the
attacker
got
those
certs,
then
the
attacker
might
have
access
a
lot
longer
than
you
think
they
do
30
in
the
third
finding
here
we've
got
a
pod
security
policy
bypass,
and
this
one
was
kind
of
fun
to
me
because
if
you've
ever
mounted
the
nodes
root
filesystem
into
a
container
and
that
containers
running
as
root,
this
is
something
we
do
in
docker
all
the
time
we're
just
playing
with
docker,
not
kubernetes.
B
You
realize
it
gives
the
attacker
great
opportunity
to
compromise
the
node
through
its
file
system,
so
the
canonical
way
to
prevent
pods
from
doing
this
is
with
pod
security
policies
and
the
source
assessment
team
found
that
an
attacker
can
get
around
this
restriction
just
by
mounting
the
nodes
filesystem
as
persistent
volumes,
instead
of
as
the
instead
of
as
host
paths
host
path
volumes.
Fourth,
one
was
a
time
of
check
time
of
useful
in
their
ability.
B
Attacker
can
chain
with
another
one
to
get
privilege
escalation
for
their
container
on
a
node
and
get
full
privilege
on
the
node.
The
fifth
one
was
pretty:
cool.
Researchers
found
a
directory
traversal
issue
in
coop
control
copy
and
because
it
uses
tar
and
here's
what
was
cool.
The
vulnerability
already
been
discovered
and
fixed
by
crew
Burnett
by
the
criminals
project,
but
the
fix
had
a
bug,
so
the
work
was
valuable
in
the
last
one.
B
B
The
other
was
the
I
scuzzy
credentials
whenever
using
I,
I
scuzzy
for
volume,
storage
being
logged
to
the
node
system
logs-
and
these
point
is
something
systemic,
not
just
with
logging,
but
overall
kubernetes
isn't
particularly
hardened
against
an
attacker
he's
able
to
break
out
of
a
container
and
take
an
administrative
control
of
a
node
once
you're
on
a
node.
There
are
privilege
escalations.
Some
are
obvious,
like
credentials
being
visible
in
the
command
lines
and
environment
variables
being
available
and
having
secrets
in
them,
but
others
are
less
obvious.
B
Like
tons
of
world
writable
world
readable
and
world
writable
file,
permissions
I
really
like
this
whole
area
for
research
and
there's
a
talk
tomorrow
by
Greg
castle
and
Tim
all
players
here,
somewhere
yeah,
there
is
so
Greg's
in
the
back
of
the
room
afterwards,
please
mob
him,
but
there's
a
great
talk
by
Greg
and
Tim.
That
looks
at
turning
a
node
compromise
into
a
wired,
wider
cluster
attack,
and
so
we
really
recommend
that
and
honestly
we
recommend
that
as
an
area
for
research,
so
I
also
want
to
show
you
share.
B
Show
you
a
quote
from
the
reports,
exact
summary,
and
it
says
we
need
to
do
three
things.
The
one
is.
We
have
to
ease
security
configuration
because
it's
too
easy
to
accidentally
configure
kubernetes
badly
and
that
leads
to
vulnerable
clusters
and
those
vulnerable
clusters
may
not
be
our
faults,
but
it's
our
responsibility
to
fix
the
situation.
B
We
can
also
improve
documentation
in
the
code,
both
by
getting
more
of
it,
but
also
making
sure
to
document
our
dependencies,
especially
when
we're
dealing
with
external
dependencies
and
then
finally,
this
is
kind
of
tie
to
that
HTTP
thing
there
are:
there
are
tons
of
places
in
the
code
where
somebody
implements
somebody
implements
functionality.
Somebody
else
has
implemented
in
another
place
and
it's
been
implemented
by
someone
else
in
another
place,
so
you've
a
ton
of
this
logic,
riri
implementation
and
they
said
listen.
B
Here's
getting
a
little
bit
more,
getting
a
little
bit
more
focused
or
systemic
years.
You
could
work
on
documentation
for
the
developers
that
avoid
these
issues.
You
can
also
focus
that
documentation
on
approvers
so
that
the
approvers
are
able
to
are
able
to
catch
more
of
this
stuff
on
its
way
into
the
codebase.
B
A
D
A
We
actually
really
do
need
your
help.
We
are
getting
started
on
the
next
round
of
audits.
We
all
have
ways
to
get
in
touch
with
us
at
the
end,
but
like
we
need
help
now
with
all
this
stuff.
There's
a
ton
of
the
qualitative
stuff
to
come
out
of
the
reports
are
like
real
and
we
need
to
fix
them.
So
Trinities
is
more
secure
and
CERN
doesn't
get
hacked.
Okay,
bones.
A
A
Not
only
helped
us
get
the
money
to
fund
it,
but
was
our
liaison
with
them
for
anything
that
we
needed
was
incredibly
helpful
and
all
everyone
who
volunteered
to
work
on
the
threat
model-
I,
don't
know
we
got
everybody
here,
but
these
people
are
very
busy
and
they
spent
a
ton
of
their
time
to
make
there's
a
quality
report
and
I
think
it
shows
some
good
results.
If.
B
We
missed
anybody
there.
It's
my
fault
will
publicly
correct
it.
We'd
also
like
to
thank
the
first
to
third-party
firms
to
participate
in
kubernetes
in
third
party
kubernetes
audits,
that's
trail
of
bits
and
at
Redis
partners.
We
want
to
give
special
thanks
to
Stefan,
Edwards
and
Bobby
tonic
for
their
great
work,
and
you
should
catch
their
talk
to
finally
well,
thank
you
in
advance.
Let's
go
fast
he's.
B
All
right,
okay,
cool
so
also
there
are
five
great
attack
and
defense
talks.
We
want
to
recommend
here
the
last
one
on
the
slide
is
my
recent
Scott
Pilgrim
themed
attack
and
defense
talk
from
blue
hat
a
few
weeks
ago.
The
others
are
talks
at
this
conference.
So
here's
your
schedule
on
Tuesday
you've
got
Tim
all
players
and
Greg's
talk
on
node
privilege
escalation.
Then
Wednesday
you've
got
a
trail
of
bits
talking
about
their
threat.
A
A
A
C
A
C
A
Talked
about
it
a
little
bit,
the
short
answer
is
no
long
answer
is
we
were
trying
to
say
as
focused
on
the
kubernetes,
as
defined
by
what's
in
the
kubernetes
repo,
as
opposed
to
kubernetes,
as
defined
as
the
thing
that
runs
your
application,
and
the
reason
for
that
was
just.
We
didn't
have
time
to
cover
every
potential
CNI
plugin,
every
potential
container
run
time
or
the
entire
attack
surface
exists
before
and
after
the
cluster
is
up
and
running.
A
B
A
B
Certainly,
the
the
what
the
gentleman
said
was,
if
you
that
goes
back
to
something
else
you
were
saying
earlier,
which
was
if
you
have
a
lot
of
repeated
code.
If
things
aren't,
if
things
are
not
as
commented
or
not
clear,
it's
a
whole
lot
easier
for
a
compromise
developer
to
to
insert
something
bad
and
we've
all
seen
that
with
Linux
kernel.
I.
Remember
the
first
big
Linux
kernel,
privilege
escalation
was
like
a
double
equal
sign.
Being
changed
to
a
single
equals
sign
meant
you
got
really
all.