►
From YouTube: The Hitchhiker's Guide to Kubernetes Vulnerabilities - Robert Clark & Micah Hausler, Amazon
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
The Hitchhiker's Guide to Kubernetes Vulnerabilities - Robert Clark & Micah Hausler, Amazon
The earth is gone. You’re the only one left. Was Kubernetes to blame? Is prior performance a strong indicator of future behavior? Is Kubernetes on a positive security trajectory? Have community efforts improved security? The speakers will present research that examines security problems throughout the entire history of the Kubernetes project. They explore patterns and trends in the data. They show a taxonomy for classifying k8s vulnerabilities; including analysis of root causes and contributing factors. They dive into each issue, examining metrics like time from commit-to-discovery, time-to-resolution, detection of similar events etc. They look to patterns of previous behavior to help the audience predict future performance. They interview leaders in the community and overlay historical security data with efforts to improve security. They show the impact SIGs, WGs, Audits, etc had on k8s security and suggest how the security posture of k8s might evolve in the future.
A
Hello,
my
name's
robert
clark
and
I'm
here
today
to
talk
to
you
about
the
hitchhiker's
guide
to
kubernetes
vulnerabilities.
I
recently
joined
the
amazon
elastic
kubernetes
service
eks,
where
I
lead
security.
I
co-wrote
this
presentation
with
michael
hauser
who's
been
working
on
eks
and
contributing
to
the
kubernetes
security
community
for
several
years.
A
Neither
myself
or
micah
are
data
scientists,
most
security,
architects,
engineers,
people
who
are
going
to
be
looking
at
vulnerability
data
and
trying
to
use
it
to
help
make
more
informed
choices,
aren't
going
to
be
a
data
scientist
either.
So
we've
taken
that
approach,
we've
pulled
the
data
together.
We've
tried
to
present
it
in
a
couple
of
different
ways
to
help
us
understand
whether
kubernetes
is
doing
better
or
worse.
With
regards
to
software
vulnerabilities
and
understand
what
we
expect,
the
trends
to
be
in
the
future.
A
In
the
spring
of
2021,
red
hat
produced
a
kubernetes
security
report,
it
was
a
short,
useful
survey
of
industry,
but
it
showed
that
vulnerabilities
were
only
the
second
most.
Concerning
issue
raised
among
the
more
than
500
devops
engineering
and
security
professionals
that
participated
in
the
survey.
A
There
were
some
interesting
call-outs
in
the
survey
94
of
respondents
experienced
at
least
one
security
incident
in
their
kubernetes
environments
in
just
the
last
12
months.
That
is
a
terrifying
statistic
and
55
percent
of
respondents
actually
had
to
delay
or
slow
down
application
deployments
into
production
due
to
container
or
kubernetes
security
concerns.
A
So
wall
vulnerabilities
might
not
be
top
of
mind
for
survey
respondents.
I
believe
that
market
pressures
over
the
next
12
years
may
change
the
results
of
a
similar
survey,
and
what
do
I
mean
by
that?
Well,
there's
lots
of
attention
being
paid
to
improving
the
secure
defaults
in
kubernetes
right.
This
is
where
everything
the
far
far
left
of
this
graph
today
is
all
about
misconfigurations
there's,
an
entire
ecosystem
of
tools
and
support
aimed
at
improving
the
full
life
cycle
of
kubernetes
deployments.
A
A
That
means
something
that
should
be
secret
is
no
longer
secret
integrity,
something
that
should
be
trusted
to
be
one
state
and
not
written
can
be
written
and
therefore
untrusted
availability
thing
that
should
stay
up
goes
down.
Authentication,
I
know
who
you
are,
and
I
can
be
reasonably
sure
that
in
that
assertion
authorization
I
know
what
you
are
able
to
do
and
a
vulnerability
that
compromises.
A
That
might
mean
that
you
can
do
things
that
you're
not
supposed
to
be
able
to
do
and
audit
what
is
actually
really
important
to
us
as
a
security
practitioner
is
to
really
understand
what's
going
on
in
a
kubernetes
system,
so
the
ability
to
disrupt
that
and
either
add
malicious
things
to
an
audit
trail
or
cause
certain
operations
to
not
be
logged.
Both
super
super
important
for
us
to
pay
attention
to
so
within
kubernetes.
The
product
security
committee,
which
is
which
was
recently
rebranded
to
the
security
response
committee,
deals
with
vulnerabilities.
A
This
rebranding
better
reflects
the
function
of
this
wonderful
group
of
volunteers,
whose
names
are
up
on
the
slide
right
now.
They've
come
from
across
the
kubernetes
ecosystem
and
they
provide
time
support
and
expertise
to
triaging
remediating
and
coordinating
the
responsible
disclosure
of
kubernetes
vulnerabilities.
A
So
what
happens
in
the
life
cycle
of
a
kubernetes
vulnerability?
Well,
a
researcher
will
research
will
discover
an
issue
and
they
will
hopefully
report
it
responsibly
to
the
the
security
response
committee.
At
that
point,
their
process
takes
over,
so
a
public
disclosure
date
is
negotiated
between
the
src
and
the
bug
submitter.
A
A
So
the
kubernetes
project
is
rapidly
evolving
with
new
features,
design
updates
and
bug
fixes.
Currently,
the
kubernetes
community
releases
a
new
minor
version,
approximately
every
four
months
and
maintains
three
active
versions
at
any
given
time.
Occasionally,
kubernetes
upgrades
change
apis
and
this
can
sometimes
break
existing
deployments
because
api
changes
will
be
required.
This
happened
from
1.15
to
1.16
and
will
happen
again
soon
with
1.21
to
1.22.
A
A
This
means
that
vulnerabilities
specific
to
an
older
version,
an
unsupported
version
of
kubernetes,
may
not
even
get
reported,
leaving
customers
exposed
with
no
notice
no
remediation
options.
In
the
case
of
a
vulnerability,
the
key
point
here
is:
you
need
to
try
and
stay
in
a
supported
version
of
kubernetes.
A
So
if
we
look
quickly
at
the
history
of
the
kubernetes
project,
we
see
that
it
was
released
in
version
1
in
july
2015,
followed
in
november
2015
by
the
first
kubernetes
cve,
which
allowed
remote
attackers
to
read
arbitrary
pod
logs
via
container
name
manipulation
in
january
2016,
the
product
security
committee
was
established
and
before
that
vulnerabilities
were
reported
direct
to
google.
So
thank
you
to
google
for
for
performing
that
function
before
the
psc
was
created
in
december
of
2018.
A
A
A
So
let's
talk
about
where
to
get
vulnerability.
Data
one
place
you
can
go
is
cve
details.
It's
an
excellent
resource.
It's
useful
for
pulling
big
batches
of
cv,
information
about
many
open
source
projects.
All
the
data
is
taken
from
xml
feeds
that
are
provided
by
nvd,
the
national
vulnerability
database.
I'll
talk
about
them
in
a
moment.
A
A
Beyond
that,
another
good
resource
is
the
national
vulnerability
database,
which
is
produced
by
the
national
institute
of
standards
and
technology,
which
we
all
know
and
love
is
nist.
The
nvd
has
been
running
in
some
form
since
1999,
where
it
was
first
known
as
icat.
The
internet
category
of
attack
toolkit
actually
has
an
interesting
history
around
2001
it
was
defunded
and
taken
over
by
the
sanders
institute.
They
stepped
in
and
provided
funding
funding,
and
it
kept
it
running.
A
A
You
know,
with
the
exception
of
severity,
which
we'll
talk
about
later,
I'm
not
going
to
go
much
further
into
vulnerability
management,
specifics
for
kubernetes,
it's
very
use
case
specific.
It
changes
from
operator
to
operator
and
vendor
to
vendor,
but
I
would
encourage
everyone
obviously
to
look
at
what
is
available.
What
is
made,
what
is
natively
in
your
platform
and
and
make
sure
you
have
a
good
understanding
of
of
the
threats
that
your
platform
may
be
facing
after
gathering
together.
What
I
thought
was
all
the
vulnerability
information
about
kubernetes.
A
I
figured
the
best
place
to
start
understanding.
The
current
security
posture
was
to
look
at
how
the
vulnerabilities
have
distributed
over
the
years.
I
believe
that
kubernetes
is
more
secure
now
than
ever
before.
So
can
I
make
this
presentation
really
quick
by
showing
a
nice
graph
that
slopes
down
and
to
the
right?
A
A
A
A
A
A
Apparently,
things
took
a
turn
around
version
1.14
if
we
just
base
only
on
when
fixes
are
released,
2019
through
2020
were
really
bad
years
for
kubernetes
and
it
and
definitely
busy
years
for
the
security
response
committee.
But
at
least
things
seem
to
have
gotten
better
recently,
with
only
one
issue
fixed
in
the
last
couple
of
releases.
A
It
didn't
that
doesn't
that
doesn't
seem
right,
I'm
pretty
sure
there
have
been
issues
recently,
but
I
checked
cv
details.
I
checked
mbd,
so
I
you
know,
do
what
everybody
does
when
they're
a
bit
confused,
just
go
to
your
favorite
search
engine
of
choice
and
have
a
look
at
the
issue
you
know
was:
is
2021
the
best
year
in
history
for
cube
security?
A
Can
we
conclude
from
the
lack
of
cvs
that
everything's
been
fixed?
Well,
google
says
no.
Google
says
there
is
more
data
that,
for
some
reason,
isn't
readily
available
or
hasn't
made
its
way
through
to
to
the
to
the
nvd
database.
Yet
so,
let's
go
ahead
and
let's
add
that
gra
add
that
data
to
our
graph.
Now
one
of
the
vulnerabilities
that
you
saw
in
the
mailing
list
screenshot
there
was
a
very
windows,
specific
client
thingy,
so
we
kind
of
grade
that
out
here,
but
either
way
it's
looking
a
little
bit
more.
A
Concerning
now
you
know.
Actually
we've
got
this
big
red
block.
I
guess
at
this
point
we
should
pause
to
make
three
key
observations.
One
kubernetes
is
clearly
doomed.
Two,
there
is
sometimes
a
significant
lag
in
kubernetes
vulnerability,
information
making
it
into
the
mitre
cde
list
which
feeds
tools
like
mbd
and
cve
details.
A
A
A
So
we
took
that
data
and
then
we
added
in
the
the
data
here
that's
in
purple,
which
is
from
identifying
the
the
line
of
code
that
introduced
a
problem
in
a
cv
and
basically
working
back
through
git
histories.
Until
we
found
roughly
when
we
think
it
was
introduced,
we
were
able
to
put
these
two
together
and
show
put
together
this
little
like
chart
that
shows
the
the
lifetime
of
vulnerabilities
and
then
their
remediation
windows.
A
Here
we
see
that
the
the
increase
in
reporting
around
1.15
didn't
really
correlate
with
long-lived
vulnerabilities
and
we
don't
see
a
bunch
of
venerability
snapping
shirt
on
that
date,
so
it
seems
likely
that
the
increased
popularity
of
kubernetes
is
resulting
in
maybe
more
people
examining
the
code
and
finding
issues
so
trying
to
understand
this
situation,
trying
to
understand
why
it
can
trend
up
like
this.
We
know
there
are
environmental
factors,
but
how?
How
are
we
doing
here
so
what
we
do
actually
do
know
about
some
environmental
factors?
A
A
So
maybe
these
things
are
actually
tracking
with
us.
So
we
looked
at
results.
We
looked
at
the
results
of
the
audit
and
actually
very
few
results.
Cbs
came
directly
from
that.
It
was
focused
more
on
secure
design
patterns
than
specific
cves,
which
is
great.
It's
just
not
super
useful
for
us
to
have
a
conversation
about
cvs.
A
What
was
in
the
data
was
a
very
interesting
demonstration
of
the
impact
of
bug
bounties.
So
here
you
see,
the
overwhelming
majority
of
vulnerabilities
that
were
found
more
recently
came
through
the
bug,
bounty
program
and
that's
really
interesting,
and
it
leads
to
leads
us
to
to
to
ask
some
more
interesting
questions
about
those
more
recent
vulnerabilities.
A
The
number
of
different
types
of
vulnerabilities
you
see,
and
to
do
that
we
need
some
common
way
of
enumerating
weaknesses,
so
mitre
the
organization
that
brought
you
cves
has
been
working
on
common
weaknesses,
enumeration
for
a
very
long
time.
Common
weaknesses
is
a
large
database
of
pathologies
of
vulnerabilities,
different
ways
in
which
people
can
make
mistakes
and
cause
these
problems
to
occur.
A
So
again,
we'll
go
and
have
a
look
and
we'll
chart
the
number
of
differences
in
cwe's,
and
you
know
we'll
start
with
this
big
ugly
graph
as
you've
gathered.
I
like
to
do
at
this
point
and
we
see
again
that
there
are
lots
of
vulnerabilities
here.
There
are
lots
of
unique
cwes
here
and
a
lot
of
them
happen
around
2019,
but
we
do
see
a
good
drop
off
around
2020
and
2021.
A
So
let's
have
a
look
at
what
that
looks
like
in
a
slightly
different
format.
So
a
number
of
the
the
categories
here
are
clipped,
but
the
the
interesting
point
here
is
that
the
distribution
is
fairly
even
like.
There
was
just
a
lot
of
vulnerabilities
going
looking
through
all
of
kubernetes
a
lot
of
vulnerabilities
in
a
lot
of
different
parts
of
the
cwe,
which
means
just
holes
everywhere,
twisties
problematic
from
a
security
point
of
view,.
A
So
we
think
the
frequency
of
reports
increasing
and
the
diversity
of
cwes
decreasing
is
an
indicator
that
the
security
audit,
followed
by
the
bug
bounty
program,
has
pruned
much
of
the
low-hanging
fruit
of
the
kubernetes
code.
Base
vulnerabilities
are
still
being
reported
at
a
reasonable
pace,
but
the
diversity
of
them
is
reducing.
A
You
need
to
go
flip,
but
I
want
to
make
you
aware
of
aware
of
mechanisms
that
you
should
be
considering
when
you're
having
to
decide
how
important
a
vulnerability
is,
because
we
all
have
to
like
this
is
probably
an
email
that
many
of
you
have
received
at
some
point
or
some
variation
of
it
for
sure.
For
some
of
us,
emails
like
this
are
almost
a
daily
occurrence
and
with
good
reason.
Vulnerabilities
in
kubernetes
are
serious
business
and,
if
not
managed
appropriately
severe
vulnerabilities
can
put
workloads
or
even
entire
clusters
at
risk.
A
So
I've
talked
to
you
a
little
bit
about
the
life
cycle.
Already,
I'm
going
to
talk
to
you
now
about
cvss,
v3
and
rescoring,
and
why
it's
important,
because
very
few
organizations
do
this
and
I'll
give
you
an
example
of
one
that
does
it
really
well
in
a
minute,
but
I'm
just
going
to
walk
you
through
it
quickly,
because
it's
very
important,
so
all
of
the
vulnerabilities
we've
talked
about,
have
had
a
cvss
score.
Take
this
mythical
vulnerability,
it's
very
red!
A
A
This
was
all
to
do
with
environmental
scoring,
so
the
environmental
score
goes
to
talk
about
how
it
is
a
vulnerability
can
change
depending
on
how
it's
deployed,
not
all
kubernetes
vulnerabilities
are
reachable
by
all
parties.
Maybe
your
control
plane
is
strongly
segregated
from
your
data
plane.
Maybe
you
run
individual
clusters
for
different
website.
A
This
vulnerability
actually
got
up,
scored
by
red
hat
red
hat,
telling
their
customers
hey
this
cbe
for
the
way
that
red
hat
believes
that
their
software
is
being
deployed
is
actually
more
important,
it's
more
important
than
or
more
severe
than
the
mvd
rating.
So
you
know
these
things
can
go
up,
they
can
go
down.
The
context
is
really
important.
You
really
do
need
to
spend
the
time
doing
the
rescoring
there's
only
a
little
bit
over
40
vulnerabilities
in
the
last
five
years
for
kubernetes,
and
it's
not
super
time
consuming
to
do
the
rescoring.
A
It's
absolutely
worth
doing
it
to
understand
which
things
you
need
to
escalate
and
take
more
seriously
and
which
things,
maybe
you
can
go
a
little
bit
slower
and
be
a
little
bit
more
cautious
with
so
this
is
really
bringing
us
towards
the
takeaways
here.
This
has
been
a
whistle-stop
tour
of
how
kubernetes
vulnerabilities
can
present
stats
are
hard.
We
think
security
is
getting
better
and
obviously
gorillas
are
great.
So
I
just
want
to
leave
you
with
some
of
these.
These
takeaways
really
2019
saw
the
highest
number
of
cves
by
by
distribution.
A
The
number
of
vulnerabilities
increased
over
the
last
several
years.
We
think
it
will
continue
to
increase,
but
we
think
that
this
increase
is
sublinear
compared
to
the
numerous
different
dimensions
in
which
kubernetes
is
just
exploding
in
terms
of
size
and
growth,
and
that
the
number
of
unique
cwes
has
consistently
decreased
over
the
last
few
years
and
in
the
last
two
years
really
did
decrease
very
significantly.
A
We're
seeing
more
issues
reported
of
the
same
type
because
of
tools
like
semgrep
and
other
things
that
are
allowing
people
to
say
hey
like
I
found
this
bad
pattern,
I'm
going
to
report
n
number
of
vulnerabilities
that
I
found
or
match
this
pattern
canonical
sources
for
vulnerability.
Information
can
sometimes
not
be
fully
up
to
date.
A
You
need
to
make
sure
that
you're
staying
on
an
up-to-date
version
of
kubernetes
staying
in
that
that
window
of
like
the
last
three
or
four
releases,
is
absolutely
the
thing
that
is
going
to
keep
you
the
most
secure
and
mean
that
you
don't
have
to
go
and
spend
a
lot
of
time
worrying
about
individual
cves
and
and
pulling
this
data
together
and
trying
to
make
these
decisions.
So
with
that
I
invite
micah
to
come
on
screen.
I
encourage
all
of
you
to
ask
any
questions
you
might
have.
Thank
you.