►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
My Container Image has 500 Vulnerabilities, Now What? - Matt Jarvis, Snyk
As security becomes a bigger concern in the world of containers and Kubernetes, using vulnerability scanning tooling in our workflows is becoming increasingly common. But many container images can show tens if not hundreds of vulnerabilities, particularly if they are built using upstream base images from public repositories. If your container has a huge amount of vulnerabilities, what do you do ? Many of us will reach information overload when faced with such a list, and struggle to work out what actions we should take. In this talk, we’ll look at how container images are constructed, understand how potential vulnerabilities can get into our images, and explore how we can prioritize and remediate the vulnerabilities we find. Take control of your vulnerabilities !
A
A
So
when
you
first
start
scanning
your
container
images,
it
can
be
a
bit
disconcerting
to
discover
that
you
might
have
large
numbers
of
vulnerabilities
in
your
images.
This
is
a
scan
I
did
last
week
on
a
vulnerable
node
image
that
I
built
it's
a
fairly
extreme
example,
but
you
can
see
that
this
image
out
of
the
box
is
showing
us
having
over
800
vulnerabilities
in
it.
A
A
A
A
So,
although
we
refer
to
this
as
a
base
image,
it's
likely
that
that
image
we're
using
is
also
constructed
from
a
parent
image
which
then
had
software
installed
into
it.
During
its
build
process,
the
parent
image
itself
was
then
constructed
in
some
way,
perhaps
even
from
a
further
parent
image
or
by
some
kind
of
root
file
system,
building
tool.
A
A
A
So,
as
we've
just
seen
to
use
upstream
images,
we
really
need
to
trust
the
entire
chain
of
build
processes
which
went
into
the
image
that
we're
consuming,
and
this
can
be
difficult
to
follow.
Clearly,
of
course,
this
is
no
different
from
how
we
consume
the
majority
of
open
source
software
and
so
many
of
the
same
quality
factors
that
might
influence
our
choices.
There
also
apply
here.
A
A
For
example,
the
official
ruby
base
image
in
docker
hub
has
lots
of
vulnerabilities
in
it
and
it's
very
big.
This
is
fairly
typical
of
official
run
time
images
because
by
design
they
need
to
be
generalized
for
every
use
case.
So
we
could
look
at
the
slim
version.
That's
smaller.
It
has
less
vulnerabilities
or
perhaps
we
look
for
another
one,
but
there
are
lots
and
lots
of
tags
in
that
repository.
So
how
do
we
choose.
A
Well,
those
generic
runtime
images
are
probably
not
what
you
want
for
production
use
cases
it's
hard
to
tell
which
framework
version
they
might
be
following
and
that
could
change
in
the
future.
But
slim
isn't
automatically
the
best
choice.
You
get
less
vulnerabilities,
but
if
you
use
slim
to
build
your
images
you're
going
to
start
to
need
to
manage
the
build
dependencies
because
they're
unlikely
to
be
in
that
slim
image.
A
So
in
this
way
we're
not
having
to
manage
our
build
dependencies
because
they're
going
to
be
in
that
generic
image
and
we
still
get
to
take
advantage
of
the
size
and
the
reduced
number
of
vulnerabilities
of
the
slim
version
and
note
in
this
example
we're
also
sticking
to
specific
runtime
versions.
So
we
know
exactly
what
runtime
environment
we're
getting,
and
we
know
it's
not
going
to
change
underneath
us.
A
A
Pin
your
apps
to
versioned
images
at
least
major,
probably
minor.
That
way
the
ground
is
not
gonna
shift.
Underneath
you
in
the
future
learn
to
love
multi-stage
builds,
so
you
can
use
slim
images
in
deployment
while
still
taking
advantage
of
proven
combinations
that
you
know
are
going
to
work
in
build
and
rebuild
pretty
often
lots
of
times.
This
is
going
to
get
you
security
fixes
as
part
of
the
build
process
and
then
finally
kind
of
consider
moving
your
pins
every
once
in
a
while.
A
Upgrading
to
new
versions
are
also
going
to
bring
in
more
security
fixes
in
those
base
images
and
fixing
these
things
isn't
usually
very
hard
and
from
some
research
we
did
at
snake
over
40
percent
of
of
docker
image.
Vulnerabilities
can
usually
be
fixed
by
upgrading
the
base
image
and
around
20
percent
can
be
fixed
just
by
rebuilding
them,
because
a
lot
of
containers
are
going
to
run
some
kind
of
upgrade
and
during
the
build
process.
A
A
Well,
if
we've
just
added
a
package
from
an
upstream
distro
repository,
perhaps
we
installed
an
rpm
or
a
deb
as
part
of
our
build
process,
then
the
same
principle
exists
as
with
base
images,
we're
not
going
to
start
building
that
package
from
source
to
fixed
vulnerabilities
in
it
we're
going
to
get
our
friendly
upstream
maintainer
to
ship
us
an
upgraded
package
or
we
might
remove
that
package.
We
don't
really
need
it.
We
might
change
it
use
something
else.
A
And
for
our
own
applications,
typically
most
modern
apps
are
based
on
small
amount
of
homegrown
code
and
lots
of
third-party
modules,
libraries,
which
are
usually
open
source.
This
is
a
pattern
that
you'll
be
familiar
with
if
you're
developing
in
java,
in
node
in
python
and
go
and
in
many
other
languages.
A
And
our
application
dependencies
are
typically
expressed
in
a
file
in
our
source
code
for
javascript.
It
might
be
a
package.json
for
python,
it
might
be
requirements.txt,
but
the
basic
principle
is
the
same:
we're
defining
which
dependencies
and
which
versions
we
need
for
our
particular
application
to
be
packaged
up
with
our
source
code
into
a
into
a
deployable
a
deployable
unit.
A
Now
this
isn't
a
bad
thing.
Having
reusable
code
means
we
write
less
code,
we
don't
have
to
reinvent
the
wheel
and
we
can
spend
more
time
on
the
functionality
we
need,
but
we
do
need
to
be
aware
of
what's
going
into
our
image,
each
of
the
things
that
we
define
as
a
dependency
can
have
a
large
dependency
tree
of
their
own
things
that
they
need
that
they'll
bring
in
automatically.
A
So
there
are
a
couple
of
different
ways
that
you
might
deal
with.
Vulnerabilities
introduced
here
depends
on
the
tooling
you're,
using
here
we're
looking
at
the
sneak
scanner
that
same
container
image
we
started
with,
and
snake
has
actually
identified
the
package.json
for
our
application
inside
the
image
giving
us
a
pretty
clear
picture
of
which
vulnerabilities
are
coming
from
the
base
image
and
which
are
coming
from
the
dependencies
of
our
application.
A
So,
whichever
way
you
end
up
separating
out
the
vulnerabilities,
it's
probably
not
realistic,
except
for
very
simple
applications
to
expect
your
container
image
to
have
no
vulnerabilities
at
all
vulnerabilities
themselves
are
not
a
zero-sum
game.
A
particular
vulnerability
may
only
be
an
issue
under
very
specific
circumstances,
on
a
specific
architecture,
specific
platform.
Without
reading
the
details
of
every
vulnerability,
how
can
we
possibly
decide?
What
is
an
issue
in
our
environment
or
not?
A
A
So
prioritization
is
is
really
not
an
exact
science.
It
can
be
based
on
a
number
of
different
factors.
Severity
alone
doesn't
really
give
us
very
much
information
other
than
potential
impact.
The
cvss
score,
which
takes
into
account
things
like
exploitability,
does
give
us
more
context,
but
we
can
also
use
information
like
the
maturity
of
available
exploit
code
and,
most
importantly,
if
a
fix
is
available.
A
A
So,
as
I
said,
the
cvss
score,
taking
into
account
things
like
exploitability
and
impact,
it
does
give
us
more
context,
including
details
on
how
the
attack
can
happen,
and
it
helpfully
provides
this
in
a
machine,
readable
format
in
this
vector
string
which
we'll
come
to
in
a
minute,
but
understanding
the
output
of
our
tools
is
also
critical
here.
So
we
need
to
be
able
to
filter
based
on
criteria,
so
we
can
just
see
the
things
that
we
should
care
about,
and
most
tools
provide
ways
of
filtering
those
discovered
vulnerabilities.
A
A
A
A
A
So
if
we
trust
our
upstream
provider,
then
we're
going
to
start
from
a
base
image,
we're
going
to
leverage
that
upstream
provider
and
go
to
them
for
fixes
in
that
in
that
base
image,
and
then
we
might
have
some
common
configuration
for
our
specific
environment,
and
this
might
be
common
to
all
of
our
images.
You
know
this
could
be
a
configuration
for
our
specific
networks
for
specific
auth,
so
this
layer,
plus
our
original
base
layer,
could
make
up
a
common
base
image
that
all
of
our
teams
are
going
to
consume.
A
So,
as
we
said,
one
way
to
achieve
this
might
be
to
establish
our
own
base
image.
So
this
is
going
to
contain
a
a
a
a
a
an
upstream
base
image,
in
this
case
the
the
python
36
slim.
Then
any
generic
configuration
for
our
environment
and
then
we're
going
to
maintain
this
as
our
base
image.
So
all
of
our
other
teams
will
take
this
base
image
and
use
it
to
build,
build
their
images.
A
So
we
can
then
establish
a
baseline
for
that
base,
image
and
anyone
else
consuming.
It
only
needs
to
worry
about
things
that
they've
introduced
through
additional
packages
or
software
added
to
this.
So
in
this
example,
we've
got
178
issues
in
our
base,
image
and
right
at
the
start
of
the
process,
but
it's
also
important
that
we
watch
out
for
new
vulnerabilities.
So
generally,
images
that
have
just
been
released
normally
have
no
fixable
vulnerabilities,
but
things
do
get
broken
things
get
fixed
stuff
changes
so
rebuild
and
set
a
new
baseline,
pretty
often.
A
And
then
finally,
we
can
use
our
middleware
image
to
to
produce
our
application
image
by
adding
our
custom
code
into
the
middleware
layer.
And
again,
we
can
then
test
our
our
application
code
can
be
tested
by
the
app
team.
In
this
case,
there
are
there's
exactly
the
same
amount
of
vulnerabilities
that
were
in
the
middleware
layer,
so
we
haven't
introduced
any
new
vulnerability
through
our
application
layer.
A
So
the
final
consideration
here
is
that,
obviously
our
containers
almost
never
exist
in
isolation.
We
typically
running
them
in
orchestration
systems,
usually
kubernetes,
and
so
our
security
isn't
just
based
on
dealing
with
vulnerabilities
in
our
applications.
The
blast
radius
of
exploits
is
almost
always
a
combination
of
application
level,
vulnerability
combined
with
infrastructure
misconfiguration.
A
So
it's
really
important
that
we
consider
our
security
as
multi-layered
and
a
container
image
which
causes
a
single
pod
to
be
exploited
is
clearly
not
a
good
thing,
but
a
container
image
that
allows
your
entire
cluster
to
be
owned
is
a
much
more
significant
issue
and
security
principles
for
kubernetes
are
pretty
well
documented.
These
days,
you
definitely
don't
want
to
be
doing
things
like
in
this
example.
Host
paths,
privilege,
pods,
not
setting
resource
defaults,
can
all
allow
an
attacker
to
significantly
increase
their
foothold
in
your
cluster
to
potentially
devastating
effects.
A
So
conclusions
from
this
talk,
if
all
you
remember,
is
define
your
trust
boundaries
so
understand
who's
responsible
for,
for
which
elements
of
things
that
have
got
into
your
images,
decide
on
a
strategy,
choose
a
strategy
and
and
execute
that
strategy
in
order
to
reduce
your
overall
vulnerability
count
and
start
with
the
low
hanging
fruit.
High
severity
vulnerabilities
that
have
a
fix
available
are
clearly
a
no-brainer,
so
start
with
those
and
then
worry
about
the
other
ones
once
you've
dealt
with
the
low-hanging
fruit.