►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Protecting the Omniverse: How NVIDIA is Securing Containers- Adam Wallis, NVIDIA
With growing use of Kubernetes, NVIDIA was increasingly delivering containerized software for external customers and internal applications. As a result, the NVIDIA Product Security team needed a scalable security process that would support diverse requirements across business units without slowing down development. They integrated security checks into their existing CI/CD Pipelines to find and fix security issues early. Session attendees will learn how NVIDIA uses open source security tools to transition to continuous container security for their Kubernetes workloads, including how to: - Automating security checks across multiple CI/CD toolchains, registries, and Kubernetes platforms - Decentralized security policies that empower development teams with the responsibility of resolving security issues - Delivering centralized reporting for business unit accountability - Providing a centrally hosted solution to support thousands of containerized apps, and hundreds of thousands of containers
A
A
I'm
going
to
talk
a
little
bit
about
nvidia's
background
in
history
and
then
also
our
ngc
container
catalog.
I'm
also
going
to
talk
about
our
approach
to
security,
sort
of
some
of
the
practical
ways
that
we're
approaching
security
with
those
containers
and
then
I'm
going
to
wrap
up
at
the
end
with
bring
your
own
policy.
A
So
many
know
us
for
the
graphics
card.
You
know
we've
been
synonymous
with
graphics
cards,
pretty
much
since
the
invention
of
the
graphics
card.
It's
been
a
really
important
core
business
for
decades.
You
know
with
a
focus
on
gaming,
but
there's
been
a
shift
over
the
last
you
know
decade
or
so.
The
importance
of
the
gpu
has
really
changed
in
that
there
have
been
massive
pushes
into
machine
learning,
deep
learning
and
artificial
intelligence.
It's
pushing
these
incredibly
complicated
systems
directly
to
the
consumer.
A
With
these
pushes
it
really
changes
the
way
that
we
look
at
the
graphics
card
and
how
we
support
the
graphics
card.
This
is
a
full
software
stack
that
is
required
in
order
to
support
the
new
development
and
it
changes
the
way
that
we
have
to
look
at
securing
this
stack
not
only
internally
but
for
our
customers.
A
A
It
allows
storage
of
helm
charts
for
your
different
kubernetes
workloads.
It
allows
storage
of
artificial
intelligence
models.
This
really
helps
you
get
off
the
ground
quickly
if
you're
new
to
ai,
with
models
that
are
proven
to
work.
This
is
all
done
with
the
goal
of
simplifying
the
effort
required
to
enable
gpu
optimized
workloads.
A
A
So
often
security
gets
a
bad
rap.
What
happens
is
you
know?
It's
three
days
to
a
release
cycle
developer
comes
to
us
and
realizes
they
have
a
bunch
of
security
problems.
They
didn't
address
or
weren't.
Even
aware
of
you
know,
they
want
us
to
drop
everything
and
help
them
right
away,
and
hopefully
this
doesn't
actually
come
out
of
our
mouths
and
security.
But
you
know
the
snarky
response
would
be
something
sure
I'm
not
doing
anything
else
I'll.
Just
you
know
help
you
and
drop
everything
else.
A
You
know
you
can
think
of
rapid
prototyping
and
development
of
software.
I'm
probably
not
even
really
testing
it.
Things
start
off
kind
of
hot
and
heavy
start
moving
down
this
hill
that
we
have
here
and
you
realize
I
should
be
unit
testing.
So
you
kind
of
bolt
that,
on
after
the
development
start
improving
your
code
coverage
a
little
bit,
you
know
you
move
on
to
integration
testing
to
make
sure
that
the
different
components
really
sort
of
interact
together.
As
expected,
you
know
start
moving
down
the
hill.
A
This
is
where
you
might
really
hit
your
first
big
hiccup
in
that,
based
on
your
different
team's
review
styles,
you
might
have
you
know
a
lot
of
feedback
that
is
based
on
opinion.
Maybe
that's
coding
style.
Maybe
true.
Problems
are
found
or
corner
cases
that
weren't
anticipated
or
brought
up,
and
you
have
to
go
back
and
now
readdress
all
the
three
phases
we
just
talked
about.
Finally,
you
make
it
through
that
phase.
You
get
the
point
where
you're
ready
to
merge,
and
then
you
think
about
security.
A
You
know
and
as
a
security
team,
we
don't
want
to
be
the
police.
We
want
to
be
a
partner,
we
want
to
provide
guard
rails
and
not
gates.
So
how
can
we
help
these
teams?
You
know:
are
we
just
plugging
in
after
the
merge
point
looking
for
security
problems?
Well,
I
think
you
know
the
answer
is
probably
no
it's
a
little
late
in
the
game
to
do
that.
A
So
some
of
the
practical
ways
we
can
look
at
hooking
in
is
described
in
this
graphic
here.
So
we'll
start
off
in
the
first
bucket
talking
about
your
code,
you
know
open
source
screening
as
described
down.
There
might
be
something
like
static
analysis
checkers.
This
is
looking
for.
You
know
basic
security
checks,
making
sure
that
you're
writing
secure
code.
This
isn't
going
to
catch
everything,
but
it's
a
really
good
thing
to
automate.
A
This
is
also
reviewing
your
open
source
dependencies.
Looking
for
known,
open
source
problems
and
vulnerabilities
directly
in
the
source
code
repository,
this
can
be
done
potentially
even
directly
at
the
developer's
workspace,
potentially
in
their
code.
Editor
so
think
about
like
a
vs
code
plugin
that
offers
some
sort
of
a
static
analysis
or
security
code
plugin.
If
they
can
be
alerted
before
they
ever
even
commit
the
code,
they
get
the
opportunity
to
address
it
right
up
front.
A
A
In
addition
to
those
security
scans.
This
is
probably
your
best
time
to
evaluate
your
your
dependencies.
This
is
not
just
your
direct
dependencies,
so
in
python,
if
you
were,
you
asked
for
the
request
library
to
be
part
of
your
package
you're,
also,
indirectly
or
transitively,
requesting
url
lib3
and
a
bunch
of
other
different
python
packages.
A
These
dependency
files
might
not
be
published
in
your
final
artifact,
and
I
would
even
argue
they
probably
shouldn't
be
so
this
might
be
your
best
and
only
time
to
really
evaluate
those
dependencies
against.
What's
expected,
moving
on
to
packaging,
you
know
we're
at
kubecon,
so
the
obvious
thought
here
is
docker.
A
A
Do
the
direct
and
transitive
dependencies
that
you
requested
did
those
actually
show
up
in
the
container
what
other
dependencies
are
now
showing
up,
for
example,
from
the
operating
system
in
the
container
that
you've
chosen
to
deploy
from
moving
on
to
release?
This
is
hopefully
just
a
set
of
checks
and
balances.
You
know
that
the
previous
stages
that
we've
already
talked
about
meet
the
certifications
that
your
team
has
to
adhere
to.
This
should
tie
back
through
all
three
of
those
stages.
A
Moving
on
to
config,
ideally,
your
configuration
is
being
stored
and
treated
as
code.
For
example,
you
know,
let's
say
you're
using
terraform
to
keep
track
of
your
aws
security
groups.
This
should
be
tracked
and
committed
as
code
that
describes
your
expected
state.
So,
as
we
move
on
to
monitoring,
you
can
check
and
see.
If
there's
any
sort
of
drift
in
your
configuration,
you
want
to
make
sure
that
there
is
no
drift
and
if
there
is
drift
you
need
to
know
do
you
do
you
know
how
to
alert
the
appropriate
parties?
A
A
This
is
just
a
quick
view
of
an
internal
tool
that
we
develop
within
my
team
called
inspect
it's
sort
of
a
product
security
catalog.
The
idea
is
to
give
you
a
single
pane
view
of
your
wrist
posture
against
your
bill
of
materials.
This
is
not
tied
to
just
one
single
scanning
tool.
On
the
back
end,
it
combines
multiple
tools
and
gives
you
this
single
pane.
It
allows
you
to
see
different
cves
against
some
of
your
open
source
dependencies,
etc.
A
It
also
allows
you
to
sort
based
on
vulnerabilities
and
give
an
idea
of
what
linked
packages
are,
providing
those
so
that
we
can
work
with
the
teams
and
see
if
those
vulnerabilities
really
apply
to
them.
Oftentimes
you'll
find
that
a
vulnerability
doesn't
apply
because
of
the
way
it
was
deployed
or
because
they're
not
using
certain
functionality
that
only
that
cve
applies
to
moving
on
to
container
security,
as
many
of
you
here
probably
dealing
with
nvidia
is
also
dealing
with
containers
at
scale.
A
You
know
I'm
talking
20,
gigabytes
or
more
and,
as
you
know,
that's
pretty
large
for
a
docker
image,
so
there's
challenges
that
come
along
with
these
large
containers,
but,
however,
having
all
these
things,
containerized
really
do
bring
some
exciting
possibilities
and
new
risks
and
considerations
to
any
developer
on
how
this
code
is
being
deployed.
This
is
definitely
not
an
all-encompassing
list
of
those
challenges,
but
just
a
few
I
wanted
to
highlight
tags
are
not
immutable.
A
This
is
probably
not
used
to
anybody
on
this
call,
but
a
tag
is
just
a
pointer
to
a
digest
and
while
that
digest
is
immutable,
the
tag
is
intentionally
changeable.
It's
a
feature
and
one
that
we
all
rely
on
really
allows
easy
updates
to
the
consumers
of
your
container,
but
based
on
the
fact
that
tags
are
not
immutable.
Reliance
on
tagging
strategies
really
can
introduce
risk
if
you're
not
paying
attention.
A
This
can
really
range
from
a
developer.
Changing
the
contents
of
a
container
out
from
underneath
of
you.
This
might
be
unintentional.
It
could
be
careless
or
even
malicious,
but
other
times.
This
might
be
a
very
intentional
change
that
the
developer
you
know
wanted
to
have
in
the
container
they've
been
broadcasting
it,
but
perhaps
your
team
hasn't
been
paying
attention
to
their
release,
notes
or
you're,
just
not
paying
attention
at
all
and
you're
not
ready
for
those
changes
in
the
in
the
context
of
kubernetes.
A
Your
different
pull
policies
can
affect
whether
or
not
you're
pulling
the
latest
and
greatest
containers
down
at
all.
You
might
be
stuck
in
a
really
old
container
image,
because
your
kubernetes
deployment
has
reference
to
an
old
image
and
you've
been
telling
it
not
to
update
it,
often
docker
daemons
that
are
shipped
with
popular
distributions,
don't
enable
user
names
facing
by
default,
so
you
might
be
over
provisioning
access
to
the
containers
that
run
within
your
infrastructure.
A
A
We've
already
talked
a
little
bit
about
some
of
nvidia's
large
docker
images,
but
you
know
scanning
these
containers
bring
additional
challenges.
It
requires
more
cpu,
more
memory
and
more
disk
space
as
compared
to
just
a
regular
static
analysis
code,
checker,
not
to
mention
secrets
and
sensitive
data
right.
I
actually
believe
it's
easier
within
containers
to
accidentally
leave
a
secret
or
something
sensitive
in
a
container
than
it
is
to
properly
leave
it
out
during
a
build
there's
a
lot
of
ways
to
mess
it
up.
A
It's
a
huge
risk
to
an
organization
if
an
api
token
or
an
encryption
key
is
accidentally
left
within
a
container
and
it
gets
published
and
a
container
digest
can
persist
forever.
But
a
tag
of
latest,
as
you
know,
induces
a
false
sense
of
security.
That
container
might
be
the
latest,
but
it
might
be
three
years
old
and
not
be
receiving
regular
updates
at
all.
A
So
scanning
these
containers
really
grants
us
some
unique
considerations.
Packages
and
artifacts
can
be
inspected
in
the
environment
in
which
it
will
actually
run.
So,
for
example,
you
can
validate
that
a
compiled
binary
in
the
linkages
so,
for
example,
the
libraries
it
relies
on
exists
within
the
container
and
meets
the
expectations
the
developer
had
scanning
those
packages.
A
I've
talked
about
this
before,
though,
before
gives
you
the
ability
to
look
through
a
lens
to
reduce
false
positives,
so
if
ubuntu
has
in
fact
patched
a
version
of
live
ssl
and
you're
using
that
container,
you
don't
want
to
see
all
the
cves
that
don't
apply
to
you
in
your
list.
It's
extra
noise
that
you
don't
want
to
deal
with
containers
should
also
be
continually
checked
against
the
feeds
for
ever-growing
lists
of
vulnerabilities,
the
different
feeds
we
have
access
to
from
nvd
from
red,
hat
ubuntu,
etc.
They're
updated
practically
daily,
sometimes
hourly.
A
A
One
of
the
exciting
changes
to
the
version
two
of
docker
registry
was
this
multi-architectural
support
under
a
single
tag.
Name,
it's
really
nice
from
an
organizational
standpoint,
because
you
can
have
a
single
tag
that
covers
both
arm
64,
for
example,
and
amd
64.,
but
that
also
introduces
the
challenge
of
knowing
the
risk
associated
with
each
of
the
container
images
that
rely
within
that
tag.
You
might
have
a
vulnerability
that
only
affects
arm
but
doesn't
affect
intel
or
amd64.
A
So
some
of
the
ways
that
we're
approaching
this
in
nvidia
I'm
going
to
switch
gears
here
a
little
bit
and
talk
about
sort
of
the
evolution
of
like
how
we've
tried
to
approach
this.
Initially,
we
really
tried
to
use
this
sort
of
decentralized
approach
directly
in
developers
pipelines,
we
used
encore's
open
source
engine
which
allowed
this
game
stored
right
in
the
pipeline,
so
it's
nice,
each
team
could
use
their
own
resources
to
handle
it.
A
So
the
next
logical
step
was
purchasing
in-core
enterprise
and
deploying
our
in-core
scanners
within
the
sas
endpoint
within
kubernetes.
A
lot
of
benefits
here
really,
like
you,
have
a
centralized
api
where
you
can
retrieve
those
results.
It's
constantly
refreshing
the
feed
databases,
so
that
you
can,
if
you
ever
need
to
check
your
security
posture
against
certain
policies
and
checks
that
you
have
deployed,
you
can
get
those
on
a
practically
real-time
basis,
but
there
were
some
drawbacks
of
this
approach.
A
A
So
some
of
what
we're
working
towards
now
is
that
kind
of
decentralized,
plus
sas
architecture.
This
is
using
the
open
source
product
by
encore
called
sift,
which
generates
a
bill
of
materials
within
teams
pipeline.
So
the
heavy
lifting
of
generating
the
bill
of
materials
is
done
within
the
team's
pipelines
and
then
sif
promotes
that
bill
of
materials
back
to
the
sas
endpoint
for
a
quick
processing
to
do
a
vulnerability
check.
A
Another
advantage
is
that
teams
don't
have
to
provide
any
credentials
at
all,
since
sift
is
operated
within
their
infrastructure
and
that's
where
the
image
is
intro
or
inspected.
I
mean
it
gives
us
the
advantage
of
not
needing
those
credentials.
So
we
sort
of
retain
the
benefits
of
sas
api
for
the
catalog
vulnerabilities
on
demand
without
having
to
have
a
whole
bunch
of
infrastructure
within
sas
to
support
it.
We
allow
each
team
to
scale
as
they
need
to
based
on
their
number
of
images.
A
So
continuing
along
this
whole
idea
of
guard
rails
and
not
gates,
you
know
we
realized
pretty
quickly
that
policies
were
not
a
one-size-fits-all
sort
of
idea
for
different
teams,
so
we
do
want
to
provide
guidance
in
a
sane
set
of
default
policies
for
teams.
You
know
this
real
life.
This
is
along
the
lines
of
vulnerability,
so
certain
cve
scores.
You
know
we
might
want
to
mark
as
critical
or
high
and
not
allow
a
pipeline
to
pass.
This
is
looking
for
secrets
and
credentials,
perhaps
stored
within
the
different
layers
of
an
image.
A
So
what
we
really
want
to
do
here,
though,
at
nvidia,
is
encourage
each
team
to
sign
off
on
their
set
of
policies
that
are
appropriate
to
their
program,
commit
these
policies
as
code
directly
within
their
source
code
repositories
and
keep
track
of
it,
and
then
we
allow
them
to
submit
that
to
our
sas
endpoint
to
evaluate
their
their
risk
posture.
This
really
allows
us
sort
of
on
the
fly
policy
and
has
really
you
know,
worked
out
well
for
our
team
so
far,.
A
So
one
of
the
first
things
we
look
at
when
we
investigate
a
new
tool
is
sort
of
its
api
posture,
so
we're
api.
First,
we
use
a
variety
of
build
tools
that
developers
are
already
comfortable
with
and
by
build
tools.
I'm
talking
more
more
along
the
lines
of
pipelines,
so
jenkins
get
lab
ci
cd,
github
actions,
team
city,
so
we're
working
in
a
variety
of
environments,
so
apis
are
really
important.
A
Teams,
don't
have
the
time
or
the
luxury
to
learn
a
new
ui
tool
for
every
single
tool
we
roll
out
and
even
worse.
We
know
they're
not
going
to
be
checking
on
any
sort
of
a
reliable
schedule
to
see
if
they're
now
vulnerable
to
some
newly
released
cve
within
that
ui.
So
we
have
to
use
the
api
within
their
pipeline
to
query
and
return
their
results.
We
need
robust
apis
for
the
services
deployed
at
nvidia.
A
We
want
to
minimize
the
impact
on
development
teams.
Scan
times
have
to
be
reasonable
if
we're
adding
15
minutes
or
even
five
minutes
to
a
pipeline
developers,
are
likely
going
to
start
looking
for
ways
to
bypass
our
scans
or
reduce
the
frequency.
I
mean
this
just
makes
sense
as
a
developer
you're
on
a
tight
schedule.
You
have
a
lot
of
work
to
get
done,
so
we
don't
want
to
slow
them
down
kind
of
combined
with
that
is
alert
fatigue.
You
know.
A
We've
really
found
that
it's
counterproductive
to
over
alert
teams
receiving
a
slack
message
for
every
different
vulnerability
report
or
every
cve
is
really
just
going
to
force
them
to
invest
more
time
in
learning
how
to
mute,
mute
channels
in
slack
than
it
is
going
to
be
with
them
trying
to
fix
their
vulnerabilities
and
then
optimizing
the
signal-to-noise
ratio.
Every
tool
is
going
to
generate
some
threshold
of
false
positives
or
even
worse,
false
negatives.
A
A
You
know
the
last
thing
I
want
to
say:
is
you
just
really
want
to
provide
these
guard
rails
and
not
gates?
You
know
we
work
carefully
and
closely
with
our
product
security
incident
response
team
and
we
help
to
encourage
teams
and
guide
them
through
the
process.
If
a
vulnerability
is
found
within
their
products,
we
also
help
to
provide
a
healthy
set
of
guidelines
to
the
team
so
that
they
don't
all
need
to
be
experts
on
every
facet
of
every
tool
that
is
provided
to
them.
A
Just
wrapping
up
here
we're
going
to
have
a
q
a
session
now,
if
you
have
any
questions,
feel
free
to
ask
them
now.
If
you
have
more
interest
in
ngc,
you
can
find
a
link
here
below
and
if
you're
interested
in
sift
for
getting
an
idea
of
your
bill
of
materials,
please
check
out
the
github
link
below
thanks
for
your
time.
Everybody.