►
From YouTube: Container Patching: Making It Less Gross Than the Seattle Gum Wall - Greg Castle & Weston Panther
Description
Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Container Patching: Making It Less Gross Than the Seattle Gum Wall - Greg Castle & Weston Panther, Google
A goal like “Production containers are patched within FedRAMP timelines” is a seemingly impossible task for many organizations. What containers do we have? Who owns them, and how can we get them patched that fast? We’ll talk about our patching strategy of “Prevent, Detect, Fix, Monitor”, discuss the opensource tools available to help in each of those steps, and share lessons learned from our customers and our own patching program. Prevention narrows the funnel: standardized images, slimming images, separating build deps, allowlisting registries, and container promotion policies all help. On detection we’ll cover discovery, recent vuln detection advances, and opportunities to reduce noise. Fixing is about automating ownership discovery, fix sequencing, and release process. Monitoring glues it all together: prioritize fixes and investigate gaps to meet your SLO.
A
I'm
Greg:
this
is
Wes
we're
from
Google
we're
going
to
talk
about
container
patching
today,
I
actually
really
like
the
gum
wall,
but
I
really
wanted
a
Seattle
theme
talk.
So
that's
why
there's
some
gumball
in
this?
It
is
gross,
though
so,
if
we
think
about
container
patching
at
like
a
really
very
simplistic
view,
you're
trying
to
hit
some
sort
of
Target,
usually
so
I
put
fedramp
Targets
on
here,
you
may
not
be
subject
to
those
ones.
You
might
have
PCI,
you
might
have
no
regulation.
A
You
might
just
have
sort
of
some
vague
notion
of
how
quickly
you
want
to
patch
things.
But
the
way
sort
of
compliance
thinks
about
patching
deadlines.
Is
a
scanner
detects
a
thing
you
go
patch
it,
and
then
your
production
needs
to
be
done
within
this.
Whatever
this
time
window
is
so,
let's
just
start
with
a
little
trip
down.
Empathy,
Lane
so
imagine
you're
the
security
person
at
an
organization,
who's
in
charge
of
patching
these
production
clusters,
or
maybe
more
likely
you're,
the
developer
who's
been
told
you're.
A
Now
the
security
person
who's
patching
these
clusters,
and
so
every
two
weeks
you
like
go
open
up.
Your
dashboard
and
you
look
at
like
the
stuff-
that's
in
there
and
you've
sort
of
already
lost
two
weeks
on
your
timer
here,
because
there's
been
two
weeks
of
vulnerabilities
accumulating
there,
while
you
weren't
looking
at
it
and
so
there's
there's
a
couple
of
things
in
there.
There's
a
web
front-end
container,
that's
missing
a
couple
of
critical
patches,
and
so
you
go
hey
web
team.
A
Hey,
what's
up
web
team
I'd
like
to
get
this
patched
to
my
dashboard
green?
Please,
and
they
come
back
and
like
oh?
Actually,
it's
not
really
our
code.
This
looks
like
it
might
be.
The
Django
based
container
so
you're,
like
okay,
cool
I,
know,
who
maintains
Django
without
orgo.
Go
talk
to
them,
so
hey
Django,
container
team.
It
looks
like
you
have
these
critical
patches
missing.
A
Can
you
patch
a
container
please
and
then
go
away
and
look
at
it
and
they're
like
oh,
but
this
is
Pearl
like
we
don't
even
use
Pearl
like
do
we
really
need
to
patch
this
thing
and
so
you're
like
yeah,
look,
look
I
just
want
my
dashboard
to
be
green,
so,
yes
or
but
even
better,
just
get
rid
of
it.
So
we
never
have
to
have
this
conversation
again
and
by
this
point
you're
out
of
your
SLO.
A
So
things
are
not
happy
but
like
eventually
they
come
back
and
like
well
I.
We
couldn't
figure
out
how
to
remove
it.
So
we
patched
it
I!
Guess
we'll
have
this
conversation
again
and
all
right,
great!
Okay!
So
now
that's
patched,
so
you
go
back
to
the
web
team.
Like
hey
web
team.
Please
go
rebuild
with
this
Django
container
and
they're
like
okay
yeah.
We
did
it
and
yeah
I
could.
But
then
you
go
look
at
the
dashboard
and
it's
still
the
old
version.
I'm.
B
A
And
you
go
back
to
the
web
team
and
they're
like
oh
yeah,
we
forgot
to
update
the
kubernetes
Manifest,
so
we
patched
the
docker
container,
but
we
didn't
update
the
Manifest
the
points
of
the
container.
Now
it's
really
done
and
you
go
to
look
at
it.
You're
like
oh,
actually,
no,
it's
still
not
done
and
also
now
there's
more
High
vulnerabilities.
So
like
we
had
these
criticals,
we
were
trying
to
patch,
but
like
there's
more
vulnerabilities
accumulating
and
and
they're
like
oh
yeah,
it
had
the
soaking
QA.
A
So
we
committed
the
Manifest
change
to
the
QA
Branch,
but
we
didn't
put
it
in
the
prod
branch.
And
so
now
it's
done
and
you
go
look
at
it.
You're,
like
okay,
it's
fixed
and
then
sort
of
a
little
voice
in
the
back
of
your
head
is
like
well
like
I
think
we
have
other
things
that
run
Django,
but
like
that's,
not
your
problem,
so
you're
going
to
just
move
on
with
your
day.
So
why
is
this
gross
like
it's
not
great,
so
there's
humans
at
every
step.
A
There's
a
human
has
to
notice
a
thing.
Human
has
to
go.
Tell
another
human
to
do
a
thing
we
don't
know
which
layer
we're
patching.
We
don't
have
any
inventory
of
containers
we're
patching
code
that
we're
not
using
and
the
vulnerabilities
are
just
accumulating
faster
than
we
can
like
pay
them
down.
So
it
just
means
this
whole
thing
is
like
slow.
It's
incomplete!
It's
not
going
to
scale
so
who
thanks
the
majority
of
the
industry,
is
doing
better
than
this
today.
A
We
use
other
options
that
you
could
use
and
talk
about
these
sort
of
four
things
prevent
detect,
fix,
monitor,
which
is
kind
of
our
way
of
thinking
about
this
problem
in
terms
of
sort
of
what
you
can
take
away
from
this
talk,
we're
not
really
talking
about
the
sort
of
vendor
containers
or
managed
service
provider
containers
like
gke,
where,
like
we've,
been
we're
just
going
to
patch
our
containers
for
you.
You
don't
need
to
worry
about
that.
A
So
take
this
as
a
case
study
for
like
where
you
own
containers
or
you
own
kubernetes,
manifest
you
need
to
update
to
point
at
rebuilt
containers.
That's
where
this
sort
of
applies.
What
do
we
know?
We've
been
patching
a
few
containers.
I
guess
a
few
thousand
across
gke
in
anthos
and
some
other
products,
but
we
have
a
a
reasonable
number
of
advantages.
So
our
environment
helps
quite
a
bit.
We
mostly
use
golang.
A
We
use
a
small
number
of
container
repositories
and
in
in
a
certain
number
of
Registries
mandatory
based
images.
We
have
a
fair
bit
of
control
over
the
release
process.
So
not
everyone
is
in
this
situation,
so
we're
going
to
try
and
give
examples
of
what
we
did
but
then
also
other
ways
to
solve
the
same
problems.
A
A
So
the
sort
of
standard
strategy
that
you
have
probably
heard
from
multiple
people
over
the
years
is
do
things
like
this
standardize
on
your
base.
Containers
make
those
containers
as
small
as
possible
the
less
code.
You
have
the
less
patching
you
do
get
rid
of
unused
code
wherever
you
can.
So
if
you
can
separate
out,
build
and
run
time
and
have
those
separate,
then
that
that
can
help
a
lot
and
there's
really
sort
of
two
approaches
to
getting
the
small
containers.
A
So
what
we
did
was
we.
We
basically
standardized
on
display
list,
as
I
said,
we're
mostly
golang,
so
we
have
a
distrostatic
container.
That's
just
enough
to
run
a
go
binary.
There's!
No
shell,
no
package
manager,
it's
like
really
minimal
little
container,
and
so
we
standardize
on
that
one
for
pretty
much
everything
with
a
few
exceptions
and
then
another
thing
that
we
did,
that
I
think
is
also
just
a
good
practice.
A
If
you
can
do
it
is
to
get
everything
into
a
single
registry
or
a
small
number
of
repositories
in
in
the
registry,
and
that
just
helps,
like
you
know,
everything's
sort
of
in
one
place,
and
it
also
that
has
a
a
good
availability
property
too,
like
if
you're,
say
you're
running
in
gke.
It's
your
your
GCR
Google
container
registry
or
your
artifact
registry.
A
So
how
do
we
do
that?
The
way
we
did
this
is
we
wrote
some
pre-submit
checks,
and
so,
whenever
a
developer
turns
up
with
some
yaml,
we
run
this
pre-subject
submit
check
on
it.
That
looks
at
what
image
it's
pointing
at
so
we're
checking
that
it's
a
GCR
or
artifact
registry
image,
and
then
we're
also
checking
that
it
was
built
on
this
distroller
space
and
you
can
actually
go
introspect
the
container
and
like
pull
out
the
file.
A
That
tells
you
what
OS
it
is
and
it'll
tell
you
that
it's
a
distross
thing
there's
other
ways
to
do
that,
though
you
don't
necessarily
have
to
do
that.
That
way,
you
could
do
it
at
build
time.
You
could
look
at
the
docker
file,
you
could
add
checks
in
between
sort
of
build
and
putting
the
container
into
the
repository.
You
could
do
the
kubernetes
piece
at
the
packaging
or
deployment
layer,
but
really
probably
one
of
the
most
common
ways
to
do.
A
This
is
with
admission
control,
and
so
here
we
have
an
example
of
doing
those
two
things
that
I
said
so
making
sure
that
all
containers
are
coming
from
a
particular
registry
and
that
they
built
off
a
particular
base
image.
So
for
the
base
image,
what
we
can
do
here
is
use
cosine
to
make
an
attestation
a
cryptographic
attestation
that
says
hey
this.
This
container
is
a
distro-less
container,
and
then
that
goes
into
your
repo
and
then
you
have
an
admission
controller
that
says:
do
the
containers
coming
in
have
have
this
disturbless
attestation?
A
B
A
What
we've
got
here
is
a
we've,
got
those
two
admission
controllers
running
and
they're,
pointing
at
this
enforcing
namespace
and
we're
going
to
try
and
run
a
container.
That's
not
a
GCR
container,
so
this
is
BusyBox,
that's
getting
pulled
from
Docker
Hub,
and
so
that
should
get
denied
by
the
gatekeeper
admission
controller.
That's
going
to
block
that.
A
Let's
draw
on
that
and
so
yeah
you
can
see
here
the
gatekeeper,
validation,
validating
Mission,
Control,
there's
a
policy
called
repo
is
GCR
and
it
didn't
let
it
in
because
the
index.docker
io
is
not
one
of
the
GCR
repos
that
we
that
we
specified
okay.
So
we'll
do
the
same
thing
again
and
this
time
we're
going
to
put
it
in
GCR.
So
it's
the
same
busy
box
container,
but
now
it's
in
GCL,
so
it
should
pass
gatekeeper,
but
it
still
fails
the
six
door
check.
A
So
we're
running
the
six
door
policy
controller
here,
and
so
that's
looking
for
a
distro-less
attestation
and
because
this
container
doesn't
have
that
attestation,
it
doesn't
get
in
there.
So
one
more
time,
this
time
we're
going
to
run
a
container
that
is
destroyless
and
a
tested
is
disturbless
using
cosine
and
that
one
will
go
through.
A
Okay,
so
just
quickly
recapping.
Here
it's
really
good.
If
you
can
look
at
what
enforcement
points,
you
have
we
like
the
pre-submit
enforcement
point,
but
you
might
have
others
if
you
can
standardize
on
those
base
containers.
It
really
helps
a
lot
and
if
you
can
get
down
to
a
smaller
number
of
Registries
for
inventory,
there's
good
security
and
availability
reasons
to
do
that.
If
you
don't
do
any
of
these
things
and
they're
all
kind
of
optional,
then
you
just
have
to
be
really
good
at
the
next
things.
C
Thank
you
Greg
all
right,
let's
get
into
it,
so
the
the
next
part
of
our
strategy
is
is
detection.
So
really
your
your
vulnerability,
Management
program
is
only
as
good
as
your
detection
methods.
If
you're,
if
you're,
not
detecting
vulnerabilities,
it's
unlikely
you're
going
to
patch
them.
So
what
are
the
problems
with
addiction?
First
is
which
containers
do
we
need
to
scan?
So
you
might
have
a
lot
of
containers,
but
not
all
of
them
are
used,
maybe
somewhere
in
Dev
or
test
or
just
old
and
outdated.
Next
is
which
scanner
should
you
use?
C
So
this
this
decision
might
have
already
been
made
by
your
organization.
You
might
already
have
a
scanner,
but
you
should
be
aware
of
sort
of
the
differences
in
the
scanners.
They
have
different
feature
sets
and
different
coverage,
mostly
owing
to
different
vulnerability,
sources
and
techniques,
and
they
may
handle
duplicates
differently
or
or
filtering
differently.
So
just
be
aware
of
these
things
and
then
last
you
need
to
know
what
layer
has
a
vulnerability.
Is
it?
Is
it
actually
in
this
container
image
or
is
it
in
some
base?
C
C
So
this
is
kind
of
like
when
a
developer
is
making
a
change.
It's
it's
the
easiest
time
to
actually
patch,
and
this
makes
sure
anything
that
is
staged
going
to
be
in
production
will
be
free
of
vulnerabilities.
Then,
second,
we
have
sort
of
this
universe
of
all
of
our
container
images
and
we
scan
through
Source.
We
can
determine
what
all
of
those
are
used
in
production
or
will
be
used
in
production
and
those
are
scanned
continuously.
C
So
as
an
alternative
here,
you,
you
have
all
of
your
images
in
some
registry,
but
you
probably
don't
want
to
scan
all
of
them.
So
what
a
lot
of
folks
do
is
they
have
something
running
on
their
hosts?
A
Daemon
set
a
an
agent
of
some
sort,
maybe
some
scanning
tool-
and
this
collects
your
inventory-
your
production,
inventory
and
scans
those
container
images.
So
this
is
like
we
said,
a
good
start
that
the
disadvantage
here
is
your
you're.
Only
catching
vulnerabilities
that
are
have
made
it
to
production
are
already
running.
C
So
next
is
the
question
of
which
scanner
to
use.
So
a
lot
of
container
image
scanners
today
use
the
national
vulnerability
database
nvd
feed,
as
well
as
a
bunch
of
os
vendor
feeds.
So
this
gives
you
the
OS
system
packages,
but
there
are
sort
of
some
new
developments.
First,
a
lot
of
scanners
are
beginning
to
support
scanning
the
actual
application
code.
So
your
your
go
or
your
python
within
the
container
image.
C
Next,
we've
heard
a
lot
about
s-bombs
the
last
couple
days
and
as
they
become
more
ubiquitous
scanners
are
beginning
to
consume
them
and
then
kind
of
optionally
use
these.
These
Vex
documents
to
help
augment
that,
and
then
I
mentioned
nvd
and
these
OS
vendor
feeds.
But
there
are
a
number
of
other
sources
where
vulnerabilities
come
from
open
source
databases,
GitHub
advisories
these
kinds
of
things,
some
more
features
here,
so
some
scanners
have
this.
This
idea
of
Base
image
detection,
either
from
metadata
in
the
image
or
pointed
at
the
docker
file.
C
C
They
try
to
determine
what
what
methods
are
you
actually
using
so
rather
than
just
returning
a
huge
list
of
cves
we're
returning
only
the
ones
that
are
in
in
code
that
that
you're
using
it
can
also
use
some
information
in
the
binary
to
do
that
and
we'll
get
into
that
a
little
later,
and
then
there
are
also
additional
scan
types
outside
of
vulnerability
scanning.
There
are
things
like
CIS
Etc
mentioned
here,
so
we
did
a
bit
of
an
experiment.
C
We
built
a
go
binary.
We
have
a
vulnerable
module.
I've
listed
it
up
here
at
a
vulnerable
version.
It's
built
with
an
old
go
Lang
version
118,
and
it's
on
an
old
W
base
from
2021,
so
I
had
a
bunch
of
vulnerabilities
and
we
scanned
that
with
two
commercial
scanners
and
two
open
source
scanners,
and
these
are
sort
of
the
results
both
fixed
and
unfixed.
C
So
then
the
question
becomes
which
one
is
correct:
well,
there's
there's
not
really
a
right
or
a
wrong
answer.
Almost
all
of
them
detected
OS
package
vulnerabilities.
Some
detected
go
module
vulnerabilities,
some
did
Google,
go
tool,
chain
vulnerabilities,
but
even
within
those
results
they
kind
of
differ
on
what
they
choose
to
hide.
What
they
choose
a
surface
and
the
the
key
Point
here
is
that
we
really
want
to
increase
coverage,
increase
the
number
of
things
we
scan,
but
we
want
to
decrease
false
positives,
so
there's
kind
of
this
tension
between
the
two.
C
So
when
it
comes
to
the
question
of
which
scanner
our
solution
is
more
than
one,
and
if
you're
publishing
images
for
consumption
for
others
to
consume,
you
have
to
sort
of
assume
they're
they're
going
to
be
scanned
by
every
every
scanner
in
existence,
and
we
found
that
to
be
true.
So
we
we
run
multiple
standards
and
compare
results,
and
this
gives
us
a
couple
advantages.
C
So
first
it
shows
us
if
there
are
any
false
positives
or
coverage
gaps
or
new
features
that
are
that
are
sort
of
coming
online
and
second,
it
gives
us
an
idea
of
what
our
customers
actually
see.
So
if
they
come
to
us
with
a
a
scan
result
from
some
random
scanner,
we've
had
some
advance
notice
and
we,
we
kind
of
know
how
to
interpret
those
results.
So
then
we
we
take
all
these
results,
compare
them
and
that
feedbacks
into
our
our
own
internal
scanner
and
hopefully
improves
things
over
time.
C
So
the
next
big
problem
in
detection
is
noise
and
we've
broadly
broken
this
up
into
two
buckets,
so
those
things
that
are
under
the
control
of
the
user
and
those
that
are
under
the
control
of
the
scanner.
So
in
the
user
control
bucket
oftentimes
we're
scanning
code,
that's
just
not
even
used
like
the
the
Pearl
vulnerability
example
at
the
top,
and
that's
just
things
we
should
rip
out
next.
There
A
lot
of
times.
C
So
next
going
into
the
application
and
golang
in
particular,
so
we
use
primarily
golang
throughout
gke
and,
as
I
mentioned,
a
lot
of
scanners
are
beginning
to
turn
up
these
module
or
tool
chain
vulnerabilities.
So
a
problem
we
often
see
is
that
they
return
all
of
the
vulnerabilities
in
that
module.
All
of
the
vulnerabilities
in
that
that
the
minor
version
of
that
go
tool
chain.
So
there's
a
tool
released
by
The
Go
security
team
go
volc.
C
This
is
kind
of
a
new
experimental
tool
in
library
and
it
tries
to
help
with
this
problem
and
we'll
there's
a
link
here
and
we'll
go
into
a
demo
of
that
all
right.
So
here
I've
got
a
small
sort
of
terrible
go
program
with
one
single
module
imported
and
you
can
see
in
particular,
I'm
calling
this
this
new
server
con
method
so
go
ahead
and
build
that
and
I've
I
went
ahead
and
I
packaged
this
up.
C
So
here
it
is,
you
can
see
there
are
a
whole
bunch
of
go
results,
and
this
is
great.
This
is
like
what
we
want.
We
want
scanners
to
start
doing
this,
but
how
many
of
these
are
actually
reachable.
That's
the
question,
so
we
run
go
voltcheck
on
that
same
binary
and
what
it
does
is.
It
actually
looks
at
the
symbol
table
in
the
binary
to
determine
what
methods
are
used,
and
so
we've
kind
of
gone
from
that
big
list
down
to
just
a
a
couple,
two
instances
and
you'll
notice
it.
C
C
So
what
is
the
problem?
Well,
really,
it's
just
a
complex
process,
and
this
is
maybe
an
example
flow
of
what
what
it
might
look
like.
So
you
have
a
scanner.
It
detects
a
vulnerability
in
some
image
and
then
you
have
to
determine.
Is
it
actually
in
this
image
or
is
it
in
some
some
parent
some
base
image?
Then
you
have
to
find
the
owner.
Hopefully
you
know
the
owner,
you
create
a
bug
or
a
PR
to
them,
and
then
once
this
is
all
complete.
C
So
our
first
solution,
originally,
we
sort
of
tried
to
determine
the
parent-child
relationships
and
images
in
the
base
images,
but
then,
more
recently,
we've
taken
a
different
attack.
So,
as
we
mentioned
earlier,
we
have
a
sort
of
a
finite
limited
set
number
of
Base
images,
so
those
are
represented
here
on
the
left
and
when
they're
built
they
get
new
tags.
So
on
the
left
is
the
oldest
tag
and
the
the
newest
is
on
the
right.
So
what
we
do
is
we
have
Automation
and
it
scans.
C
The
latest
of
these
continuously
and
then
say
we
find
a
vulnerability
in
Debian
base
the
middle
one
here,
so
we
go
ahead
and
we
automatically
create
a
new
one,
make
a
new
tag
and
then
we
scan
the
latest
again.
We
just
do
this
for
eternity,
and
this
ensures
that
when
you
need
to
patch
your
container,
you
always
have
a
patched
base
image
available.
So
there's
really
no
question
of.
Do
I
need
to
go,
find
the
base
image
patch
the
base
image
and
do
this
whole
dance.
C
Next
is
the
question
of
ownership.
So
again,
we've
got
to
pre-submit.
What
happens
here
is
whenever
a
developer
introduces
a
new
image.
This
check
will
look
and
say:
is
there
an
owner
defined
for
this
image?
So
this
this
ensures
that
we,
we
don't
have
any
question
of
who
owns
the
thing?
Who
should
be
on
the
hook
for
doing
the
actual
patching
so
taking
that
earlier
example
and
pulling
out
those
those
two
pieces
it?
This
might
be
a
simplified
process,
so
your
scanner
to
detects
a
CV
in
an
image.
C
We
already
know
that
the
base
image
is
patched.
So
if
there's
a
new
one
available,
we
just
update
that.
We
know
that
we
have
an
owner
to
send
the
bug
to
so
we
go
ahead
and
do
that
and
then
Etc
we
wait
for
a
build
and
and
so
on
until
we're
done.
Something
else.
You'll
note
here
is
that
we
also
aggregate
bugs
that's
sort
of
what
we
think
is
the
appropriate
level
of
abstraction.
So
say
you
find
15
cves
in
your
your
Cube
proxy
image.
C
It
doesn't
make
sense
to
make
15
bugs
to
various
teams
or
15
times
the
number
of
deployments.
Hundreds
of
bugs.
We
found
that
when
you
have
a
large
number
of
bugs
or
the
larger
number
of
bugs
you,
you
have
the
less
likely
they
are
to
get
fixed,
or
rather
the
more
likely
they
are
to
get
ignored.
So
we
create
one
bug
for
the
image
once
that
image
is
patched
that
fixes
all
of
them
and
that's
another
way
we
sort
of
simplify
things.
C
So
the
title
back,
if
it's
useful
to
you,
track
the
container
parent-child
relationships.
So
you
know
what
base
image
relates
to
what
what
image,
as
a
complement
or
alternative,
just
automate
patching
based
images
consistently.
So
you
know
you
know
you
always
have
one,
it's
very
useful
to
have
comprehensive,
inventory
and
track
your
ownership.
So
we
know
who
owns
the
image
and
then,
where
possible,
sort
of
use
the
existing
systems.
You
have
to
track
these
things,
your
your
issue,
tracker
or
jira,
whatever
you
use.
C
So
the
last
part
of
our
strategy
is
monitoring
our
visibility
and
this
sort
of
cross-cuts
all
the
others,
it's
kind
of
integral
to
to
all
of
the
rest
and
it's
complementary
to
each
of
them.
So
as
we
know
what
gets
measured
gets
managed.
So
the
problems
here
are
first,
there's
kind
of
this
question
of
what
is
the
holistic
view
of
the
fleet
given
a
cve?
Is
it
patched
and
then
further
has
that
patch
been
deployed?
C
Next?
Is
this
sort
of
composition
question?
So
we
know
we
have
a
cve,
but
what
containers
actually
have
that
cve
and
where
is
that
container
used
and
so
on
up
the
stack,
the
next,
inevitably
bugs
or
PRS
won't
get
merged
or
fixed,
and
what
do
we
do
about
that?
So
who
is
watching?
How
do
we
measure
those
things
and
then
how
do
we
escalate
so
having
those
processes
in
place
and
then
last
we
want
to
measure
all
of
this.
We
want
to
know
how
we're
doing
against
our
timelines,
and
then
we
also
want
to
know.
C
So
our
solution,
kind
of
a
simplified
view
of
our
solution
is
something
like
this.
It
may
burn
some
ideas
in
your
mind,
so
our
standard
detects
of
cve,
if
it's
new
finding
we
we
create
a
bug
kind
of
like
described
before.
If
it's
an
existing
bug,
we
we
check
it
against
our
stated
SLO
and
we,
if,
if
it's
nearing
the
SLO
end
date,
we'll
say
hey
50
of
your
SLO
is
gone.
You
really
need
to
take
a
look
at
this
and
then
once
it
gets
past
SLO
we
have
escalation
processes.
C
C
So
then
we
we
create,
but
we
create
these
bugs.
We've
got
these
processes
in
place
to
add
comments
and
and
nag,
and
then
we've
also
got
processes
for
sort
of
dashboarding
and
email
reporting,
and
that
kind
of
thing
another
optimization
here
might
be
if
the
detection
is
is
no
longer
around.
You
could
close.
The
bug
have
some
automation
to
do
something
like
that.
C
The
next
problem
is
composition,
so
NGK
we've
built
a
system
all
in
spanner
that
has
these
relations,
so
we
have
a
GK
version.
We
know
it
has
one
or
more
applications,
one
or
more
containers
that
have
one
or
more
cves
packages
with
CVS
in
them.
So
then,
let's
say
your
security
engineer
comes
along
and
says:
hey
are
we
affected
by
this
cve?
C
C
C
This
is
just
all
example
data,
but
we
track
images
on
a
number
of
axes
so
by
the
number
of
cves
by
the
the
team
by
that
that
sort
of
thing,
so
we
know
which
teams
are
getting
the
most,
which
images
have
the
most
CVS
Etc.
So
in
this
example,
we
have
fake
image.
101
and
103
have
a
lot
of
CVS.
This
is
somewhere.
You
might
look
into
the
prevention
methods.
C
We
talked
to
the
beginning,
moving
something
to
disturbless
or
slimming
it
down,
and
the
top
right
I
think
something
that's
often
overlooked
in
a
lot
of
scanners.
Is
it's
it's
important
to
track
over
time?
We
want
to
know
how
we're
doing
right
now,
but
we
also
want
to
know
how
we're
trending.
So
this
gives
us
an
idea
of
sort
of
that
that
Trend
and
then
it
also
tells
us
if
we're
meeting
our
goals.
So
we
have
these
stated.
C
Slos
are
we
are
we
meeting
them
and
you
can
kind
of
find
Trends
in
the
data
perhaps
like
in
this
example?
Maybe
maybe
our
low
vulnerabilities
are
not
being
prioritized
as
well
as
they
could
at
the
bottom
here.
This
is
sort
of
an
example
of
how
we
track
the
life
cycle
of
a
single
vulnerability.
So
we
look
at
from
the
time
it's
detected
to
the
time
it's
rolled
out
everywhere.
What
are
as
granularly
as
possible?
C
We
try
to
say
what
actions
took
place
and
how
much
time
was
spent
between
them
and
those
might
be
good
candidates
for
further
automation
or,
for
maybe
you
know,
improving
QA
processes
or
release
processes.
Simplification,
some
Alternatives
here
for
inventory
you're.
If
you're
on
a
cloud
provider,
they
might
provide
some
some
systems
for
that.
But
a
lot
of
scanners
provide
this
for
the
kind
of
composition.
Piece
Lyft
has
a
really
good
article
about
how
they
use
the
open
source
cartography
graph
database
to
do
something
similar,
something
similar
to
what
we
did.
C
So
to
tying
this
all
back,
it's
very
important
to
track
your
slos
over
time,
not
just
kind
of
the
current
status
tracking
those
those
parts
of
your
release
process
can
help.
You
identify
these
bottlenecks
to
see
where
you
can
sort
of
prioritize
investment
and
then,
where
possible,
use
your
existing
systems
to
do
this
escalation.
Dashboarding,
monitoring
piece
all
right
and
with
that
I
will
give
it
back
to
Greg.
A
Cool
all
right,
so
just
a
reminder
what
we
talked
about.
So
we
talked
about
standardizing
on
Registries.
Getting
those
minimal
containers
getting
as
far
left
as
possible.
I
know
everyone
would
love
to
say,
shift,
left
and
loves
hearing
it,
but
we
found
like
you
know
that
that's
just
the
cheapest
easy,
cheapest
way
you
can
do
it.
Keep
those
vulnerabilities
out
of
production
scanners,
get
you
inventory
and
visibility
and
the
record
of
ownership
of
containers
is,
is
really
critical,
so
you
know
who
to
who
to
nag
to
patch
them.
A
If
you
can
automatic
patching
is
definitely
better.
I
think
you'll
probably
want
bugs
and
ticketing
systems
anyway,
but
to
help
you
build
dashboards,
but
if
you
can
Auto
patch
and
send
PRS
instead
of
just
sending
bugs,
then
that's
definitely
going
to
be
better
and
yeah
ticketing
systems
or
escalation,
and
just
really
like,
if
you
can
do
rather
than
tell
that,
will
make
a
big
difference.
A
B
A
Yeah,
so
the
question
is
like
scanning
in
the
in
the
repo
is
nice,
but
like
isn't
it
isn't
a
good
idea
to
like
sort
of
rescan
in
case
like
I,
don't
know
you
brought
in
a
dependency
that
had
a
vulnerability
in
it
or
or
some
other
malicious
change
yeah
like
definitely
so.
We
have
like
multiple
points.
What
I
was
scanning
I
think
maybe
there's
not
too
many
places
to
do
it.
I
I,
don't
know
a
bit
like
it.
I
think
you
can
sort
of
the
earlier.
A
You
do
it
and
that's
sort
of
the
least
cost
on
the
developer
to
fix
it.
If
they're
already
built
their
release,
they've
already
gone
through
their
QA
and
they're
like
about
to
put
it
on
a
cluster
and
you're
like
wait,
that's
like
the
most
annoying
place
to
do
it
so,
but
yeah
I
think,
like
the
visibility
in
multiple
places
along
that
pipeline
is
useful.
Yeah.
A
Question
is:
is
where's
the
ownership
information
right
now
effectively.
What
happens
is
it's
like?
We
don't
have
like
a
huge
number
of
images,
so
it's
just
a
file,
that's
checked
in
and
there's
a
preset
that
basically
looks
at
your
manifest
and
if
the
image
your
image
is
not
in
that
file
with
an
owner,
then
we
won't
let
you
put
the
Manifest
in
so
in
the
future.
This
could
maybe
be
something
more
fancy,
but
that's
definitely
doing
the
job
at
the
moment,
yeah.
A
C
Don't
know
I,
don't
think
so,
I,
don't
think
so
we
have
a
number
of
additionalist
variants
and
some
of
them,
certainly
don't
they
have
only
you
know
Temp
and
some
Etsy
files
I
mean
we
have
sort
of
different
variants
for
different
use
cases.
Some
may
include
glossy
to
my
knowledge.
None
of
them
have
have
open
SSL,
but
I
know
that
the
open
source
District
list
provided
by
Google
some
of
those
do
have
openness
associated.
A
Example
yeah,
so
the
comment
was
that,
even
if
you
don't
have
a
a
CLI
in
your
container,
then
there
are
some
maybe
sneaky
ways
to
get
something:
that's
kind
of
close
to
a
CLI.
If
you
have
things
like
openssl,
there's
a
whole
website,
I
forget
what
the
I
forget.
What
the
name
is,
but
there's
entire
website
dedicated
to
this
premise
of
just
like
I,
have
a
small
number
of
tools
that
aren't
shells.
A
How
can
I
turn
them
into
shells
and
there's
a
huge
and
fairly
surprising
list
of
of
things
that
you
can
do
that
with
yeah?
But
you
know,
like
I
I,
think
that's
like
everything
that
not
having
the
shell
is
making
the
attacker's
job
harder.
It's
also
making
my
job
as
a
Defender
easier,
because
I
have
less
things
to
patch.
So
it's
just
win-win
anyway,
even
if
there
is
a
like
there's
still
a
way.
D
A
Yeah
yeah,
so
what
do
we
do
about
dependencies?
So
I
think
like
for
us
in
particular,
and
the
example
was
like
you
know
if
there's
a
critical
somewhere
sort
of
like
deep
in
the
chain,
but
there's
maybe
a
low
that
will
like
fix
that
critical
and
sort
of
hidden,
so
I
think
it
probably
comes
down
to
like
what
your
scanners
can
tell
you
for
like
in
terms
of
like
the
some.
A
A
So,
like
you
know,
the
Google
has
these
sort
of
like
release
processes
where,
like
best
practices,
are
sort
of
like
rolling
out
over
a
week,
rolling
out
Zone
by
Zone
and
like
lots
of
like
availability
safety
built
into
things,
so
that
if
we,
if
there's
something
wrong
in
that
release,
then
we
affect
a
small
number
of
people
to
start
with,
and
we
hopefully
notice
and
then
we
can
roll
it
back
quickly.
And
so
you
went
into
really
urgent
fix.
A
Then
you're
talking
about
a
different
different
playing
field
and
so
like
the
security
gets
dialed
up
and
the
availability
gets
dialed
down
and
sort
of
like
how
how
far
you
dial
each
of
those
dials
is
sort
of
depends
on
how
bad
it
is.
But
if
it's
really
bad,
then
you
go
really
fast
and
you
hope
you
don't
break
stuff.
Yeah.
C
C
Some
can
just
roll
out
as
usual,
so
we
sort
of
map
it
to
those
those
outcomes
and
I'd
say
what
we're
talking
about
here
is
sort
of
the
the
happy
path
and
we've
we
sort
of
try
to
optimize
that
happy
path,
and
we
found
that
it
also
indirectly
helps
when
we
have
like
a
break
glass
situation,
because
those
tools
and
presses
are
in
place.
It's
just
a
matter
of
you
know,
making
the
the
deployment
take
one
day
instead
of
seven
okay,
that
sort
of
thing.
C
A
That's
interesting,
so
the
question
is:
how
often
does
automated
patching
break
test
or
or
break
prod
or
whatever,
like
I
I,
think
the
for
us,
like?
We
mostly
go
and
we're
like
on
these
really
tiny
containers.
So
we
don't
like
have
a
whole
ton
of
like
dependencies.
So
we're
mostly
talking
about
like
go
dependencies
breaking
us,
and
so
there
have
been
a
couple
of
cases
where
a
go-minded
version
changed
changed
some
things,
and
so
we're
like
pretty
careful
about
how
we
like
look
after
good
minor
version
changes.
A
There's
also
implications
for
that
inside
of
kubernetes
We've,
recently,
Jordan's
been
doing
a
whole
ton
of
work
to
move
kubernetes
onto
onto
modern
versions
of
go
and
do
that
within
a
kubernetes
minor
release
and
that's
a
bit
sort
of
I
guess
more
risky
than
what
we've
been
doing
before,
which
was
we'd.
Only
move,
kubernetes
upper
version
only
move
up
a
Go
version
when
kubernetes
moved
up
a
minor
version
as
well,
but
that's
not
really
keeping
patch
Pace
with
the
with
the
the
patches
that
we
need
and
so
like
in
our
experience.
B
C
C
The
majority
here,
like
Greg,
said
these:
these
go
patches
it
typically
when
we
update
a
container,
it's
just
a
matter
of
moving
to
a
new
base,
image
that
has
slightly
newer
system
packages,
maybe
running
an
apt
update,
Etc
and
those
are
pretty
well
vetted
by
the
vendor.
The
the
only
cases
I
can
remember
is
when
we
moved
like
an
image,
an
entire.