►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Using Open Policy Agent to Meet Evolving Policy Requirements - Jeremy Rickard, VMware
Our team runs a Kubernetes platform for 30+ teams in a variety of commercial and government environments. Each of these environments has different security and compliance requirements, such as PCI and FedRAMP. We must deal with evolving requirements as our tenants pursue new accreditations. While we could implement a variety of mutating and validating webhook implementations to meet our needs, we instead turned to Open Policy Agent (OPA). OPA has allowed us to quickly develop and deploy new policies as these requirements shift and evolve. In this talk, we will look at several concrete examples of how we used OPA to implement our changing kubernetes policy requirements and help our tenants achieve a variety of compliance certifications, while at the same time striving to make these security policies as unobtrusive to their existing CI/CD pipelines and workflows.
https://sched.co/ekEP
A
My
name
is
jeremy
rickard,
I'm
a
software
engineer
at
vmware
and
my
team
has
really
been
focused
on
doing
things
in
the
kubernetes
space
for
the
last
few
years.
I'm
also
the
kubernetes
120
release
lead
and
I've
worked
on
a
number
of
open
source
projects
like
virtual
cubelet
and
service
catalog
for
kubernetes.
A
We're
called
the
vmware
developer
platform
and
we've
got
this
long
collection
of
words
that
describe
what
our
team
does,
but
if
I
boil
it
down
to
a
really
simple
explanation,
our
team
provides
managed
kubernetes
to
vmware
sas
services,
along
with
supporting
infrastructure,
like
maybe
vault
or
creation
of
resources
in
aws.
That
might
support
what
those
teams
are
doing.
A
This
project
has
been
around
since
mid-2018
kind
of
generally
available
for
vmware
sas
teams
and
the
the
genesis
of
this
thing
was
really
focused
on
deploying
clusters
for
multi-tenant
use,
so
we
would
deploy
kind
of
shared
clusters
and
these
clusters
would
have
multiple
tenants
on
them.
These
were
deployed
into
amazon,
so
running
in
public
clouds
and
available
to
the
sas
teams
to
deploy
their
workloads.
A
Since
we
are
running
multi-tenant
clusters,
we
kind
of
use
name
spaces
as
the
the
level
of
isolation
where
we're
enforcing
that
multi-tenancy
and
we
use
our
back
pretty
extensively
in
order
to
to
make
that
happen.
So
to
really
facilitate
that
we
also
have
a
data
plane
that
we
run
in
something
we
call
our
management
clusters.
So
when
a
user
is
going
to
use
vdp,
they
use
a
cli
that
we've
written
to
maybe
create
namespaces
or
label.
Namespaces.
A
Do
things
like
that,
then,
once
they
have,
the
namespace
created
they're
able
to
do
pretty
much
whatever
they
want
inside
of
that
namespace,
and
it
belongs
to
them.
They
can
define
network
policies
things
like
that,
but
we
very
quickly
came
to
see
that
that's
not
really
sufficient.
For
all
the
use
cases
there
were
teams
that
needed
to
do
more
than
just
exist
within
one,
maybe
two
namespaces.
They
needed
the
ability
to
create
more
namespaces,
do
other
things
on
the
cluster
and
doing
that
doesn't
really
fit
into
that
multi-tenant
cluster
kind
of
kind
of
model.
A
So
we
also
have
evolved
to
support
non-shared
clusters.
We
call
these
tenant
cluster
pretendants
and
in
those
cases
they
get
much
more
access
to
the
cluster.
They
can
do
things
like
create
namespaces.
They
they
get
nearly
clustered
admin
access
instead
of
having
to
rely
on
our
data
plane
to
do
a
lot
of
that
stuff.
But
at
the
same
time,
we
have
resources
that
we
deploy
that
are
really
responsible
required
for
the
cluster
to
operate
successfully.
So
we
come
to
this
challenging
point
of
tenants
really
having
what
amounts
to
cluster
admin.
A
They
can
do
a
lot
of
things
that
might
impact
the
operation
of
our
system,
but
we
also
have
resources
that
we
want
to
protect
and
our
back
wasn't
really
sufficient
to
handle
all
of
those
things.
Maybe
we
need
to
validate
that
somebody
can
do
something
based
off
of
their
org
membership
in
an
external
system.
So
thinking
about
how
we
might
solve
that
problem,
we
realized
pretty
quickly
that
webhooks,
you
know
the
mutating
dynamic
and
mission
control.
A
A
Finally,
it
moves
on
through
the
chain
until
it
gets
to
the
validating
workbooks
in
there
it
makes
decisions
about
whether
the
the
request
should
be
allowed
or
not.
You
know,
maybe
it's
checking
parts
of
the
the
request
against
other
things
in
the
system
to
make
sure
that
that
should
be
allowed,
or
maybe
it's
checking
something
against
an
external
system.
So
we
ended
up
writing
a
web
hooked
to
do
some
of
this
and
it
was
focused
on
protecting
the
resources
that
vdp
manages
in
the
clusters,
but
allowing
them
to
do
most
everything
else.
A
So
when
a
request
comes
into
a
vdp
managed
namespace,
where
we
have
deployed
things,
we
can
can
analyze
who's,
making
the
request
and
either
allow
it
or
not
allow
it.
This
really
became
a
great
just
general
extension
point
for
us
to
add
new
functionality
that
we
couldn't
directly
express
with
with
role
based
access
control.
But
as
our
scope
started
to
grow,
you
know
we
onboarded
more
tenants.
We
also
started
to
pick
up
some
additional
places
where
we
needed
to
run.
A
So
we
recently
just
completed
an
effort
to
help
the
vmware
cloud
and
aws
team
secure
a
fedramp,
high
certification
and
doing
that
meant
that
we
needed
to
evaluate
a
lot
of
what
we
were
doing
and
you
know
look
at
the
requirements
for
that
certification
process
find
the
gaps
in
what
we
had
deployed
already
and
start
to
evolve
to
fix
those
things.
Shortly
after
we
started
to
support
a
pci
certification
effort
for
vmc
again
each
one
of
these
new
environments
brought
new
requirements.
A
When
you
consider
fedramp
high
there's
over
400
different
controls
that
you
have
to
meet
in
order
to
get
that
certification,
pci
has
a
completely
different
set
of
requirements.
A
lot
of
them
are
similar,
but
there's
also
differences
between
them.
You
need
to
really
review
each
one
of
these
things
against
what
you've
deployed
and
how
you're
operating
to
make
sure
that
you're
fitting
into
that
those
requirements,
does
kubernetes
directly
meet
all
of
those
things,
probably
not,
and
in
our
case
we
we
didn't
try
to
to
justify
each
one
of
those
things
with
kubernetes.
A
And
for
us
that
was
a
great
way
for
us
to
take
the
kubernetes
clusters
and
the
other
stuff
that
we've
deployed
for
our
tenants
and
figure
out
how
we
can
augment
those
things,
maybe
with
policies
or
maybe
some
additional
things
we
deploy.
That
can
help
to
really
reduce
those
risks
and,
as
we
looked
at
each
one
of
these
things
and
considering
that
we
have
lots
of
different
clusters,
you
know
we're
playing
in
the
commercial
regions-
us
west,
2,
u.s,
east,
one,
various
apac
or
europe
regions.
A
When
we
look
at
those
and
compare
them
to
the
govcloud
deployments,
they're
pretty
different
and
the
requirements
for
them
are
pretty
different.
We
do
have
a
base
set
of
security
things
that
we
have
to
follow
for
vmware
security.
Obviously
any
vmware
service-
that's
going
to
be
deployed,
has
to
go
through
a
set
of
security
validation
and
to
make
sure
that
it's
going
to
meet
our
internal
requirements.
A
But
when
we
move
to
the
other
environments,
there's
more
and
more
restrictive
things
put
in
place,
so
we
obviously
don't
want
to
force
all
of
these
requirements
onto
the
tenants
that
don't
need
them,
because
that
would
make
their
jobs
harder.
We
want
to
be
an
enabling
feature
for
them.
Help
them
be
successful.
A
That
doesn't
seem
like
it
fit
really
well
with
our
web
hook
model
because
we
would
be
adding.
You
know
different
features
that
we
would
have
to
probably
feature
flag
in
different
clusters,
keep
track
of
all
those
different
things.
It's
additional
code
we'd
have
to
write
and
test
every
time
we
wanted
to
make
one
of
these
new
features
available.
Then
it
would
have
to
go
through
our
whole
rollout
process
and
just
be
a
little
bit
more
complicated
than
we
think
would
be
great,
and
we
also
thinking
about
this
problem.
A
We
want
to
make
sure
our
users
don't
hate
us.
Additionally,
some
things
we
we
really
wanted
with.
You
know
this
change.
These
new
things
we
wanted
to
apply
was
that
we
didn't
really
want
to
require
new
code
for
each
one
of
these
things
we
didn't
want
to
have
to
make
changes
to
our
existing
webhook
code.
It's
written
in,
go.
We
build
it
into
a
docker
container.
We
deploy
it,
it
rolls
through
our
pipelines.
A
It
goes
through
a
full
upgrade
process.
If
we
wanted
to
make
individual
changes
to
the
to
that
thing,
every
time
we
had
to
identify
one
of
these
new
policies
that
we
needed
to
enforce,
that
would
get
a
little
bit
complicated.
So
we
wanted
to
require
something
that
didn't
really
require
new
code.
We
also
wanted
to
make
it
easy
for
the
team
to
learn.
So
we
wanted
to
to
not
require
them
to
learn
a
brand
new.
You
know
programming
language
from
the
ground
up.
A
A
We
did
a
search
across
the
cncf
landscape
and
we
really
identified
something
that
we
think
would
help
us.
We
thought
would
help
us
quite
a
bit,
but
it
turns
out
that
was
opa
open
policy
agent,
open
policy
agent
is
pretty
extensible.
It
provides
its
own
language
for
defining
what
policies
look
like
and
we'll
look
at
that
in
just
a
second
and
it
turns
out,
it
integrates
pretty
well
with
kubernetes
there's
a
project
called
gatekeeper
that
highly
recommend
you
take
a
look
at.
A
We
ended
up
not
using
gatekeeper
for
a
few
reasons
that
I'll
get
into
as
we
go
through
the
talk,
mostly
because
when
we
started
this
journey,
it
was
pretty
early
days
for
gatekeeper
and
we
ended
up
going
with
the
cube
management
approach,
which
runs
queue,
management
and
opa
together,
like
in
a
sidecar
manner.
It
ends
up
looking
something
a
little
bit
like
this.
Just
like
we
wrote
our
own
web
hook,
validating
admission
controller,
this
plugs
in
pretty
much
the
same
way.
A
So
when
you
deploy
opa
and
cube
management
together,
you
can
register
them
as
validating
and
mutating
web
hooks.
They
plug
into
the
api
server,
just
like
any
other
web
hook.
Would
so
when
a
user
is
making
requests
with
cube,
ctl
cicd
pipelines
maybe
are
using
the
api
directly,
maybe
using
cube
ctl
themselves
or
when
controllers
inside
of
the
cluster
are
making
changes
to
objects
and
resources
via
the
api
server.
Everything
goes
through
that
normal
admission
process,
an
admission
request,
hits
opa.
A
Opa
looks
at
that
request
determines
if
any
of
the
policies
you've
applied
should
should
result
in
a
deny
or
a
block,
and
then
it
sends
that
response
back
and
the
api
server
handles
that
appropriately.
So
let's
look
at
a
really
simple
example
of
what
a
policy
might
look
like
here.
We
want
to
deny
any
request
that
comes
in
that's
labeled
with
a
certain
value,
so
you
can
see
that
this
is
really
a
declarative
language.
A
We're
saying
a
series
of
facts,
or
in
this
case
really
just
one
fact,
and
then,
if
that
fact
is
true,
then
we're
setting
a
variable
value
of
this
getting
returned.
So
we
start
this
off
with
a
deny
block,
so
the
keyword
deny
and
then
that
is
a
message,
that's
going
to
be
returned
and
then
the
first
line
in
this
is
really
the
statement
that
we're
checking
the
policy
we're
enforcing.
A
So
in
this
case,
if
the
metadata
has
a
label
called
pants
with
the
value
of
sweatpants,
then
the
message
we're
going
to
send
back
is
you
can't
sit
with
us
if
you
notice
in
that
line?
Input.Request.Object,
that's
really
coming
from
the
kubernetes
admission
request.
If
you
look
at
the
json
that
makes
up
a
kubernetes
admission
request,
it's
got
those
pieces
of
it,
so
it's
really
great
in
this
policy
you're
able
to
say
I
want
to
look
at
the
metadata
of
this
object.
A
That's
coming
in,
or
maybe
I
want
to
look
at
the
spec
of
this
object.
That's
coming
in.
Maybe
I
want
to
look
at
the
the
verb.
Is
this
a
create
or
an
update?
Maybe
I
want
to
apply
policies
differently
that
way
it's
really
flexible
and
gives
you
a
lot
of
power
without
having
to
go
write.
You
know
new
code,
it's
still
code.
Obviously
you're.
Still,
writing
some
declarative
statements
and
you
still
have
to
end
up
putting
those
in
the
cluster
somehow,
but
it's
it's
a
much
simpler
path
forward
to
test.
A
This
opa
provides
a
lot
of
tooling
and
you
can
actually
take
this
stuff
and
put
it
into
a
opa
playground.
I
have
a
link
to
that
at
the
end
of
the
presentation
just
to
test
these
things
without
having
to
run
anything
locally
on
your
machine,
you
can
you
can
build
out
a
sample
test
document
and
build
out
your
sample
policy
and
just
and
run
the
validation
in
this
playground,
and
it's
pretty
cool.
A
So
with
all
of
that
in
mind,
let's
talk
about
a
few
use
cases
that
we
have
solved
with
open
policy
agent
in
rego
and
for
each
one
of
these
I'm
going
to
go
through
three
examples.
I'm
going
to
loosely
tie
this
back
to
some
control
or
some
rule
that
we
found
in
fedramp
or
pci
that
we
needed
to
apply
to
our
system
and
the
first
of
those
is
the
use
of
external
information
systems.
A
Inside
of
this
requirement,
there's
a
whole
bunch
of
different
rules
and
a
lot
of
different
individual
control
points,
but
the
one
I'm
going
to
focus
in
on
is
information
systems
that
are
outside
of
the
authorization
boundary
really
qualify
as
those
external
information
systems.
So
we
deploy
our
in
govcloud.
We
deploy
kubernetes
into
those
fedramp
environments
and
we
play
a
lot
of
other
things
in
there.
A
We
try
to
minimize
our
reliance
on
external
resources,
things
that
are
outside
of
that
authentication
boundary
and
one
of
those
things
is
a
docker
registry,
so
in
production,
in
our
commercial
environments,
we're
using
a
hosted
service
from
jfrog,
that's
not
available
to
us
to
use
directly
as
part
of
our
fedramp
offering,
so
we
needed
to
run
our
own
registry
in
boundary.
So
inside
of
that
govcloud
environment.
We
have
our
own
docker
registry
that
we're
running,
and
we
push
all
of
our
images
to
that.
A
So
then,
when
we
want
to
deploy
stuff
into
the
cluster,
we
need
to
reference
those
images.
We
also
want
to
make
sure
that
the
cluster
isn't
running
things
that
it's
directly
pulling
from
the
internet.
There
is
some
connectivity,
or
there
was
originally
we've
locked
it
down
since
then,
but
originally
you
were
able
to
pull
things
from
docker
hub
or
pull
things
from
our
jfrog
hosted
solution.
So
the
first
thing
we
looked
at
with
opa
was:
how
do
we
restrict
the
use
of
those
other
registries?
A
We
really
block
it
down
to
just
the
one,
so
we
want
to
make
sure
that
requests
that
are
coming
in
only
come
from
that
registry
that
we
want
them
to
come
from.
So
one
of
the
first
policies
we
built
was
a
pretty
simple
one.
That
would
look
at
the
the
image
that's
being
used
by
containers,
so
this
policy
really
is
cool
and
it
lets
us
restrict
the
any
request,
that's
coming
in
to
only
those
that
come
from
certain
repositories.
A
So
in
this
case
we
start
off
again
with
that
deny
block,
and
the
first
thing
we
look
at
is:
does
this
kind,
the
the
the
kind
of
this
request?
So,
just
like
you
know,
you
deal
with
kinds
of
kubernetes
we're
checking
that
here,
so
an
admission
request
will
come
with
whatever
type
of
object
you're
dealing
with,
so
we
really
only
want
to
apply
this
to
pods,
and
you
know
we
could
do
this
at
different
levels.
We
could
look
at
the
deployments
replica
sets.
A
This
was
the
simplest
for
us
to
just
look
at
when
a
pod
is
created.
Is
the
container
using
something
using
the
registry
that
we
expect
it
to
use.
It's
a
pretty
simplistic
check,
so
we
iterate
through
all
of
the
images.
So
obviously
you
can
have
multiple
images
in
in
a
pod
spec
and
we
want
to
make
sure
that
each
one
of
those
things
is
is
is
valid,
so
we
start
off
with
the
second
line
or
of
the
block
some.
A
I
so
like
just
for
every
image
that
exists
in
this
array
of
input
that
requests
that
object.spec.containers.
Let's
grab
that
thing
and
validate
it,
so
then,
for
each
one
of
those
images
we
we
basically
just
say:
does
this
image
start
with
what
our
you
know,
our
gov
repo
is,
I
replace
it
here
with
vmware
is
awesome.
Just
for
for
notional
purposes,
but
you
can
see
the
you
know
we're
making
a
little
bit
more
complex
policy
here
by
calling
into
that
function.
A
When
this
evaluates
true,
when
the
first
line
is
it's
a
pod
and
when
this
is
not
a
govcloud
image,
we're
going
to
return
the
message
pod's
container
is
not
allowed
to
use
the
image
from
a
non-approved
repo
in
gov.
So
what's
that
look
like
in
practice,
so
using
this
deprecated
functionality
of
creating
a
pod
with
cube
ctl
run
we're
still
running
fairly
old
clusters,
so
I
can
still
do
this.
A
I'm
going
to
try
to
run
an
mq
test
client
from
my
personal
docker
hub
account,
so
I
run
that
with
cube,
ctl
run
that
cr
actually
behind
the
scenes
right
now
in
that
version
of
kubernetes,
creates
a
deployment
and
that
deployment
that
will
then
spin
up
pots.
I
don't
get
an
error
here,
though,
because
my
policy
was
really
applied
to
just
the
pod,
so
to
kind
of
work
around
that
or
to
see
how,
like
what
feedback
you
get.
Let's
take
a
look
at
the
events.
A
We
can
run
cube,
ctl,
get
events
and
filter
that
down
to
open
policy
agent
in
the
string,
and
you
can
see
that
we
can't
actually
create
the
pod,
and
if
I
did
a
cube,
ctl
get
pods
here.
You
would
see
that
there
were
no
pods
created
for
this
deployment
and
it's
specifically
showing
that
error
message
that
I
created
before.
A
So
this
was
great
and
we
were
able
to
lock
all
of
the
registries
down,
make
sure
that
we
weren't
deploying
anything
from
you
know
the
non-controlled
things
that
were
inside
of
the
boundary,
but
now
we
have
some
fairly
unhappy
users
and
our
goal
all
along.
I
mentioned
this
at
the
beginning-
was
to
make
sure
that
the
users
didn't
hate
us.
We
wanted
to
make
sure
that
things
were
as
easy
as
possible
for
them
and
not
every
one
of
our
tenants
is
super
versed
in
kubernetes
they're
using
kubernetes.
A
They
realize
the
benefits
of
deploying
their
stuff
under
the
platform
they're
along
for
the
ride
for
govcloud,
but
us
adding
this
constraint
makes
it
a
little
bit
more
difficult
for
them.
They
either
have
to
go,
maintain
a
separate
set
of
values,
files
if
they're
using
helm
or
some
other
tool
that
does
a
kind
of
templating
and
overlaying,
maybe
their
helm
chart,
doesn't
even
allow
them
to
really
template
that,
because
they've
they've
not
done
a
super
great
job
of
templating
that
stuff
out.
So
there
were
changes
that
had
to
be
made
there.
A
So
we
thought
what
can
we
do
to
help
with
that
situation?
And
I
mentioned
this
earlier,
but
you
can
actually
run
opa
as
a
mutating
web
hook
in
addition
to
a
validating
web
hook.
So
what's
the
big
difference
there?
Well,
when
it
runs
as
a
validating
web
hook,
we
have
those
blocks
and
they
started
with
deny
and
what
happens
there
is
when
all
of
the
rules
match
for
a
deny
the
validating
web
hook.
Functionality
will
say
this
request
is
not
allowed.
A
Here's
the
error
message,
but
just
like
every
other
mutating
web
hook,
opa
can
also
update
your
resource
and
it
does
that
by
generating
json
patches,
the
syntax
gets
a
little
bit
more
complicated
and
I'm
not
going
to
show
you
the
entire
thing
here,
but
I'm
going
to
show
the
pretty
like
the
relevant
parts.
So
here
we've
defined
two
variables,
vdp
repo.
So
it's
going
to
map
you
know
be
whatever
our
upstream
public
managed.
Jfrog
thing
is
and
then
also
whatever
our
govcloud
the
host.
You
know
the
host
name
for
our
our
glove
cloud.
A
Govcloud
repo
is
and
then,
instead
of
using
the
deny
block,
we're
going
to
define
a
patch
block
which
is
going
to
return
whatever
json
patch
needs
to
be
applied.
So
in
this
case
we
have
a
couple
of
extra
things
here.
I
probably
should
have
removed
from
the
example,
but
we
first
want
to
make
sure
is
mutation
allowed,
so
we
want
to
to
validate
that
the
type
of
resource
that
we're
going
to
mutate
is
something
we
want
to
mutate.
We
have
some
rules
built
around.
A
A
We
don't
want
to
mutate
like
this
because
could
lead
to
unexpected
consequences,
but
we
we
essentially
check
to
see
if
that
exists
or
not
and
then
move
on
and
then
just
like
the
deny
rule
we're
going
to
iterate
over
all
the
containers
and
then
we're
going
to
check
to
see
if
that
container
matches
the
upstream
public
repo
and
replace
it
with
the
downstream
value.
If
any
of
that
generated
a
new
value,
then
we're
actually
going
to
make
the
json
patch
here.
A
So
I've
removed
some
of
the
bits
about
actually
making
the
json
patch
and
I'll
link
to
the
documentation
at
the
end
of
the
talk.
But
what
will
happen
here
is
that
when
we
make
a
request,
so
maybe
we're
going
to
helm,
deploy
some
deployment
and
it's
going
to
reference
our
upstream
jfrog
repository
this
code
will
actually
get
invoked.
It'll.
Look
at
that
request.
That's
coming
in
and
say:
oh
hey,
you're,
using
the
upstream
version.
We
can't
use
that
in
govcloud.
A
Let
me
go
ahead
and
mutate
that
for
you,
so
when
this
actually
hits
the
api
server
or
sorry
ftd,
it's
going
to
actually
get
the
gov
repo
instead
of
the
upstream
repo.
So
it'll
try
to
pull
that
down
and
it'll
work
just
like
we
would
expect
it
to
work,
but
we've
made
it
a
little
bit
transparent
to
the
to
the
end
users.
So
that's
the
first
use
case
that
we
had
that
we
solved
with
opa.
A
A
As
we
went
through
this
initial
process,
we
found
a
lot
of
containers
that
we
were
deploying
that
actually
had
a
number
of
vulnerabilities.
So,
as
I
mentioned,
these
things
are
based
off
of
cvss
scores.
So
in
the
pci
case
anything
that's
mod,
medium
or
higher,
which
is
a
cvs,
has
score
of
four
or
more.
You
actually
have
to
remediate,
or
you
will
fail
your
pci
audit
and
any
re
reinvestigation
or
you
know
subsequent
audits.
You
go
through.
You
have
to
demonstrate
that
you've
been
doing
these
things
and
fixing
these
things,
and
you
can
do
this.
A
A
Just
a
really
quick
example
of
what
that
might
look
like.
I
scanned
one
of
the
images
that
we
have
deployed
in
our
commercial
environment
and
you
can
see
that
it
found
a
number
of
vulnerabilities.
Two
of
them
are
critical.
Three
were
high,
four
were
medium
and
three
were
low,
so
we
definitely
have
to
fix
those
criticals
those
highs
and
for
pci
we
need
to
fix
those
mediums.
A
What
kind
of
things
do
you
find
inside
of
that?
So
these
can
be
os
level
vulnerabilities.
You
could
say,
like
the
version
you're
running
an
ubuntu
based
container
and
it's
got
glibc
vulnerability
inside
of
it.
That'll
come
up
in
these
scanners
and
that
will
get
flagged
by
one
of
the
auditors.
So
may
not
really
be
a
problem,
but
it's
best
to,
for
us
at
least
the
least
amount
of
effort
is
to
fix
the
problem.
It
could
also
be
application
level
things.
A
So
in
this
case
this
is
a
java
application
and
the
problem
here
is
actually
the
version
of
log4j
that
it's
using.
So
how
do
we
fix
this
process?
How
do
we?
How
do
we?
You
know
the
vdp
team
handle
fixing
these
things?
How
would
anybody
else
really
handle
these
things?
Well,
you
generally
need
to
build
some
new
container
to
do
this.
A
Updating
libraries
may
be
applying
os
updates
inside
of
the
container,
if
you're,
using
something
like
photon,
if
you're
a
vmware
person
or
ubuntu
or
wn
as
the
base
images.
So
we
built
a
little
process
around
that
for
ourselves
and
that
involves
taking
whatever
the
base
image
that
we
have.
So
maybe
it's
some
upstream
component,
that's
based
off
of
alpine
or
some
upstream
component.
That's
based
off
of
ubuntu.
We
write
a
new
docker
file.
We
take
the
old
one
as
the
the
from
line.
A
So
if
you're
building
a
docker
file
out
first
line
is
from
whatever,
then
we
run
whatever
the
os
appropriate
updates
are
just
to
make
sure
that
we
apply
those
things
and
then
we
add
in
something
we've
built.
We
add
in
you
know.
We
make
sure
that
we've
rebuilt
that,
with
whatever
the
updated
libraries
are
and
voila
that
results
in
a
new
tag,
hopefully
without
any
vulnerabilities,
then
we
need
to
deploy
that
to
the
cluster.
So
we
go
and
maybe
we
run
a
helm,
update
or
k
app
deploy.
A
A
We
build
all
of
the
images
we've
identified
in
our
inventory
file,
which
is,
of
course,
the
ammo
file,
and
we
update
those
things
to
generate
new
tags.
Then
we
update
the
inventory
file
and
then
we
somehow
need
to
deploy
that
to
the
kubernetes
cluster,
and
we
want
to
do
this
in
a
not
really
manual
way,
so
we
were
able
to
pretty
pretty
easily
automate
the
front
half
of
that
process
where
we
would
rebuild
these
containers,
we've
already
written
the
docker
files
for
them.
A
We
have
this
the
skeleton
of
the
inventory.
How
do
we
then
take
that
and
deploy
it?
So
one
of
the
cool
things
is
that
opa,
especially
when
you're
using
the
cube
management
sidecar,
is
that
you
get
access
to
other
resources
in
kubernetes.
It
acts
just
kind
of
like
any
other
kubernetes
client
would
where
it
establishes.
A
watch
sees
things
from
the
api
server,
so
we
put
that
into
a
config
map
in
the
cluster.
A
It
lists
out
the
name
of
the
image
and
then
what
version
we
want
to
run
and
then
the
opa
sidecar
cube
management
sees
that
those
things
have
changed
and
makes
them
available
as
a
data
field
to
opa.
You
can
access
it
like
this.
So
next
we
can
write
a
pretty
simple
policy
that
looks
at
that
inventory
and
compares
that
to
what's
deployed
and
then
mutates
the
tag.
So
our
we
again
start
with
a
patch
block.
We
iterate
through
all
the
images
in
the
in
this
case
the
unit
containers.
A
So
now
our
inventory
file
gets
deployed
to
kubernetes.
We
play
that
as
a
config
map
that
gets
reloaded
by
opa,
and
then
we
run
a
small
job
that
just
labels
touches
the
labels
on
all
our
deployments,
which
then
forces
them
to
go
through
the
emission
process
again
that
forces
the
mutating
web
hook
in
this
case
opa
to
update
all
the
tags
and
then
start
up
again
with
all
those
new,
hopefully
vulnerable,
vulnerability,
free
images.
A
So
the
last
policy
that
we
really
wanted
to
enforce
was
running
as
non-root
and
running
is
non-root,
we're
doing
mostly
with
pod
security
policies.
Cloud
security
policy
works
pretty
well,
and
it's
pretty
easy
for
our
tenants
to
understand,
except
when
they
don't
pay
attention
to
the
notifications
we
send
and
don't
make
changes
to
their
their
their
3ml.
A
A
The
great
thing
here
is,
though,
when
the
users
are
making
these
calls
and
they're
deploying
their
stuff,
they
can
actually
specify
whatever
security
context
they
want
and
we
won't
mutate
it.
This
is
just
a
nice
add-on
for
them
when
they
don't
have
that
done
so
recapping.
What?
If
we
lo?
What
have
I?
What
did
we
learn
with
vdp
team?
Learn,
opa
is
really
flexible.
A
Validation
can
get
you
pretty
far.
You
can
write
a
lot
of
deny
rules
to
lock
your
clusters
down
and
do
a
lot
of
things,
but
mutation
can
get
you
even
further.
You
can
do
a
lot
of
things
when
you
combine
these
two
things
together.
Rego
is
pretty
pretty
easy
to
learn
the
declarative
nature
of
it
just
makes
it
pretty
easy
for
people
to
pick
up,
and
we
have
found
that
it's
pretty
easy
for
all
the
team
members
to
really
learn
it
and
start
writing
new
policies
or
fix
problems.
A
But
let's
go
back
for
a
second
and
talk
about
the
the
mutation
aspect
of
this.
I
have
mixed
feelings
about
mutating
web
hooks
and
if
you
read
the
documentation,
the
kubernetes
documentation,
there's
actually
some
call
outs
to
say
hey.
You
should
probably
be
aware
of
these
things,
and
maybe
it's
not
the
greatest
case,
and
one
of
them
is
that
users,
don't
necessarily
know
what's
happening.
They
may
be
confused
by
like
what
this
thing
that
I've
created
doesn't
look
like.
What's
in
the
cluster
now
what
happened?
A
There's
a
great
tutorial
linked
here
as
well
and
like
how
to
validate
ingress
in
the
cluster
and
then.
Finally,
one
of
the
reasons
we
didn't
use
gatekeeper
was:
it
doesn't
do
mutation
yet,
but
there's
an
open
issue
here
and
maybe
by
the
time
cubecom
comes
around.
This
will
be
done.
I
would
totally
advise
you
to
follow
this
and
give
gatekeeper
a
try
if
you're
going
to
look
at
opa,
especially
if
you
want
to
do
just
validating
sort
of
things
at
the
moment.
So
at
that
point
I'll
turn
it
over
to
questions.