►
From YouTube: Kubernetes Policy Enforcement Using OPA At Goldman Sachs - Miguel Uzcategui & Tim Hinrichs
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes Policy Enforcement Using OPA At Goldman Sachs - Miguel Uzcategui, Goldman Sachs & Tim Hinrichs, Styra
Managing state on multiple shared Kubernetes clusters may sound scary. The Goldman Sachs Kubernetes team uses OPA to manage that state using two different applications of policy. The first is the validating admission control policies that prevent unsafe resources on the cluster. The second, and novel, application goes beyond simple yes/no decisions and uses OPA policy to provision new resources on the cluster to implement a common baseline, e.g. RBAC, Volumes, ResourceQuotas, and LimitRanges. This talk focuses on the architectural design that allows GS to run OPA at scale in production. Along the way we discuss best practices and lessons learned, highlighting how GS reduced policy deployment times from days to under 10 minutes. The audience will learn how to create their own policy pipelines using popular open-source tools to enforce OPA policy across multiple Kubernetes clusters.
https://sched.co/UaaX
A
All
right
we
made
it
last
of
the
day
my
show
hands.
Can
you
please
tell
me
who
uses
kubernetes
in
production
on
a
multi-tenant
model?
Go
who
needs
some
sort
of
automated
provisioning
workflow
for
new
namespaces,
for
application
teams,
cool
and
out
of
you
who
uses
OPA
all
right.
You
raise
your
hand,
you
didn't.
This
talk
is
for
everybody
from
discriminate
when
it
means
me
get
this
category.
This
is
my
colleague,
Tim
Hendricks
and
we're
here
to
talk
about
policy
enforcement
using
OPA
and
police
u.s.
code
at
common
Sachs.
A
A
It's
a
little
bit
of
what
we
are
going
to
talk
about
today.
Well,
it's!
Why
are
we
having
this
talk,
then
we'll
see
how
the
Q&A
policy
at
Goldman
Sachs
will
then
look
at
how
we're
using
OPA
to
solve
this
will
step
into
the
in
through
the
architecture.
Some
of
the
future
work
lessons
learned
and
then
we'll
open
up
for
questions
so
we're
we
have
in
this
talk
to
answer
this
question.
We
first
need
to
look
at
how
Gorman
Sachs
is
currently
running.
Q
Benares.
A
So
a
little
bit
little
statistics
we're
currently
running
12
total
share
clusters.
As
I
said
before,
we
use
multi-tenancy,
in
which
we
isolate
our
users
by
namespace
a
government
namespace
half
meanings.
We
can
basically
tie
a
namespace
name
to
any
inventory
system
at
Goldman
Sachs.
We
also
call
them
deployments.
I
know
it
conflicts
a
little
bit
with
a
given
se
ponen,
but
we
currently
run
on
VMs.
We
have
an
average
of
159
spaces
per
cluster
and
just
like
everybody
else,
we
run
common
software
like
prometheus
graph,
an
i7
rule,
core
DNS
or
PI
etcetera.
A
So
not
too
deep
ago,
man,
we
love
inventory
systems.
Everybody
runs
around
an
example
of
this
would
be
a
group
Rose
inventory,
basically,
who
has
admin
access
or
view
access
on
certain
namespaces
compute
capacity
inventories
what
names?
What
people
can
request
for
the
application
names
namespaces
in
terms
of
course
and
memory,
and
then
we
have
a
production,
access,
control,
namespaces
production,
access,
control,
inventories,
storage,
inventories,
etc.
A
So
a
little
bit
some
examples
of
how
government
needs
policy
for
their
Kira
Nerys
environment.
We
have
our
back,
we
need
to
know
who
will
get
admin
and
viewer
access
on
each
of
our
namespace.
Obviously,
not
everybody
can
have
unlimited
access
across
the
entire
Cavanaugh's
cluster
right.
So
we
need
to
limit
this
based
on
some
inventory
data
we
have.
We
obviously
don't
want
to
give
unlimited
resources
to
everybody
in
our
cluster,
so
we
need
to
manage
these
through
resource
folders
storage.
A
Even
though
we
use
a
safe
and
route,
some
people
need
to
manifest
shares
onto
their
application.
So
who
can
mount
these
NFS
shares
on
the
containers
we
need?
We
need
to
manage
it
on
upper
namespace
races.
Finally,
another
example
would
be
ingresses.
We
definitely
had
situations
in
the
past
where
application
teams
define
ingresses
with
same
host
names
in
the
same
cluster,
and
this
happened
to
break
and
the
nginx
ingress
controller
for
everybody,
which
means
the
cluster
was
basically
down.
B
Yeah,
so
the
media
talk
is
gonna,
be
focused
on
how
we
implement
those
policies.
How
helped
me
go
set
this
up
at
Goldman,
and
so
really
we're
gonna
be
seeing
two
different
ways
of
implementing
those
some
of
those
policies
that
you
saw
in
the
last
slide.
The
first
is
well-known
and
in
both
we're
gonna
be
using
open
and
in
the
admission
control
case,
the
first
kind
of
implementation.
B
People
have
heard
about
this
for
quite
a
while.
The
way
that
this
works
is
really
the
same
way
that
you've
heard
before
I
would
imagine
where
the
kubernetes
api
server
on
every
request
it
receives,
has
a
pipeline
that
it
sends
that
request
through
starting
with
authentication,
moving
on
to
authorization
and
then
finally,
to
admission
control,
and
so
what
we're
gonna
do
is
that
that
that
admission,
control
step,
we're
going
to
go
ahead
and
enforce
those
policies
that
ensure
that
there
are
no
conflicting
ingress
is
on
the
cluster.
B
All
right,
so
that's
gonna,
be
one
kind
of
enforcement
and
we'll
we'll
talk
a
little
bit
about
that,
but
because
we've
talked
about
it
so
many
times
and
you've
heard
about
it
so
many
times
we're
gonna
focus
more
so
on
the
second
kind
of
policy
implementation,
and
we
call
we're
calling
this
provisioning
today.
So
the
idea
here
is
that
each
and
every
time
somebody
let's
say,
creates
a
new
namespace
on
the
cluster.
There
are
a
number
of
resources
that
need
to
be
provisioned.
B
They
need
to
be
created,
and
so
here
what
we'll
see
is
that
we're
gonna,
be
creating
rules
and
role
bindings
will
be
creating
volumes
and
quotas
as
well
and
to
drive
this
point
home.
You
can't
solve
the
provisioning
problem
with
admission
control
and
the
reason
for
that
is
admission.
Control
is
designed
to
allow
you
to
either
accept
or
reject
a
resource,
or
maybe
at
the
most.
You
can
mutate
that
resource
that
comes
into
the
cluster.
B
B
That
say
things
like
ensure
that
whatever
hostname
is
being
used
in
this
new
ingress
that
the
users
trying
to
create
isn't
the
same
and
doesn't
overlap
with
any
of
the
other
ingress
--is
that
are
already
on
the
cluster
and
so
as
an
example.
What
we've
actually
shown
here
is
the
OPA
policy
that
would
that
you
would
write
in
order
to
enforce
that.
The
enforce
that
only
that
there,
no
conflicting
ingress,
is
and
so
I
won't
go
into
all
the
details,
but
I
will
sort
of
talk
through
it
here
briefly.
B
The
idea
is
that
you're
going
to
be
writing
a
bunch
of
deny
statements.
That's
the
the
typical
pattern
that
we
see,
and
so
what
you're
going
to
do
here
is
we're
going
to
deny
an
incoming
a
new
ingress.
Well,
we're
going
to
first
check
that
it's
an
ingress
and
then
what
we're
going
to
do
is
we're
going
to
iterate
over
all
of
the
new
hosts.
B
All
of
the
hosts
that
show
up
in
that
new
ingress
that
perhaps
someone's
trying
to
create
we're,
also
going
to
simultaneously
go
out
and
grab
all
of
the
hosts
that
already
exist
in
the
cluster
for
ingress
is
that's
what
the
next
line
is
doing
and
then
we're
gonna
check.
If
the
old
host
in
the
new
host
are
the
same
here,
I'm
playing
a
little
fast
and
loose
with
what
the
same
means
you
could
make
this
more
sophisticated.
B
If
you
wanted,
but
for
the
sake
of
a
of
a
slide,
this
is
a
good
enough
check
and
then
the
last
line
there
is
just
constructing
an
error
message
that
will
return
to
the
user.
So
the
right
way
to
interpret
this
statement
is
that,
if
all
of
those
conditions
hold
then
in
fact
the
request
for
a
new
ingress
will
be
rejected
and
the
error
message
that's
returned
to
the
user
is
the
one
that
we're
constructing
here,
so
host
conflicts
with
the
ingress
that
we
found.
B
B
Cases
here,
we're
all
obviously
talking
about
kubernetes,
but
here,
even
within
this
talk,
we're
talking
about
two
different
use
cases:
two
different
kinds
of
policy
implementations,
one
with
an
in
control
and
one
within
within
provisioning,
but
generally
the
way
it
works
is
that
you've
got
a
service.
Any
service
it
could
be
be
the
admission
control
or
it
could
be
a
micro
service.
It
could
be
really
anything
that
you
like,
and
that
service
decides,
that
it
needs
a
policy
decision.
B
So
what
it
does,
is
it
cobbles,
together
all
of
the
information
that
it
has
that
it
needs
a
decision
about
encodes
it
as
JSON
and
hands
it
over
to
Opa
Opa
returns
a
decision.
The
service
is
then
responsible
for
enforcing
that
decision.
Hopa
makes
a
decision.
The
service
enforces
it.
One
of
the
important
things
that
you'll
see
later
today
is
that
OPA
when
it
receives
the
input
that
JSON
object.
It
doesn't
really
understand
the
semantics
of
that
object.
B
It
doesn't
know
anything
about
kubernetes,
pods
or
ingress
is
not
natively,
but
is
a
policy
author
when
I'm
writing
my
policies,
I
know
what
that
JSON
object
represents.
I
know
that
it
represents
a
pod
and
I
know
that
a
pod
has
containers
and
images
within
it.
I
know
that
an
ingress
has
hosts
and
so
I
can
write.
The
policy
much
like
I
showed
you
on
the
last
slide.
B
The
second
thing
that
I'll
call
out
here
is
that
OPA
allows
you,
when
you're
making
a
decision
to
return,
not
just
allows
and
deny
it's
not
just
true/false,
it's
not
just
authorization.
It
allows
you
to
return
technically
any
JSON
object.
You
can
use
that
JSON
object
to
represent
anything.
You
like
you
could
use
it
to
return.
True,
false
meaning,
allow
deny
you
could
return
a
string
representing
a
host
name,
a
number
representing,
maybe
at
a
rate
limit.
You
can
return
an
array
of
the
clusters
to
what
you
want
to
deploy
an
application.
B
You
could
return
an
entirely
deeply
nested
JSON
object.
It
represents
an
admission
review
response.
You
could,
as
we'll
see
later
today,
return
an
actual
kubernetes
resource
description
in
JSON
that
should
be
provisioned
on
the
cluster
and
so
we'll
see
how
that
works
a
little
later.
The
other
thing
I'll
call
out
here
is
it
we'll
also
allows
you
to
inject
external
data
in
order
to
make
policy
decisions.
B
You
heard
Miguel
talk
about
the
inventory
management
systems
that
Goldman
uses,
and
this
is
really
important,
and
what
we'll
see
happen
is
that
those
in
the
data
from
those
inventory
management
systems
is
going
to
be
injected
into
OPA
in
order
to
compute
the
resources
that
need
to
be
provisioned
in
any
new
namespace.
So
that's
going
to
be
really
crucial
and
what
that
allows
us
to
do
is
have
those
separate
teams
that
are
responsible
for
the
group.
B
The
other
thing
I'll
mention
here
about
OPA
is
that
architectural
II
it
is.
It
is
an
in-memory
policy
engine.
It
was
designed
to
run
at
the
edge
it
was
designed
to
run
next
to
whatever
software
system
needs
decisions.
It
is
lightweight
and
does
not
persist
policy,
and
what
that
means
is
that
you
will
need
a
way
of
a
source
of
truth
for
your
policy.
Get
as
a
is
a
common
one
will
again
see
that
later.
B
But
what
you
also
need
is
a
way
to
distribute
policy
from
that
source
of
truth
down
to
the
OPA,
so
that
the
open
knows
exactly
what
the
policy
is,
and
so
that's
pretty
important.
There
are
a
number
of
ways
of
doing
that.
There
is
a
push
API
that
you
can
use
and
just
push
it
right
into
OPA,
or
you
can
configure
open
to
pull
that
policy
and
actually
external
data
as
well
through
something
we
like
to
call
the
bundle
API.
Also
along
with
management
OPA,
does
allow
you
to
log
all
the
decisions
it
makes.
B
B
Here's
an
example
just
to
drive
this
home
again,
showing
you
the
sort
of
functional
inputs
and
outputs
that
OPA
provides
here
on
the
top.
We
have
our
policy,
the
policy
that
I
talked
through
a
moment
ago.
That
is
one
of
the
inputs
that
you
hand
to
two
oppa,
there's
also
the
in
this
case
the
in
the
admission
control
case.
There
would
be
an
admission
review
input
that
gets
handed
to
opus.
Well,
that's
literally
called
input
in
the
open
language.
There's
external
data
in
this
ingress
admission
control
example.
B
That
would
be
all
of
the
existing
ingress
--is
that
already
exist
on
the
cluster,
we'll
see
later
how
we
can
use
the
inventory
management
systems
as
well.
Those
three
things,
input
policy
and
data-
are
all
the
are
all
the
the
information
that's
fed
into
open
and
open
makes
a
decision
decision.
It
produces
an
output.
In
this
admission
control
ingress
example
we're
returning
the
list
of
of
error
messages
that
open,
eventually
gets
returned
to
the
to
the
end
user
and
just
to
drive
this
point
home.
B
Remember
to
uplevel
this
again
in
this
setting
of
oppa
oppa
is
a
building
block
that
can
be
used
for
many
different
things.
We've
already
shown
how
you
can
use
it
to
solve
the
admission
control
problem
to
eliminate
to
implement
the
ingress
policy
that
the
Goldmann
uses,
but
that's
not
enough.
It's
not
enough
for
the
other
three
classes
of
policies
that
Miguel
talked
about
the
arbok
policies,
the
volumes
of
quotas,
those
all
are
new
resources
that
need
to
be
provisioned
into
the
namespace
whenever
that
namespace
is
created.
B
That
is
not
something
that
Mission
Control
can
do
in
Michigan.
Troll
allows
you
to
accept
or
reject
the
resource,
or
maybe
even
mutate
it.
It
does
not
allow
you
to
create
new
resources
anytime
in
response
to
a
resource
like
a
namespace
being
created.
So
these
are
definitely
different
problems,
and
that
is
what
we'll
focus
on
for
the
rest
of
the
talk.
A
A
Sadly,
called
controller
manager
like
the
kubernetes
controller
manager
that
basically
what
he,
what
he
was
doing
in
the
past,
was
getting
all
the
information
from
all
this
inventory
systems
beatrice
api,
getting
a
bunch
of
a
cuban
areas,
cluster
information
using
sharing
formers
and
having
all
the
policy
definer
in
code
in
go
code.
Then
it
was
making.
Then
what
he
was
doing
is
continuously
running
this
put
this
in
code
defined
policies
and
applying
it
to
through
basic
cuban
as
a
standard,
API
libraries.
Now
this
approach
happened
to
be
very
inflexible.
Why
the
rules
were
in
unclear.
A
In
addition
to
these
users
had
no
input
into
this
process
whatsoever,
we
define
our
own
baseline
policies
in
this
controller
manager
and
nobody
had
visibility
into
and
the
moment
somebody
needed
something
special.
We
needed
to
create
it
and
manually
in
the
cluster.
So
if
something
happens
to
this
cluster-
and
they
object
somehow
disappears,
then
either
somebody
raised
a
ticket
because
in
their
applications
and
longer
working
or
we
recreate
it
manually,
in
addition
to
these
production
reasons,
we're
needed
every
time
we
needed
to
do
any
small
change
into
some
API
definition.
A
If
you
look
at
key
bernetta,
so
objects
they're
what
yamo
basic
Djamel
definitions
or
json
definitions,
you
shouldn't
be
too
hard
to
change
and
anytime
we
needed
to
manage
any
doing
new
type
of
resources.
For
example,
day
one
we
were
managing
row
bindings,
they
two
we
needed
to
manage
for
security
policies,
the
we
needed
to
add
a
bunch
of
code
to
make
this
happen.
A
Fortunately,
we
know
we
knew
what
we
wanted
to
get.
We
we
knew
that
policy
should
be
simple
to
understand
and
granular
policies
should
be
independent
from
one
another
should
back
up
all
of
our
releases.
A
change
in
policy
should
only
take
minutes
to
apply
to
all
the
clusters
if
need
to,
and
we
want.
This
is
important.
We
want
to
give
the
users
ability
to
define
their
own
policies,
we
as
a
platform
providers.
A
We
define
a
set
of
baseline
rules,
let's
cut
it
away
that
everybody
needs
to
follow,
but
we
understand
that
certain
applications
teams
might
have
special
names
like
running
privilege,
containers
phrases,
so
we
need
to
enable
them
and
empower
them
to
to
define
such
policies.
Of
course,
they
define
something
that
we
review
right.
A
However,
this
is
a
very
trim
down
version
of
the
old
controller
manager
which
no
knowledge
that
these
runs
on
GS
whatsoever
and
we
basically
offload
all
the
decisions
about
what
resources
should
be
created
in
the
cluster
to
appa.
So
here's
how
the
interaction
works,
controller
manager
will
continuously
get
the
namespaces
from
urban
areas
API
using
sharing
formers.
So
every
time
you
create,
update
or
delete
a
namespace,
Control
Manager
will
get
a
notification
after
it
does
is.
A
It
will
basically
ask
up
a
simple
question:
hey
what
resources
I
need
to
create
for
this
namespace
oppa
will
evaluate
a
bunch
of
rules,
and
you
will
ultimately
return
a
JSON
list
where
each
of
the
Jason's
represent
a
key
bernetta
subject
and
there
might
be
different
types.
Controller
manager
gets
situation
and
at
the
end
it
says
okay
I'm
going
to
apply
it
and
it
runs
a
simple
cube.
Ctl
apply
command
on
this
JSON
objects.
So
what
are
the
benefits
of
this
well
dynamism?
A
To
start
with,
we
can
change
policy
dynamically
without
having
to
roll
any
role
out
any
new
code.
Since
it's
continuously
being
applied,
the
cluster
state
will
be
enforced
every
so
often
they
depends
on
what
interval
we
define
and
it's
totally
configurable
and
then
well.
We
have
poly
the
all
of
our
policies
defined
in
code
in
some
sub
in
some
submergence
is
an
operation
version
control
system
that
we
can
run
unit
tests
for
and
finally,
we
can
better.
A
So
here's
a
basic
example
of
what
a
baseline
rule
would
look
like
and
I'm
not
gonna
go
too
much
into
it,
but
if
you
see
that
the
first
line
is
basically
validating
that
a
namespace,
what
is
valid
or
not,
we
don't
want
to
create
this.
For,
for
our
own
manage
names
namespaces
these
only
applied
to
application
namespaces,
and
if
it
is,
we
were
basically
going
to
create
a
resource
quota
like
that
looks
like
this
JSON
now
this
is
a
standard
resource
board
and
nothing
special.
A
The
real
special
part
is
that
the
actual
data,
or
the
actual
number,
of
course,
that
you
can
use
or
memory
is
gonna,
come
from
some
capacity
inventory,
which
oppa
will
have
on
its
data
cache.
Essentially,
let's
look
at
another
example
again.
This
is
an
example
where
we
have
teams
that
requires
something
extra
on
their
name
spaces
and
the
sukkah
row
bindings.
For
instance,
we
obviously
have
our
baseline
rule,
in
which
we
need
to
create
an
admin
row
binding
for
those
users
that
need
to
create
resources.
They
need
to
create
things
on
that
namespace.
A
So,
if
I'm
part
of
a
random
namespace,
that
is
not
part
of,
let's
call
it.
The
special
team,
X
I
will
just
get
this
place
an
object,
but
the
moment
that
the
namespace
belongs
to
one
of
the
namespace
from
a
special
team
X.
They
will
get
a
combination
of
these
of
these
of
these
two
objects.
How
does
that
happen?
Well,
it's
very
simple.
A
Oppa
has
this
very
cool
feature
in
which,
if
you
define
rules
that
have
the
same
name,
they
will
open
what
it
will
do
when
you
queried
it
will
evaluate
both
simultaneously
and
if
they
both
happen.
To
evaluate
to
true
you'll
get
two
objects.
If
one
of
them
evaluates
which
one
the
other
one,
doesn't
you
get
one
object?
A
So
in
this
case
of
the
application,
application,
X
you'll
get
the
base
baseline
object
again
and
you'll
get
a
special
row
binding
that
serves
your
business
needs
all
right.
So,
back
to
what
Tim
said
about
about
oh
about,
you
basically
need
three
things:
to
make
up
a
producer,
an
accurate
policy
decision.
You
need
both
policies.
Obviously
you
need
data
which
we're
gonna
divide
into
cuban
s.
Cluster
data
like
the
ingresses
namespace
and
deployments
are
you
have
etc,
and
you
have
your
unique
and
the
company
proprietary
data
right
all
of
these
external
inventory
systems.
A
Finally,
you
need
a
query
and
then
you'll
get
your
Apple.
So,
let's
start
with
a
policy
and
cue
Burnett
as
a
cluster
data
to
sing
this
to
these
two
pieces
of
data
into
into
a
power
using
this
tool
called
queue
management.
It
is
a
tool
developed
by
Stara,
completely
open
sourced,
with
the
sole
purpose
of
running
as
a
cycle
container
with
APPA
and
continuously
upload
policy
and
data
into
it,
using
a
standardized
API
on
localhost.
So
how
does
it
work
for
the
cubanelles
cluster
data?
A
You
basically
have
to
only
define
a
complete
a
command
line
argument
specifying
which
resources
you
wanna
continuously
sync
right,
and
you
can
control
this
via
the
deployment
channel,
for
open
key
management
and
for
the
policy
what
queue
management
will
do
is
it
will
look
at
any
name
space
of
your
choice
and
it
will
look
at
the
config
Maps
inside
of
it.
If
any
config
map
in
that
namespace
happen
to
have
Rago
compatible
syntax,
it
will
say:
ok,
these
are
policy,
I'm
gonna
upload
it
into
a
pass,
a
policy.
A
So
if
we
refer
to
the
length
of
the
picture,
imagine
this
guy
here
is
either
me
as
a
platform
provider
or
system
administrator
or
some
application
developer.
I
need
something
special.
They
can
simply
commit
their
policy
into
some
version
control
system.
In
this
case
we
use
a
key
lab
and
we
use
standard
gillip,
CCD
pipelines
to
push
all
these
policies
into
cube
analysis.
Config
Maps,
when
you
management
detects
that
there
is
a
new
config
map
or
something
has
been
updated.
It
say
AHA.
This
is
policy
of
loading
to
open
up
data
policy
right.
A
A
What
essentially
means
is
that
Appa
has
the
capability
of
querying
external
systems,
be
a
REST
API
these
external
systems,
the
only
thing
that
they
need
to
comply
with
is
to
reply
with
a
turbo
when
that,
when
OPA
get
said
it
will,
it
will
basically
enter
it
and
upload
as
either
data
or
policy
depending
on
what
it
or
what
it
is
of
on
what
that
turbo
has
so
okay,
so
we
need
a.
We
need
some
sort
of
observer
for
this
and
we
created
our
own.
We
created
our
own
component
call.
A
We
call
it
GS
feeder
that
it's
very,
very
simple.
The
only
thing
it
does
is
again
query:
Q
Benares.
We
are
sharing
formers
to
get
all
the
namespaces
that
we
have
in
our
clusters
and
then
for
as
many
inventory
systems
as
we
want
cosine
query
each
and
every
single
one
of
them
either.
We
are
bulk,
API,
requests
or
singles.
It
depends
on
this
on
each
inventory
system.
A
It'll
get
data
from
them,
they'll
put
them
in
some
sort
of
JSON
that
we
like
and
you'll
tell
them,
and
that's
it
you'll
serve
them
to
oppa,
so
every
every
so
often
oppa
will
last
years,
Peter
hey.
Do
you
have
any
data?
You
have
any
data.
Do
you
have
any
data?
If
something
has
changed,
GS
Peter
will
reply
with
the
full
scholar
bundle,
which
is
again
it's
a
toughie.
But
if
nothing
has
changing
will
reply,
nope,
there's
nothing
new
and
Papa
will
just
carry
along
all
right.
So
we
have
again.
A
Let's
recap:
we
have
policy,
we
have
Q
Benares
data,
and
now
we
have
our
own
proprietary
data.
The
last
step
is
query
and
provision-
and
we
have
explained
this
before
right.
We
use
the
trim
down
version
of
controller
manager
to
essentially
continuously
query
OPA
and
since
up
already
has
the
data
and
the
policy
that
we
need,
it
will
provide
a
response
to
controller
manager,
which
is
a
list
of
your
JSON
objects,
and
controller
manager
will
apply.
Those
objects
via
standard
cube,
CTL
apply
into
cue
burners.
A
So
some
things
to
note
about
this
approach,
one
it's
almost
completely
open-source.
If
you
exclude
the
left
part
of
this
picture,
there
is
not
special
about
these
right.
You
have
killaby
of
cuban
areas.
You
have
open
of
queue,
management
and
controller
manager,
we're
actually
thinking
on
open
source.
So
who
knows?
A
The
second
thing
is
sorry,
these
eventually
consistent,
we
don't
only
run
one
or
by
instance.
We
run
three
or
four
depending
on
the
cluster
right
just
for
scalability
purposes,
and
this
OPA
controllers
might
and
might
not.
Meyer
may
not
be
in
sync
at
all
times
right
because
they
all
get
data
from
this
GS
feeder
at
their
own
interval,
but
at
some
point
they'll
they'll
be
in
sync
with
each
other.
A
A
Another
takeaway
from
this,
and
to
me
the
most
important,
is
that
this
is
failure,
tolerant,
tolerant
it
doesn't
matter
if
any
of
the
GS
feeder
controller
manager
of
our
queue
management
dies.
Because
what
will
happen,
let's
say:
GES
feeder
queue,
management
die
OPA
either
won't
have
data
or
policy.
To
reply
with
every
time
we
query,
OPA
and
Apollo,
saying
on
the
data,
will
essentially
return
an
undefined
I.
Don't
know
what
to
do.
Control
Manager
will
look
at
this
one.
This
is
undefined
I,
don't
know
what
to
do
either.
I'm
not
gonna,
do
anything.
A
A
The
last
thing
to
know
about
this
is
that,
if
there
is
any
changes
in
the
environment
that
come
from
us
like
we
detect
that
we
need
to
change
something
in
policy,
they
will
again,
since
we
query
continuously,
it
will
be
applied
to
the
environment
within
two
to
five
minutes.
A
little
bit
about
metrics
are
monitoring
a
steam
mention,
op
exposes
promethium
promises,
metrics
that
you
can
graph.
A
You
can
do
things
like
round
three
time
by
HTTP
method,
garbage
collecting
time
how
your,
how
as
you're,
responding
to
requests
mean
API,
request
latency,
and
you
can
do
graphs
and
alerts
for
any
single
one
of
these
up
to
you
so
evaluation
of
result.
Okay,
we
had
a
problem,
we
did
something
they
actually
work
and
the
answer
is
actually
yeah
pretty
successfully.
We
now
have
five
minutes
turn
around
turnaround
time
on
global
policy
application.
So,
basically,
if
something
changed,
we
can
deploy
very
quickly
after
we
push
the
deploy
button,
I
mean
we
commit
our
policy.
A
We
push
the
deploy
button.
It
happens
within
two
to
five
minutes
on
every
namespace.
The
scale
that
we
have.
We
have
a
min
request,
round-trip
time
of
less
than
30
milliseconds.
There
are
24
rules
currently
in
our
environment,
meaning
that
every
time
we
query
namespaces
OPA
will
fire
24
rules
for
that
namespace,
potentially
returning
24
different
Cuba
NetID
subjects
that
doesn't
even
like
that
those
happen
all
the
time,
because
very
very
special.
We
have
an
average
data
size
of
like
one
megabyte
of
GS
data
and
2
megabytes
of
cubanelles
data.
A
Something
very
important
to
us
is
that
we
can
fix
any
issues
in
the
cluster
within
two
to
five
minutes
pertaining
these
objects.
Obviously,
maintenance
wise
is
not
needed.
I,
don't
look
at
this
at
all
unless
I
have
broken
something
on
that
GS
pillar.
That
I
wrote
myself,
however,
whenever
I
need
to
upgrade
the
OPA
or
the
queue
management
version,
but
it's
pretty
standard
updated,
make
sure
everything
is
working,
fine,
run
some
integration
tests
or
whatnot
and
release,
and
then
we
have
received
multiple
feedbacks
that
this
is
that
our
turnaround
times
have
gotten
better.
A
Overall,
everybody
is
quite
happy.
Alright,
some
of
the
lessons
learned
to
model
using
OPA
bundles
is
more
scalable
than
a
push
model
using
OPI.
We
used
to
have
a
push
model
while
we're
the
feeder
used
to
push
this
these
GS
data
into
OPA.
This
basically
was
not
scared
scaleable
for
us.
If
we
wanted
to
scale
open
key
management,
we
will
have
to
deal
with
issues
like
synchronization.
What
happens
if
I
know
the
controller
dies?
We
didn't
want
to
deal
with
that
in
our
case
waiting
two
to
five
minutes
for
namespace
provisioning.
A
It
was
more
than
obviously
create
metrics
and
secure
europa
deployments.
This
is
very
well
documented
in
the
style
of
documentation.
So
please
read
that
do
not
have
note
static
data
strago.
This
will
impact
your
policy
upload
times,
make
sure
you
performant,
performant,
use
and
unit
test.
Your
poll
is
as
much
as
you
can
and,
needless
to
say,
use
version
control.
You
see,
ICD
pipelines
or
your
friends
are
gonna
help
you
to
roll
out
and
roll
back.
Your
changes
when
you
need
to
some
of
the
future
works,
we're
still
expanding
on
this
provisioning.
A
Future
implementations
on
these
spaces
say:
if
we
feel
there
is
an
appetite
for
these,
we
might
consider
open
source
in
the
controller
manager,
which
will
mean
we
need
to
things
like,
maybe
creating
an
operator
for
to
set
up
all
of
this
architecture.
For
you
having
good
metrics
like
be
a
permutation
graph
and
make
sure
we
can
horizontally
scale,
this
controller
manager
etc.
D
A
D
A
B
D
E
A
Let's
say
a
new
name:
space
was
created
and
this
is
name
spaces
call
Miguel.
This
happened
to
somehow
tie
down
to
GS
inventory
systems.
Right
they'll
be
information
about
me.
Go
in
there.
The
namespace
me
go,
Jesus
fira
will
get
this
update
from
the
Cuban
s
API
and
it
will
say:
okay
I
need
to
query
the
group
roles
inventory,
the
capacity
inventory,
the
storage
inventory
and
others.
However,
how
many
we
configure?
Let's
say
we
do
four
groups.
A
It
will
basically
ask
the
arrests,
API
to
our
groups,
inventory
again
its
proprietary,
it's
our
own
input
output
format,
but
it's
a
Jason
right,
Jason,
Sena,
Jason,
say:
hey
the
namespaces
Miguel.
Tell
me
what
information
do
you
have
on
your
records
for
this?
The
group
rolls
inventory
will
respond.
Well,
the
admins
for
the
namespace
Miguel
are
going
to
be
Miguel
on
team
and
the
viewers
are
going
to
be
here.
No
Jack
and
Rodrigo
right,
we'll
get
that
information,
we'll
find
it
once
we
get
it
from
this
inventory
systems
into
whatever
JSON
format.
A
We
think
we
want
to
query
in
our
policies
and
will
basically
turn
that
up
into
into
a
tar
file.
At
that
point,
like
he
just
sits
there
right.
This
feeder
is
a
server.
Oppa
will
query
periodically
his
feeder
service
and
he
will
detect
that
a
change
has
happened.
Right
like
this
is
not
the
same
tired
that
I
had
before
essentially
and
you'll
pull
it
when
he
pulls
it
and
say:
okay,
there
is
new
data.
A
A
Sure,
let's
say
that
happens
first
of
all
in
our
environment.
That
really
cannot
happen
because,
before
you
create
a
namespace,
it
needs
to
be
in
the
inventories.
I
like
the
our
provisioning
work,
like
our
orchestration
workflow,
don't
won't
let
you
do
that,
but
let's
say
it
happens.
What
it
will
happen,
then,
is
that
when
controller
manager
queries
OPA
about
namespace
Miguel,
which
is
not
every
anywhere
OPA,
will
try
to
run
the
policies
and
it
will
say:
well,
you
know
what
I
don't
have
the
data
that
I
need
to
compute
this
policy,
I'm
gonna
return
undefined.
A
So
at
this
point,
what
remedy
will
say?
Oh
I,
don't
know
what
to
do
with
this.
I'm,
not
gonna,
create
any
object.
So
how
would
you
fix
this?
Well,
you
actually
put
your
data
in
this
inventory
system
and
as
this
thing,
this
thing
is
self-healing.
The
field
will
continuously
ask
about
names.
Face
Miguel
ear
is
often
so
when
that
data
becomes
available
in
the
inventory
system,
Peter
will
get
it.
Oppa
will
get
it,
and
next
timing
require
it
be
a
controller
manager.
Your
your
resource
will
be
provisioned.
A
A
What
is
queue
management
doing
with
us
sure
so
there?
The
role
of
queue
management
is
to
sync
policies
right:
let
the
feeder,
basically
syncs
GS
data,
just
specific
data
queue
management,
basically
syncs
policies
that
are
defining
config
maps
in
our
inactive
NS
namespace,
and
it
also
syncs
Hueber
net,
a
cubanelles
specific
data
that
we
need.
Some
of
our
policies
required
to
know
the
current
state
of
the
cluster,
like
the
one
that
team
showed
about
the
ingress,
for
instance,
that
needed
to
know
like
which
ingresses
are
in
the
cluster
and
what
host
names
they
have
right.