►
From YouTube: Design Patterns for Extendable, Scalable K8s Extensions - Max Smythe, Google & Rita Zhang, Microsoft
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Design Patterns for Extendable, Scalable K8s Extensions - Max Smythe, Google & Rita Zhang, Microsoft
OPA Gatekeeper is a customizable Kubernetes admission webhook that helps enforce policies and strengthen governance. The Gatekeeper project is capable of dynamically creating, managing and destroying new custom resources which are used to customize the webhook. This unique model has led to some interesting design patterns. In this talk, Gatekeeper's maintainters explore the CRDs-that-create-CRDs and leaderless horizontal scalability design patterns that allowed us to create an extendable, scalable Kubernetes extension.
https://sched.co/ekBY
B
We
are
both
maintainers
of
the
gatekeeper
project
and
today
we
wanted
to
take
a
look
at
some
of
the
design
patterns
underlying
gatekeeper
and
how
they
allowed
us
to
create
an
extensible,
scalable,
kubernetes
extension,
because
there
are
a
lot
of
topics
to
cover.
This
talk
will
mostly
cover
high-level
concepts.
A
B
Gatekeeper
supports
both
audit
time
and
admission
time
checks,
but
today
we're
going
to
focus
on
admission
time,
specifically
serving
the
kubernetes
admission
web
hook.
This
diagram
is
a
high
level
model
of
how
gatekeeper
serves
admission
requests
when
something
makes
a
request
to
the
api
server.
So,
for
example,
a
user
making
a
cube
cuddle
request.
B
The
api
server
receives
that
request
and
sends
an
admission
request
to
gatekeepers
validating
webhook.
It's
asking
gatekeeper,
hey.
Should
I
allow
this
request
to
proceed
right
and
gatekeeper
must
answer
either?
Yes
or
no,
and
to
do
this
gatekeeper
consults
its
policy
configuration
which
is
the
large
gray
box
at
the
bottom
of
the
slide.
B
The
green
objects
that
are
labeled
constraint
tell
the
web
hook
what
checks
an
admin
wants
to
perform
and
constraints
rely
on
templates,
which
are
the
blue
documents
to
tell
the
system
exactly
how
to
perform
the
check.
For
example,
they
might
say
labels
live
in
metadata.labels,
so
that's
where
you
should
validate
labels.
A
The
notions
of
gatekeeper,
the
web
hook
constraints
and
templates
underlie
gatekeepers
philosophy.
That
policy
is
a
team
effort.
Separating
these
roles
into
discrete
areas
allows
for
specialization
where
work
in
each
area
benefits
the
others
increasing
its
scale.
For
example,
a
change
to
gatekeepers
platform
has
the
potential
of
increasing
the
utility
of
all
templates
across
the
board,
with
zero
extra
work
on
the
template.
Authors
part.
B
A
A
Our
goal
for
ammons
is
that
they're
able
to
define
different
types
of
constraints
that
share
universal
behavior.
For
instance,
all
constraints
can
use
label
selectors.
We
do
this
by
creating
a
parent
class
for
the
constraint
that
implements
universal
behaviors
template
authors
are
able
to
create
subclasses
by
injecting
their
enforcement
logic
and
its
function.
Signature
by
declaring
a
constraint
template
resource
the
constraint
template
is
combined
with
this
universal
behavior
to
create
a
new
constrained
kind.
How
does
this
look
inside
gatekeepers
code
when
gatekeeper
receives
an
admission
request
from
the
api
server?
A
B
B
B
The
first
was
that
we
had
inefficient
memory
usage
because
the
controller
runtimes
watch
cache
was
duplicated
across
the
two
managers
and
a
larger
problem
for
users
was
that
we
needed
to
use
finalizers
so
that
we
could
catch
delete.
Events
that
may
have
been
missed
because
the
sub
manager
was
restarting
and
we've
since
been
able
to
move
on
from
this
specific
model,
because
oren
shamron
wrote
a
dynamic
watcher
that
allows
us
to
change
the
set
of
watched
resources
without
restarts,
cache
duplication
or,
of
course,
finalizers.
A
Let's
zoom
in
a
bit
and
discuss
how
we
interact
with
this
dynamic
watch
manager.
Gatekeeper
has
two
sets
of
dynamic,
watches
one
for
constraints
and
the
other
to
sync
arbitrary
data.
For
each
one
of
these
we
have
a
main
controller
and
a
dynamic
controller.
The
main
controller
is
responsible
for
telling
the
dynamic
controller.
What
to
watch
the
dynamic
controller
is
a
generic
controller
that
can
understand
a
duct
type
resource.
A
A
A
The
watch
manager
can
then
take
the
union
of
all
the
intents
across
all
registers,
to
figure
out
what
resources
to
watch
by
adding
a
layer
of
indirection
and
name
spacing
intent.
We
have
made
it
significantly
easier
to
write
multiple
dynamic
controllers
that
watch
potentially
overlapping
sets
of
resources.
B
Now
we've
touched
a
lot
on
meta
topics:
right
controllers,
that
control
controllers
crds
that
create
crds.
You
know
ducts
that
walk
like
constraints.
We
could
take
this
process
to
the
logical
conclusion
right.
We
probably
should
we
should
go
full
meta
all
right.
So
let's
take
a
look
at
the
policy
enforcement
as
a
phenomenon
right.
It
follows
a
pretty
standard
pattern.
B
B
A
So
here
we
have
the
constraint
framework
constraint
framework
is
the
library
that
underlies
gatekeeper.
It
coordinates
all
of
the
doctyping
logic
we've
covered
so
far.
It
provides
the
execution
flow
gatekeeper
uses
to
render
a
decision
to
the
api
server.
It
also
provides
two
abstractions
that
allow
us
to
define
constrained,
templates
and
constraints
enforcement
points
and
targets.
A
B
A
B
A
B
B
A
B
That
means
they
need
to
be
responsive
and
therefore,
generally
intolerant
of
downtime
and
web
hooks
also
scale
their
availability
and
serving
capacity
by
increasing
the
number
of
serving
pots,
and
each
of
these
pods
are
peers,
which
means
there's
no
leader
or
follower
relationship,
and
everything
is
a
flat
hierarchy
where
every
pod
is
equally
able
to
serve
any
given.
Request
controllers,
on
the
other
hand,
observe
and
reconcile
resources
and
come
together
to
create
an
eventually
consistent
system.
B
B
A
Because
of
this
tension
between
how
web
plugs
and
controllers
usually
scale,
it
seems
like
they
may
be
incompatible
models.
This
apparent
conflict
can
be
resolved
by
observing
that
idempotent
controller
processes
don't
need
to
be
singletons
if
more
than
one
controller
is
watching
the
same
resource
and
they
both
agree
on
the
end
state.
The
first
controller
to
write
the
output
will
win.
The
right.
A
Other
controllers
will
either
have
not
yet
processed
that
resource
or
will
have
their
rights
rejected
on
retry.
Those
controllers
will
see
the
correct
state
and
will
not
attempt
to
reconcile
further.
This
is
similar
to
how
kubernetes
leader
election
works.
This
can
lead
to
some
extra
traffic,
but
only
when
controllers
need
to
write
a
change.
B
Now
there
are
some
limitations
this
model
imposes
on
us
right.
We
need
to
be
sure
that
non-idempotent
operations,
like
audit,
run
in
a
separate
singleton
pod.
We
also
should
avoid
scaling
right
contention
quadratically
with
the
number
of
pods,
which
can
happen
if
each
pod
needs
to
write
its
own
pod
specific
state.
A
B
B
If
we
have
three
web
hook,
pods
to
enforcing
a
new
policy
one
not
and
an
api
server,
that's
going
to
choose
a
web
hook
bob
randomly
the
policy
only
really
has
a
66
percent
percent
chance
of
being
enforced
and
in
order
to
reason
about
whether
a
constraint
is
enforced.
Therefore,
we
need
to
know
whether
it
is
recognized
by
all
pods.
A
B
B
A
So,
in
order
to
make
these
observations
more
meaningful,
we
need
to
enforce
some
invariants
in
our
project,
one.
If
we
expect
to
have
n
pots,
we
must
never
have
n,
plus
one
pot.
Otherwise,
counting
in
observations
leaves
the
possibility
that
there
is
still
one
pod
that
has
not
observed
a
new
resource.
A
Two
pods
cannot
serve
until
they
have
blue
strapped
all
resources
present
during
startup.
Otherwise,
a
new
pod
would
put
the
assumption
that
n
observations
means
in
enforcing
pots
into
question,
and,
lastly,
we
must
design
our
resources
such
that
a
missing
resource
has
a
known
impact
on
the
system.
In
this
case,
a
missing
constraint
means
that
policy
is
enforced
more
loosely
than
it
would
otherwise
be.
A
We
should
also
know
here
that
writing
referential
constraints,
which
rely
on
cash
data,
potentially
violates
this
principle.
For
one
thing
whether
data
has
been
cached
is
unreported
for
another,
it's
impossible
to
know
the
significance
of
missing
data
without
knowing
the
specifics
of
a
constraint
logic.
B
B
A
A
If
we
assume
one
serving
pot
is
sufficient
to
serve
all
inbound
traffic
and
that
each
pot
fails
independently
of
every
other
pod.
The
probability
of
downtime
decreases
exponentially
with
the
number
of
running
pods.
On
the
other
hand,
because
we
only
consider
a
constraint
as
being
enforced
when
all
pots
have
observed
it,
we
lengthen
the
mean
time
to
enforcement,
as
reported
by
the
system
as
a
whole.
Unfortunately,
without
knowing
the
exact
distribution
of
ingestion
times,
it's
hard
to
say
how,
by
how
much.
B
A
And
with
that,
thank
you
for
attending
this
session.
We
just
want
to
thank
the
gatekeeper
community,
all
the
users,
all
the
contributors
for
your
feedback
and
all
the
feature
requests
to
make
gatekeeper
the
project
it
is
today,
and
we
also
want
to
send
our
thank
you
to
the
cube
builder
controller
runtime
community
for
all
the
awesome
work
that
has
bootstrapped
the
gatekeeper
project
and,
last
but
not
least,
our
wonderful
audience
for
attending
this
session.