►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Managing Kubernetes Webhooks with SPIRE - Faisal Memon, F5 Networks
Webhooks are one of the most useful resources in Kubernetes. They can be used for admission control, validation, and injecting sidecars into Pods. The are also one of the most challenging to operate. You have to manage and rotate a certificate, key, and root CA certificate. With some new features in SPIRE you can now automate certificate management for Webhooks like any other workload. The SPIRE issued certificates for Webhooks are tied into the same trust chain as regular workload certificates.
A
All
right,
so
I'm
really
glad
to
be
back
here.
First
off,
you
know
doing
an
in-person
presentation,
in-person
conferences,
it's
been
so
long
and
it's
it's
one
of
those
things.
I
never
thought
I'd
missed
until
it
was
taken
away.
So
I'm
just
really
glad
to
be
here,
I'm
actually
going
to
take
off
the
mask
for
a
bit
a
lot
easier
to
talk.
I'm
happy
to
be
here
talking
about
this
subject:
managing
kubernetes
web
hooks
web
hooks
have
been
kind
of
a
thorn
on
my
side.
A
So
I'm
faisal
lemon,
I'm
a
software
engineer
at
f5
networks.
Previously
I
worked
at
nginx
and
nginx
was
acquired
by
f5.
Nginx,
of
course,
is
the
open
source
web
server
load,
balancer
solution
that
a
lot
of
you
guys
are
familiar
with
and
I'm
sure
are
using
in
your
deployments
the
picture
there.
Of
course
my
daughter
salma
and
our
cat
marley.
A
A
So
let's
go
ahead
and
dig
into
it.
So
a
little
bit
of
a
refresher
on
the
web
hook
side
so,
first
off
what
is
a
web
hook
so
in
the
world
of
kubernetes
a
web
hook
is
a
callback
and
it
runs
before
kubernetes
objects
are
created,
updated
deleted,
so
any
sort
of
modification
of
the
kubernetes
state
a
web
hook
runs
and
then
once
that
rep
hook
runs,
you
have
the
opportunity
to
do
something,
and
so
what
is
that
something
that
you
can
do?
And
they
typically
fall
into
these
two
buckets?
A
A
A
very
common
use
case
of
the
validating
web
hook
is
in
the
custom
resource
definition
or
crd,
and
when
you
create
a
crd,
you
obviously
have
your
custom
resource
and
there's
custom
rules
as
to
what's
acceptable
and
what's
not
acceptable
and
so
you're
validating
webhook.
You
have
the
opportunity
to
reject
invalid
configs.
A
A
A
So
what
makes
webhook
so
challenging
and
I
think
the
first
one
really
is
that
you
have
to
provide
a
certificate,
a
key
as
well
as
a
root
ca
certificate.
So
you
have
to
provide
all
three
of
these
pieces
of
information,
two
kubernetes
and
they
all
have
to
be
kept
up
to
date,
kept
fresh
rotated
and
it's
up
to
the
operator
to
do
that.
A
The
certificate
and
key
have
to
be
saved
on
disk,
which
is
another
challenge,
move
towards
more
in-memory
certificates.
The
web
hook
system
still
requires
certificates
to
be
on
disk
and
that's
another
challenge,
another
step
in
the
process,
and
if
anyone
knows
how
to
do
web
hooks
without
storing
stuff
on
disks,
I
would
love
to
hear
how
you
accomplish
that
and
then
the
root
ca
certificate
has
to
be
provided
into
a
kubernetes
object.
The
validating
web
hook
or
the
mutating
web
configuration
in
the
ca
bundle
field,
and
you
have
to
keep
that
up
to
date.
A
A
A
We're
guilty
of
doing
this
as
well
internally,
not
the
best
way
to
do
it,
but
it
does
work
challenges.
There
is
manually
you
have
to
rotate
it
manually,
but
if
you
keep
the
expiration
date
like
in
the
year,
3000-
probably
not
a
big
problem
there,
but
the
biggest
problem,
of
course,
is
that
if
those
certificates
leak,
it's
a
security
liability.
How
do
you
revoke
it?
A
The
standard
solution
that
people
use
for
managing
kubernetes
web
hooks
is
cert
manager.
If
you
follow
the
coup
builder
book
and
the
steps
they
put
in
there,
they
have
a
section
on
cert
manager
and
how
to
use
cert
manager
to
manage
kubernetes
web
hooks.
It
does
the
auto
rotation
for
you,
which
is
great.
A
A
lot
of
you
know
a
new
tool
to
learn
new
logs,
and
so
we
wanted
to
avoid
going
down
that
path,
and
so
the
solution
that
we
came
up
with
in
working
with
this
fire
team,
the
contributors
and
maintainers,
is
to
use
fire
to
manage
certificates,
and
the
solution
that
we
have
on
spire
is
based
a
lot
on
how
strip
manager
does
it
as
well
so
moving
forward?
How
do
we
actually
use
spire
to
manage
those
certificates?
A
There's
three
basic
steps
that
we
need
to
do
and
we'll
go
through
them
in
detail.
The
first
is
that
we
need
to
keep
that
root.
Ca
certificate
fresh
within
the
validating
web
hook,
mutating
webhook
configuration,
let's
see,
bundled
field,
so
we
have
to
have
a
way
of
rotating
and
maintaining
that
field,
and
then,
of
course,
we
need
to
create
an
entry
on
the
spire
server
for
the
actual
webhook
pod.
A
Okay,
the
first
part
of
the
solution
is
keeping
the
ca
bundle
fresh.
This
is
actually
one
of
the
easier
parts
of
it
and
the
way
we
do
that
is
using
the
k-8s
bundle,
notifier
plug-in,
and
so,
if
you're
not
familiar
with
this
plug-in,
what
it
does
is
it
takes
the
spire
root
ca
certificate
and
pushes
it
into
a
config
map
which
then
you
can
use
to
bootstrap
your
spire
agent.
A
We
push
that
root
ca
certificate
also
to
the
web
hook
configuration
and
so
the
configuration
is
webhook
label
and
then
the
enable
or
sorry
the
name
of
the
label
that
you
want
to
use
and
then
what
spire
server
is
going
to
do
is
going
to
filter
on
that
label
and
pods
or
sorry,
not
pods
web
hook.
Configurations
with
that
label
a
minute
spire
will
manage
the
ca
bundle
field.
A
So
the
first
box
there
shows
a
spire
configuration,
so
you
just
specify
the
actual
label
you
want
to
use
and
then
the
second
box,
with
the
actual
web
hook
you
put
in
in
the
label
field,
whatever
label
you
specified
in
this
case,
is
spiffy
io,
slash,
webhook,
coal
and
true,
and
then,
if
that
is
there,
when
you
deploy
the
validating
webhook
configuration,
you
just
keep
that
ca
bundle,
field
clear
and
then
once
it's
deployed
we
have
a
watcher
on
the
spire
server
it
filters
on
that
label.
So
anything
with
that
label.
A
Spire
is
going
to
push
the
root
ca
certificate
into
the
ca
bundle
field.
It
rotates
and
manages
it
for
you.
So,
whenever
the
root
ca
certificate
is
about
to
expire,
it
goes
through
all
the
validating
web
hooks
mutating
web
hooks
that
have
that
label
set
and
rotates
the
ca
bundle
for
you
and
on
startup.
If
you
deploy
the
validating
web
hook,
first.
A
A
So
that's
step
one,
and
so
that
manages
the
root
ca
field
within
the
webhook
configuration.
The
next
thing
we
need
to
do
is
create
an
entry
on
the
spire
server.
There's
two
ways
you
can
do
that
you
can
just
do
it
manually.
You
can
go
onto
the
spire
server
cli
and
just
do
the
create
entry
as
you
normally
would.
A
A
The
kubernetes
api
server,
expects
that
dns
name
to
be
in
the
certificate
other
than
that.
Whatever
you
create,
your
entry
can
be,
you
know
whatever
you
normally
create,
whether
it's
pod
name
pod,
uid
whatever
it
is
that
you
need.
So
as
long
as
the
pod
is
associated
with
that
entry,
you
can
also
use
the
kubernetes
workload
registrar
and
that
automatically
will
also
add
that
dns
name
that's
required
to
the
certificate
for
you,
the
crd
version
of
it
has
that
configuration
enabled
by
default.
A
B
A
Created
the
entry,
the
most
complex
part
of
the
the
whole
solution
is
saving
the
certificate
key
to
disk.
There's
a
lot
of
code
here.
Just
first
thing
to
keep
in
mind,
though,
is
this
is
just
based
on
the
ghost50
example:
the
watcher
example
sorry
and
the
ghost50
v2
directory.
So
what
you're
doing
here
is
we're
creating
a
watcher
that
watches
for
new
certificates
using
the
ghost50
client
and
then,
when
the
new
certificate
comes
in,
we
save
it
to
disk.
A
B
A
A
A
Okay,
so
that
starts
up
the
ghost
fifty
client.
The
thing
to
keep
out
for
one
more
thing
to
keep
out
for
is
this
x
509
watcher
field,
so
we're
passing
the
directory
that
we're
saving
the
certificates
to
the
next
part
of
this
slide,
which
is
the
watcher
struct,
and
so,
if
you're
familiar
with
the
ghost50
client.
A
What
happens?
Is
that
every
time
that
there's
a
new
certificate,
this
on
context,
x509
x,
onyx,
509
context,
update,
gets
called,
and
so
what
we're
going
to
do
is
when
that
gets
called.
All
we
do
is
just
write
the
actual
s-vid
out
to
disk,
and
so
that's
also
just
kind
of
encapsulated
in
this
library
function,
but
it
just
takes
extracts
the
default
s-fid
and
then
just
saves
it
to
disk
in
the
directory.
You
specify
the
file.
A
Names
are
static,
tls.crt
tls.key,
because
every
webhook
api
that
I've
seen
in
kubernetes
just
requires
it
to
be
on
those
files
and
the
only
function
or
only
configurability.
You
have
is
actually
so
you
just
pass
in
the
directory,
and
then
we
save
it
here
to
those
file
names
and
the
webhook
server
to
do
it
and
then.
B
A
A
Another
option
that
I
think
would
be
much
more
clean
and
nicely
integrated
is
just
to
have
spire
save
the
certain
key
to
a
kubernetes
secret
object,
and
then
your
pod
mounts
that
secret
object
volume
mounts
it
and
then
the
certificates
will
live
on
disk
because
that's
how
the
secrets
are
mounted,
so
it
would
work
out
nicely
and
that
would
also
parallel
what
the
cert
manager
does.
Cert
manager
saves
the
key
and
disk
into
a
certificate
crd,
which
is
basically
a
secret
object.
So
it'll
be
a
nice
nice
parallel
there.
A
A
A
And
so
where
all
this
is
coming
from
is
our
nginx
service
mesh?
So
that's
what
we're
working
on
at
f5,
so
we
embed
spire
and
deploy
it
as
part
of
the
service
mesh.
We
deploy
the
spire
server
the
agent
and
then
we
use
spire
to
manage
all
aspects
of
identity,
mtls,
webhook
certificates,
api
service
certificates,
secure
nas
for
communication,
secure
communication,
sorry,
secure
control,
plane,
communication
we
secure
through
nats
using
certificates
provided
by
spire.
B
C
A
A
B
C
A
Oh
so
the
three,
the
three
minute
timeout
is
for
just
one
spire
delivers
the
certificate
to
the
workload,
and
so
that
hasn't
come
up.
I
mean
it
hasn't
been
a
problem
for
us.
The
testing
that
we
did
was
just
to
see
that
we
put
like
a
short
lifetime
on
the
ca
bundle
and
then
to
make
sure
that
it
keeps
keeps
rotating,
but
we
haven't
done
a
whole
lot
of
testing
if
we
have
like
5
000
web
hooks
or
anything
like
that,
we
haven't
tested
at
that
scale.