►
From YouTube: Customizing Kustomize with Client-Side Custom Resources - Katrina Verey, Apple & Jeff Regan, Google
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Customizing Kustomize with Client-Side Custom Resources - Katrina Verey, Apple & Jeff Regan, Google
Kustomize is a popular tool for declarative, template-free management of Kubernetes configuration. Under the hood, it works by defining a graph of Kubernetes resources and applying a pipeline of declaratively specified transformations to its nodes. A lesser known fact is that users aren't limited to the transformations that Kustomize comes with out of the box: it's extensible! The way Kustomize extensions work has evolved over time towards a model that closely resembles Custom Resources, but on the client side! In this talk, Katrina and Jeff will briefly review the history and future of Kustomize extensions, and then dive into practical applications. The talk will showcase how these CR-style extensions can address common yet idiosyncratic application needs, as well as how platform builders can leverage them to manage declarative configuration at scale. Attendees will leave with practical knowledge of what makes a good extension and how to use Kustomize’s kyaml tools to start building.
B
A
A
A
Okay,
hello,
quick
housekeeping
items
at
the
end
of
the
session.
You
may
raise
your
hand
and
ask
you
a
question.
I
will
be
running
around
with
microphone
and
helping
to
answer
the
questions.
If
you're
watching
us
online,
you
can
also
submit
your
questions
in
the
q,
a
box
and
we'll
make
sure
your
question
is
answered.
Thank
you.
C
Thank
you
thanks
for
the
introduction,
okay,
so
I'm
jeff-
and
this
is
katrina-
we're
going
to
talk
about
customizing
customize
with
custom
resources
thanks
to
the
cncf
for
bringing
us
all
here
to
I'm
really
glad
to
be
at
a
live
event.
This
time
around
the
talk's
gonna
have
three
sections
we're
gonna
talk
about
what
is
customized,
at
least
in
the
context
of
extending
it.
B
C
C
It
is
a
set
of
go
modules
right
and
it
is
also
a
command
in
coupe
cuddle.
So
back
in
late
2009
or
early
2019,
late,
2018,
brian
grant,
phil
wittrock
and
others
noticed
that
there
were
many
missing
features.
Outstanding
issues
include
cuddle:
that
kind
of
centered
around
doing
edits
to
the
client
side,
data,
either
edits
or
creating
data,
as
opposed
to
what
kubecuttle
normally
does,
which
is
edit
the
live
cluster.
C
So
we
wanted
to
fix
or
provide
a
fix
for
these
features,
but
it
was
complicated.
We
felt
it
was
kind
of
big,
so
we
decided
to
do
it
outside
of
the
contents
of
cube
cuddle,
because
we
figured
we'd,
probably
throw
away
a
few
versions.
So
this
resulted
in
some
modules,
which
are
now
called
by
customizers
there's
a
relatively
thin
cli
and
the
same
modules
are
called
by
a
coupe
cuddle,
so
customizes
configuration
stream
editor
works
with
kubernetes
yaml,
as
I
just
said.
It's
also
very
good
with
working
with
variants.
C
Now
everybody
has
multiple
environments,
multiple
clusters,
multiple
namespaces,
maybe
multiple
namespaces
and
clusters
and
customize-
is
really
good
at
expressing
the
differences
between
these
different
environments,
if
you've
got
production,
staging
and
development,
and
they
all
share
like
90
of
it
in
common
customize,
allows
you
to
succinctly,
describe
the
differences
between
these
different
environments
and
it
plays
next
we'll
get
so.
It
can
read.
Information
from
git
and
you're
encouraged
to
write
the
information
back
into
a
git
repository
and.
C
Which
is
what
the
core
of
this
talk
is
going
to
be.
So
what
are
the
guiding
principles
when
you're
thinking
about
customize
and
thinking
about
writing
extensions
first
configuration
is
at
rest
is
just
data.
It
should
be
directly
usable
by
kubernetes,
so
we
issue
templating,
we
eschew
domain
specific
languages.
We
just
want
to
see
kubernetes
ammo
second
edits
or
generating
again.
What
should
be
done
that
that
operation
should
be
described
itself
using
kubernetes
example.
So,
no
matter
where
you
look,
you
see
kubernetes
example.
C
Not
too
close
okay,
so
this
is
an
example
of
a
layout
for
a
very
simple
customized
situation.
Where
you've
got
three
you've
got
a
three
normal
kubernetes
files.
You
want
to
make
edits
to
them,
you
you
can
edit
them
with
vi
or
emacs
or
whatever,
or
you
can
drop
a
customization
file
in
there
to
describe
the
edits
that
you'd
like
to
perform,
and
this
makes
sense
in
in
many
different
contexts
which
we're
not
going
to
go
into
today.
C
The
customization
file
is
the
center
file
on
this
screen.
It's
saying,
add
a
name
prefix
add
some
labels,
do
it
to
the
service.yaml
file
so
that
the
service.yaml
file
on
the
left
comes
out
standard
out.
Looking
like
the
on
the
left,
your
left
comes
out,
looking
like
the
file
on
the
right,
so
you
invoke
this
by
writing.
Customize
build
and
then
the
path
to
the
folder
that
has
a
customization
file
in
it.
C
C
C
It
reads:
the
ammo
from
disc
runs
generators
to
create
more
ammo
feeds
all
that
yaml
into
a
pipeline
or
transformers,
and
some
of
those
transformers
are
listed
up
on
the
right.
There.
You've
got
a
patch
transformer
for
you
can
do
json
patching.
You
can
do
strategic,
merge,
patching,
you
can
change
the
namespace
modify
the
names.
C
So
getting
down
into
how
we're
going
to
extend
customize
here
are
two
customizations
files
that
do
exactly
the
same
thing
on
the
left
is
a
short
customization
file
with
one
field
in
it
that
actually
does
stuff.
It's
the
name,
prefix
field
and,
it
says,
add
a
name
prefix
bob
hyphen
and
that's
going
to
add
that
name
prefix
to
all
of
the
resources
in
the
view
of
the
customization
units.
In
this
per
view
on
the
right,
there's,
no
name,
prefix
field,
there's
a
transformers
field,
and
that
has
a
list
of
transformer
specifications.
C
C
They
could
also
add
kind
specifications
to
limit
the
transformation
to
only
apply
to
say
deployments
or
cron
jobs
and
leave
ingress
and
services
alone.
So
you
get
the
full
power
by
using
the
specification
on
the
right,
it's
just
more
verbose.
So
at
90
of
the
time
people
are
going
to
use
the
thing
on
the
left,
but
this
is
the
pathway
to
expansion.
C
So,
on
the
left,
you
see
the
same
thing
again.
That's
the
prefix
suffix
transformer.
It
represents.
There's
a
chunk
of
code
specified
there
and
there's.
How
do
you
configure
that
chunk
of
code
to
run
and
on
the
right?
There's
a
proposed
custom
resource?
That's
doing
the
same
thing:
it's
specifying
a
chunk
of
code,
in
this
case
a
containerized
image
and
there's
going
to
be
data
in
that
custom
resource
spec,
telling
that
image
telling
that
code.
What
to
do.
C
So
it's
helpful
for
us.
It's
helpful
for
me
anyway
to
think
about
how
what
problems
we're
going
to
solve,
and
we
with
customer
resources
or
extensions,
and
we
put
those
into
three
different
categories:
one
is
generation
and
then
transformational
validation
with
generation.
You
might
take
it.
You
might
want
to
create
a
config
map
from
a
basic
amv
file.
You
might
want
to
create
a
kubernetes
secret
by
exchanging
your
single
sign-on
token
for
some
passwords
that
you're
pulling
from
hashicorp
vault
or
you
might
want
to
do
something
really.
C
This
is
like
a
transformer
that
doesn't
do
any
transformation.
It
just
simply
fails
with
good
error
messages,
telling
you
what
you've
done.
This
might
be
a
way
to
say
constrain
the
number
of
replicas
to
some
bound
okay.
So
this
slide
is
meant
to
trigger
just
ideas
about
the
kinds
of
extensions
you
might
want
to
write.
C
B
B
Most
of
these
come
from
the
fact
that
extension
follows
so
exactly
in
the
footsteps
of
core
customized
features
as
we're
just
seeing.
This
means
that
the
users
of
your
extension
will
retain
customized
properties
that
they
know
and
love.
That
is,
they
retain
a
purely
declarative
configuration
that
is
fully
encapsulated
by
their
customization
directly
directory,
and
they
can
use
the
familiar
customized
resource
model
exclusively
in
making
their
edits.
B
B
Well,
there
are
many
benefits
to
moving
more
of
your
configuration
management
to
the
client
side
and
therefore,
earlier
in
your
software
development
life
cycle,
a
really
obvious
one
is
that
client-side
custom
resources
naturally
do
not
require
you
to
install
anything
in
your
clusters,
which
can
be
a
huge
benefit,
especially
in
some
multi-tenant
situations
where
crd
installation
might
be
not
desired
or
not
even
possible
from
an
abstraction
designer
perspective.
There's
also
an
advantage
to
the
fact
that
the
end
user
retains
control
over
the
final
result.
B
B
If
you
do
it
on
the
client
side,
your
end
users
are
going
to
be
able
to
use
the
familiar
language
of
customize
to
modify
the
deployment
arbitrary
fields
of
that
deployment.
Even
after
you
generate
it,
and
you
don't
have
to
require
you
don't
have
to
expose
any
additional
fields
to
let
them
make
those
arbitrary
modifications,
so
your
abstraction
stays
tight,
and
yet
they
get
the
power
to
modify
whatever
they
need
related
to
that
once
you
have
the
results
of
your
extension
and
the
modifications
that
your
end
users
make.
B
All
of
that
is
going
to
still
end
up
as
gamble
in
your
source
control
if
you're
using
github's
best
practices.
Of
course,
this
provides
greater
auditability
and
change
management,
change,
management
capabilities
and
then
server-side
implementation
of
the
same
thing
would
and
then.
Finally,
the
fact
that
we're
fully
finalizing
our
configuration
on
the
client
side
also
means
that
any
problems
with
it
can
be
surfaced
earlier
in
the
development
life
cycle.
That
is
say
they
can
be
surfaced
even
before
commit
or
maybe
nci.
B
So
that
sounds
great
right:
let's
shift
everything
to
the
client.
Well,
no,
there
are
many
use
cases
for
server
side,
custom
resources
that
are
really
not
suitable
for
being
ported
to
the
client
side,
notably
server
side.
Custom
resources
should
be
completely
side
effect,
free,
which
means
that
they
can
only
be
used
for
abstractions
that
simply
resolve
to
a
set
of
resources.
B
For
example,
let's
say
you're
designing
a
custom
resource,
that's
going
to
cause
a
database
to
be
configured
in
a
third-party
cloud
somewhere.
Well,
that
is
an
enormous
side
effect
and
it
should
definitely
not
be
happening
from
a
client
side
cr
also,
although
customized
extensions
are
really
great
way
to
validate
kubernetes
resources,
they're
generally
not
suitable
for
doing
any
sort
of
enforcement
of
either
policies
or
standards,
because,
as
we
were
just
mentioning
in
the
benefits
section,
the
flip
side
of
giving
your
users
a
control
over.
The
final
modifications
is
that
they're.
B
B
Have
full
control
in
typical
workflows
again
over
the
final
result
after
your
extension
runs
then?
Finally,
you
need
to
keep
in
mind
when
you're
building
extension,
that
using
extensions
is
going
to
add
dependencies
to
your
customized,
build.
That's
really
important
to
keep
in
mind,
especially
if
you're
not
already
committing
the
results
of
the
build
to
get.
B
B
First,
an
important
note
about
extensions
and
customize
the
type
of
extension
that
we're
talking
about
today,
isn't
actually
the
only
one
that
customize
currently
supports.
It's
the
one
that
we
recommend,
though
customize
actually
has
five
different
types
of
extensions,
all
of
which
have
been
in
alpha
for
quite
some
time.
B
B
B
We
decided
that
extensions
are
really
important
and
we
really
want
to
graduate
them
out
of
alpha,
but
having
these
various
ways
of
doing
things
having
to
maintain
all
of
this,
it's
really
not
sustainable
and
it's
pretty
confusing.
So
we
have
a
proposal
open
that
deprecates
the
legacy
plug-in
styles
and
promotes
the
style
that
follows
that
newer
specification.
B
B
So,
as
I
was
just
saying
to
recap,
the
recommendation
we're
highly
encouraging
extension
authors
who
are
getting
started
today
to
build
containerized
care
and
function
specifically
and
if
you
know,
go,
there's
an
easy
way
to
do
this
as
part
of
customize.
There
is
a
package
called
k,
yaml
and
it
has
a
function
framework
package
that
can
be
used
to
help
you
easily
author
extensions,
so
we're
gonna
look
through
we're
going
to
walk
through
some
examples
of
how
to
use
that
today.
B
Let's
get
started
with
an
example
where
we're
going
to
build
a
transformer
that
does
something
really
simple.
What
it's
going
to
do
is
hide
a
single
annotation
that
has
a
static
key
and
a
user
controlled
value,
so,
in
other
words,
we're
determining
what
the
key
is.
The
user
is
giving
us
the
value.
That's
all
they
get
to
do
we're
going
to
insert
that
annotation
into
all
the
resources
they
give
us
simple.
B
So
in
our
customization,
we
want
to
write
something
like
this.
This
is
what
the
end
user
should
get
to
write.
As
you
can
see,
we
have
a
transformer
configuration.
It
has
its
own
kind,
the
value
annotator
and
we're
using
an
api
version
that
is
unique
to
our
company
in
the
transformer
configuration
the
user
is
going
to
be
supplying
our
extension
with
that
very
important
data
that
they
want
us
to
put
in
the
annotation
so
understand
what
this
extension
is
going
to
need
to
do.
We're
going
to
first
take
a
step
back
and
review.
B
B
B
It's
going
to
specifically
pass
the
resources
that
is
accumulated
so
far
from
that
pipeline
that
we're
looking
at
to
the
docker
container
in
a
really
specific
format.
We're
going
to
take
a
look
at
next
so
that
input
format
is
a
kubernetes
resource
model
api,
of
course,
because
everything
customize
is,
and
it's
called
a
resource
list.
B
This
resource
list
is
going
to
be
passed
to
the
extension
on
standard
in
and
it
looks
it
looks
like
this.
It's
got
two
fields
that
we
care
about
today,
and
these
fields
are
determined
by
the
way
by
that
karen
function,
specification
that
I
keep
mentioning
so
the
first
field
that
we're
interested
in
is
the
items
field
items
simply
contains
that
list
of
resources
that
we've
accumulated
so
far
and
that's,
in
other
words,
the
resources
that
we
need
to
process.
In
our
extension,
the
second
part
is
called
the
function,
config
field
and
the
content.
B
B
It
turns
out
that
resource
list
is
actually
both
the
input
and
the
output
format
required
by
the
karen
function
specification.
So
when
we
receive
that
input
on
the
left
in
the
form
of
a
resource
list,
we're
going
to
need
to
do
our
extensions
work
and
then
we're
going
to
need
to
output
a
resource
list
as
our
result.
B
B
So
would
you
actually
implement
this?
No
no
way?
Customize
has
a
really
powerful
annotation
feature
built
right
in
so
you
wouldn't
bother
doing
this.
The
purpose
of
this
example
is
to
just
get
started
with
showing
how
easy
it
is
to
reproduce
simple,
customized
style
transformations
using
the
kml
sdk.
B
B
B
B
So
what
we
decided
to
do
is
create
a
client-side
custom
resource
for
use
in
customize.
It's
going
to
be
a
customized
extension
notice
that
we're
going
to
be
asking
the
user
to
give
us
two
inputs,
in
this
case
the
image
that's
containing
their
code,
plus
the
domain
that
they
want
us
to
expose
their
application
at.
In
all
likelihood,
a
real
implementation
of
this
would
have
a
bunch
more
options,
but
to
keep
it
simple,
we're
just
going
to
go
with
these
fields.
B
Let's
start
once
again
by
looking
at
the
input
and
the
output
specification
required
by
the
krm
function
specification,
so
we
have
a
resource
list,
of
course,
on
both
sides,
but
in
this
example
we
see
the
client
side,
custom
resource
that
we
just
looked
at
from
our
customization.
So
now
it's
the
spring
boot
up
and
then
we're
going
to
work
with
an
empty
items
in
this
case,
because
if
you
think
about
what
we're
doing
here,
we're
not
transforming
anymore
this
time
we're
a
generator.
B
So
if
you
look
at
the
output
that
we
need
to
create,
we
have
some
items
in
our
items
list.
Specifically:
we've
got
a
deployment,
a
service,
an
ingress
network
policy,
we're
going
to
have
to
generate
those
from
scratch
and
we're
gonna
have
to
make
sure
they're
all
wired
up
together.
The
way
the
user
asks
for.
B
The
next
thing
that
we're
going
to
need
to
do
is
implement
a
filter
method
on
that
type.
That
has
that
same
signature
that
we're
just
looking
at.
In
the
basic
example,
it's
going
to
take
in
a
list
of
items
coming
from
the
resource
list,
we're
going
to
do
whatever
we
need
to
do
to
those
items,
then
we're
going
to
return
the
resulting
items
and
or
an
error
as
applicable.
B
B
B
B
So
this
is
also
going
to
be
called
automatically
by
the
framework
before
filter
is
called
the
framework
even
helps
you
make
your
custom
resource
api
sustainable
by
versioning
over
time.
This
version,
api
processor
type
that
we're
showing
here
essentially
wraps
related
implementations
and
acts
as
a
dispatcher
based
on
api
version
and
kind.
B
The
firmware
has
even
more
tools
than
we
have
time
to
show
off
today,
notably,
it
supports
patching,
including
for
crds
and
at
the
container
level,
which
can
help
you
do
a
bunch
of
really
handy
operations.
It
also
has
a
suite
of
selector
and
matcher
tools
that
can
help
you
target
exactly
the
resources
that
you
want.
It
can
even
help
you
kick-start
containerization
by
generating
your
docker
file.
C
C
The
output
should
be
deterministic
so
think
in
terms
of
unit
tests.
If
your
input
doesn't
change
your
output
shouldn't
change
either,
so
you
want
to
have
some
predictable
outcomes
right,
as
katrina
just
mentioned,
leverage
the
api
version,
one
of
the
reasons
we
want
to
use
custom
resources
rather
than
inventing
some
other
new
way
of
doing.
This
is
to
leverage
all
these
concepts
that
we
all
know
so
use
the
api
version
field
to
keep
the
your
existing
users
of
your
extensions
happy.
C
C
So
if
you're
working
in
go-
and
of
course
you
can
use
any
language
you
want,
because
the
the
underlying
format
is
a
container
but
if
you're
using
go,
you
can
use
this
where
you
see
the
word
processor
just
put
in
transformer
or
generator
or
whatever,
and
you
can
specify
the
inputs
into
that
thing.
Exp
and
specify
the
expected
outputs
and
you
can
also
specify
inputs
that
generate
errors.
So
you
can
check
that
the
errors
are
what
you
want.
That's
important
for
validators,
because
that's
the
whole
business
in
life
is
to
report
errors.
C
Stepping
back
to
the
bigger
picture,
customize
is
part
of
a
cd
pipeline
and
it
wants
you
work
with
git
and
we
recommend
using
what's
called
a
hydrated
repository.
So
imagine
you
do
a
deployment
you
run.
Customized,
build
and
you're
using
all
these
extensions
and
now
you're
using
containers
in
your
pipeline.
C
You've
just
introduced
some
risk,
there's
the
usual
risk
which
we've
all
been
bearing
for
many
years
now
pulling
down
the
containers
of
the
internet.
You
mitigate
that
by
using
a
trusted
catalog
right,
but
the
new
risk
is
now
you've
got
an
extra
piece,
that's
running
in
your
cd
pipeline
and
something
could
go
wrong.
So
what
you
do
is
you
put
you
run
customize,
build
you
put
the
output
into
a
repository,
usually
with
a
robot
right
and
then
a
robot
also
reads
a
repository
on,
say,
a
tag
event
and
deploys
to
the
cluster.
C
C
Slide:
that's
where
customize
lives,
there's
extension
documentation
there.
There
is
the
kml
live
framework
that
we
were
just
talking
about.
As
listed
there
there's
some
caps
that
are
talking
about
future
plans,
especially
the
catalog,
which
is
a
cool
project,
there's
a
contributing
file
at
the
site.
I
would
recommend
anybody
here
who
is
interested
in
writing,
go
or
contributing
to
kubernetes
in
general
to
come.
Take
a
look
at
the
contributing
file.
Look
at
issues
that
are
marked
help
wanted
turns
out.
C
A
A
All
right,
thank
you,
katrina
and
jeff
for
amazing
presentation.
We
are
now
have
few
minutes
for
your
questions.
So
actually
we
have
a
few
questions
that
our
users
asking
online
first
question
from
daniel:
how
do
you,
how
would
we
combine
customized
with
home
charts
yeah
is?
Is
there
any
ways
that
they
can
be?
You
know
used
together.
B
Sure
yeah,
I
can
take
it,
we
have
a
built-in
helm
generator
function
right
now
that
you
can
use
that
it
has
evolved
over
time
and
the
latest
version
of
it
requires
you
to
set
additional
flags
on
the
command
line
invocation
of
customize,
but
it
works
as
a
generator.
So
we
talked
about
generators
today.
Part
of
that
generator
pipeline
is
just
going
to
read
your
your
home
chart
and
send
the
results
through
to
the
next
thing
in
line.
B
Another
thing
that
you
could
do
is
you
if,
if
the
built-in
extension
doesn't
expose
the
home
options
that
you
want,
you
can
build
an
extension
generator
extension
like
we
were
talking
about
today
that
that
renders
home.
In
fact,
the
kept
folks
have
created
an
extension
generator
for
helm
that
follows
the
care
and
function
specification.
So
that's
yet
another
option
that
you
can
use
to
to
bring
those
two
tools
together.
C
B
I
guess
a
caveat
on
that:
yeah,
the
cameo,
and
that
was
that
you
might
have
noticed
we
had
template
processor
as
part
of
the
extension
guide
there
so
template.
If
you
want
to
use
templating,
we
recommend
that
you
hide
it
behind
a
formal
api
right,
so
you're
defining
your
formal
type.
It
has
the
fields
your
end.
Users
are
working
with
pure
kubernetes
data
like
they
would
always
with
customize,
but
then,
within
your
extension,
if
you
so
choose,
you
can
use
templating
in
there
and
you're,
not
exposing
it
to
your
end
users
at
that
point.
A
There's
actually,
I'm
gonna
take
last
question
from
virtual
attendees
and
then
I
will
be
asking
you,
but
actually
I
can
answer
that
question.
Would
it
be
possible
what
is
possible
the
compatibility
of
customizable
tools
like
argo,
cd
or
flux?
So
if
you
look
into
the
documentation
of
flux
or
argo
cd,
there's
actually
a
the
whole
chapter
around
how
to
use
customize
so
get
up,
storing,
fully
supporting
right
now,
the
customized.
D
I
wanted
to
ask
so
I
happen
to
have
the
sad
task
of
maintaining
the
capability
of
installing
manifest
across
a
very
large
set
of
kubernetes
versions.
So
today
I
support
all
the
way
from
117
to
like
the
head
of
master
on
kk
does.
Is
there
any
provisions
for
making
it
easier
to
like
as
a
for
example,
you
know
certain
apis
get
deprecated
and
removed
and
later
kubernetes
versions,
new
versions
get
added?
It
would
be
nice
to
have
some
capability
to
express
that,
like
hey.
C
C
B
If
you
wanted
to
sort
of
extract
over
the
different
like
api
changes
for
for
some
end
users
to
say-
or
if
you
want
to
do
this
yourself,
you
could
do
that
in
extension,
because
extension's
one
of
the
tools
that
we
have
allows
you
to
select
on
on
different
resources
that
exist
in
the
source.
So
you
can
even
theoretically
identify
that
the
input
stream
has
the
older
api
versions
and,
as
part
of
your
extension
spec
you
could.
You
can
include
that
cluster
version
you
could
like
inside
your
extensions
business
logic.
B
You
could
look
up
well
what
api
versions
are
supported
here
and
you
could
even
do
the
conversion
internally,
since
you're
writing
a
full
program
there.
You
can
do
essentially,
whatever
whatever
you
would
like.
You
just
got
to
expose
the
data
that
you
need
to
on
to
form
the
basis
of
those
decisions
in
in
the
api
that
that
that
you're,
configuring
and
then
and
then
you
can
take
care
of
that
within
the
extension
and
since
you're
you're
writing
a
full
program
there
you
can.