►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Higher-Order Deployments: Reducing Boilerplate With Kubernetes Custom Resource Definitions - James Bowes, Manifold (Intermediate Skill Level)
Custom Resource Definitions (CRDs) with associated Controllers can provide a more powerful alternative to templating YAML manifests in Kubernetes. This talk will cover how we use CRDs and Controllers at Manifold to define higher-order composite Kubernetes resources for our deployments at Manifold. These CRDs combine the definition of Deployments, Services, and Ingresses into a single resource that is continually reconciled by Kubernetes, preventing accidental deletion or modification of one of the components. With the associated Controllers, we are able to define new styles of deployment rollouts, and trigger automatic deployments when secrets or container images change, based on definable policies.
About James
James Bowes is the Technical Lead at Manifold. Over his 13 year career he has worked for companies like Red Hat and Salesforce as a senior member of the technical staff. James has scaled early stage startups and also managed and sustained large organizations. James is mainly a backend developer, focusing on thorny concurrency issues, but whenever he can, he likes to build infrastructure, try out cool new software, help his peers learn, and make people laugh.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Good
morning,
everyone
it's
11:10,
so
let's
get
started.
I'm
gonna
talk
to
you
today
about
custom
resource
definitions,
one
of
my
favorite
parts
of
kubernetes
and
I'm
James,
Bo's
I'm,
the
technical
lead
at
manifold
I'm
gonna
get
my
promotion
out
of
the
way
up
front,
we're
a
marketplace
for
cloud
services
and
my
clicker
is
not
working.
A
It
is
it'll
go
so
summer
months
ago
we
migrated
to
kubernetes
and
I
want
to
tell
you
a
bit
about
the
process
that
we
went
through
the
pain
points
we
hit,
how
we
sort
of
got
around
them,
and
hopefully
it's
something
that
you
might
be
interested
in
and
you
can
use
as
well.
You
know
if
you're
going
through
migrations
or
just
sort
of
experiencing
these
things
yourself,
so
we
started
with
a
you
know,
fairly
typical
setup
right.
We
had
a
monolithic
application
and
we
had
an
infrastructure
that
matched
it.
A
You
know
single,
auto
scaling
group
that
ran
multiple
instances
of
this
application.
One
load
balancer
in
front,
so
we
started
off
at
least
in
a
better
place
because
we
had
already
began
was
containerized
to
applications
right.
So
it
was.
You
know
we
had
an
a
on
ami
that
we
built,
and
then
we
used
cloud
on
it
to
sort
of
inject.
A
Of
course,
we
start
building
more
and
you
know
we
get
a
more
complicated
architecture
for
our
system
right.
So
we've
got
more
features,
we
add
in
as
as
new
services
right
and
take
some
of
the
existing
architecture
and
sort
of
features
and
pull
it
out
into
different
services,
and
this
was
great
for
the
application
level.
But
you
know
I
guess
we
heard
sort
of
earlier
today
having
a
service
where
you've
got
like
one
container
per
VM
and
then
we've
got.
You
know
like
tens
of
load
balancers,
it's
just
not
great
right.
A
So
we
began,
like
I,
said,
with
some
a
pretty
simple
setup
where
we
had
Packer
building
our
ami
is
to
start
with,
and
then
we
use
terraform
for,
like
everything
from
you
know,
setting
up
DNS
to
setting
up
the
auto
scaling
groups
and
actually
running
the
deploys
as
well.
So
we
could
do
you
can
just
do
like
blue
green
deploys
with
terraform
when
you're
using
auto
scale
groups
and
then
some
good
old
shell
script
for
sort
of
gluing
together
a
bit
of
the
the
ami
layer.
A
Well,
we've
got
a
like,
you
know,
runs
some
some
commands
on
one
sort
of
one
startup
of
the
machine,
and
so
we
moved
over
to
a
set
up.
That
was
just
terraforming
kubernetes,
with
using
cops
to
sort
of
you
know
set
up
this
generate
all
this
terraform
for
us,
so
we
had
kind
of
like
reduced
the
number
of
moving
parts
we
had
to
use.
A
Even
though
you
know
there
was
not
much
bash,
but
you
look
at
like
a
lot
of
the
examples
and
everything
for
kubernetes
and
you
see
the
the
manifest
files
and-
and
it
looks
a
lot
like
configuration
management
right,
so
you're
thinking
like
this.
This
is
what
I
want
to
use
right,
I
can,
and
especially
with
all
of
these
sort
of
secondary
projects
in
the
larger
ecosystem.
You
can
control
a
lot
more
than
just
sort
of
your
application
code
in
your
cluster
right.
You
can
configure
SSL
Certificates
in
DNS
and
so
you've
you're
doing.
A
But
what
we
started
to
see
was
that
you
know
if
you
look
at
all
these
services
that
we
have
that
are
very
similar
in
structure.
Everything
has
a
well
an
actual
service
right.
It
has
a
deployment,
and
most
of
them
have
ingress
is
since
they're
publicly
facing,
and
the
only
thing
that
varies
would
like.
We
have
also
got.
A
You
know
obviously
sort
of
the
actual
container
image
is
different
between
them
and
then
we
had
them
repeated
across,
come
on
Bluetooth
repeated
across
a
lot
of
environments
right,
so
we
had
our
stage
in
our
fraud
in
vironment
and
we
have
a
devil
of
a
so.
We've
got
like
three
different
directories
that
all
have
the
same
yamo
for
the
for
all.
These
applications
that
also
sort
of
have
similar
groupings
of
llamó,
and
it's
a
lot
of
coffee
paste
and
it's.
A
This
is
a
this?
Is
you
know
what
you
pass
in
to
keep
control,
but
it's
a
direct
mapping
to
an
API
object
that
exists
in
kubernetes
right.
This
is
a
deployment
object
and
essentially
what
you
have-
and
this
is
sort
of
the
good
part
right-
is
that
what
you
have
in
a
manifest
file
is
then
what
exists
in
cluster?
A
A
A
A
Multiple
replica
sets
form
a
deployment
right
as
you
as
you
do.
Your
deployment
you
bring
up
the
new
replica
set
and
you
scale
up
and
down
the
old
and
new,
and
so
this
depart
is
then
this
sort
of
base
unit
that
these
other
parts
build
on
and
the
pod
is
existed
for
a
long
time.
Sort
of
you
know.
Different
versions
of
of
these
concepts
have
sort
of
been
built
and
replaced
on
top
of
it
additional
ones.
Are
there
too
right,
like
Damon's?
Had
some
staple
sets
and
then
sort
of
looking
out
wider?
A
You
know
you
look
at
sorry
docker
for
Mac
and
Windows
when
they
added
kubernetes
support
and
one
of
the
sort
of
you
know
neat
features
of
this
was
hey.
You
can
use
a
compose
file
with
kubernetes
and
it's
really
just
taking
this
compose
file
and
sort
of
putting
your
your
sort
of
standard
kubernetes
like
header,
on
top
of
it
getting
your
metadata
in
there
and
then
pushing
it
up
through
ivett
like
so.
You
could
take
this.
Your
compose
file
and
sort
of
hand.
Add
these
these
fields
or
use
docker
stack
and
the
dr.
A
stack
command
and
run
it
as
well,
and
it'll
just
go
right
to
a
kubernetes
api
server
and
then
through
api
server
aggregation.
You
know
this
bounces
over
to
these
darker
specific
compose
api
server,
which
sees
us
and
says:
okay,
cool,
I'm
going
to
make
a
this
compose
object
that
now
lives
within
kubernetes.
A
A
So
this
is
all
happening
within
the
cluster
right
and
you
could
either
use
docker
stack
or
you
could
use
cube
control
to
actually
have
this
happen
and
then
another
recent
example
is
a
gonna
is
which
I'm
sure
I'm
mispronouncing,
but
it's
a
project
to
sort
of
build.
This
general
purpose
tool
for
doing
Game,
Servers
by
Google
and
Ubisoft,
and
so
there's
sort
of
your
application
specific,
like
Lobby
or-
and
you
know
in
a
client,
connects
this.
A
You
have
multiple
clients
connect
to
this
Lobby
or
like
matchmaking
server,
and
this
specific
logic
then
says:
ok,
cool
I've
got
enough
people
to
start
a
game,
I've
sort
of
paired
them
all
up
and
I'm.
Just
gonna
speak
to
the
kubernetes
api
server
and
I'm
gonna
say
make
a
game
server.
This
makes
a
new
kubernetes
resource,
that's
a
custom
resource
definition
and,
and
the
echo
known
as
controller
comes
along
and
looks
at
this
and
sort
of.
A
Does
the
the
specific
logic
needed
to
then
make
this
into
an
actual
pod
right,
and
the
purpose
here
is
that
they
have
these
sort
of
somewhat
ephemeral,
pods
that
are
specific
for
the
the
players
in
this.
In
this
game
and
there's
a
direct
connection
to
the
outside
world,
like
instead
of
going
through
a
service
or
ingress
or
anything
because
they
want
to
maintain
this
sort
of
low
latency
of
a
direct
connection.
A
So
look
at
all
these
things
and
then
sort
of
some
points.
Kind
of
struck
for
us
right
is
that
kubernetes
is
all
about
composition
right
so
taking
these
base
components
and
sort
of
building
abstractions
and
additional
features
on
top
of
them.
And
the
other
kind
of
weird
thing
is
that
in
some
ways
kubernetes
is
a
database
as
well
right.
A
You
have
the
API
server
with
custom
resource
definitions
or
even
the
API
server
aggregation
and
it's
responsible
just
for
taking
in
objects
and
persisting
them
and
then
there's
other
components
that,
like
controllers,
that
run
that
sort
of
do
something.
With
this
information
either
they,
you
know,
add
more
into
the
API
server
or
eventually
you
get
down
to
like
actually
scheduling,
pods
or
you
get
down
to
creating
DNS
entries
and,
like
a
you,
know,
completely
different
cloud
service.
A
So
what
we
did
was,
we
said,
a
we're
gonna
make
our
own
custom
resource
definitions
and
we're
going
to
build
a
controller
to
handle
this,
so
we're
going
to
model
our
sort
of
specific
logic
around.
These
are
around
our
services
as
Sierra
T's
and
we're
going
to
handle
that
you
know
template
expansion
in
kubernetes
itself,
and
so
the
recipe
for
this
is
you
need
a
custom
resource
definition.
You
need
a
controller
and
then
you
can
add
admission
controllers
to
taste.
A
A
Specifically,
sort
of
references
to
other
objects
isn't
going
to
work
Oh
but
defaults
as
well,
don't
sort
of
fit
in
here
and
we'll
come
back
to
defaults
anyways.
It
is
a
good
place
like
if
you
wanted
to
say
that
a
policy
of
your
organization
is
now.
We
have
to
have
at
least
two
of
these
things
running.
A
Then
you
can,
you
know,
enforce
that,
there's
going
to
be
two
replicas
and
then
the
next
part,
of
course,
is
the
controller,
and
if
I
was
going
to
have
a
bunch
of
code
on
my
own
in
my
slides,
it
would
all
be
around
this,
because
this
is
where
we
spend
most
of
our
time.
But
the
idea
here
is
that
you
take
your
application.
You
know
our
application
definition,
which
we're
where
we
would
basically
say.
A
This
is
the
container
image
that
you're
going
to
use,
and
these
are
some
additional
environment
variables
and
the
controller
knows
that
I'm
going
to
need
to
make
a
deploy.
Out
of
this
to
actually
you
know,
run
the
code,
have
a
service
map
to
this
and
then
include
an
ingress
with
it,
and
our
logic
here
is
unable
to
sort
of
take
our
our
default
a
default
config
map.
We
have
that
sort
of
contains
a
base
environment,
variables
and
inject
this
in
as
well
for
everything.
A
But
what
the
controller
does
is
it
says:
okay,
I
know
that
I
need
these
parts.
What
exists
in
the
system
right
now
and
I'll
just
create
everything.
That's
missing
right,
so
this
is
sort
of
where
your
template
expansion
happens
again.
I
said
you
know.
This
is
where
all
the
code
would
be,
and
this
is
the
takeaway
myself
for
this.
If
we
were
gonna,
go
back
and
do
it
again,
I
would
say
to
just
start
with
meta
controller.
A
So
if
you're
not
familiar
with
it,
meta
controller
is
a
quite
nice
project
that
sort
of
wraps
a
lot
of
the
controller
logic
and
uses
sort
of
lambda
functions
or
web
hooks
that
you
provide
to
to
actually
give
like
your.
You
know,
specific
logic
for
so
one
of
the
sort
of
flavors
of
controller
that
you
can
do
with.
This
is
a
component
controller.
A
Where
you
are,
you
know,
you're
given
like
in
this
sort
of
web
book
you're,
given
a
pair
and
children,
and
so
this
models
like
that,
you
know
the
way
of
taking
this
application
object
and
expanding
it
out
into
the
other
resources.
So
your
code
is
able
to
just
say,
like
look
at
the
the
parent
that's
passed
in
and
return
like
always
just
return,
the
the
children
that
you
expect
to
come
out
of
that
and
if
they
exists
already-
and
you
know,
someone's
tampered
with
one.
A
A
For
all
these
different
objects
and
making
API
calls
to
actually
like
reflect
them
back
into
the
API
server
versus
just
returning.
You
know
a
single
sort
of
result
out
of
some
JavaScript
or
Python
or
whatever
so
yeah
I
would
definitely
like,
as
a
first
place
to
start.
You
know
prototyping.
If
you
want
to
look
at
doing
this
kind
of
thing.
This
is
the
place
to
begin
for
sure.
So
moving
right
along
to
the
optional
pieces.
A
A
So
you
know
the
contrived
example.
You
could
say
you're
not
allowed
to
set
the
debug
environment
variable
for
production
deploys,
so
we
can
check
the
you
know
check
to
what's
passed
in
and
if
it's
yes
and
you
know
return
a
nice
result
with
a
sort
of
readable
error,
it
says
no
and
the
API
server
will
then
return
this
result
to
whoever
called
the
API.
So
you
can
see
if
you're
running
like
cube
control
directly.
You
can
see.
Oh
this.
This
wasn't
permitted
as
an
aside.
A
This
is
sort
of
another
place
where
you
could
implement
some
interesting.
You
know
policies
because
you
are
able
to
see
who
made
that
request.
So
you
could
say
something
like
you
know
on
weekends,
only
this
person
is
allowed
to
deploy
and
they
can
only
deploy
if
it's
like
we're
using
semper
and
it's
just
a
patch
version
bumper,
or
something
like
that,
so
you
can
start
to
do
sort
of
some
more
advanced.
You
know:
policy
enforcement.
The
other
type
is
a
mutating
admission
web
book,
where
you're
actually
sort
of
able
to
take
the
the
full
object.
A
That's
come
in
and
modify
it.
So
if
you
did
want
to
use
defaults,
this
is
a
place
to
to
implement
that.
For
us,
we
just
did
it
in
the
in
the
controller,
because
we're
only
expecting
our
controller
to
actually
read
these
objects,
but
if
you
implement
it
sort
of
as
part
of
a
web
hook,
then
once
it
hits
the
API
server.
It
already
has
these
values
like
in
the
resource
itself,
so
other
controllers
that
may
be
doing
different
things
can
actually
read
them
as
well.
A
A
Why
would
you
want
to
go
down
this
path
instead
of
using
you
know,
ansible
or
pop
out
or
chef
four
or
anything
else?
For
us?
It
was
because
you
know
we
didn't
want
to
use,
bring
in
an
additional
tool,
but
well,
but
we
did
want
to
get
rid
of
all
this
duplication
and
the
CR
DS
exists
already
right.
This
is
something
that
we
were
able
to
build
on
top
of
so
some
other
points.
A
You
know
if
you
ran
a
puppet
master
or
whatever
you
would
have
the
same
kind
of
periodic
reconciliation,
but
this
is
running
within
your
cluster
and
using
a
loop
that
sort
of
exists
in
this
cluster
already,
instead
of
bringing
in
an
additional
component.
So
this
sort
of
prevents
you
know
some
some
mistakes
that
can
be
made.
If
you,
if
you
delete
a
service
by
accident,
the
reconciliation
live.
You
know
in
your
controller
is
going
to
see
that
this
service
is
missing,
whereas
your
sort
of
application
CRD
expects
expects
it
to
exist.
A
A
But
it's
really
simple
to
just
switch
this
over
to
using
your
own
custom
resource
definition.
So
you
can
take
these
sort
of
built
within
the
cluster
and
then
have
other
consumers
use
them
right.
So
if
there's
another
part
of
your
team
that
wants
use
ansible
or
they
want
to
build
a
slack
bot
or
whatever
they
can
speak
to
the
kubernetes
api,
and
they
can
do
this-
you
can
also
build
on
top
of
it
within
the
cluster.
So
there's
there's
nothing
preventing
you
from
building
a
you
know,
a
CR
D
that
is
composed
of
other
CRTs.
A
So
a
few
sort
of
points
of
caution
is
that
CRTs
are
still
relatively
new
and
you
know
the
big
potential
pain
point
here
is
no
multi
versions,
support,
I,
think
four,
and
then
this
is
why
the
the
docker
example
from
earlier
is
done
with
API
server
aggregation,
because
you
know,
there's
sort
of
a
need
for
being
able
to
advance
versions
and
do
it
right
now,
but
for
like
internal
company
use
I.
Think
it's
it's
okay
to
especially
with
something
that's
not
sort
of
super
complex
to
just
say.
You
know
this.
A
This
isn't
an
issue
for
us.
If
we
need
to
have
a
new
version
of
this
resource.
Well,
we'll
just
give
it
a
different
name.
You
know,
and
then
people
can
use
that
we
don't
need
to
sort
of
worry
about
the
migration,
especially
because,
if
it's
in
your
own
cluster,
then
you
can
sort
of
see
what
exists
that
sort
of
the
the
old
versions.
A
Even
if
you
get
into
meta-chem
troller
like
you've,
got
to
deploy
medic
controller,
then
you
have
to
deploy
your
code
that
will
actually
receive
this
webhook
for
that
a
controller,
and
then
you
have
to
deploy
the
configuration
resource
that
will
actually
handle
sort
of
firing.
This
now,
in
our
case,
where
we
did
write
everything
in
go
and
we
use
client
go,
you
know,
custom
resource
definitions
are
just
another
type
of
API
object
in
kubernetes.
A
A
And
you
know
I
the
temps
to
sort
of
make
this
a
little
easier
to
do
or
still
early
got
Mehta
controller
a
cubic.
It
is
something
that
appear
mine
is
working
on
just
sort
of
taking
some
of
our
logic
and
then
extracting
it
and
then,
of
course,
operate
our
sdk.
You
know
announced
this
week,
which
is
looking
quite
promising
for
for
any
kind
of
operator
or
sort
of
higher
level
sort
of
sort
of
resource
dealing,
the
abstraction
in
the
controller
so
anyways,
I'm
I'm
pretty
excited
about
all
this,
like.
A
A
And
we've
got
a
bit
of
time.
If
there's
any
questions,
yeah
pardon
me
the
presentation
or
the
the
presentations
up
on
the
on
the
on
sched
and
for
code,
there's
a
there's
so
there's
links
in
them.
So
for
code
you
can
go
and
take
a
look
at
cube
kit
for
our
actual,
like
a
for
our
organization,
specific
stuff
I
would
look
at
examples
of
existing
existing
sort
of
controllers,
because
the
the
specific
parts
there
are
sort
of
you
know
specific
to
us
and
not
super
interesting.
C
A
So
why
wasn't
how
a
good
fit
so
having
a
standard
template
right
with
helm
hell
makes
it
simpler
to
sort
of
modify
your
single
application
right,
but
we're
talking
about
many
applications.
So
it's
really
like
kind
of
a
trivial
difference
and
I
think
that
you
could
even
begin
with
helm.
Actually
I
think
they
do
complement
each
other
quite
well
is
to
start
with
helm
and
say
things
that
I'm,
manipulating
sort
of
on
my
laptop
I
make
those
simpler
and
then
have
more
of
this
logic
in
my
cluster
for
performing
reconciliation.
Thank
you.
You're
welcome,
oh.
A
A
When
we
started
working
on
the
controllers,
did
we
use
meta
controller?
No,
we
started
with
client
go
so
we're
yeah
yeah,
and
that
I
mean
it
has
a
lot
of
nice
primitives
for
building
controllers
right,
that's
sort
of
its
purpose,
along
with
being
a
client
of
the
kubernetes
api,
but
it's
it
is
relatively
low-level.
So
we
had
to
write.
You
know
several
thousand
lines
of
code.
That
was,
you
know
if
we
had
written
it
with
meta
controller.
To
start
with,
it
would
have
been
on
the
order
of
tens
of
lines
of
code.
C
A
A
You
know
I
want
to
create
a
deployment
and
just
go
ahead
and
take
care
of
that.
For
me,
Matt
a
controller
it's
and
that
begins
to
feel
more.
Like
you
know,
templating
versus
writing,
code,
cool,
okay,
I
think
we're
out
of
time.
So
any
more
questions.
Let's
take
them
offline
here
all
right
thanks.
Everyone.