►
Description
Extending Kubernetes 101 [A] - Travis Nielsen, Quantum Corp
Kubernetes provides the ability to extend the platform with your own custom types and controllers. We will walk through a tutorial to write a custom controller, also known as an operator. Patterns will be reviewed that will make your application a natural extension of the platform through CRDs and desired state management, all with the same security, lifecycle management, and API surface that native Kubernetes applications expect.
About Travis Nielsen
Travis Nielsen is a Principal Software Engineer for Quantum Corporation where he works on Rook – a software defined storage initiative based in Seattle. Prior to Quantum, Travis was the storage platform tech lead at Symform, a P2P storage startup acquired by Quantum. Before joining the startup world, he was an engineering lead for the Windows Server group at Microsoft.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
right
welcome.
Everyone
sounds
like
this
is
working
glad
to
have
you
here
at
extending
kubernetes
101
I'm
Travis
Nielson
I'm,
a
developer
for
quantum
corporation
working
on
the
rook
project,
I've
been
working
with
custom
resources
and
kubernetes
for
about
the
past
year,
while
working
in
on
rook.
So
I'll
tell
you
some
about
my
journey
along
the
way
and
and
then
we'll
it
will
walk
through
some
some
examples
here
about
getting
started
alright.
Well.
A
First
of
all,
this
is
really
the
first
talk
after
a
great
keynote
this
morning
by
yen
on
extensibility,
what
you
can
do
with
it
in
kubernetes,
so
I
encourage
you
to
look
at
these
other
talks
today.
If
you
don't
didn't,
learn
what
you
need
to
from
this
presentation,
there's
lots
of
other
opportunity
today,
so
a
lot
of
great
talks
that
I
hope
to
go
to
as
well
so
agenda.
Since
this
is
a
101
course
I
wanted
to
start
off
by
well.
Why
would
you
want
to
extend
kubernetes
and
then
how
would
it
happen?
A
What
are
the
patterns
surrounding
kubernetes
and
and
then
let's
just
get
right
into
the
code?
You
know
that's
what
I
do
for
my
day:
job
I
write
code,
and
so
that's
why
I
want
to
work
through
my
slides
pretty
quickly,
so
we
can
get
to
that
that
demo.
At
the
end,
if
we
have
a
couple
minutes
for
Q&A
I'll,
look
forward
to
your
questions.
A
First
of
all,
the
most
important
pattern
that
I
believe
kubernetes
has
gotten
right
and
why
it's
successful
is
because
resources
are
declarative.
If
you
want
to
create
something,
you
say
you
know
I'm
going
to
just
declare
it.
There's
no
code
required,
you
defined
your
your
manifest
in
Y,
Amal,
usually
and
kubernetes
will
go,
create
it
for
you.
A
A
Another
example
a
pod
you
know
pod
is
the
most
basic
resource
that
allows
you
to
run
code
in
kubernetes.
Any
application
you
want
so
I,
say:
I've
got
a
pod,
here's
my
container
and
here's
the
image
you
want
to
run.
In
this
case,
my
hello
world,
1.0
container
and
boom.
You
have
a
container
kubernetes,
goes
and
starts
it
for
you.
So
then
you
start
to
ask
the
question.
Well
what
happens
if
I
want
to
run
my
own
type
of
thing?
What
do
I
do?
Well?
I
can
create
a
pod
that
goes
and
runs
my
application.
A
But
what,
if
I
want
to
declare
my
own
type
of
resource?
What
do
I
do?
Well,
kubernetes.
The
creative
team
has
done
a
great
job
of
making
this
very
natural
and
a
very
integrated
thing.
So
they
have
this
concept
called
custom
resources,
CR
DS,
which
you
can
define
and
we'll
follow
exactly
the
same
pattern
as
in
kubernetes
last
years,
I
was
working
with
custom
resources
known
as
third-party
resources.
At
the
time
I
felt
like
I
had
this
light
come
on
like
oh,
that's.
A
So
there
are
lots
of
it's
it's
a
pretty
new
concept
as
far
as
you
know
how
plugging
into
kubernetes
and
last
year,
khoob
khun,
a
couple
of
examples
were
announced.
The
etsy
d
operator
was
one
of
those
and
that's
the
first
example
that
I
found
for
for
custom
resources.
So
I
just
thought:
I'd
start
there.
What
does
that
look
like
and
why
do
they
create
it?
So
I
went
to
their
github
page
and
an
extract
they're
an
example
of
their
CRD.
So
just
some
things
to
point
out.
A
From
from
this
example,
you
see
okay,
they've
defined
an
API
version.
That
whoops
is
it's
just
it's
their
own:
they
define
okay
they're
from
core
OS
comm
they've,
they
define
their
version
v
1
beta
2
they've
got
their
own
kind,
it's
a
net
CD
cluster,
and
but
when
you
create
this
thing
now
me
as
the
user
I
get
to
decide.
Well,
what's
my
etsy
D
cluster
called
I,
give
it
a
name-
and
these
other
properties
in
here
are
very
specific
to
the
FC
D
cluster.
A
You
want
three
of
their
Etsy
DS
servers
and
here's
what
version
and
now
their
Etsy
D
operator
is
going
to
go
orchestrate
the
sed
cluster
for
you,
you
declare
it
and
then
they
go
make
it
happen
and
as
a
consumer
of
an
Etsy,
D
cluster,
you
say
wow.
This
is
great.
It's
all
integrated
and
kubernetes
in
a
way
that
I
would
expect.
A
Another
example.
Prometheus
follows
the
same
pattern.
They
have
a
Prometheus
operator
that
goes
and
applies
these
things
I
kind
of
truncated.
It
ran
ran
out
of
some
room
here,
but
again
everything
down
here
in
the
spec
section,
the
bottom
half
of
the
the
operator
or
the
yeah
Mille
there.
That's
all
something
that
the
Prometheus
team
decided
hey.
We
need
these
properties,
we'll
specify
them.
Our
operator
will
go
act
on
them,
but
you,
the
user,
don't
have
to
write
any
code.
It's
just
a
simple
declaration
on
the
rook
team.
A
Again
same
pattern:
we
define
our
own
type,
we
have
a
cluster
and
our
own
properties
which
I
won't
get
into
today,
so
we
can
get
into
the
sample.
But
really
that's
the
point.
We
have
properties,
you
define
them,
you
declare
them
and
then
the
code
behind
it
worries
all
about
it
and
treats
them
as
Cooper
native
these
sources
for
you.
A
A
We
need
to
monitor
our
our
cluster
to
make
sure
your
data
is
safe.
We
need
failover.
That
needs
to
happen
a
certain
way
we
need
upgrade.
You
know
when
you
have
data
that
needs
to
be
upgraded.
The
data
path,
kubernetes
just
doesn't
know
anything
about
those
details.
So
this
we,
we
were
able
to
answer
all
these
questions
by
using
the
custom
resource
patterns
and
then
operator
to
back
that
up.
So
a
traditional
approach
and
actually
where
I
started
was
oh
well.
A
How
would
I
do
this
well,
I'll
just
start
a
pod
it'll
run
my
service
I'll,
implement
it
as
a
REST
API,
so
create
my
REST
API
great
now,
I
have
a
service
endpoint
where
whoever
needs
to
configure
my
service,
they
can
connect
to
that
and
they
can
call
my
my
rest
endpoints
and
okay.
They
can
manage
it,
but
then
it
start
to
think
wait
a
minute.
It's
not
integrated
with
the
kubernetes
api
or
coop
cuddle.
You
know
how
how
how
would
they,
how
do
they
create
my
resource?
A
A
You
know
what
why
can't
I
just
use
our
back
security
and
do
it
the
the
kubernetes
way
so
the
traditional
approach
for
us
at
least
it
wasn't
a
good
fit.
We
wanted
to
manage
kubernetes
resources,
and
you
know
this
just
didn't
cut
us
cut
it
for
us.
So,
instead
of
the
traditional
approach,
the
extension
of
approach
where
kubernetes
just
makes
it
makes
it
natural,
so
they're
designed
from
the
ground
up
to
say,
hey.
This
is
part
of
kubernetes
they're
building.
It
looks
like
feels
like
smells
like
built-in
resources.
A
If
I
want
to
change
the
specification,
the
properties
that
are
in
that
llamo
file
I
can
well
I
can
do
it
with
coop
cuddle
edit
or
you
know,
however,
I
added
edit,
my
yeah
no
files
when
I'm
done
I
can
delete
the
thing
and
I
can
use
the
name.
The
proper
name
of
my
object,
I
can
say,
delete
my
cluster
and
it'll
be
gone.
A
So
simple,
right,
the
but
the
power
in
this
is
that
it's
your
custom
resources
can
be
consistent
across
all
of
kubernetes.
If
you
want
to
be
taken
seriously
be
consistent,
so
that
means
all
the
tools
whether
it's
cube,
cuddle
or
helm,
whether
it's
the
client
go
API,
whether
it's
security
and
and
you
know
any
other
endless
number
of
tools
and
kubernetes.
It's
going
to
be
consistent
with
with
how
that
works.
So.
A
Again,
we've
already
talked
about
this.
Basically,
it's
declarative
and
behind
each
of
these
resources,
there's
a
controller
in
kubernetes.
The
controller
is
what
makes
it
happen
and
yen
this
morning
talked
about
this.
You
had
a
much
much
prettier
picture
than
me,
but
basically
the
controller
sits
there
and
it
observes
for
what
you
want
to
happen,
or
it
waits
for
you
to
declare
your
your
manifest
files.
The
controller
will
see
when
you
create
it,
it
will
analyze.
A
Oh
what
do
you
want
to
change
in
your
cluster
and
then
it
will
go
ahead
and
act
and
apply
changes
in
the
cluster.
Whether
you
want
to
you
know,
add
a
new
resource,
whether
you
want
to
update
it,
whether
you
want
to
remove
that
that
resource.
That
is
the
controller's
job
to
make
that
happen
all
right.
So
so
that's
the
basic
kubernetes
pattern
that
then
we'll
go
through
for
custom
resources,
so
I
want
to
walk
through
what
does
it
take
to
write
code
and
get
a
sample
operator
and
custom
resource
running?
A
A
Actually
the
kubernetes
team
created
this
tool
that
will
generate
code
for
your
CR
DS
so
that
you
will
have
strongly-typed
resources
so
I'll
get
to
that
in
a
minute
to
show
you
what
you
know
what
that
takes,
or
what
that
looks
like,
so
that
you
generated
code,
it's
wonderful,
it
makes
it
feel
again
like
kubernetes,
and
you
know
the
kubernetes
types
inside
the
built-in
types
use
these
same
code.
Generators
as
I
understand
so
the
third
step
after
you've
defined
and
generated
some
code.
A
A
So
we
there's
three
basic
steps
to
implement.
You
need
to
first
register
the
CRD
with
kubernetes
so
that
it
knows
that
this
controller
will
be
watching
and
applying
these
changes.
You
need
to
implement
three
simple
methods
for
add,
update
and
delete,
and
then
you
know
you
start
watching
the
CRD
tell
kubernetes
hey,
I'm
ready
to
be
notified
when
there's
a
change
here
when
you
create,
update
or
or
delete
anything
alright
when
you're
done
writing
code.
A
Of
course
you
need
to
build
it,
and
that
means,
in
my
case
build
a
go
binary,
as
shown
shown
this
morning
in
the
keynote
you
can
also
do
it
in
other
languages
on
my
demo
will
all
be
about
the
the
going
implementation
but
the
so
you
build
your
binary.
You
create
a
doctor
container
with
it
and
then
we'll
need
to
run
that
so
once
you've
built
it.
What
comes
next
at
runtime?
A
First
of
all,
since
it
is,
it
is
a
pod
that
you
will
need
to
run
a
controller,
though
we
will
define
its
manifest.
It
will
have
some
are
back
rules
that
say:
hey
I
need
to
access
kubernetes
resources,
depending
on
what
your
operator
is
going
to
do
if
it,
if
it's
going
to
be
creating
other
namespaces
or
pods
or
replica
sets
it
just
needs
to
declare
that
so
coober
days
will
will
allow
the
operator
to
do
that
role.
Bindings,
define
the
basic,
manifest
second
go
ahead
and
run
the
operator.
A
It's
a
simple
coop
cuddle
creep
command,
no
new
pattern
there
and
then,
when
you're
ready
to
actually
create
one
of
your
custom
resources,
you
just
say:
coop
cuddle,
create
your
llamo
file
all
right.
So
let's
jump
right
into
what
you
know.
What
all
this
code
looks
like.
We
will
exit
the
slides
and
get
to
the
fun
part.
A
A
So
you
get
you
see
all
that.
You
know
the
properties
here
that
that
you
get
to
define.
So
this
is
my
project
I.
Oh,
that's
my
project
name
I
can
declare
what
version
this
is
a
the
very
first
thing.
So
we
call
it
alpha,
I'm
gonna
call
it
sample,
you
can
call
it
anything.
You
want
we'll
call
it
sample
and
then
the
name
of
this
instance
I
want
to
create
in
my
sample
and
then
the
interesting
part,
though,
is
when
you
get
down
to
the
spec.
A
So
in
the
spec
we
get
to
define
any
properties
here
that
we
want
to
consume
in
our
CRD
hello
world.
Alright,
let's
change
this
a
little
bit
to
make
it
a
little
more
interesting.
I,
don't
want
to
say
hello
world.
Let's
say
hello,
coupe
con
we're
here
and
well.
We
still
need
a
world
property,
so
our
world
is
Austin
for
the
day.
A
Alright,
so
we've
defined
a
simple,
simple
llamo
file,
but
in
this
spec
you
can
have
any
any
sort
of
structures,
complex
structures,
lists
of
things.
You
know
this
is
completely
flexible
as
far
as
what
you
can
do.
What,
if
you
can
do
it
with
a
yamo
file,
you
should
be
able
to
put
it
in
the
spec
all
right.
So
now
that
we
have
our
spec,
we
need
to
get
into
the
land
of
code
I'm
going
to
switch
over
to
go
Ling
and
see
how
this
translates
into
go
code.
A
A
So
if
I
want
to
deserialize
that
world
property
that
I
added
now
I
have
to
add
world
and
it's
going
to
be
of
type
string
and
if
I
want
to
deserialize
it,
it's
called
world
in
the
yam,
oh,
and
if
I
put
the
quotes
all
correctly
there,
we
go
all
right.
So
we
have
our
two
properties,
hello
and
world
loops,
and
that's
all
I
need
to
get
them
into
the
land
of
go.
Now.
A
There
are
some
other
things
surrounding
this
that
will
be
used,
that
you
need
to
declare
so
the
for
the
code
generation
as
well.
So
let's
look
at
that
now,
so
the
spec
is
contained
by
our
sample
type,
so
this
sample
is
the
same
name
as
as
what
we
saw
at
the
top
of
our
our
CRD
ml
file.
So
the
sample
has
to
define
a
couple
of
standard
properties,
the
type
meta
and
object
maida.
A
That's
what
brings
us
in
the
name
and
namespace
properties
and
and
some
other
things,
and
then,
of
course
the
spec
is
what
we.
What
contains
our
hello
world
properties
above
the
sample,
these
Jen
client
and
deep
copy
attributes
are
what
we
need
to
add
to
our
sample
struct,
so
that
we
can
run
the
code
generation
tools
to
generate
the
strongly
typed
objects
for
people
to
consume
from
the
client
go
client,
alright,
so
that
you
know
that's
pretty
much
it
down
here.
There
is
a
sample
list,
see
RDS.
A
Another
pattern
with
them
is
that
you
can
create
any
number
of
them
so
so
that
you
can
list
them.
We
have
a
simple
list
type
with
some
items
in
them
and
again
for
cogeneration.
You
need
to
be
able
to
deep
copy
them
and
that's
that's
pretty
much
it.
We
have
our
types
defined
with
the
cogeneration
attributes.
A
Now,
let's
see
in
this
file,
we
have
about
sixty
lines
of
code,
so
a
little
bit
more
just
to
show
what
it
takes
to
register.
So
we've
defined
our
CR
D.
Now
what
does
it
take
to
register
our
type
now
in
this
library
that
that
we've
got
a
simple
library
that
lets
you
define
custom
resources
and
register
them?
A
It
takes
all
these
properties
that
you
saw
in
the
yamo
file
will
will
be
defined
here,
and
so
it's
named
a
sample
when
you
have
multiple
of
them,
you're
listing
them.
You
need
the
plural
name
samples
our
group
name,
my
project
at
I/o,
our
version
alpha
yep.
Those
are
all
the
properties
that
that
we
need
for
our
custom
resource.
A
So
the
operator
kit
now
will
register
those
for
us.
I,
don't
have
time
really
to
dig
in
and
there's
just
kind
of
some
framework
around
here,
these
other
few
lines
of
code
that
you
can
go
refer
to
on
the
github
later.
But
basically
you
know
we
register
these
two
types
sample
and
sample
list
and
kubernetes
will
know
away.
We
go
now
in
our
sample
project.
A
A
A
So,
after
the
context
we
go
ahead
and
ask
our
operator
kit
to
create
custom
resources,
so
this
takes
our
sample
resource
and
registers
it
until
scrip
enemies,
hey
I'm,
listening
for
this
object
and
now
some
basic
things
to
watch
for
signals
in
the
pod
to
manage
the
the
pod
lifecycle.
But
then
the
important
part
is
okay.
Let's
watch
for
watch
for
the
custom
resource,
so
this,
if
I
jump
into
this-
and
you
see
I'm
gonna
watch
all
namespaces
for
for
our
new
resource.
A
So
when
I
tell
cam
raise
I
want
to
start
watching.
Well,
we
say:
I've
got
some
handler
functions.
I've
got
an
ADD
method.
I've
got
an
update.
I've
got
a
delete.
Okay,
so
here
there'll
be
below
here
implemented,
I've
got
a
strong
rest.
Client
I'm
going
to
this
is
the
one
that
was
generated
and
I'm
going
to
watch
this
resource.
A
So
I
tell
tell
go:
ok,
I'm
ready
to
watch
this
watch
my
sample
now
to
make
it
a
simple
hello,
world
application
when
I
add
the
resource,
I'm
gonna,
call
this
helper
deep
copy
and
all
I'm
gonna
do
is
simply
can't
get
much
simpler
than
just
printing
to
the
console
or
printing
it
to
a
log
file.
Ok,
so
I
add
the
sample
with
hello
equals
whatever.
A
A
Okay,
so
ad
will
basically
just
print
the
values
of
what
I,
just
added
now
in
a
real
operator.
I
would
want
to
you
know,
go
make
some
change
in
the
cluster.
I
would
want
to
go
start.
Some
pods
start,
you
know,
create
whatever
communities.
Resources
I
need
to
create
now
update
is
just
gonna,
be
something
similar.
It's
going
to
update,
update
from
some
old
to
some
new
value
and
delete
again
will
just
print
the
message
very
simple
basic
thing:
to
get
us
going
all
right.
A
A
Okay.
So,
let's
run
this
so
code
code
gin
will
run
pretty
quickly
generate
the
deep
copy
functions.
The
client
set
solicitor's,
The
Informers,
and
now
we
have
generated
code.
If
I
go
look
at
my
project
again,
we've
got
a
few
new
files
highlighted
in
green
here.
Let's
see,
if
I
expand
this
a
little
bit,
you
see
this
deep
copy
go
all
generated
and
the
client
set
that's
generated
if
I
just
glanced
quickly
into
some
of
these
files.
A
A
A
A
A
A
A
A
A
A
A
A
A
There
we
go
so
inside
the
amal
here.
We've
got
hello,
coop
gun
world
austin
great.
Now
we
still
don't
really
know
if
the
controller
responded
to
it
and
if
you
remember
the
controller
was
simply
going
to
print
to
the
the
log.
So
we
let's
look
at
that-
that
log
now
so
cuddle
logs
and
I
I
put
a
label
on
it,
so
we
could
find
it
sample
operator.
So
the
log
says
yes
added
sample
my
sample
with
hello
equals
cubed
on
and
world
equals
austin.
Alright,
it
works,
and
then
you
can
do
other
things
like
edit.
A
A
A
Alright,
it
saved
it
and
now,
if
I
look
at
the
log
again,
it
would
say:
oh
I
changed
from
ku
come
Austin
to
coupe
con
Seattle
and
there
we
go.
We
have
our
custom
resource
and
if
I
did
the
same
thing,
I
could
I
could
go
ahead
and
delete
delete
samples
by
the
way
you
can
fully
qualify
it.
If
something
else
has
a
else
has
a
name
like
that
samples
dot.
My
project
I
owe
my
sample
and
it
would
go
ahead
and
delete,
and
the
operator
would
then
print
to
the
log
file
again.
A
So
there
we
go.
We
have
a
running
sample
operator
that
we
created
from
scratch
with
the
with
the
cogeneration
and
everything.
If
I
had
more
time-
and
you
know,
I
would
have
spent
more
time
on
the
client
to
see.
Well,
what
are
those
strongly
types
look
like
and
well?
We
would
see
that
they
look
exactly
like
other
kubernetes
resources.
A
Okay,
so
we
just
got
a
couple
minutes
left
to
go
back
to
the
slides.
Here
we
went
through
the
demo
already
so
takeaways.
You
know
really
CRTs
make
Kuber
days
extensible.
They
follow
all
the
same
patterns
that
you'll
find
with
other
resources.
Crts
I
really
have
a
low
overhead.
Most
of
your
time,
once
you
get
set
up,
is
going
to
be
inside
your
business
logic,
whatever
you
need
to
do
when
your
resources
are
created
since
I
work
for
the
the
rook
team.
A
If
you
want
to
see
a
bigger
example
where
a
CRTs
are
in
are
implemented,
rook
does
rook
defines
five
CRTs.
Actually,
we've
got
cluster
pool,
object,
file,
volume,
attachment
and
all
of
these
are
anyway
fully
implement.
You
can
go
look
for
for
more
examples
there
rook
by
the
way
it's
so
it's
file,
object
and
block
storage
for
kubernetes,
built
on
SEF,
where
we've
submitted
to
the
to
CN
CF
and
we're
excited
about
about
getting
that
reviewed
there.
So,
thanks
for
your
support,
there
take
a
look
at
it
and
I'll
be
in
the
rook
booth.
A
Today,
if
you
have
any
questions,
here's
the
links
to
the
projects,
the
the
operator
kit
at
CD,
Prometheus,
rook,
just
and
there's
lots
of
other
examples
out
there,
the
and
it's
starting
to
roll.
If
you
search
for
operators
on
github,
you'll,
probably
find
a
number
of
them,
it's
a
community.
This
is
just
growing
more
rapidly
all
the
time.