►
From YouTube: Kubernetes Feature Prototyping with External Controllers and Custom Resource Definitions
Description
Kubernetes Feature Prototyping with External Controllers and Custom Resource Definitions [I] - Tomas Smetana, Red Hat
Getting patch into Kubernetes might be difficult. Getting a new feature into Kubernetes is... even more interesting experience. When working on the persistent volume snapshotting feature we realized that the straightest path might not lead us where we wanted, Our original idea of adding few API objects and a controller become more complicated when we presented it to the community. So we took a small detour by creating the feature out-of-tree first.
In the talk I will describe the journey of the volume snapshotting feature, how do the external controllers work, what are Custom Resource Definitions and how to add features to Kubernetes without changing its code base.
About Tomas Smetana
Tomas is a an Engineering manager in Red Hat. He is an Open Source enthusiast who used to work on various userspace Linux components contributing to several FOSS projects. For the past one year he is active in the Kubernetes Storage SIG.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
A
Okay,
so
just
wave
the
hands?
If
you
can't
hear
me
because
I
know
it's
all
too
softly,
usually,
and
if
you
don't
understand
just
shout
I
will
for
a
red
hat
on
openshift
project.
That
means
I
have
to
contribute
to
kubernetes
as
well.
I'm
active
in
the
storage
sig
group
because
of
my
previous
engagements
and
yeah
started
working
on
kubernetes
about
about
a
year
ago
and
son.
Don't
sure
if
you
realize
how
large
project
it
is
I
mean
it's
not
just
a
matter
of
amount
of
code,
it's
mostly
about
the
size
of
the
community.
A
If
you
look
around
yourself
this,
this
is
a
huge
conference
for
one
software
project
and
the
community
is
not
only
contributing
code.
There's
many
users,
they
run
kubernetes
in
production.
That
means
they
don't
want
this
thing
to
break
so
there's
many
processes
established
around
kubernetes.
You
cannot
just
in
you
know,
go
and
change
things
there.
If
you
want
to
contribute
a
culture,
how
many
of
you
have
ever
tried
to
send
a
patch
or
a
contribute
patch
to
kubernetes
yeah?
A
So
few
of
you,
so
everyone
know
that
it
takes
some
time
for
the
patch
to
be
merged.
It
has
to
be
tested
thoroughly.
It
has
to
pass
a
review,
it
has
to
be
approved
and
depending
on
which
part
of
kubernetes
you're
contributing
to
there
are
only
certain
people
who
are
allowed
to
approve
a
patch.
So
if
you
create
something
slightly
bigger
that
touches
more
parts
of
kubernetes,
you
have
to
chase
the
approvals
you
have
to
convince
the
community
that,
yes,
this
is
what
you
want,
and
no
it's
not
going
to
break
anything
really.
A
Things
get
more
interesting
if
you
not
only
fixing
bugs,
because
people
want
you
to
fix
bugs,
but
if
you
are
trying
to
add
a
new
feature
to
kubernetes
I
think
a
new
feature
usually
means
doing
all
these
things
that
go
against
these
stability.
You
create
a
new
API
objects
or
you
want
to
change
API.
You
want
to
interact
with
the
other
objects,
and
you
have
to
prove
that
things
will
not
change
that
the
behavior
will
not
change.
So
you've
got
to
go
to
community
meetings.
A
You
got
to
propose
the
feature
you
get
to
create
formal
documents.
That's
describing
your
feature.
How
will
it
work?
How
will
it
interact
with
the
other
objects?
You've
got
to
go
to
the
community
meetings
where
they
regularly
track
the
progress
of
the
development
of
the
feature
it's
very
difficult,
and
my
goal
was
to
teach
kubernetes
to
be
able
to
take
snapshots
of
persistent
volumes.
So
you
know
there's
a
persistent
volume
is
basically
a
disc
that
your
pot
can
use
to
store
data
and
the
disc
will
survive
the
pot
once
it
goes
down.
A
The
data
stays
there,
and
so
it's
like
a
normal
disk
and
people
who
are
using
disks
on
computers.
They
do
things
like
backups
and
snapshot
is
part
of
this
workflow.
We
take
a
snapshot
of
a
disk
and
your
application
can
continue,
and
then
you
backup
the
data.
For
example,
one
of
the
use
cases
perform
snapshots.
It's
a
very
frequently
used
feature,
so
we
thought
it
would
be
really
good
if
kubernetes
would
have
an
API
to
take
a
snapshot
of
persistent
volume.
So
all
the
applications
could
use
the
same
API,
no
matter
where
they
run.
A
So
we
proposed
the
feature
and
we
thought
it
was
simple.
It's
not
right.
Snapshot
is
basically
a
tiny
thing
and
I
come
from
Linux
background.
So
for
me,
snapshot
this
I'll
be
create
yes
and
there's
nothing
exciting
about
it.
So
I
thought
it
would
be
simple
all
right.
So
this
was
the
plan.
This
is
how
it
should
look
like
when
you
propose
a
feature.
So
you
present
the
idea
to
the
community
somehow
sent
a
mail
to
a
mailing
list
go
to
a
meeting
or
whatever
create
the
former
proposal.
The
proposal
has
to
pass
a
review.
A
A
So
what
we
did
I
thought
the
snapshot
is
really
easy.
So
how
would
you
do
the
snapshot?
You
just
tell
the
system
to
take
a
snapshot.
Okay,
so
with
kubernetes
I
would
create
an
object.
It's
how
we
talk
to.
One
of
these
features.
Declare
I
want
this,
so
my
idea
was
user
accounts
creates
and
volume
snapshot,
object
and
kubernetes
will
take
snapshot
of
the
PV.
That
volume
snapshot
object.
References
simple
is
a
good
work.
I
presented
the
idea
to
the
community
by
you
know
calling
for
meeting.
So
whoever
is
interested
in
developing
this
feature.
A
Please
join
my
conference.
Call
that
time
that
day
and
to
note
I
was
right.
Goober
snapshot
was
a
very
demanded
feature
according
to
the
attendance
of
the
call.
Unfortunately,
if
you
have
this
many
people,
they
have
many
ideas.
Many
opinions
on
how
the
things
should
look
like
what
should
it
do,
what
it
should
not
do,
so
they
asked
questions
like
what
the
snapshot
object
be
named,
spaced
or
not.
Of
course,
it
would
be
because
user
has
to
be
able
to
do
that.
A
So
how
do
I
move
the
snapshots
to
other
namespace,
then
no
idea
I
mean
I,
didn't
think
about
that,
and
then
people
were
asking
like
and
how
do
I
stop
the
application
before
we
start
taking
the
snapshot,
no
idea,
I
didn't
think
about
that,
and
do
we
need
that?
Can
we
do
that?
Can
we
even
do
that
can
be
so
Universal
that
every
application
or
every
every
parts
would
be
able
to
tell
communities
what
to
do?
You
know
to
make
the
snapshot
consistent?
It's
not
not
easy,
so
the
first
meetings
were
really
dramatic.
A
First
change
we
made
was
that
snapshot
cannot
be
just
one
object.
We
decided
to
make
it
two
objects
like
PV
and
PVC
similar,
so
the
object
that
would
the
user
be
able
to
create
or
delete
would
be
the
following:
snapshot:
object
and
somewhere
outside
of
the
namespace
like
PV.
There
would
be
another
object,
we
call
it
snapshot,
data
doesn't
matter,
and
this
arrangement
would
eventually
allow
us
to
move
snapshots
between
namespaces,
because
there
is
some
way
how
to
make
the
non
namespaced
object,
accessible
somewhere
else.
A
One
interesting
thing:
I
work
from
Czech
Republic,
that's
Central,
European,
Time,
Zone,
most
of
the
people
that
I
interacted
with
at
meetings
work
from
the
States
or
from
different
time
zones.
So
you
know
we
were
taking
notes,
I
allowed
them
to
suggest
changes,
and
the
meetings
look
like
we
agreed
that
we
will
do
this
this
and
that
all
right
I
went
home
when
I
come
back
in
morning.
I
opened
the
documents
and
there
was
this
big.
You
know
list
of
suggested
changes
we
agreed.
We
will
do
this,
no,
no!
No
again!
A
So
as
time
progressed,
we
didn't
actually
move
forward
that
quickly
as
I
wanted
to,
and
we
still
didn't
write
a
line
of
codes.
We
were
still
talking
about
the
proposal
only
so
we
had
to
come
with
some
Plan
B.
How
do
we
do
that?
How
we
really
give
the
users
the
feature
and
the
plan
B
was:
let's
not
go
through
this
official
path
of
creating
a
proposal,
creating
implementation
and
have
it
all
reviewed
and
have
it
all
merged
afterwards,
as
they
the
other
way
round.
A
Let's,
you
know,
be
agile,
create
MVP,
create
this
small
thing
that
would
take
the
snapshot,
give
it
to
the
users
and
see
how
they
use
it.
What
do
they
do
with
that?
And
if
it
works
for
them
or
not,
if
it
does
not
make
the
changes
and
rinse
and
repeat,
and
once
we
are
sure
we
have
something
the
users
want,
describe
it
and
call
it
a
proposal,
and
then
we
should
be
able
to
get
it
in
the
main
tree
be
sure
we
have
a
useful
feature
that
it
works.
A
So
what
did
we
do
is
obvious,
create
a
custom
resource,
so
we
started
with
external
implementation
and
here's
the
awkward
part
of
my
presentation,
because,
if
you've
been
here
before,
then
you
saw
all
the
details
of
how
customer
resources
are
being
created.
How
are
they
being
compiled
hardly
being
used?
A
Moreover,
if
you
are
interested
in
details
times
have
changed,
because
we
started
working
on
snapshots
few
months
ago,
maybe
maybe
longer
even
right
and
since
then
there's
many
projects
using
custom
resources,
custom
resource
definitions,
external
controllers,
here's
one
of
the
examples,
there's
kubernetes
incubator
organization
on
github,
where
you
can
see
many
repositories
where
you
can
basically
take
a
look
and
get
some
inspiration.
If
you
want
to
want
to
develop
something
on
your
own
I
think
it's
the
easiest
way.
You
know
just
copy
the
other
code.
That's
why
the
open-source
it
sound
great
so.
A
Let's
do
the
volume
snapshots
outside
of
kubernetes
first,
so
okay
I
was
too
quick
a
custom
resource
for
those
who
have
not
been
here
before
it's
basically
a
custom
data
type
that
you
can
create.
It's
like
persistent
volume,
note
ports.
These
are
the
types
known
to
API
server
in
terminology
of
API
server.
They
call
kinds,
so
you
create
your
own
kind,
that's
basically
unknown
to
the
API
server
and
you
can
dynamically
add
it
to
API
several
register
it.
So
it
recognizes
it.
It
can
store
it.
It
can
validate
it.
A
A
Implementation
detail
that
I
consider
to
be
important,
because
it's
probably
one
constraint
of
your
new
data
type
it
has
to
implement
the
runtime
object
interface.
That
means
you
have
to
use,
or
you
should
use
the
code
generators
again.
You
might
have
seen
it
in
the
in
the
previous
presentation
to
get
you
the
necessary
methods
like
deep
copy
methods
for
for
your
new
object,
so
custom
resource
is
your
new
data
type
custom
research.
A
A
definition
is
the
built-in
API
that
allows
you
to
register
your
new
data
type
in
the
API
server
and
teach
it
to
to
handle
it.
So
you
can
store
your
new
object
on
the
API
server.
You
can
delete
your
new
object
on
the
API
server.
You
can
edit
it
and
that's
everything
the
customers
definition
usually
is
being
used
by
the
external
controllers
on
startup
to
register.
A
A
This
is
our
volume
snapshots
made
a
type.
This
is
piece
of
goal
and
code.
It's
snippet,
don't
try
to
compile
it.
It's
just
taken
from
the
snapshot
controller.
So
there's
nothing
interesting
in
there
you've
seen
before
we
define
some
strings.
That
would
allow
you
to
access
the
data
types.
So
the
plural
is
volume
snapshots.
When
you
do
cube
cut'
or
get
volume
snapshots
the
API
server
knows
it
should
retrieve
this,
and
you
also
give
it
a
group
name
and
version.
A
So
it's
position,
but
also
accessible
the
data
type
or
the
kind
is
again
if
you
know
how
the
PV
or
PVC
look
like
this
is
exactly
at
that
we
got
inspired
by
it.
So
there
is
some
spec
which
says
how
I
want
my
snapshot
to
look
like,
which
is
the
desired
state.
There
is
the
status
which
describes
what
is
the
current
state
of
the
object
and
kubernetes,
or
your
new
controller
makes
some
action
to
make
the
desired
state
and
current
state
equal.
A
It
should
be
quite
self-explanatory.
You
just
specify
what
I
want
the
customer
service
to
look
like.
So
what
is
its
API
group
name?
What
is
its
API
version?
There's
a
scope
so
for
the
volume
snapshot
I
said
we
decided
to
make
it
a
namespace,
so
the
scope
is
Ming,
spaced,
scoped
and
again
names.
Plural
kind
again
should
be
self-explanatory,
and
now
we
have
it.
A
If
you
register
this
in
the
API
server,
the
API
server
is
able
to
store,
delete,
validate
the
new
objects,
but
it's
not
very
useful
on
its
own,
because
you
will
store
your
object.
You
will
retrieve
your
object,
but
we
want
something
to
take
action
when
you
create
the
object.
That's
why
you
have
to
write
the
external
controller.
This
is
the
biggest
part
of
the
code,
so
control
talks
to
the
API
server
and
watches
for
changes
on
the
objects
again.
A
I
repeat
what
my
predecessor
has
already
shown
you
in
great
detail
and
great
presentation.
So
you
just
register
your
handlers
for
addition
of
object,
updates
of
object,
deletions
of
objects
and
your
controller
issue
then
can
act
so,
for
example,
if
I,
if
the
user
creates
a
new
volume
snapshot
object,
the
controller
sees
that
talk
to
the
backend
tells
it
create.
The
snapshots.
Talks
back
to
the
or
gets
from
the
backend
some
status,
like
ok,
I've
started,
so
it
updates
the
object
once
the
snapshot
is
taken.
It
update
the
object
with
success.
A
A
If
somebody
deletes
the
volume
snapshot
before
it's
actually
taken
on
the
back
end
and
all
the
corner
cases
that
up
to
you,
although
the
controller
is
usually
the
part
that
takes
care
about,
registering
the
custom
resource
definition
in
the
API
server,
so
the
users
don't
have
to
do
that
manually
or
you
don't
have
to
do
something
just
too
difficult
and
now
snippets
of
code
that
I
consider
to
be
again
the
important
ones.
So
you
get
the
client
set
again.
That's
the
thing
that
the
code
generator
creates
for
you,
client
set
talks.
A
The
API
server
increase
the
customer
service
definition.
This
is
appalling,
routine.
That
waits
for
the
customs
custom
resource
to
be
available
in
the
API
server.
So
we
can
start
using
it
once
the
customer
service
is
available.
We
installed
the
handlers
again,
we
need
just
the
rest
client
the
scheme,
so
we
can
access
the
objects
and
we
can
go
to
the
realm.
We
API
server
and
install
the
handlers.
A
Okay,
there's
a
lot
of
syntactic
sugar
around
it,
but
the
gist
of
it
is
this.
You
say
that
my
SC
on
snapshot
ad
is
the
addition
handler
on
snatcher
that
day
and
there's
basically
nothing
nothing
complicated
and
that's
that's
the
point
of
it.
It's
really
simple.
If
you
want
to
create
a
simple
controller,
that
would
do
something
for
you
that
would
act
on
some
of
your
new
object
changes.
It's
really
easy
and
again
in
the
previous
talk
you
might
have
seen
it
in
great
detail.
Demos.
A
So
advantages
for
us
because
I'm
talking
about
not
extending
the
kubernetes
with
external
thing,
mind
you,
our
goal
is
still
to
eventually
get
the
feature
in
kubernetes
to
get
it
entry,
but
now
we
are
not
bound
to
the
kubernetes
development
cycle.
We
can
we
can.
We
can
change
our
code
as
we
want
it.
We
can
say
to
the
users
that
this
is
an
experimental
feature,
something
that
you
don't
see
in
kubernetes
or
you
don't
want
the
kubernetes
or
is
more
difficult
to
implement
inside
of
kubernetes.
A
A
Good
way
or
easiest
way
of
forgetting
features
like
this
into
kubernetes,
if
you
can,
if
you
have
something
that
isolated,
then
today,
I
probably
wouldn't
go
the
the
path
we
started
with.
I
wouldn't
start
with
the
community
meetings
and
formal
proposals.
I
would
really
try
it
out
myself
and
show
the
code
to
the
others.
First
until
them
see
I
have
this
works.
Is
it
useful
for
you?
Yes,
no,
if
no
change
redo
and
it
would
be
much
much
faster
and
we
would
be
talking
about
specific,
concrete,
real
thing.
I
believe
this
is
important.
B
A
There's
obviously,
some
disadvantages
what
I
found
in
our
code?
Maybe
it's
because
we
don't
have
it
right,
but
I
believe
it's
a
consequence
of
having
an
external
controller.
You
don't
have
things
like
shared
informers
because
there's
nothing
to
share
them
with,
because
it's
your
own
tiny
piece
and
it
has
to
talk
to
the
API
server
and
the
API
calls
is
basically
the
only
way
you
can.
You
can
interact
with
kubernetes,
so
the
API
calls
might
be
more.
A
My
needs
to
be
more
frequent
and
this
might
be
somewhat
less
performant
and
would
have
been
if
it
would
have
been
implemented
in
the
communities
itself.
But
what's
more
problematic
is
this
is
not
what
the
user
gets
with
kubernetes
if
they
install
kubernetes.
If
they
you
know,
do
the
heck
local
cluster
up,
it's
not
better,
so
you
also
have
to
take
care
about
deployments
or
good
documentation
of
the
deployments
for
the
users.
Probably
admins
are
always
role
based
access
control.
Again.
A
This
is
something
that
you
don't
have
some
any
access
control
by
default,
like
in
kubernetes
and
depending
on.
How
is
the
cluster
set
up?
The
access
control
might
slightly
change,
so
you
have
to
really
be
mobile.
Balls
in
the
documentation.
Tell
your
users
how
to
do
that.
Also,
that
turns
you
into
a
release
engineer
and
the
package
maintainer,
because
you
have
to
take
care
about
updating
your
upstream
images.
A
You
should
really
do
some
marketing
around
your
project
because
it's
not
in
kubernetes,
it's
not
so
closely
watched,
so
you
must
make
sure
that
people
really
know
what
you're
doing
and
that
they
will
test,
because
it's
the
point
of
doing
that.
I
believe
that
this
non-technical
part
is
actually
the
most
difficult
one.
When,
when
you
try
to
take
this
path,
you
really
quickly
get
some
working
codes,
but
getting
that
code
to
the
users
is
not
so
easy.
It's
not
for
me.
A
Another
thing
we're
running
into
is
the
dependencies
vendor,
rink
and
golang
I,
don't
know,
you've
seen
that
it
might
become
really
really
difficult
because
there's
dependency
circles,
and
now
you
depends
on
kubernetes,
which
is
large
and
once
the
kubernetes
once
kubernetes
dependencies
changes,
somehow
change.
Somehow
you
got
to
reflect
it
in
your
code,
so
we
spend
a
lot
of
time
fighting
with
glide
and
updating
the
vendor.
Ink
directories.
A
Again,
this
is
something
that
I
hope
somebody
will
eventually
solve
in
the
future,
because
I
don't
think
it's
bearable.
Yeah
I've
talked
about
less
of
your
visibility
for
potential
users
and
contributors.
One
more
thing
that
I'm
seriously
afraid
of
is
because
snapshot
is
such
a
cool
feature,
I'm,
not
sure
if
somebody
else
in
the
world
is
not
developing
it
in
parallel,
that's
the
thing
that
you
would
probably
avoid,
if
you
would
do
it
right
away
in
in
communities.
So
it's
it's
like
it's
mostly
a
social
thing.
A
You
know,
if
it's
this
big
project,
somebody
else
would
see
that
and
join
you
if
you're
developing
it
in
isolation
somewhere
else,
it's
pretty
possible
there'll,
be
the
splitting
forces.
I,
don't
know,
I
hope
that
nobody
else
is
working
on
snapshots.
If
you
know
someone
who's
Kanaan
snapshots,
let
me
know
so
we're
getting
to
the
end
of
my
presentation.
So
just
sum
it
up
see
our
days
are
really
really
easy
way
on
how
to
extend
kubernetes
and
that's.
Why
I
think
the
really
great
way
of
experimenting
with
kubernetes?
A
If
you
know
that
your
feature
is
well
isolated,
then
there's
basically
no
reason
to
start
with
forking
goober,
not
easy
and
changing
changing
kubernetes
code
itself.
You
can
do
it
with
custom
resource
definitions
with
external
controllers
and
then
try
to
merge
it
back
in
kubernetes
and
one
more
thing:
if
you
heard
the
keynotes
in
the
morning,
people
talked
about
slimming
down
kubernetes
much
of
the
functionality.
That's
now
part
of
the
quantities
itself
is
going
to
get
external
anyway.
A
So
if
they
way,
if
we
decide
not
to
merge
my
new
feature
in
kubernetes
itself,
there's
no
problem.
I
still
have
not
failed
I.
Have
it
as
an
external
component
same
as
many
other
features
that
eventually
will
exist
in
communities
in
the
future.
So
this
is
basically
a
way
how
you
can
ensure
that
you
don't
fail
or
not
fail
that
easily
lowers
the
probability
of
failure
and
I
still
hope.
A
C
A
A
C
A
We
started
at
the
end
of
1.6
I
think
it's.
We
actually
started
with
third-party
resources
and
change
that
later
so
yeah
there
was
a
common
that
we
can
use
the
API
aggregator
and
it
would
be
also
working.
This
is
high-level
close
to
the
code.
Yes,
it's
true,
you
might
have
used
that
other
questions.
Yes,.
B
A
A
It's
still
very
alpha
beta
quality,
so
it
might
be
unstable.
It
works
for
Amazon,
it
works
for
GCPD,
it
works
for
a
house
path
and
we
have
new
patches
for
cinder,
that's
OpenStack
and
we
have
patches
for
blaster,
but
these
are
very
new,
so
be
careful!
Please
are
there
any
other
questions?
Yes,
please.
A
Again,
we
think
this
is
some
something
like
persistent
ball.
It
belongs
to
volumes.
So
if
you
have
persistent
volumes,
you
should
be
able
to
take
snapshots.
For
me,
it's
actually
quite
core
feature.
That's
why
I
want
to
have
it
in
three
and
it's
up
to
the
maintainer
and
community
to
tell
me
whether
I'm,
right
or
wrong,
so
my
plan
is
to
really
show
them
that
it's
this
useful
it
does
these
things.
Do
you
want
to
have
them
in
tree?
A
A
Yes,
custom
resources
can
be.
You
can
create
roles
that
would
basically
define
access
control
for
custom
resources
as
well
custom
resources
in
this
manner.
They
behave
like
standard
API
object
that
already
exists
in
the
API
server,
so
you
can
define
role
based
access
even
to
see.
Are
these
some
more
questions
all
right?
Thank
you
very
much
for
listening.