►
From YouTube: Writing a Custom Controller: Extending the Functionality of Your Cluster [I] - Aaron Levy
Description
Writing a Custom Controller: Extending the Functionality of Your Cluster [I] - Aaron Levy, CoreOS
Much of the functionality in a Kubernetes cluster is managed by a reconciliation pattern within "controllers". The node, service, or deployment controllers (just to name a few) watch for changes to objects, then act on those changes to drive your cluster to a desired state. This same pattern can be used to implement custom logic, which can be used to extend the functionality of your cluster without ever needing to modify Kubernetes itself.
This talk will cover how to implement your own custom controller, from contacting the Kubernetes API to using existing libraries to easily watch, react, and update components in your cluster. By building on existing functionality and following a few best practices, you can quickly and easily implement your own custom controller.
About Aaron Levy
Aaron Levy is a software engineer at CoreOS, working on all things Kubernetes. He is also the lead maintainer of bootkube, a kubernetes-incubator project that enables launching self-hosted kubernetes clusters.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Thank
you.
Everyone
thanks
for
coming.
My
name
is
Aaron
levy,
I'm,
a
software
engineer
at
core
OS
and
today,
I
want
to
talk
about
writing
a
custom
kubernetes
controller.
So
first
I
want
to
talk
about
what
exactly
is
a
controller
whoa
we're
going
to
go
back
a
little
bit,
so
a
controller
is
ultimately
a
active
reconciliation
process.
A
So
in
simplest
sense,
we
want
to
look
at
the
desired
state
of
your
cluster,
the
actual
state
of
your
cluster
and
then
you're
going
to
make
changes
so
that
those
come
in
to
in
the
Union
and
so
in
pseudocode.
This
would
look
something
like
a
loop
that
just
constantly
is
getting
the
desired
state,
getting
the
current
state
and
then
making
the
changes
to
bring
those
into
sync
inside
of
kubernetes.
The
canonical
example
of
this
would
be
the
kubernetes
controller
manager
which,
inside
of
it
it
has
a
whole
bunch
of
different
kinds
of
controllers.
A
There's
the
deployment
controller,
daemon
node
service,
endpoint,
many
others
and
they're
kind
of
all
doing
this
act
of
reconciliation,
loop
internally,
culture,
applications
outside
of
of
the
controller
manager
follow
the
same
pattern
like
cube.
Dns
is
watching
services
and
pods
that
are
being
created
and
destroyed
in
the
cluster
and
then
updating
DNS
records.
Based
on
that,
other
tools,
like
flannel,
is
going
to
watch
the
network
cider
that
gets
assigned
to
pods,
and
then
it's
going
to
translate
that
into
network
settings
on
the
nodes
themselves,
or
even
things
like
SPD
operator,
Prometheus
operator.
A
These
are
going
to
create
entirely
new
concepts
inside
of
the
cluster
like
a
net
CD
cluster,
and
then
it's
going
to
take
that
that
third
party
resource
object
and
then
it's
going
to
translate
that
into
your
desired
state
and
constantly
doing
the
same
resolution
loop.
So
it's
kind
of
this
pattern
is
used
all
over
kubernetes.
So
how
can
we
you
know,
building
new
custom
controllers
use
these
main
things
are
that
you're
able
to
extend
the
functionality
of
your
cluster
of
existing
objects.
A
So
you
could
add
new
information
like
new
annotations,
new
labels
and
then
act
off
of
those
and
build
a
new
functionality.
You
could
add
entirely
new
concepts
that
don't
actually
exist
in
the
core
of
kubernetes
itself.
Like
I
mentioned
the
sed
operator.
The
idea
of
being
able
to
unlike
just
create
a
cluster
from
you
know,
you
create
an
object.
You
say
I
want
to
three-note
it
a
cluster,
that's
not
something
that's
built
in
the
kubernetes
core,
but
it's
something
that
you're
able
to
build.
On
top
of
you
can
automate
administrations
of
normal
tasks.
A
You
could
wholesale
replace
existing
components
you
could
have.
Maybe
you
know
you
have
specific
needs.
The
tube
proxy
doesn't
doesn't
address,
and
so
you
could
write
your
own
kind
of
proxy.
That
follows
the
same
loop
and
you
can
do
all
of
this
without
actually
modifying
the
core
kubernetes
codebase.
So
it's
actually
pretty
powerful
and
pretty
extensible.
A
So
I
want
to
talk
about
building
an
example
and
the
example
I'm
going
to
walk
through
is
building
a
node
reboot
operator,
not
sure
how
useful
this
actually
is,
but
I
think
it
touches
on
excusing
a
few
concepts
that
are
make
discussing.
That's
pretty
easy,
so
this
this
operator
would
allow
an
administrator
to
trigger
a
node
reboot
just
by
a
cube,
CTL,
so
you're
not
having
to
SSH
into
the
box.
You
can
just
do
it
via
kubernetes
directly
and
then
also
support
rolling
updates
across
the
cluster.
A
So
this
is
just
a
hypothetical
thing
that
we
can
talk
about.
It's
fairly
easy
to
build,
so
the
operator
itself
is
going
to
be
composed
of
a
couple
components:
it's
going
to
be
a
reboot
agent.
This
is
going
to
run
on
all
nodes,
there's
going
to
be
a
reboot
controller,
and
this
is
a
single
instance
and
then
they're
going
to
communicate
their
actual
and
their
desired
State
through
annotations,
so
reboot
now
reboot
in
progress
or
reboot
needed.
So
the
just
a
general
architecture.
We
have
our
reboot
controller
three
nodes.
A
A
Well,
they
all
need
a
reboot
we're
just
going
to
roll
a
reboot
across
the
cluster
and
our
reboot
controller
is
going
to
look
at
the
cluster
and
it's
going
to
look
at
the
current
state
and
say
well:
I'm
allowed
to
have
a
maximum
number
of
unavailable
nodes
of
one
and
currently
they're
all
available,
so
we're
safe
to
actually
reboot
a
node.
So
we'll
go
ahead
and
reboot
that.
But
then
we
stop,
because
the
maximum
number
of
unavailable
nodes
has
been
reached.
So
we
wait
that
node
comes
up.
A
It's
now
safe
again
to
reboot
the
next
node.
We
read
that
and
on
and
so
we're
able.
The
controller
is
just
a
simple:
it's
just
coordinating
the
reboost
across
the
cluster
in
a
very
simple
way.
So
the
way
that
we're
going
to
build
this
is
that
we're
going
to
have
the
agent
is
going
to
be
a
daemon
set,
and
it's
going
to
run
a
pod
on
on
the
note
in
the
cluster,
and
it's
only
job
is
really
to
watch
for
an
annotation
on
the
notes.
A
A
We're
going
to
use
an
object
called
an
informer,
and
this
handles
object,
caching
and
handling
for
us.
How
many
people
here
show
of
hands
are
familiar
with
informers.
Okay,
I'll
be
going
into
a
bit
cool
so
and
then
we're
going
to
be
communicating
our
desire
or
actual
state
of
the
cluster
by
annotations
on
the
node
object.
A
So
I
have
a
functional
example
of
this,
so
sometimes
dense
code
is
hard
to
do
a
presentation
on
so
github
Aaron
Levie
cube
controller
demo.
So
it's
basically
the
same
exact
thing.
We're
talking
about
some
of
this
code
has
to
be
pseudocode
or
a
bridge,
but
that
just
has
the
full
example,
and
then
these
are
following
some
general
upstream
patterns,
but
there's
no
singular
right
way
of
writing
controllers.
There's
many
different
ways
of
doing
it.
We're
just
going
to
kind
of
follow
an
easy
path
and
take
advantage
of
a
lot
of
client
library
stuff.
A
A
So
v2
is
currently
recommended
and
v3
the
pr2
update
it
was
this
morning
and
so
v2
is
targeted
against
1/5,
but
it
can
work
against
one
six
clusters,
there's
just
some
new
objects
that
it
may
not
know
about,
and
then
v3
should
come
soon,
but
all
of
these
examples
are
using
v2
and
then
I'll
update
the
the
demo
repository
when
v3
comes
out
to
keep
up
with
that.
So
the
first
thing
we're
going
to
build
is
the
reboot
agent.
So
the
first
thing
we
want
to
do
is
we
need
to
determine
ourself.
A
So
the
reason
for
this
is
that,
when
we're
running
the
agent
on
every
node,
we
don't
want
to
have
to
create
a
special
pod
for
each
node
and
say
this
is
pod
one.
That
knows
it's
on
node
one.
We
want
to
create
cattle,
not
pets,
and
so
we
need
some
way
of
determining
ourselves
and
the
way
to
do
this
in
this
case
is
we're
going
to
use
the
downward
API.
A
So
when
the
we're
able
to
create
a
deployment-
and
this
pot
is
going
to
be
run
on
a
bunch
of
different
nodes
and
at
runtime
when
the
couplet
runs
that
pod,
it's
going
to
inject,
that
environment
variable
with
the
correct
information
and
then
we,
as
as
the
authors
of
this
controller,
can
consume
it
in
a
couple
different
ways.
One
way
would
just
be
to
in
our
code,
get
that
environment
variable
and
then
use
it.
A
In
our
case,
we're
just
going
to
pull
it
directly
from
the
the
environment,
so
now
that
we've
determined
ourselves,
the
next
thing
we
need
to
do
is
create
a
client
so
that
we
can
speak
securely
to
the
API
server
and
there's
two
main
things
that
I
that
I
want
to
touch
on
here.
So
the
middle,
if
block,
we're
creating
a
config,
a
client
config
in
two
different
ways,
one
is
the
in
cluster
config
and
another
is
creating
a
config
using
a
reference.
A
Coop
config
file,
the
in
cluster
configure
going
to
be
using
when
you
launch
a
pod
in
the
cluster,
it's
going
to
get
a
service
account
injected
into
it
automatically,
and
that's
going
to
have
all
of
the
information
you
need
to
be
able
to
securely
contact
the
API
server.
It's
going
to
inherit
any
permissions
that
you
gave
that
service
account.
Generally,
that's
just
the
right
way
of
allowing
your
pod
if
it's
in
the
cluster
to
contact
the
API
server.
A
All
of
these
external
ones,
they're
just
clients
of
the
API.
So
if
it's
running
locally,
while
you're
developing
it's
going
to
behave
the
same
more
or
less
as
if
it
was
in
the
cluster
and
then
once
you've
created
the
config,
you
just
there's
a
new
config
method
and
it
gives
you
back
a
kubernetes
client.
So
at
this
point
we
have
enough
information.
We
can
securely
contact
the
API.
We
know
who
we
are.
We
could
write
a
simple
loop.
A
Maybe
our
controller
is
the
node
information
printing
controller,
and
so
we
have
we
print
out
the
the
labels
that
are
on
our
own
node
and
we
print
out
the
number
of
nodes
that
exist.
So
pretty
pretty
straightforward.
To
do
this
you
can.
The
client
is
actually
pretty
intuitive,
pretty
easy
to
use,
but
those
kinds
of
operations.
A
If
you're
constantly
saying
go
to
the
API
server
and
give
me
back
all
of
these
objects,
it
can
actually
become
pretty
expensive,
you're
extracting
information
out
of
that
CD
deserializing
it
possibly
doing
server-side
filtering
of
it
realized
it's
sending
it
to
the
client
deserializing
it
again.
So
you
know
for
doing
these
kind
of
get
and
list
operations
a
lot
of
times
in
code.
A
What
we're
going
to
end
up
doing
is
we
need
to
build
a
cache,
and
the
other
option
is
that
you
know
we
may
not
want
to
have
to
pull
the
API
and
say
well,
what's
the
full
state
of
all
the
nodes
in
the
cluster
every
10
second,
and
over
and
over
again-
and
so
in
this
case
we
just
want
to
be
notified.
Has
a
node
been
added?
Has
it
been
deleted?
A
Has
it
been
updated
and
so
that's
building
and
watches,
and
in
this
case
we
already
get
this
from
something
called
an
informer
and
so
a
simple
overview
of
it.
You
know
calling
new
and
former
and
I'll
go
over
each
of
these
in
more
depth,
but
you're
going
to
provide
it.
A
lip
swatch
object.
The
type
of
objects
that
you
care
about
operating
on
in
this
case
we're
going
to
care
about
nodes,
a
resync
period
and
event
handler
functions,
so
the
first
one
I'm
going
to
go
over
is
the
list
watch
object.
A
So
let's
start
here
at
the
the
return
statement.
So
if
you
see
here
we're
still
using
the
client
like
we
did
before
we're
saying
we
want
to
get
the
nodes
and
we
want
to
list
them.
But
in
this
case
we're
talking
about
our
agent
and
our
agent
only
cares
about
itself.
It
doesn't
care
about
anything
else,
any
other
nodes
in
the
cluster.
So
we
want
to
filter
this
down.
To
only
show
us
updates
about
our
own
node,
so
we
see
at
the
top
there
we're
creating
a
field
selector.
A
That
says,
we
only
care
if
the
metadata
and
the
name
of
that
actual
pod
or
of
that
that
node
actually
matches
matches
ourself.
So
then
we
use
that
selector
when
we're
passing
that
into
the
list
function
and
the
watch
function,
and
so
that
just
allows
you
to
restrict
the
set
of
information
that
you're
actually
operating
on.
This
could
be
labels.
This
could
be
field.
This
could
be
all
kinds
of
information,
so
it's
pretty
flexible
in
how
you
can
portion
out
what
you
actually
want
to
work
on.
A
A
So
the
other
part
is
the
resync
period,
and
this
is
this
is
important
towards
that
kind
of
constant
rethinking
of
current
versus
desired
state.
What
this
means
is
that
at
that
interval,
it's
going
to
riku
all
of
the
objects
that
you
have
in
your
cache
and
fire
the
update
function
over
again,
and
so
what
this
allows
you
to
easily
tie
into
is
maybe
you've
missed.
A
You
know
some
kind
of
some
kind
of
event
adds
leaves
or,
like
you
know,
maybe
you
were
trying
to
perform
something
and
it
actually
failed
and
you
weren't
actually
able
to
do
that
update
what
this
means
is
that
maybe
you
set
it
to
every
30
seconds.
I
just
want
to
go
through
all
the
objects
again
and
recheck
the
state,
and
so
that's
a
way
that
you
can
just
constantly
be
in
this
loop
of
like
okay,
everything
is
still
good.
A
I,
don't
have
to
like
make
sure
that
everything
has
has
successfully
gone
through,
because
I
know
that
I'm
just
going
to
check
it
again
soon
so
given
if
we
were
to
pass
in
those
things
that
we
just
that
we
just
talked
about
into
the
new
Informer.
What
we
would
get
back
is
a
store
and
a
controller,
and
one
of
the
things
we're
going
to
do
is
just
as
a
helper.
There
are
the
store
to
node
list
or
store
to
pod
Lister.
This
just
makes
takes
away
some
boilerplate
of
doing
type
conversions.
A
So
we
just
use
that
because
it
makes
listing
everything
out
of
out
of
the
cash
easy,
and
then
we
tell
the
controller
we
want
it
to
start
and
what
that
means
is
it's
going
to
reach
out
to
the
API
service
or
populating
the
local
cache
and
then
start
triggering
these
update
functions,
and
then
we
can
use
the
cache
normally
by
saying
you
know
list
or
get,
and
it's
just
going
to
pull
from
our
local
state.
So,
given
that
information,
we
can
actually
start
building
this
controllers,
so
the
first
is
the
agents
itself.
A
So
what
we're
going
to
tie
in
to
is
that
that
update
function
in
the
informer
itself,
because
we
don't
care
when
nodes
get
deleted,
because
if
they're
deleted,
we
don't
care
about
rebooting
them
and
when
nodes
are
added,
we
don't
really
just
want
to
try
and
reboot
them
immediately.
So
we're
only
going
to
tie
into
the
update
function
and
really
it's
only
job
is
to
look.
Does
it
have
the
annotation
that
says
reboot
now
and
if
it
does
have
that
annotation?
A
That
means
it
should
reboot
and
so
we're
going
to
first
just
get
rid
of
that
annotation.
We
don't
need
it
anymore,
then
we're
going
to
update
that
in
the
API
server,
so
that
that's
reflected
and
then
we're
going
to
reboot
the
node,
and
so
the
the
client
logic
is
essentially,
you
know,
get
self
determined.
A
Who
I
am
create
a
client
and
then
we're
going
to
create
an
informer
and
so
pretend
that
this
took
self
took
the
the
update
function
and
then
inserted
that
into
the
informer
we
talked
about,
and
then
we
tell
the
controller
to
run
and
we're
just
going
to
start
getting.
Those
update
functions
are
automatically
triggered
it's
going
to
walk
through
this
logic,
and
then
it
will
reboot
your
note
for
you
just
by
adding
an
annotation.
So
the
note
object.
A
So
then,
the
next
piece
would
be
the
reboot
controller
itself,
and
this
what
we
actually
care
about
is
is
tying
into
all
three
add
update
and
delete,
because
it's
possible
that
maybe
the
maximum
number
of
available
notes
has
changed
if
we
added
new
nodes
to
our
clusters-
and
maybe
we
can
reboot
more
of
them
now,
it's
safe
to
actually
reboot.
So
we're
going
to
write
a
handler
function
that
can
be
used
for,
add,
delete
and
then
update.
We
only
care
about
the
new
copy
of
the
object.
A
We
don't
care
about
the
old,
so
we
ignore
it
and
we
just
passed
a
new
object
directly
on
so
to
go
over
that
handler
function.
What
we're
going
to
do
is
the
first
thing
is
if
the
reboot
needed
annotation
exists.
So
this
was
that
first
step
that
we
saw
where
we
just
sprayed
across
the
cluster
and
said
every
node
we
want
to.
We
want
to
reboot
every
node
in
the
cluster,
but
we
don't
want
to
have
to
coordinate
it.
A
We
just
want
to
say
that
should
happen,
and
then
let
this
controller
coordinate
that
action
for
us
and
do
it
safely.
So
we
just
check
well,
this
node
changed
in
some
way
this
node
object
changed.
Does
it
have
that
reboot
needed
annotation?
If
it
doesn't,
then
just
move
on,
we
don't
care
about
this
update
at
all.
The
next
step
would
be
if
it
does
have
that
annotation.
We
know
it
needs
to
reboot,
but
we
need
to
determine.
A
Is
it
actually
safe
to
do
so,
and
so
we're
going
to
get
the
number
of
unavailable
nodes
and
we're
going
to
pair
that
events,
our
configurable
of
the
max
number,
and
if
we're
over
that
limit,
then
we're
just
going
to
return
we're
going
to
ignore
this,
because
we're
know,
based
on
the
recent
period
we're
going
to
come
back
later
and
we're
going
to
check
this
node
again.
So
we're
not
super
worried
that
we're
just
ignoring
it
for
now.
We
know
it
needs
to
be
rebooted
later,
we'll
figure
it
out
so
that
constant,
just
reconciliation
mode.
A
But
if
we
get
to
this
point,
it's
safe
to
reboot
a
node,
so
we
can
delete
the
reboot
needed
annotation.
We
add
the
reboot
now
annotation,
and
this
is
what
ultimately
will
coordinate
with
the
with
the
agents
itself,
and
then
we
update
the
object
in
the
API,
and
then
the
agent
will
see
this
change
and
that
will
reboot
the
node.
So
the
pretty
simple
logic
between
the
two
and
we
didn't
even
really
have
to
write
that
much
code
at
all.
All
of
the
client
logic
is
kind
of
handled
for
us.
A
A
What
does
what
this
could
happen
is
that
you
know
we're
getting
this
update,
we're
getting
this
update
event,
and
it
says:
okay,
it's
got
a
reboot,
now
annotation,
but
then
every
time
you
see
that
you're
going
to
say
well,
then,
give
me
every
node
in
the
cluster,
so
I
can
determine
whether
it's
safe
to
reboot
or
not,
and
that
could
be
an
expensive
operation
if
you're
going
to
the
API.
Every
time
instead,
because
our
Informer
gives
us
both
those
events
as
well
as
the
local
cash
we
use,
ask
the
local
cash.
A
So
we
get
the
event
and
then
we
get
that
node
store
and
we
say
node
store
list,
and
then
we
just
get
all
of
the
nodes
and
it's
not
an
expensive
operation.
So
it's
cheap
for
us
to
say
just
go
relook
at
the
state
over
and
over
again,
and
we
know
that
the
Informer
has
kept
it
up-to-date
for
us.
So
in
this
case
we
just
get
them
all.
And
then
we
determine
is
the
node
rebooting
or
is
the
node
in
a
not
ready
state?
We
should
add
it
to
this
count
of
unavailable
nodes.
A
So
even
though
those
those
examples
seem
pretty
simple,
we
did
do
some
things
that
we
probably
shouldn't
have
done
and
so
I
want
to
touch
on
those
one
is
you
shouldn't
actually
modify
cash
objects
and
I
was
doing
this
and
those
examples
for
brevity.
But
this
is
an
example
of
something
we
did.
We
were
getting
an
object
from
the
cache
and
then
we
were
mutating
it.
We
were
changing
the
annotation
and
then
we
were
updating
it
in
the
API.
A
But
if
that
update
call
fails,
then
our
cache
is
now
incorrect
until
it
gets
resync
a
lot.
Another
common
pattern
here
is
that
you
might
have
multiple
routines
that
are
all
operating
on
the
cache
at
the
same
time,
so
you've
just
invalidated
the
cache
for
other
routines
that
may
be
expecting
to
operate
off
of
that
object.
So
the
solution
to
this
is
fairly
straightforward.
A
So
this
is
relatively
easy
to
check
once
you
start
your
controller,
you
can
ask
it:
has
it
synced
yet
and
then
just
don't
start
any
of
your
processing
until
that
happened,
but
it's
just
a
common
pattern
to
follow
or
something
to
be
aware
of
that.
You
could
start
up
your
program
and
it
can
make
some
bad
decisions
based
on
a
partial
cache
state,
some
of
the
other
things
that
I
want
to
go
over
some
helpful
tools
that
you
know
we
didn't
actually
use
so
leader
election
work,
use
third-party
resources
and
shared
informers.
A
So
we
go
over
each
of
those,
so
leader
election
I
didn't
use
because
it's
not
in
the
client
go
library.
Yet
there's
an
open
issue
about
that.
It
can
be
vendor
directly
from
the
kubernetes
repository,
but
that
does
mean
that
your
vendor
in
a
whole
lot
more
code,
but
it
is
a
really
useful
construct.
So
in
the
scheduling,
the
controller
manager,
for
example,
these
are
both
leader
elected
components
and
you
can
actually
copy
the
the
code
out
of
those.
A
If
you
want
a
reference
example,
but
so
you
can
run
multiple
copies
have
have
hots
payers
running
and
that
you
know
that
they're,
their
leader
elected.
So
in
our
case
for
the
reboot
controller,
when
we
set
replicas
equals
one,
that's
because
we
don't
want
them
fighting
over
the
decision
of
how
many
nodes
to
reboot
you
could
get
into
two
bad
situations,
so
leader
election
would
be
one
method
of
protecting
against
that
another
is
work
use.
A
So
these
are
not
in
client
go
v2,
but
they
are
and
they
they
should
be
in
v3
pretty
soon,
and
these
just
allow
you
to
easily
parallel
eyes
processing.
They
have
nice
cue
types
like
rate
limiting.
So
if
you
get
in
a
hot
loop
of
trying
to
do
some
action
on
an
event
and
it
fails,
it's
going
to
take
care
of
backing
off
riku
event
for
you,
you
can
also
just
simple
things
like
collapsing:
multiple
updates
into
the
same
into
the
same
work
item,
and
so
the
reboot
controller
can
MIT
take
advantage
of
this.
A
Maybe
the
note
has
been
updated
multiple
times,
but
we
only
want
to
pop
that
off
once
and
take
action
on
it
and
then
the
other
thing
with
with
the
work
uses
that
pattern
where
you
need
to
wait
for
your
seeing
your
your
cash
flows
thing.
It
just
makes
that
pattern
super
easy,
because
your
event
handlers
are
all
essentially
just
pushing
onto
a
work
queue
and
then
you're,
spinning
up
workers
that
are
now
just
consuming
off
of
that
queue.
A
So
you
just
don't
start
them
until
you
said:
okay,
my
cash
is
synced,
even
though
these
events
are
filing
and
filling
up
the
queue
now
I
can
start
popping
them
off
and
processing
third-party
resources.
This
actually
is
a
presentation
unto
itself,
but
it's
a
really
important
topic,
if
you're
unfamiliar,
essentially
what
they
allow.
You
to
do
is
dynamically,
create
new
objects
in
in
the
API,
and
so
this
can
be
really
powerful,
because
in
our
controller,
what
we
did
was
we
were
adding
annotations
to
nodes,
because
that
fits
the
model.
A
We're
talking
about
rebooting
nodes,
so
adding
an
annotation
to
a
node
kind
of
makes
sense.
But
maybe
the
action
that
you
want
to
take
doesn't
have
anything
to
do
with
nodes
or
an
existing
object.
So
third-party
resources,
you
can
create
an
entirely
new
object,
and
so
an
example
of
this
is
the
sed
operator.
What
it
does
is
there's
a
new
third-party
resource
object
called
a
TD
cluster
and
you
can
stamp
out
new
sed
clusters
and
it
has
a
data
type.
A
Where
you
can
say
you
know
the
number
of
replicas
that
I
want
are
three
and
the
version
is
X,
and
then
it's
going
to
take
that
object
and
then
use
that
to
then
start
up
a
cluster.
But
there
was
no
existing.
You
know
objects.
It
would
be
well
that
liked
to
be
used
to
well
describe
that
need.
So
this
is
a
really
powerful
thing
in
terms
of
like
our
reboot
controller,
something
that
we
could
do
is
you
know,
maybe
adding
reboot
groups
third
party
resource,
and
so
you
could
define
new
groups,
maybe
by
label.
A
A
Another
thing
are
the
shared
informers.
So
what
this
means
is
that,
when
we're
stamping
out
these
informers
are
creating
a
local
cache
for
that
one,
Informer
and
so
you're
saying
this
is
a
set
of
objects
that
I
care
about
I
care
about
this
set
of
nodes
and
you're
going
to
create
a
cache
for
that
set
of
nodes.
But
what
can
end
up
happening
is
that
maybe
you
care
about
a
whole
bunch
of
different
set
of
nodes,
but
they
may
be
overlapping
sets.
A
A
This
is
a
common
pattern
found
in
coop
controller
manager
because
it
had
so
many
internal
controllers,
but
one
of
the
things
that
I
found
is
that
I
don't
actually
end
up
using
this
in
practice
because,
most
of
the
time,
what
I'm
writing
is
going
to
be
a
single
task.
Controller
like
in
this
case
the
the
controller
that
we
go
over,
is
that
I
just
care
about
one
node
or
I
care
about
all
nodes,
and
those
are
the
only
things
that
I'm
watching.
A
So
in
general,
if
you
are
going
to
try
and
write
some
some
custom
controllers,
some
helpful
resources
under
the
community
repository
there
is
a
controllers
document
that
describes
a
lot
of
the
stuff
that
I'm
talking
about
best
practices
things
you
should
do
things
you
shouldn't.
Do.
There's
also
in
terms
of
there's
some
design
principles
about
control
logic.
Just
these
are.
A
These
are
internal
things
for
kubernetes
itself,
but
they're,
very
good
patterns
to
follow
so
they're
good
things
to
take
a
look
at
when,
if
you
want
like
reference
examples
under
the
kubernetes
repository
package,
controller,
just
have
pretty
much
every
internal
controller
code
in
there,
and
so
there's
a
lot
of
good
examples
that
you
can
you
for
use
for
reference.
The
first
time.
I
was
writing
this
kind
of
pattern.
I
just
looked
at
how
Kubb
dns
did
that?
How
did
it
create
a
client?
How
did
it
ask
the
API
server
for
services?
A
How
did
it
keep
them
updated?
So
same
kind
of
thing
you
can
just
follow
these
patterns
pretty
easily
the
client
go.
Repository
itself
has
a
lot
of
good
examples.
It
shows
in
the
examples
directory
there's
how
to
interact
with
third-party
resources,
how
to
create
different
kinds
of
clients,
as
I
said
before,
the
tube
controller
demo
there's
a
functional
copy
of
this
in
in
that
repository,
and
then
also
I'm
talking
about
using
client
go,
but
it
just
isn't
restricted
to
just
using
go.