►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Why Are We Copying and Pasting So Much? - Solly Ross & Philip Wittrock, Google
If you’ve contributed to the one of the Kubernetes controllers or built a custom controller (such as an operator), you’ve probably had to copy a large amount of boilerplate code. Other bits of the controllers may look similar, but have small variations, making it difficult to separate what is bespoke and what is common logic. The copy/paste approach makes code harder to refactor, understand, and debug. Additionally, neither bug fixes nor improvements to the original are not propagated to the copies. So what can we about it? Enter controller-runtime, a new foundation library for the next generation of controllers which replaces the boilerplate code with a few function calls. In this session, we’ll look at common controller patterns across both the Kubernetes codebase broader ecosystem, identifying the common elements, simplifying them with the utilities in controller-runtime.
To learn more: https://sched.co/GrUR
A
Thank
you
all
for
coming
out
here
before
we
get
started
just
a
brief
disclaimer.
So
there's
a
couple
of
spots
in
this
presentation
where
we
stripped
the
license
text
off
the
top
of
some
blocks
of
code
to
make
it
easier
to
read
the
code
on
the
slide
in
all
those
cases.
Unless
there's
an
explicit
license
text
at
the
top
of
the
file,
it's
all
Apache
to
copyright,
the
kubernetes
authors,
all
right
so
I'm
Solly
Ross,
and
this
is
Phil
Whitlock
and
we
want
to
do
a
quick
survey.
A
A
We
were
wondering
Phil
and
I
at
one
point:
why
are
we
copying
and
pasting
so
much
so,
for
those
of
you
who
are
not
as
familiar
controllers
are
the
one
of
the
basic
building
blocks
of
kubernetes,
so
kubernetes
api
is,
are
not
imperative.
They're
decorative,
they
just
store
state.
Very
simple:
the
controllers
are
the
loops
that
actually
check
the
chain
for
changes
to
those
objects
that
state
and
and
check
the
real
world
and
figure
out
the
difference
in
fix
it.
A
So,
hypothetically,
a
controller
should
be
pretty
simple:
we
watch
some
resources,
we
do
some
business
logic.
We
reconcile
the
differences
between
the
real
world
and
the
desired
state,
and
we'd
have
a
little
bit
of
plumbing
to
wire
things
together
right.
How
complicated
could
it
be
so
as
developers
probably
one
of
the
first
things
that
we
do
when
we
start
in
a
new
environment?
If
we
say
we
want
to
write
a
controller,
is
we
look
for
a
sample
right
and
there
is
a
sample
controller
for
kubernetes
Kate's
thought
is
live
sample
controller.
A
So
if
we
look
at
the
main
file
of
the
sample
controller
controller,
we
can
see
that
it's
429
lines
long
and
if
we
take
a
look
at
the
rest
of
sample
controller,
not
even
including
the
vendor
directory,
we
can
see
that
there's
28
directories
and
52
files,
which
seems
rather
large
and
cumbersome
for
a
sample.
But
you
know-
maybe
it's
all
just
comments.
A
Maybe
if
we
just
rotate
that
a
little
bit
perfect.
So
that's
simple
right.
You
guys
can
all
read
and
understand
that
perfectly
so
I'm
another
serve
quick
survey,
Cheryl
fans.
How
much
do
you
think
this
is
boilerplate
plumbing
raise
your
hands?
If
you
think
it's
like
a
quarter
plumbing
raise
your
hands.
If
you
think
it's
half
plumbing
all
right,
let's
take
a
look.
A
Everything
Lisa
has
good
comments,
though,
that
are
clearly
understandable
if
you've
never
written
a
controller
before
so.
You
guys
couldn't
read
that
and
I
couldn't
read
that,
but
thankfully
Phil
went
and
did
some
work
ahead
of
time
and
he
has
prepared
an
overview
of
what
that
sample
controller
does
for
us.
So
let's,
let's
take
a
look
at
it.
So
the
first
thing
we
do
is
we
copy
clients
into
a
reconciler
struct.
Then
we
create
a
queue
for
reconcile
requests
and
then
we
add
event
handlers
for
all
the
object.
A
A
A
A
That
must
have
changed
so
then
we
once
we
have
all
that
we
have
to
write
logic,
to
turn
our
objects
into
keys,
that
we
can
add
to
the
queue
and
typecast
those
requests
into
a
string
and
do
error
handling
and
for
some
reason
we
can't
typecast
those
requests
into
a
string,
and
then
we
start
some
worker
threads,
but
not
too
many
worker
threads.
And
then
we
start
some
informers
and
we
wait
for
a
caches
to
sync
and
I
think
we
probably
should
be
done
at
that
point.
It's.
B
B
A
Cool
so
I'm,
clearly
very,
very,
very
simple,
and
unless
you
all
think
that
this
is
just
limited
to
the
sample
controller,
another
quick
quiz
for
you
guys
for
you
all
to
think
about
in
your
heads.
One
of
these
is
a
sample
controller.
One
of
them
is
a
built-in
kubernetes
controller,
and
one
of
them
is
a
third-party
operator
for
what
for
a
popular
database
and
all
that
read,
that's
all
boilerplate,
so,
as
you
can
see,
they
all
have
a
lot
of
boilerplate.
So,
let's
break
that
down
into
numbers.
A
A
So
what
what
do
we
do
when
we
have
a
bunch
of
code
that
we're
repeating
over
and
over
again
we
put
in
a
library
and
when
that
code
is
used
across
all
of
our
projects
and
becomes
the
de-facto
library?
We
call
it
a
standard
library.
So
we
talked
to
the
NGO
core
maintainer
z--,
and
we
got
them
to
put
a
standard
library
for
controllers
and
go
1.12,
which
you
can
just
import
as
controllers
just
kidding.
A
But
seriously,
though
we
we
wrote,
we
wrote
a
library,
a
standard
library,
even
for
writing
controllers
and
it's
called
controller
runtime.
So
let's
take
a
look
at
how
controller
runtime
has
captured
some
of
this
boilerplate
some
of
these
patterns
and
has
refactored
it
out
into
reusable
chunks
that
can
make
it
easier
for
you
to
build
a
controller.
B
Yeah
so
the
first
thing
you're
gonna
want
to
do
when
you
credit,
when
you're
creating
controller
is
create
a
controller
and
so
over
on
the
right.
What
you
see
is
the
controller
runtime
version
of
creating
a
controller
which
basically
says
just
set
one
up
for
me
and
start
it.
Here's
the
signal
handler
to
use
on
the
left.
You
have
the
sample
controller
in
version
of
the
same
thing,
which
is
manually,
the
Stan
shading
cues,
in
setting
up
various
machinery
to
get
started.
B
A
This
looks
all
great
and
awesome
Phil
and
it
looks
like
it's
really
easy
to
get
started,
but
I'm
a
problem
with
libraries
sometimes,
and
that
is
I,
sometimes
like
to
do
crazy,
complicated
stuff
and
sometimes
I,
get
to
a
library
and
it's
great
for
getting
started,
and
then
I
go
to
do
my
crazy,
complicated
stuff
and
the
library
won't.
Let
me
do
it
the
abstractions,
it
provides.
A
B
That
problem
makes
me
cry
Solly.
This
is
the
third
iteration
of
us
trying
to
solve
this
problem.
The
first
iteration
we
started
two
years
ago
was
something
called
API
server
builder
and
there
we
looked
at
all
the
boilerplate
code.
When
you
say
you
don't
need
to
hand
copy
that
and
write
it,
we'll
just
code
generator
for
you
because
we're
generating
so
much
code
anyway,
and
so
we
did
that
for
a
while,
and
then
we
looked
at
it
again
kind
of
sideways
we're
like.
Why
are
we
generating
that?
Why
don't
we
just
make
it
a
library?
B
B
But
the
problem
was
that
library
called
directly
the
underlying
API
machinery,
and
so
as
soon
as
you
wanted
to
do
something
that
the
builder
didn't
support,
you
were
back
to
the
bad
old
days
of
cobbling
it
together
yourself.
So
when
we
redesigned
for
the
controller,
runtime
library
and
cupola
window,
oh,
we
thought
this
and
said
you
know.
B
What
we
really
need
is
layers
and
you
should
be
able
to
break
through
these
layers
and
do
more
and
more
complex
things
while
still
having
abstractions
on
top
of
some
of
the
stuff
API
machinery
is
provided,
and
so
we
start
out
at
the
top
of
the
pyramid,
with
a
very
simple
pattern
that
satisfies
what
most
people
want
to
do,
especially
in
their
first
few
weeks
of
work.
Right,
which
is
listening
to
a
couple.
B
B
Here,
at
that
same
layer
is
the
owns
relationship
represented,
and
this
one
actually
allows
a
little
more
functionality.
You
can
say
control
whether
do
I
care,
if
it's
a
controller
or
not,
and
then
compose
these
pieces
together
to
use
again
that
same
source
on
a
different
type
with
a
different
event.
B
This
one
you
haven't
seen
yet
because
this
is
not
so
simple,
it
belongs
in
the
simplest
use
case,
but
it's
still
relatively
common
right
and
and
frequently
we'll
have
people
come
to
us
with
scenarios
such
as
hey
I
wrote
this
thing
that
needs
to
trigger
an
event
when
a
when
a
config
map
changes
for
some
pots
right.
The
deployment
has
a
reference
to
a
config
map.
Config
Mac
gets
updated
now,
I
want
to
do
something:
that's
not
a
knowns
relationship.
B
That's
this
bespoke
relationship
that
the
user
kind
of
defines,
even
though
that
one
is
actual
it'll
more
common,
and
so
we
can
allow
you
to
provide
a
mapping
function
to
say,
given
this
object
map
it
to
another
set
of
objects,
and
this
has
access
to
a
client,
so
you
can
even
look
up
from
the
cache
particular
indexes
or
various
things
like
that.
Another
common
use
case
is
actually
we
have
something
that
needs
to
be
triggered
whenever
a
note
is
added
or
removed
to
a
cluster
and
scaled,
or
something
like
that.
B
So
we
do
see
these
arbitrary
relationships
happen,
but
we
still
don't
want
to
make
you
go
all
the
way
back
to
API
machinery.
If
you're,
like
you
know
what
I
do
have
something
a
little
more
bespoke
than
that
and
so
those
pieces
we
pass
in
or
actually
interfaces,
there's
a
source,
that's
an
interface,
and
then
there
is
an
event
handler.
That's
an
interface,
and
so
you
can
swap
out
the
source
to
be
something
like
a
pub
sub
event
source
or
a
web
hook.
Event
source.
If
you
were
to
implement
that
or
maybe.
B
B
And
say:
okay:
I
got
this
event
now
what
do
I
do
with
it?
For
the
cue
in
theory,
you
shouldn't
need
to
actually
write
your
own
event
handler.
The
mapping
function
is
actually
pretty
flexible,
but
we
designed
it
this
way,
because
we
know
that
we're
not
going
to
know
ahead
of
time
all
these
cases
people
are
going
to
encounter,
and
so
we
want
to
provide
that
flexibility,
so
I'm
going
to
say,
don't
use
this.
A
The
common
cases
should
be
easy
and
the
uncommon
cases
should
be
possible.
So
what
is
that?
What
does
this
get
us
right?
So
it
makes
it
really
easy
to
get
started.
It
makes
it
easy
for
us
to
build
what
we
want,
but
what
does
it
get
us
in
the
long
term?
Well
for
developers
in
addition
to
getting
started
easily,
we
get
the
ability
to
pull
in
new
features
performance
improvements
without
having
to
rewrite
all
of
our
code.
A
They
were
all
changes
to
underlying
implementation
details,
but
once
we
moved
beyond
kind
of
the
getting
started,
adding
new
features
move
to
like
day
two
operations,
things
get
even
more
interesting
for
ops
and
sre.
We
get
standard,
observability
and
behavior,
so
we
can
monitor
all
of
our
controllers
in
similar
ways.
We
can
write
a
baseline
set
of
dashboards
and
alerting.
A
Beyond
that,
we
get
a
stable
base
layer
for
people
to
build
on
top
of.
Instead
of
reinventing
the
wheel,
people
can
build
up
on
controller
run
time,
so
you
get
projects
like
queue
builder
and
operator
SDK,
which
helps
you
get
started.
Writing
your
operators
and
controllers,
and
they
didn't
have
to
reinvent
everything
from
scratch,
and
they
we
can
both
focus
on
writing
the
best
possible
abstractions
over
API
machinery
and.
B
One
thing
that
we
didn't
plan
for,
but
we've
actually
found,
is
that
different
folks
discovering
the
controller,
runtime
and
COO
builder
say
hey.
This
is
pretty
cool
and
we
like
the
top
end
of
this
pyramid,
but
for
this
specific
domain
we
could
actually
build
even
a
higher
layer
on
top
of
that
pyramid
right.
A
And
those
folks
get
to
focus
on
building
awesome
stuff
and
they
don't
have
to
deal
with
the
nitty
gritty
arcane
API
machinery.
So
let's
tie
this
all
back
together.
You
remember
this
one.
Before
this,
our
sample
controller
I
lied
a
little
bit.
I
actually
forgot
to
tell
you
about
this
piece
too:
that's
the
main,
funk
it
wires
up
clients
and
pulls
in
flags
and
and
loads
tokens
for
talking
to
the
cluster.
So
there
was
more
so
if
we
look
at
what
this
looks
like
a
controller
runtime,
it's
actually,
it
fits
on
the
slide.
A
It's
almost
readable
at
this
size.
So
there's
three
parts
here.
The
first
part
is,
is
basically
the
entirety
of
that
main
file.
It
just
loads
up
and
declares.
What
we
want
to
do.
Second
part
is
our
business
logic
and
that
third
part
over
there
on
the
edge
is
just
the
structure
for
our
deployment
that
we
wanted
to
create.
That's
what
the
sample
controller
does
it
creates
deployments.
A
So,
in
conclusion,
controllers
can
be
simple
and
we
don't
need
to
copy
and
paste
as
much
if
we
think
about
when
we
find
ourselves
copying
and
pasting
what
our
abstractions
should
be,
and
we
think
about
how
we
can
refactor
those
out
into
reusable
building
blocks
that
are
flexible
enough
to.
Let
us
get
our
work
done,
but
simple
enough,
so
that
we
can
understand
them
easily.
A
Okay,
so
cute
builder
and
operator,
SDK
are
kind
of
two
sides
of
the
same
coin.
They
have
a
little
bit
of
a
different
focus,
but
they
still
use
the
same
underlying
libraries.
So
they
scaffold
out
your
initial
project
differently,
but
they
use
the
same
underlying
mechanics.
They
both
use
controller
run
time.
Yeah.
B
I
can
add
a
little
bit
that
to
that
too.
So
originally,
these
were
built
independently,
COO
builder
pre-blended,
oh
and
the
operator
SDK,
and
they
both
did
not.
Neither
one
looked
like
that
pyramid.
We
showed
you
right.
It
was
like
just
API
machinery
and
then
one
layer
on
top
of
it
and
both
of
us
looked
at
both
projects.
Looked
at
what
the
other
project
had
done
and
said:
hey,
there's,
actually
some
cool
stuff
in
there,
I
hadn't
thought
about,
and
I
would
like
to
copy
that.
B
B
B
C
B
B
The
so
one
way
of
doing
this
is
not
to
generate
controller
runtime
for
every
language
and
it's
to
expose
a
web
hook
based
approach,
for
instance,
and
so
there's
a
project
called
meta
controller
that
takes
that
approach.
That
is
probably
the
more
likely
direction.
I
see
the
language,
agnostic
approach,
taking
and
kind
of
is
in
theme.
B
With
the,
as
you
have
more
and
more
specific
things,
you
actually
have
give
funneled
into
a
more
specific
route,
and
so,
if
you're
doing
something
where
that's
good
enough
for
you
with
the
web,
what
the
web
hook
model
supports
go
for
it,
then,
when
you
actually
find
out,
you
need
some
piece
that
happens
to
be
written
only
and
go
go.
Do
that
I'd
be
also
interested
in
seeing
other
folks,
maybe
copying
our
API
is
our
approach
in
porting
it
to
other
languages.
That
would
be
interesting
to
me
as
well.
It.
A
Turns
out
maintaining
a
bunch
of
libraries
across
all
the
languages
that
everybody
wants
to
use
is
a
big
problem
like
takes
a
lot
of
people,
and
it
takes
a
lot
of
people
who
are
good
at
the
individual
languages
right.
One
of
the
things
that's
important
in
any
project
like
this
is
that
the
code
should
feel
idiomatic.
So
if
you
just
you
know,
if
you
try
to
have
someone
who
didn't
say,
no
Python
very
well
just
knew
the
surface
kind
of
right,
controller,
runtime
and
Python.
It
might
come
out
more
cumbersome.
A
So
the
question
is:
to
what
extent
are
the
test,
fakes
and
and
common
test
harnesses
for
common
use
cases
supported
by
queue
builder?
So
we
do
have
some
fakes
for
a
lot
of
our
stuff.
One
of
the
things
that
we've
found
out
during
testing
cubular
itself
and
kind
of
the
the
queue
builder
maintainer
is
when
we,
when
we
wrote
controllers
ourself
before
we
wrote
Q
builder,
is
that
it
turns
out.
A
Eventually,
you
end
up
with
a
test
fake,
that
just
kind
of
looks
like
the
API
server,
but
like
missing
a
bunch
of
features,
and
maybe
with
some
more
bugs
in
it,
and
so
the
approach
that
we
encourage
instead
is
we
make
it
really
really
really
easy
to
just?
Have
your
test
harness
run
just
the
kubernetes
api
server
in
that.
B
B
A
We
have
not,
we
have
not
done
super
extensive
performance
testing,
but
we
have
written
a
number
of
kind
of
real-world
controllers
that
have
been
using
controller
runtime
pretty
successfully.
It
is
one
of
the
things
on
our
list
kind
of
things
that
we
would
like
to
do
going
through
and
and
doing
more
extensive
performance
profiling,
but
we
expect
that
the
impact
should
not
be
terrible
and.
B
Welcome
but
yeah
I
mean,
and
we
we
have
looked
at
it
like
one
thing
we've
done
is.
If
you
take
the
long
code
you
saw
originally,
that
was
read
with
the
with
the
spiky
stuff,
like
we
refactor
that
and
massaged
it
like.
We
didn't
just
start
over
and
say
we're
gonna
write
something
else.
It's
like
okay.
How
do
we
pull
that
into
a
lie,
or
how
would
he
pull
that
in
the
library?
B
So
in
most
areas
we
don't
expect
there
to
be
huge
performance
changes,
because
all
we
did
is
like
factors
stuff
out
into
a
library.
There
are
a
couple
places
where
we
did
do
stuff.
That
was
kind
of
novel,
which
is
in
the
client,
and
we
do
some
a
bit
of
reflection
in
there
in
a
couple
places.
So
that
would
be
probably
one
area
to
look
at
one
possible
performance
improvement
that
we
would
see
is.
The
client
is
very
friendly
towards
reading
from
the
cache
in
the
old
sample
controller
way.
B
You
have
a
Lister
and
you
have,
which
is
separate
from
from
your
client,
and
so
engineers
have
to
make
sure
they
get
the
programming
right
to
read
from
the
cache
separately
from
the
from
the
API
server,
and
so
by
getting
that
sort
of
simple
integration.
You
might
save
yourself
a
bug
that
would
cause
force
performance.
B
A
So
the
question
was:
if
I
do
things
this
way,
how
much
of
my
code
is
typecasting
and
the
answer
is
ideally
none.
We
have
attempted
to
write
things
in
a
way
that
either
do
not
require
typecasting
or
we
abstract
away
the
typecasting
from
you
in
in
a
way
that
we
attempt
to
actually
do
some
checks.
So
like
the
way
our
client
works.
We
have
what's
closer
to
a
dynamic
client,
except
that
you
pass
in
a
concrete
struct
and
we
use
the
type
of
that
struct
to
look
up
what
we're
supposed
to
fetch
right.
A
So
your
if
you
pass
in
a
pot
object,
you're
guaranteed
to
get
a
populated
pot
object
same
thing
with.
If
you
look
at
the
reconcile
key
so
in
traditional
handwritten
controllers,
especially
those
in
Korakuen
eddie's,
they
take
a
key
in
as
a
string
and
then
you
have
to
manually
split
that
into
namespace
and
name,
and
you
hope
you
do
it
right.
Otherwise,
you'll
get
errors
in
controller
runtime.
We
pass
you
in
a
priest,
get
split
key
in
a
struct
so
that
you
just
have
a
struct
with
namespace
and
name,
and
you
don't
have
to
typecast.
A
B
A
It
would
be
super
awesome
if,
eventually,
we
could
get
this
to
the
point
where
the
cork
and
controllers
were
also
using
controller
runtime.
But
for
the
moment
we
are
looking
at
mostly
just
third-party
controllers
and
also
kind
of
supporting
gradual
refactoring.
So
if
you
want
to
do
stuff
like
you're,
writing
a
new
cloud
controller-
and
you
want
to
run
some
of
the
cloud
controllers
from
core
kubernetes
plus
some
of
the
stuff,
you
wrote
your
own
self
with
controller
runtime,
you
can
do
that
and
run
them
in
the
same
controller
manager,
binary
I'd.
B
Like
to
add
something
to
that,
but
just
like
part
of
your
question
might
be
like,
is
the
project
being
built
with
this
right?
To
what
extent
is
it
and
like
the
kubernetes
as
a
project
is
probably
moving
towards
a
direction
of
more
components
or
independence
of
these
components?
And
so,
as
that
happens,
we
see
those
projects
running
on
controller
runtime.
B
So
one
example
is
the
cluster
API
uses
controller
runtime,
and
that's
because
it
was
started
outside
of
kubernetes
kubernetes
repo
and
had
a
fresh
start
taking
right,
for
instance,
these
very
complicated
deployment
controller
right,
each
of
which
was
copy
and
pasted
and
changed
a
little
bit
here
and
there.
That
would
be
like
a
pretty
big
endeavor,
I
think
and
and
not
clear
that
we,
you
would
see
not
clear
that
it's
the
best
investment
of
our
time
right
now.
It's.
A
So
your
your
custom,
it
uses
like
your
custom
resources,
are
defined
in
terms
of
throwing
types,
and
you
can
also
like
cue
builder,
is,
is
designed
from
the
perspective,
like
the
scaffold
that
it
sets
up,
is
I.
Have
some
going
types
I
want
to
turn
those
into
a
custom
resource
definition,
so
I
can
use
them
as
custom
resources
on
the
cluster
and
I
want
to
reconcile
them,
but
you
can
also
use
built-in
types
as
well
with
control
or
runtime
like
deployment,
and
so.