►
From YouTube: Controllers: Lambda Functions for Extending your Infrastructure - Ross Guarino & Terin Stock
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Controllers: Lambda Functions for Extending your Infrastructure - Ross Guarino & Terin Stock, Cloudflare (Intermediate Skill Level)
This session demonstrates how to leverage Kubernetes Controllers and Initializers as a framework for building transparent extensions of your Kubernetes cluster. Using a live coding exercise and demo, this presentation will showcase the possibilities of the basic programming paradigms the Kubernetes API server makes easy. As an SRE build custom integrations directly into the Kubernetes api that transparently enhance the developer experience. Ross Guarino will talk about how this pattern is used extensively at Cloudflare to provide a feature rich internal platform for our development team.
About Ross
Ross Guarino is a Systems Engineer at Cloudflare in charge of the technical direction of the internal platform. He’s determined to improve the lives of developers building and maintaining everything from a simple function to complex globally distributed systems.
About Terin
Terin is a long-time engineer at Cloudflare, currently working on building an internal Kubernetes cluster. By night, he hacks on building new hardware projects.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
B
Our
Twitter,
if
you
want
to
follow
us
later,
we'll
link
these
slides
and
the
example
code
that
we
publish
so
yeah.
First,
we
need
to
talk
about
kubernetes
a
cobbler
before
kubernetes.
We
have
a
bunch
of
ways
of
managing
processes
and
configuration
and
business
code
review
comments
or
on
developer
laptops
and
developers
don't
seem
to
like
having
their
laptops,
go
to
production
and
email
change
in
pieces.
We
kind
of
wanted
to
consolidate
all
of
these
into
one
source
of
truth,
which
we
think
we
can
do
with
kubernetes
and
kind
of
all.
B
B
B
And
another
thing:
that's
kind
of
different
than
many
communities
clusters
that
we
talk
to
people
here
at
coop
Carnivale
is
we
run
all
of
our
own
physical
infrastructure,
so
we're
doing
kubernetes
in
the
salt
way.
It's
pretty
much.
Our
only
option,
there's
no
G
cloud.
Salt
start
kubernetes,
and
so
since
we
don't
have
a
child
provider,
we
need
to
bridge
the
gap
between
what
kubernetes
expects
the
cloud
provider
to
provide
and
what
we
have
on
bare
metal.
B
So
we've
built
a
couple:
different
services,
there's
no
cloud
load
balancer
without
the
cloud
as
we
bought
Leifer,
which
is
a
load
balancer
for
about
bare
metal
hardware
at
a
much
slower
rate
than
what
you
can
do
in
the
cloud.
Also,
we
built
pile
Iowa
to
automate
user
control
and
namespace
creation
in
our
back,
so
that
we
can
share
Asuma
cluster
across
teams
and
projects,
and
we
already
have
an
existing
Paloma
triometrics
platform
for
our
previous
system
and
also
our
edge
network.
A
B
So,
to
fully
appreciate
the
service
movement
functions
as
a
service.
More
specifically
I
had
to
first
understand
the
role
event-driven
architecture
of
play
in
modern
day
computing.
It's
not
just
IOT.
That
kind
of
brings
us
to
what
this
presentation
is
about,
which
is
controllers
they're,
pretty
much
event-driven
responses
to
things
that
kubernetes
doing
so
that
you
can
extend
your
kubernetes
cluster
in
an
automated
fashion.
B
So
our
example
problem.
We,
as
we've
mentioned
what
a
namespace
for
every
developer
and
the
communities
cluster
there's
a
couple
ways
that
we
can
implement
this
I'll
type
kubernetes.
We
can
have
IT
provision
namespace.
When
people
start,
we
can
have
people
file
tickets
when
they
first
start
in
the
first
week.
B
B
It's
unfortunately,
kind
of
small
but
fairly
simple,
yeah,
most
structure
that
defines
a
new
festive
resource
or
third-party
resource,
and
you
can
also
automatically
on
admission
make
sure
that
this
matches
the
schema.
If
you
want
DeFede
to
find
one
for
code
generation
converter,
you,
ships
with
a
coach
in
tool
that
helps
them,
build
the
client
code
and
farmers,
listeners
and
automatic
deep
copping
support
so
that
you
can
easily
change
these
objects
without
changing
all
our
presentations
of
them.
And
so
here
we
have
an
example:
type,
a
user
type
which
embeds
a
user
spec
type.
B
Then
you
can
depend
on
the
code
gin
tool
in
your
favorite
package
manager
for
go,
and
then
you,
once
you
there's
a
script
that
can
set
that
all
up
and
run
it
and
we'll
actually
to
write
the
generator
code
for
you
and
that
pretty
much
we
take
types.
If
that
turns
it
into
types
and
Zizi
generated
deep
copy
and
similar
things
for
Lister's
and
Informer's.
B
A
The
next
part
about
writing
a
controller
is
hooking
up
your
business
logic
to
events
that
are
happening
in
the
cluster
and
one
of
the
things
that
we
realized
after
we
had
finished.
Writing
these
these
four
or
five
extensive
x/y.
A
different
services
that
extend
our
kubernetes
cluster
was
that
we
really
didn't
write
that
much
code
that
wasn't
related
to
the
business
logic,
and
it
came
up
that
we
might
actually
be
able
to
fit
all
of
the
code
that
we
wrote
that
wasn't
business
logic
into
a
presentation.
A
So
the
first
thing
first
bits
of
code
that
we
need
to
write
ourselves
are
hooking
up
into
the
event
framework
for
kubernetes
and
there's
a
ton
of
stuff.
That's
already
written
for
this
one
of
the
first
things
is
informers,
so
there's
a
concept
of
informers:
that's
written
in
the
client
go
big
shout
out
to
all
the
people
that
work
on
the
API
machinery,
because
this
is
awesome
and
it
makes
your
life
so
much
easier.
A
These
Informer's
lets
you
react
to
changes
in
resources
in
the
communities.
Cluster
they've
reduced
the
burden
on
the
API
server
by
removing
all
the
polling
essentially,
and
they
also
do
another
nice
thing,
which
is
populate
a
read-only
cache.
So
whenever
you
need
to
read
a
resource
from
the
API
server,
you
don't
incur
a
round-trip
to
the
API
server.
You
essentially
just
read
it
from
the
cache
that
was
populated
efficiently
by
the
watch
mechanism.
So
then
the
the
next
part
in
your
arsenal
is
a
Lister.
This
is
the
read-only
cache
that
is
populated
by
the
informer.
A
A
A
really
simple
and
straightforward
interface
to
implementing
a
asynchronous
queue,
and
some
of
the
big
selling
points
is
that
these
queues
are
stingy,
which
means
that
they,
if
you
add
a
single
item,
twice
to
the
queue
it
will
only
be
processed
once
and
then
they
are
fair,
which
means
that
items
are
processed
in
the
order
which
they're
added
and
then
they
support
multiple
consumers
and
producers.
So
you
can
it
allow
us
a
item
to
be
recued
while
something
is
currently
processing
the
item.
A
So
that's
really
super
useful
for
handling
errors,
which
we'll
see
in
a
few
slides.
So
the
first
question
that
you
well
that
usually
comes
up
is
what
to
actually
put
on
the
cue
it.
It's
a
really
free
form,
but
we
do
have
some
guidelines
for
this.
We've
tried
it
a
couple
of
different
ways,
and
this
seems
to
be
the
best
way.
Is
the
cues
use
go?
A
Go
map
equivalents
to
actually
determine
whether
or
not
an
item
is
match
matches
to
an
item,
that's
already
in
the
queue,
so
keep
it
as
simple
as
possible
and
the
simplest
key
that
we've
been
able
to
find
that
works.
Reliably
is
actually
just
a
metadata
that
name:
we've
tried
other
things
like
the
UID,
the
revision
and
UID,
but
really
those
don't
they
kind
of
throw
away
a
lot
of
the
guarantees
of
the
cue.
A
It
won't
stop
you
from
doing
this,
but
just
try
not
to
because
the
cues
work
way
better
when
you're
using
something
that
doesn't
change
every
time
you
try
and
process
a
an
item.
So
here's
the
first
bit
of
code.
So
in
this
code
example,
we
we
set
up
a
cue.
This
is
a
rate
limiting
cue
and
we
set
up
an
informer.
So
the
cue
there
for
processing
events,
the
informer
is
there
for
for
getting
updates.
So
it's
really
simple.
A
These
two
lights
essentially
get
you
set
up
watching
kubernetes
resources,
and
then
this
function
that
we
have
defined
below
in
cue
user.
This
is
really
all
it
takes
to
take
an
event
from
a
user
and
actually
just
put
it
into
the
queue.
So
this
cache
delete,
yeah,
delete,
handling,
meta,
name,
space,
key
func,
a
really
long
name.
All
that
does
is
if
an
object
is
deleted
in
the
time
since
the
last
sync
period,
and
you
didn't
see
the
deletion
for
whatever
reason
due
to
a
network
failure
or
something
like
that.
A
You
look
at
the
object
tombstone,
instead
of
an
actual
object,
and
this
function
just
unpacks
the
object
tombstone
for
you
and
extracts
the
name
for
you
and
by
default.
It
assumes
that
the
key
that
you
want
is
just
the
name.
So
once
you
get
the
key
back,
you
just
do
Q
add
and
it's
off
to
be
processed,
so
we
have
to
hook
up
the
informer
and
that
function
that
we
just
showed.
A
So
it's
really
simple
to
do
and
in
forward
produces
three
types
of
events,
creation,
update
and
delete,
and
one
of
the
weird
things
here
is
that
we
handle
all
of
these
things
the
same
way.
So
we
found
this
to
be
actually
really
kind
of
empowering.
It
makes
your
function
on
that
function.
It
makes
writing
the
code
later
on
way
easier,
and
it
means
that
you
have
way
less
code
paths
to
worry
about.
So
every
time
we
see
a
creation,
updation
or
deletion
for
an
event.
A
So
in
this
case,
you
can
see
that
we
always
just
call
the
same
user
func
and
queue
user
function
from
before
and
that
just
takes
the
queue
and
the
object
that
was
just
operated
on.
So
the
next
thing
now
that
we're
watching
our
actual
user
objects
so
yeah.
Now
that
we're
actually
watching
the
user
CRD
that
we
had
before
we
actually
want
to
watch
our
children.
A
So
one
of
the
things
that
we
found
well,
that
was
confusing
was
that
you
might
have
processed
the
initial
creation
of
a
user
properly
where
you've
mapped
the
user
to
a
namespace,
and
it
appears.
But
then,
if
someone
with
permissions
to
goes
and
deletes
the
namespace,
it
may
take
well
up
to
your
resync
period
before
the
namespace
comes
back,
and
that
was
confusing
and
a
little
bit
strange
to
users.
So
the
way
that
we've
gotten
around
this
is
by
triggering
updates
reprocessing,
the
user.
Every
time
one
of
the
children
is
updated
as
well.
A
So
kubernetes
has
a
metadata
owner.
References
filled
in
every
object,
and
this
is
essentially
just
a
list
of
every
kubernetes
object
or
ya
object.
That
can
is
considered
to
be
a
parent
of
the
other
object.
So
in
the
kubernetes
namespace
we
set
the
owner
references
to
be
the
user
and
then
what
we
do
is
we
just
watch
for
events
on
all
the
namespaces,
and
if
that
namespace
has
an
owner
reference
to
a
user,
we
add
that
user
to
the
queue.
A
So
that
means
that
if
someone
deletes
or
mucks
around,
with
a
namespace
that's
owned
by
one
of
our
things,
we
have
the
chance
to
immediately
go
back
out
and
fix
it
and
make
sure
that
it's
not
broken.
So
that's
the
input
side
of
the
queue
for
listening
for
events
from
both
the
parent
and
the
child
in
this
relationship
and
we're
adding
the
item
to
the
queue
based
on
the
name
of
the
parent,
and
now
we
actually
need
to
go
out
and
edit
or
actually
go
out
and
do
the
work.
A
A
Is
it
just
sits
in
a
loop
of
popping
items
off
of
the
queue
and
then
calling
a
work
function
and
it
one
of
the
things
that
it
does
is
it'll
never
block
forever
on
one
item,
because
if
you
take
forever
on
ten
items
or
so-
and
you
only
have
ten
go
routines
and
you
stop
processing
good
items,
so
we
make
a
point
of
avoiding
that
situation
altogether
by
timing
out
work
functions.
So
then
a
work
function
in
this
model.
A
It
needs
to
handle
deletion,
since
we
didn't
do
that
earlier,
and
it
also
needs
to
be
Adam
potent
because,
since
we
watch
our
children
resources,
if
there's
a
potential
for
a
loop
here,
if
you
update
the
child
and
then
add,
go
back
to
processing
the
item
again
and
then
update
the
child
again
and
then
go
back
to
processing
the
item
again.
It'll
never
end.
So
it's
really
important
that
if
there's
no
work
to
be
done,
the
work
function
doesn't
do
any
work,
which
is
easy.
A
It's
potentially
problematic
sometimes
so.
The
first
part
we
have
here
is
this
process
for
Chrome.
This
is
essentially
the
same
thing
across
every
single
one
of
these
that
we've
written
should
open
source
this.
It
is
a
function
which
you
call
once
and
it
processes
an
item
from
the
queue.
If
it
returns
true,
then
you
call
it
again.
If
it
returns
false,
you
don't
call
it
again,
so
you
just
do
for
process
work
item
and
sit
forever
in
your
girl
routine,
but
this
function
it
does
a
little
bit
of
work
it.
A
The
first
thing
it
does
is:
it
gets
the
item
from
the
queue
and
when
you
get
an
item
from
the
queue
you
get,
both
the
key
and
a
indication
of
whether
or
not
the
queue
is
shutting
down.
So
if
the
queue
is
shutting
down,
then
we
just
don't
do
anything
here
and
return
false.
This
means
that
this
go
routine
will
exit.
A
So
no
matter
what
what
no
matter,
whether
it's
a
success
or
a
failure,
you
still
need
to
indicate
that
you're
done
and
then
the
next
important
bit
is
this.
We
call
the
work
function
here,
so
remember
it's
idempotent
and
it
handles
deletes.
So
really
it's
really
simple
in
this
sort
of
go
routine
code.
All
you
do
is
call
the
work
function
and
return
an
error
capture,
the
air.
If
there's
an
error.
Well,
then
we
didn't
succeed.
A
So
we
need
to
add
the
Q
back
or
add
the
item
back
to
the
Q
at
a
later
time
and
the
actual
the
work
queue
functions
do
this
for
you.
They
have
this
ad
rate
limited
key,
which
will
mean
that
your
you
won't
sit
in
the
hot
loop,
trying
something
that's
family
and
over
and
over
and
over
again
it'll
it'll,
essentially
back
off
up
to
a
certain
point,
and
you
can
tweak
the
settings
here.
We
just
used
the
default,
so
this
seemed
to
work
pretty
well.
A
So
if
you
have
an
error,
though,
or
if
you
don't
have
an
error,
you
have
to
tell
the
Q
that
it
was
successful,
so
you
do
Q
dot
forget,
and
that
means
that
the
Q,
if
it
hasn't
already
been
re,
adding
well
just
forget
that
the
item
was
in
the
queue
in
the
first
place.
When
you
call
done
so
in
the
happy
path,
you
have
to
call
Q
dot.
Forget
it
doesn't
seem
it's
a
weird
function
name,
but
it's
absolutely
required.
A
So
that
those
slides
of
code-
that's
pretty
much
it
you.
Those
are
really
only
the
only
functions
and
the
only
code
that
you
need
to
write
that
isn't
business
logic
code.
It
was
kind
of
really
surprising
to
us
that
we
were
able
to
write
so
little
code
and
essentially
have
like
event-driven
code,
that's
resilient,
that
handles
failures,
and
that
is
fast
and
doesn't
use
a
poly
mechanism.
A
A
So
one
of
the
things
that
we
we
we
started
out
doing
was
handling
our
errors
in
retrying
and
trying
to
do
all
this
fancy
like
diff
and
fetch
the
object
again
and
make
sure
that
you
have
the
proper
revision
and
then
we
just
decided
to
you
know
what?
Let's
not
even
worry
about
it.
If
we
get
an
error
anytime,
just
return
it
you
don't
panic
and
you
don't
make
sure
that
you
catch
the
air
but
just
return
it
and
let
the
queue
retry
them.
So
it's
actually
really
incredible.
A
If
you
have
the
two
questions
that
I
ask:
whenever
we're
writing
a
new
controller
is,
do
you
have
external
state
and
do
you
need
to
guarantee
that
you
witness
a
deletion
like?
Are
you
keeping
track
of
a
state
really
just
two
ways
of
asking
the
same
question?
Sometimes
you
get
different
answers,
though.
So,
if
you
have
well
I
guess
don't
handle
onto
the
differently,
which
is
the
first
part.
A
So
if
you
handle
on
delete
code
differently,
you're
essentially
duplicating
your
deletion
code
in
two
different
places,
because
it's
still
possible
that
even
if
you've
received
an
update
or
create
or
a
refresh
that
by
the
time
you
go
to
process
it,
the
thing
will
have
been
deleted,
so
you're
going
to
have
to
handle
the
deletion
in
the
worker
code
anyway.
So
just
skip
that
whole
thing
and
then
check
in
your
worker
code
to
be
to
see
if
the
deletion
is
necessary.
A
If
you
need
to
handle
deletes,
if
you
don't
need
to
handle
the
leads,
then
just
use
the
curve
garbage
collection.
So
earlier
we
said,
talked
about
building
the
parent-child
relationship.
If
you've
done
this,
you
essentially
get
garbage
collection
for
free.
The
garbage
collector
will
delete
objects
that
once
had
an
owner
but
no
longer
have
an
owner.
So
if
you've
set
up
that
owner
relationship
to
your
parent
resource,
then
when
the
parent
resource
is
deleted,
which
is
when
you
would
be
handling
the
delete,
kubernetes
will
go
around
and
delete
any
child
resources
for
you.
A
So
this
way
you
don't
actually
have
to
write
any
code
at
all,
two-handled
leash
and
kubernetes
will
do
it
for
you.
So
no
code
is
best
I,
think
you've
all
seen,
Kelsey
Hightower's
framework
no
code
is
definitely
better
than
code,
so
it's
essentially
free.
But
unfortunately,
if
you
answered
yes
to
either
those
questions,
you
should
use
finalizes
for
deletion.
A
So
in
reality,
what
a
finalizar
does
is
that
when
you
delete
a
resource,
kubernetes
sets
the
object
as
having
a
deletion
timestamp
and
then
will
prevent
any
other
things
from
actually
updating
the
object
and
adding
more
finalizes
to
the
list.
And
then
once
the
list
of
finalize
errs
is
empty,
the
resource
will
be
removed
from
the
cluster
automatically.
A
A
This
means
that
you
can
run
you
can
handle
restarts
without
worrying
about
missing
the
delete,
because
if
your
service
starts
back
up
in
time,
then
you
can
still
process
the
delete,
even
if
you
weren't
running
at
the
time
that
someone
deleted
the
parent
resource.
So
this
is
an
example
of
the
worker
function.
A
We
have
a
whole
bunch
of
arguments
here,
because
these
are
the
things
that
we
interact
with,
but
essentially,
if
we
have
what
we
do
is
we
have
essentially
an
if
statement
is
one
of
the
first
things
in
this
worker
function
that
checks
to
see
if
the
deletion
timestamp
is
zero
and
if
the
finalize
errs
list
has
our
finalizar
on
it.
So
if
that's
true,
that
means
that
the
object
is
deleted
and
we
haven't
yet
done
the
work
that
we
need
to
do
so
we
need
to
handle
the
delete.
A
So
this
is
different
from
depending
on
whatever
your
business's
needs
are,
so
I
don't
have
any
code
here
for
it,
but
at
the
end,
what
you
do
is
you
need
to
remove
example.com
from
your
or
whatever
your
finalized
name
is
called
from
the
list
of
finalized
errs
and
then
return.
If
neither
of
these
are
true,
that
means
you
have
nothing
to
do
either
the
object.
Well,
yeah
isn't.
If
neither
of
them
are
true,
then
you
just
continue
on
with
the
happy
path
of
actually
creating
and
updating
the
resource
and
then.
A
A
Yes,
in
the
front
down
in
front
okay,
the
question
was
tell
us
more
about
Leifer.
Okay,
so
Leifer
is
our.
Since
we
run
on
bare
metal,
we
have
no
cloud
load
balancer.
So
Leifer
is
our
approach
at
doing
load,
balance
or
service
IPS,
and
it
works
it's
very
similar
to
metal
lb
from
google,
except
for
it
makes
no
sort
of
assumptions
at
all
about
your
networking
environment
and
it
you
can
plug
it
into
whatever
you
want.
B
A
There
are
yeah,
they
have
some
fun
quirks,
but
they
are
there.
One
of
the
things
to
be
to
be
worried
for
the
file
with
an
initializer
is
that
you
can't
delete
an
object
that
hasn't
finished
to
no
civilization
yeah,
which
is
confusing
a
little
bit.
But,
yes,
they
do
exist.
I
think
they've
existed
for
longer
than
finalized
use.
Even
it's.