►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
The Kubernetes Control Plane for Busy People Who Like Pictures - Daniel Smith, Google
Let’s dive into the modular design of the Kubernetes control plane together. We’ll talk about the specific resources and controllers which make Kubernetes what it is, and how this pattern generalizes--which problem domains are well solved by the Kubernetes control plane architecture? This talk will speak to people ready for a 201-level introduction to the Kubernetes control plane, and also to people contemplating using Kubernetes API Machinery for their own purposes.
https://sched.co/MPa5
A
Hi
everyone
thanks
for
coming
to
the
Cooper
dendi's
control
plane
for
busy
people
who,
like
pictures.
If
you
saw
my
previous
talks
or
you
looked
at
the
slides
online,
you
know
what's
in
store,
okay,
okay,
a
little
bit
about
me:
I'm
Daniel,
Smith,
I'm,
a
staff
software
engineer
at
Google,
I've
been
working
on
kubernetes
since
before
it
was
open-source
I
am
the
co-chair
coati
of
segi
API
machinery,
which
owns
API
server
and
controller
manager
and
like
a
lot
of
a
lot
of
the
control
plane
of
kubernetes.
A
So
who
is
this
talk
for
this
is
for
two
groups
of
people.
If
you
have
ever
considered
building
your
own
control,
plane,
I'm
sure,
like
everybody,
does
this
right.
If
you've
ever
thought
about
this,
this
this
talk
is
gonna,
be
a
survey
of
the
existing
controllers
in
kubernetes.
You
can
look
at
those
and
together
we're
gonna,
learn
what
patterns
are
good
and
what
patterns
are
a
little
harder
to
understand.
A
Maybe,
and
you
can
decide
whether
your
problem
domain
is
a
good
fit
for
this
style
of
control,
plane
architecture,
the
other
group
of
people
that
I'm
targeting
this
at
is
like
you
know
what
a
pod
is.
You
know
what
a
deployment
is:
you're
ready
for
you're
ready
for
201,
so
we're
gonna.
Do
that
we're
gonna
go
through
like
every
controller
in
the
system
and
say
what
it
does
we're
gonna
have
fun.
We've
got
a
lot
of
slides
to
go
through.
A
A
Don't
worry,
you
won't
have
any
problem
understanding
us.
So
if
you've
seen
those
you've
probably
heard
me
say
something
like
the
kubernetes
api
is
about
humans
and
machines
working
together.
Well,
you
can't
do
that
without
some
machines.
So
this
talk
is
about
controllers.
Controllers
are
the
things
that
make
it
all
work
the
ghosts
in
the
machine,
the
the
person
behind
the
curtain.
You
get
the
idea
and
before
before
we
like
dive
into
their
controllers
and
kubernetes
I
I
want
to
you,
know
everybody.
A
Let's
control
theory
right
I
want
talk
about
control
theory
for
just
a
little
bit
and
and
I'm
going
to
start
off
by
rehashing
the
age-old
debate
of
state
machine
versus
control
loop-
and
the
argument
goes
like
this
and
we
we
actually
rehash
this
argument
in
a
number
of
places
early
on
kubernetes
and
we
came
down
pretty
firmly
on
the
control
loop
side
of
things.
But
but
the
argument
is
like,
if
you're,
if
you're
constructing
your
system
in
terms
of
state
machines
for
every
like
binary
variable,
you
could
you
double
the
number
of
states.
A
So
if
you've
got
to
write
things
that
get
every
action
perfect
you've
got
so
many
states
like
you
can't
possibly
get
it
all
right.
The
number
of
states
grows
really
big,
as
you
add
variables,
and
to
make
it
worse
like
not
all
the
variables
in
the
system
are
binary
right.
We've
got
real
numbers,
it's
not
good!
That's
the
technical
term,
so
if
you
instead
factor
your
logic
in
terms
of
controllers,
now,
you're
not
talking
about
States
you're
talking
about
like
clusters
of
variables.
A
This
is
the
input
to
your
controller,
the
status,
the
other
things
going
on
in
the
cluster,
and
now
you
just
have
to
add
an
additional
controller
to
to
to
handle
every
cluster
variables
right.
So
the
number
of
controllers,
just
scales,
sort
of
linearly,
with
with
with
the
number
of
things
going
on
in
the
cluster
in
a
prior
talk,
I
talked
a
lot
about
the
kubernetes
resource
model
like
the
API,
how
it
lets
you
gives
you
a
constant
way
of
performing
API
actions
on
the
cluster,
and
that
was
about
integration.
Complexity.
A
Right,
if
you
know
how
to
if
you
know
how
to
drive
one
kubernetes
resource,
you
know
how
to
drive
them
all,
because
the
API
is
the
Cotten's
constant,
there's
even
constant
metadata,
whereas
the
the
controller
half
of
it.
The
half
that
I'm
talking
about
in
this
talk
is
the
implementation
complexity
right.
So
so
there's
the
the
API.
We
factor
everything
to
make
it
simple,
but
controllers.
We
also
factor
the
logic
of
the
controller
to
concentrate
the
the
hard
part
of
the
controller
into
one
place.
A
This
makes
it
globally
easier,
like
there's
no
single
state
that
you
have
to
worry
about,
like
there's.
No,
to
make
a
change
in
kubernetes,
you
don't
have
to
have
every
single
as
possible
state
in
mind
to
make
a
correct
change,
but
it
makes
it
locally
harder
because
your
controller
has
to
be
written
with
all
possible.
It's
like
everything.
B
A
A
In
doing
this,
survey
came
up
with
a
list
of
things
that
I
think
make
a
good
controller.
Only
do
one
thing
have
some
clear
input
source
like
people
need
to
know
how
to
how
do
you
make
it
work?
How
do
you
drive
your
controller?
Have
a
place
to
write
status.
People
did
you
need
to
know
if
your
controller
is
happy
or
healthy
or
unhappy,
or
you
know
what
what
how's
it
feeling
you
need
to
have
an
output
place
like
what
are
you
actuated?
A
What
are
you,
what
does
your
controller
doing
and
then
the
last
two
I
think
are
maybe
the
least
obvious
and
possibly
the
most
important.
The
controller
has
to
anticipate
its
own
effects
like
the
effects
of
its
own
actions
on
the
rest
of
the
system,
and
when
you
are
having
a
problem
break
things
a
little
bit,
don't
bring
things
a
lot
just
just
a
little
bit
and
don't
pretend
things
are
not
broken.
A
So
if
you
zoom
in
on
a
controller
you'll
find
like
there's
a
lot
of
plumbing
and
there's
flags
and
configuration
and
there's
all
this
stuff
like
informers
like
who
knows
what
commented
out
code,
you
don't
what
your
co-workers
to
read,
but
there's
also
some
some
logic
in
there
right.
There's,
there's
some
there's
some
brains
like
there's.
A
I
have
200
pods
delete
hundred
pods
and
you
look.
You
still
have
to
look.
You
lost
like
right,
so
you
have
to
have
a
model
of
what
you're
doing
to
the
environment
and
some
simple
controllers:
don't
necessarily
need
to
do
too
much.
So
it's
like
okay,
I
set
the
thing.
The
thing
is
set
like
that's
it,
but
there
is
some
sort
of
implicit
model.
A
So
if
you
zoom
into
the
like
the
controller
in
the
model
that
doesn't
have
to
be
anything
in
there,
you
don't
have
to
go
that
far.
Take
a
deep
breath:
you're,
okay,
okay
and
so
the
last
thing
about
break
things.
Just
a
little
bit.
Well,
I
want
to
talk
about
like
how
two
changes
propagate
in
a
kubernetes
cluster.
A
So
your
country,
you
do
something
as
a
human
or
as
a
CATV
system
or
something
and
then
your
controller
goes
out
and
does
something
else
and
then
then
maybe
more
controllers
go
out
and
do
other
things,
and
so
that
that
propagates
and
eventually
like
the
system
quiesce
is
it
comes
to
a
quiet
state?
It's
happy.
Nothing
needs
to
change
to
get
into
a
good
state.
That's
but
that's
one,
that's
one
way
changes
can
propagate
another
way.
A
Changes
can
propagate
is
like
you
trigger
a
controller
which
then
triggers
something
else
which
then
triggers
something
else
which
is
like
you
can
goes
into
a
circle
right,
that's
another
way
that
your
change
can
ripple
through
the
system
after
you,
after
you've,
ready
to
change.
There's
another
two
way
which
is
like
you
make
a
change
and
then,
like
you
know,
you've
got
a
cascading
like
failure
and,
like
controllers,
keep
riding
until
API
server
can't
keep
up
anymore.
So
you
don't
want
to
do
this
like.
If
you
do
this,
your
control
plan
can't
keep
up
something.
A
So
it
gives
signal
to
the
rest
of
the
rest
of
the
system.
If
there's
self-healing
components,
they
can
fix
your
problem
for
you
without
the
controller
author
really
having
understand
what
it
is
as
fixing
the
problem.
Okay,
so
in
summary,
ya,
break
things
a
little
bit
anticipate
your
own
effect.
So
that's
really
important
to
anticipate
your
own
effects.
Okay,
that's
enough
theory.
Take
a
deep
breath.
We're
done
with
the
theory
it's
time
for
control
practice
and
what
we're
gonna
do
is
we're
gonna
spend
like
a
few
seconds
on
every
controller
in
the
system.
A
I've
classified
them
here
into
three
three
categories:
the
classic
controllers,
like
the
things
that
really
look
like
a
controller.
If
you
go
read
the
Wikipedia
control
theory
page,
which
I
recommend
is
good,
the
sanding
query
or
table
join
controllers
and
the
injection
or
by
ejection
enforcers
okay
well
clarify
what
those
means
in
a
second.
Let
me
give
you
just
a
few
disclaimers
before
we
dive
in
many
controllers,
write
events.
A
If
you've
used
kubernetes,
you
know
what
events
are
I'm,
not
going
to
show
that
on
the
slides.
It
makes
everything
really
messy.
Likewise,
many
controllers
have
some
way
of
writing
status.
Sometimes
I
showed
it
because
it's
important,
sometimes
I
didn't.
If
you
need
to
know
more,
you
can
always
go
read
the
code
or
the
API
or
whatever
I'm
gonna
go
fast
on
some
of
these
controllers,
I'm
going
to
go
slower
and
others
just
depending
on
whether
I
think
it's
instructive
or
interesting.
So
here's
our
first
group.
It
is
the
garbage
collectors.
A
This
is,
this
is
like
the
scenario
is
you've
got
too
much
of
something
and
you
need
to
have
less
of
it.
So
these
this
is.
This
is
the
theme
for
this
group
of
controllers.
First
one
is
the
pod
garbage
collector.
It's
basically
like
you've
got
too
many
finished
pods
and
your
cluster.
So
let's
go
get
rid
of
something.
This
is
a
pool.
Skimmer
like
you've
got
too
much
water
in
your
pool.
Let's
drain
some
water
next
is
the
namespace
lifecycle.
A
Next
is
the
like
actual
garbage,
collector
and
and
I'll
spend
a
minute
on
this
one,
because
this
is
really
great.
It
frees
your
regular
controllers
from
having
to
tear
down
the
structures
that
they
make,
because
you
set
up
the
the
parent
pointers
on
your
objects.
There's
a
there's,
a
metadata
field
and
then
the
garbage
collector
is
just
sits
there
and
watches,
and
when
the
the
parent
objects
go
away,
it'll
it'll
delete
the
child
objects
and
free
up
your
whole
hiring
or
hierarchy
for
you.
A
If
you've
looked
at,
like
the
I,
don't
know
replica
set
code
or
deployment
code
before
we
added
the
garbage
collector
like
it
actually
is
a
significant
amount
of
work
to
clean
up
everything
correctly.
So
this
this
is.
This
is
great.
Please
use
it
in
your
own
controllers,
yeah,
okay!
So
next
we
have
this
triplet
of
certificate.
Signing
request
controllers
in
this
first
one.
A
These
these
three
controllers
work
together
to
provide
a
feature,
and
that
is
signed
certificates,
so,
like
user,
makes
a
certificate
signing
request
and
then
various
things
happen
to
it,
and
this
first
controller
that
I'll
talk
about
is
the
is
the
CSR
cleaner
that
basically
deletes
certificates
CSRs
after
they've
expired
or
yeah?
That's
really
they've
expired,
whether
or
not
they
were
signed.
The
these
controllers
are
really
good
examples
of
the
like
do
only
one
thing
principle,
so
yeah,
okay,
next
up
is
a
pair.
The
persistent
volume
and
persistent
volume
claim
protection
controllers.
A
We
have
this
tool
in
kubernetes,
the
finalizar.
How
many
of
you
have
heard
of
a
finalizar
okay
good
about
finalizar
is
a
basically
you
put
a
tag
on
an
object
and
the
object
can't
be
removed
from
the
system
until
something
else
comes
and
removes
that
tag
right.
It's
called
the
finalizar,
so
the
persistent
volume
controller
and
claim
controllers
they
set
up
these
tags
because
you,
you
have
a
matching
volume
and
claim
these
have
these
have
references
to
each
other.
A
A
A
Okay,
so
that's
it
for
garbage
collectors,
I'm
sure
you
are
ready
for
something
else.
This
is
the
replica
set
controller,
which
I
think
is
like
the
first
like
really
true
controller,
it's
exactly
like
a
thermostat
like
you
can
get
make
it
make
less.
Pods
make
more
pods
yeah,
but
this
is
like
the
first
thing
you
see
when
you
learn
kubernetes,
so
I'm
sure
you
all
know
about
this
one.
So,
let's
go
to
the
deployment,
which
is
a
little
bit
more
interesting,
I.
A
Think
of
this
one
as
like
a
a
thermostat
programming,
robot
right,
he
says
it's
like
the,
where
the
replica
set
automates
like
keeping
the
collection
alive,
the
deployment
automates
like
rolling
between
two
collections
right.
So
it's
like
it's
like
a
robot
that
turns
down
one
and
turns
up
the
other.
Okay
and
a
similar
one
is
the
horizontal
pod
autoscaler,
which
reads
metrics
about
pods
and
uses
that
to
decide
like
do.
I
need
more
of
these
things
or
less
of
these
things
like.
A
Are
they
loaded
or
not,
and
you
can
work
on
resources
that
support
the
scale
step
resource?
So
again,
it's
like
a
generic
like
if
you,
if
you,
if
you
use
the
if
you
use
the
thing
this,
this
HPA
can
scale
your
collection
and
finally,
is
the
modern
office
final
or
not
for
this
group,
but
this
is
the
cluster
autoscaler
and
basically,
if
you've
got
unscheduled
pods
it
will.
It
will
add,
nodes
I,
think
it's
not
available
in
all
cloud
providers.
There's
no
reason
you
can't
write
one-
and
this
is
this-
is
under
the
theory
like.
A
The
deposit
implement
the
surface.
Okay
next
up,
I've
got
a
pair
that
work
very
similarly.
These
are
the
controllers.
Are
the
pod
disruption,
the
disruption,
controller
and
the
resource
resource
quota
controller,
and
they
both
work
by
observing
things
and
updating
a
status?
And
then
they
have
a
API
server
component
that
acts
on
that,
so
the
disruption
budget
account
spot
disruptions
like
getting
evicted
or
something
like
that
and
I
think
I
labeled
it
an
admission
plugin,
but
it's
actually
a
sub
resource.
A
The
evict
sub
resource
looks
at
that
thing
to
know
like,
is
it
safe
to
perform
another
disruption
on
this
pot?
The
resource
quota
does
something
very
similar.
It's
like
the
the
easiest
one
to
explain
is
just
counting
things
right,
like
you
put
a
resource
quota
object
in
that
says,
you
can
only
have
100
pods
and
it
like
counts
up
the
pods
and
says
okay.
A
Before
admitting
a
pod
right,
you
need
to
catch
that,
that's
what
these
controllers
are
for,
and
next
up
we
have
the
CSR
again.
This
is
the
CSR
approver.
If
a
CSR
comes
in
and
it
is
from
a
node,
this
thing
approves
it:
why
do
we?
Why
did
we
do
it
this
way?
It's
so
that
you
can
use
the
system
with
sources
other
than
nodes,
so
you,
like
you,
can
as
a
human.
You
like
you,
can
make
a
CSR
and
you
could
have
like.
Maybe
your
system
administrator
could
could
approve
it
all
right.
A
So
the
controller
is
like
it
approves
the
CSR
that
it
knows
about,
but
you
can
use
the
system
for
other
things
and
the
this
particular
CSR
Singh
question
is
we're
getting
cubed
access
tokens
to
to
API
server
so
can
so
can
participate
next
up.
Another
part
of
the
auth
system
is
the
cluster
role,
aggregation
and,
and
so
cluster
roll
is
part
of
the
our
back
system
and
you
can
put
in
a
list
of
rules
like
like
this
roll
like
the
the
reader
of
the
viewer.
A
It
can
like
get
pods,
say
and,
and
but
it's
not
nice,
if
you
have
to
like
somebody
adds,
adds
a
C
or
D
and
they're
like
well
really
I,
the
author
of
the
roll,
didn't
know
about
my
C
or
D.
So
how
can
this
work?
We
have
this?
We
have
this.
It
basically
takes
a
selector
and
it
looks
up
cluster
wolves
that
match
the
selector
and
copies
them
into
the
the
cluster
role
with
a
selector
right.
So
it
can
either
have
a
rule
list
or
a
selectors
like
copy
rules
from
these
guys.
A
This
one
I
think
III
would
say
it's
it's,
maybe
not.
The
best
example
of
the
of
the
like
it
uses
its
input
and
output
location
is
the
same
I
think
it's
a
little
confusing,
so
III
would
recommend
like
if
you're
going
to
learn
something
from
it.
Learn
like
maybe
maybe
this
would
have
been
better
as
like
a
two
step
object.
A
Okay,
next
up
are
the
injection
or
by
Jackson
enforcers
and
I'm.
Probably
gonna
use
the
word
injection
just
because
I
don't
want
any
math
majors
to
come.
Give
me
a
hard
time
afterwards.
Safer
first
up
is
the
daemon
set,
which
is
maybe
just
the
classic
example
of
this,
like
a
demon
set
is
like
this
is
a
node
description.
This
is
a
pot
description
and
then
the
controller
all
nodes
that
match
the
label.
It
adds
a
pod,
all
nodes
that
don't
match
the
label
and
make
sure
that
this
pod
doesn't
exist
there
right.
A
A
So
this
is
maybe
I.
Don't
know
this
is
maybe
the
most
entertaining
header
of
any
file
in
the
kubernetes
codebase
is
the
the
persistent
volume
binder.
It's
got
this
very
entertaining
blob
about
how
it's
written
in
space-shuttle
style
and
all
branches
are
considered,
and
so
this
is
not
a
good
example
of
the
like.
A
It's
a
it
basically
matches
up
like
you
make
a
persistent
volume
claim.
We
have
to
find
a
volume
to
to
bind
to
the
claim
it
finds
a
volume
buts,
but
you
can
actually
like
the
system,
can
allocate
volumes
volumes
for
you.
So
then
you
have
to
allocate
the
volume
in
the
availability
zone
where
the
pot
is
going
to
schedule.
So
it's
like.
Does
this
dance
with
a
label
with
the
scheduler?
It
works.
A
Okay,
and
now
we
have
a
sequence
on
what
I
call
the
the
candy
wrappers
they're
like
a
bisection
of
or
an
injection
of
one
one
thing
basically
and
the
so
see
us
are
signing
like
every
time
it
sees
a
we're.
Gonna
go
through
pass
through
these
every
time
it
sees
a
CSR
that
is
approved,
it
signs
it,
the
node,
the
sorry,
the
TTL
controller,
which
is
the
the
name,
is
not
helpful.
It's
actually
managing
how
long
nodes
will
cache
secrets
and
config
maps
based
on
the
cluster
side.
A
A
It
observes
conditions
on
the
node
and
sets
taints,
which,
like
keep
pods
from
scheduling
and
also
if
the
node
goes
to
unhealthy,
it
will
evict
the
pods
like
if
the.
If
the
node
isn't
around
to
like
manage
itself,
we
have
to
have
a
backup
controller
come
in
and
and
manage
things
for
it
and
I
will
point
out
at
this
one.
It's
a
good
example
of
like
bridging,
like
understanding
the
no
conditions
and
turning
those
into
a
set
of
things
at
this
that
the
scheduler
can
understand
right.
A
So
the
scheduler
doesn't
have
to
understand
like
like
low
disk
state
or
something
on
the
on
the
node.
It
can
just
look
at
the
tank.
The
cloud
node
controller
is
for
initializing
nodes.
They
come
up
with
the
cloud
taint.
This
thing
takes
it
off
and
it
puts
cloud
specific
properties
on
the
root
publisher
is
yet
another
part
of
the
the
security
process.
Your
pod,
when
it
comes
up
it
has
to
be
able
to
trust
api
server.
Api
server
presents
assert.
A
Next
up
is
the
scheduler,
which
I
called
the
candy
wrapper.
It's
maybe
the
most
complicated
controller
in
the
system.
It
is
a
controller
I
just
listed
the
things
that
it
looks
at.
Maybe
it
looks
that
more
that
I
didn't
see,
but
it
is
it's.
It's
I'm
not
gonna,
explain
that
here,
okay,
we're
getting
close,
bear
with
me.
There's
a
lot
of
controllers:
this
is
the
scheduled
job
controller,
which
is
like
you,
you
give
it
a
job
and
like
a
like
a
frequency,
and
it
like
just
waits
for
that
to
come
up.
A
I
like
to
think
of
this,
as
injection
with
the
phase
of
the
moon.
Next
is
job
itself,
which
is
the
injection
is
complicated.
The
the
thing
is
just
running
a
pod,
so
that's
not
too
complicated,
but
you
have
to
run
it
in
like,
like
you
want
in
total,
and
you
can
only
do
em
at
a
time,
so
it's
kind
of
like
baking
cookies
into
the
oven
like
you
can
only
fit
so
many
sheets
on,
and
maybe
sometimes
the
cookies
burn
and
yeah.
A
You
know
three
of
these
things
or
whatever,
but
the
thing
that
you're
doing
is
very
complicated
because
you've
got
to
make
a
persistent
volume
claim
and
a
pod
and
like
match
them
up
and
like
what,
if
they
go
down
and
you
restart
the
disorder
so
yeah,
that's
like
that's
a
complex
thing
similar
to
cubelet,
which
is
also
doing
a
very
complex
thing
with
plugins
and
the
node
OS
and
like
it's
got
a
update,
pod
status
and
node
status
and
read
secrets
and
config
maps.
That's
complicated!
Okay,
almost
done
this
is
the
the
operator
pattern.
A
You
may
recognize
it
from
my
one
of
my
previous
talks,
and
this
is
this
is
like
how
this
is
like
the
canonical
way
from
bridging
a
legacy
system
and
providing
an
interface
that
works
natively
with
kubernetes
API
clients
and
like
we
actually
have
some
controllers
that
are
essentially
this.
The
first
one
is
the
service
controller,
which
updates
allocate
some
updates
load,
balancers,
the
route
controller
programs,
the
network
fabric
in
the
in
the
cloud
provider,
the
node
life
site
with
a
cloud
node
life
cycle
provider
controller
it.
If
your.
A
If
your
node
goes
unhealthy,
it
checks
to
see
like
does
the
cloud
provider
still
think
that
this
node
should
exist
and,
if
not
like,
maybe
I'll
just
delete
it,
and
the
no
problem.
Detector
also
falls
into
this
category,
where
it's
like,
observing
the
operating
system
on
the
node
and
writing
corresponding
conditions
to
the
to
the
node
object.
Okay,
that's
all
of
it
most
of
them.
Do
you
feel
ready
to
write
your
own
controller.
I
really
appreciate
you
coming
to
this
talk.
I
think
this
is
a
really
fun
problem
space.