►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Writing Kube Controllers for Everyone - Maciej Szulik, Red Hat (Beginner Skill Level)
With the introduction of Custom Resource Definition, as well as external API servers everyone is now able to create its own resources and store them inside kubernetes cluster. But that is only half of the work that needs to be done. During this presentation Maciej, who co-authored both Job and CronJob controllers, and is a frequent reviewer of controller and apiserver-related code will guide you through basic set of steps that are necessary to write a simple controller. As an example he will discuss his recent work around improving CronJob controller, as well as introduce the 11 rules that needs to be fulfilled when writing a good controller. Topics to be covered include: 1. How controllers work, loop over:* monitor resources * react to changes 2. Shared Informers. 3. Owner references. 4. Emitting status. 5. Handling errors.
About Maciej
Maciej is a passionate developer with over 10 years of experience in many languages. He's working on OpenShift and Kubernetes for Red Hat. In his free time he enjoys hacking on bugs.python.org and CPython's IMAP library. He's a frequent speaker at various events and meet ups, including both previous KubeCon EUs, multiple PyCons and DevConfs.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Welcome
it's
about
time.
It's
11:55
I
know:
I
am
right
before
lunch,
so
I'll
promise
to
speak
fast
enough
for
you
to
pick
up
something
from
my
talk
of
course,
but
at
the
same
time,
to
share
my
knowledge
as
much
as
I
can
with
you
about
the
controllers.
My
name
is
ma
che
I
work
for
Red,
Hat
and
I'm
part
of
the
both
kubernetes
core
team
and
OpenShift
core
team.
A
A
Better
controllers,
how
many
of
you,
if
you
hear
the
word
controller
and
thinks
about
this?
Nobody
like
seriously
people
people,
okay,
so
you
are
at
least
there's
a
handful
of
you.
There,
a
very
happy
person
and
fun
person.
Unfortunately,
I'm
I'm
a
little
bit
off
the
hook,
because,
for
me
controller
is
this
it's.
This
is
actually
a
a
cron
job
controller
I
managed
to
fit
in
this
four
hundred
lines
of
codes.
But,
to
be
honest,
it's
almost
four
four
hundred
lines
of
code.
A
If
you
count
the
additional
utilities
etc,
it
will
be
more
close
to
a
thousand
lines
of
code
and
but
the
good
thing,
it's
probably
the
smallest
controller
that
we
have
in
kubernetes
world
so
but
still
for
a
newcomer,
it
might
be
quite
overwhelming
to
to
look
through
it
and
figure
out.
What's
going
on,
there's
a
lot
of
use
cases
that
we
are
covering
in
every
single
controller.
A
So,
during
the
this
presentation,
I'm
hoping
to
give
you
the
core
components
of
every
single
controller,
I'm,
hoping
that
after
the
presentation,
you
will
feel
comfortable
reading
the
code
of
a
controller
or
even
writing
your
own
controller.
Let
me
raise
your
hand,
whoever
wrote
a
controller
or
seen
a
source
code
over
controller.
This
doesn't
count.
Okay,
there's
some
of
you.
That's
that's!
That's
nice!
So,
okay,
these
four
hundred
lines
of
code.
A
Let
me
try
to
squash
it
up,
so
that
it's
more
comprehendible
for
you,
that's
what
the
controller
actually
looks
like
this
is
the
core
piece
of
a
controller.
So
after
we
created
the
controller
structure,
we
initiated
with
some
stuff
that
I'll
be
talking
about
in
a
few
moments,
we
do
Girl
teen.
That
is
basically
doing
just
this.
It's
processing
every
single
item
from
the
queue
picking
it
up,
do
whatever
you
want
to
do
with
your
controller,
push
it
back
to
the
queue
or
ignore
it,
update
the
status
and
continue.
A
This
is
basically
the
entire
logic
of
every
single
controller
within
the
kubernetes
world
and
I'm
not
talking
when
I'm
talking
about
controllers.
This
applies
to
see.
Are
these
as
well
as
the
core
controllers?
All
of
them
are
working
in
the
same
way,
but
then
again,
if
you're
a
newcomer,
it's
pretty
overwhelming.
Where
do
you
start
how
to
write
your
own
controller
or
where
to
look
for
stuff
like
the
one
that
I
just
mentioned?
A
So
you
can
start
like
just
like
me
when
I
was
writing
my
first
controller,
the
job
controller
back
in
the
old
days
when
kubernetes
was
small
and
not
that
many
folks
knew
about
it.
I
actually
opened
replication
controller
and
reading
the
replication
controller
I
was
replicating
the
same
logic
within
the
job
controller.
It
worked
pretty
well,
but
my
advantage
was
at
the
time
that
I
was
already
a
kubernetes
contributor
for
some
time
and
I
knew
more
or
less
how
to
go
through
the
entire
kubernetes
stack.
A
Besides,
there
was
nothing
else
at
the
time
that
I
could
actually
pick
it
up.
So,
thankfully,
for
you,
the
stuff
is
so
much
easier.
These
days
you
have
two
possibilities
and
numbers
blog
posts,
tutorials
or
the
simplest
thing
just
reach
out
to
myself.
Personally
on
slag,
there
was
a
Twitter
handle
github
and
all
like
everything
on
the
first
slide
and
I'm
gonna.
Repeat
it
on
them.
A
On
my
last
slide,
I'll
be
more
than
happy
to
help
you
with
anything
and
additionally,
like
I,
said,
there's
so
many
already
written
controllers,
starting
from
simple
all
the
way
to
the
super
advanced
controllers,
so
this
is
per
actually
part
of
the
main
kubernetes
repo,
it's
being
synced
with
the
main
repo,
and
we
ensured
that
this
is
always
working.
This
sample
controller,
the
sample
controller,
is
very
simple.
It
is
supposed
to
help
you
get
started.
Unfortunately,
writing
controllers.
A
There's
other
project
that
sets
up
the
boilerplate
for
you,
if
you
use
queue
builder
queue
builder,
is
very
helpful.
If
you're,
starting
with
building
your
own
CR
these,
it
will
generate
that
the
types
that
go
for
you,
it
will
generate
the
the
framework
for
your
controller
and
you
can
actually
focus
only
on
filling
in
the
logic.
But
then
again
it
is
nice
to
know
the
base
details
for
what
fits
with,
which
block
and
how
the
the
blocks
should
interact
with
each
other
that
that
make
sense
so
far.
A
Okay,
so
ahead
of
us,
there's
nifty
grifted
details,
we're
gonna
be
looking
at
some
code,
I
promise.
It
won't
be
that
hard
and
but
I'm
not
going
to
be
throwing
at
you
tons
and
tons
of
code
as
well,
because,
like
I
said,
there's
the
sample
controller
and
the
queue
builder
that
are
very
helpful
too,
to
get
started.
So
let
me
rish.
Oh
you
back
of
the
main
control
loop
and
within
that
loop.
There
are
three
elements
that
I
can
already
point
you
that
we
will
be
focusing
on
the
first.
One
is
actually
we're.
A
Looking
at
the
process.
Next
item
is
the
queue.
The
second
one
will
be
the
sync
handler,
and
this
has
to
be
something
that
is
filling
in
the
queue.
So,
let's
start
with
the
Q,
the
Q
is
pretty
simple
topic.
Are
you
familiar
with
queues?
Generally,
that's
a
first
hand.
Fist
up
first
out
queue,
there's
no
magic
behind
it.
You
can
set
it
up
pretty
easily.
There
are
several
rate
limiters
available
within
the
cube
project.
A
The
default
one
and
the
other
ones
are
exponential
failure,
rate
limiter,
there's
a
fast,
slow
rate
limiter,
and
there
you
can.
You
can
also
combine
the
limiters
using
Max
of
rate
limiters,
although
personally
I'm
always
using
the
the
exponential
fail
rate
limiter.
So
that's
pretty
obvious,
there's
no
magic
behind
it.
You
put
stuff
into
the
queue
you
put
it
you
take
it
out
and
the
in
the
sync
loop
and
the
work
keeps
going
on
over
and
over
again.
So
the
most
important
piece
of
every
single
controller
is
this:
the
sheraton
formers.
A
Remember
the
cron
job
controller.
That
I
showed
you
at
the
beginning
of
all
the
controllers
that
I
showed
you.
Please
do
not
take
the
cron
job
controller
as
an
example.
I
wrote
it,
but
it
has
a
major
flaw
that
major
flow
is
that
it
is
not
using
the
Sheraton
formers
and
the
reason
for
that
there
is
a
like,
very
important
reason.
The
cron
job
controller
was
written
before
the
Sheraton
formats
were
introduced.
I
need
to
find
the
time
to
rewrite
this.
A
So
it
takes
a
little
more
time
for
me
to
actually
pick
it
up
and
and
improve
it.
If
you
have
the
time
if
you're
willing
to
do
it,
I'll
be
more
than
happy
to
help
you
with
this
as
well.
So
let
me
know-
and
you
know
we
can
get
some
code
cranking
going
on
so
the
Sheraton
format,
what
it
is.
That's
the
funny
thing
I
was
looking
through
the
entire
kubernetes
repo,
every
kubernetes
repo
of
every
coronaries
developer.
If
you
ask
him
yeah,
yeah
I
know
what
I
sure
it
informer
is
everybody.
A
Everyone
should
be
using
it
and
everyone
are
using
it.
But
there's
to
be
honest,
there's
no
documentation
about
what
I
shared
Informer.
Is
it's
it's,
this
type
of
tribal
knowledge
that
everyone
has
that
everyone
possess,
but
nobody
is
actually
competent
enough
to
write
a
dog
about
it
or
even
explain
you
in
like
human
words,
what
it
does
so
I'm
gonna
I'm
gonna,
try.
A
A
Picture
that
is
summarizing
the
interactions
in
the
controller,
and
that
is
actually
focusing
a
lot
on
the
Sheraton
formers
a
lot
so
I'm
hoping
that
that
that
will
help
you
to
to
get
the
I
and
get
the
idea
more
or
much
better.
So
the
most
important
part
of
the
Sheraton
formers
is
the
event
handlers.
This
is
how
you
register
your
interest
in
specific
of
object
updates,
whether
that
will
be
a
a
addition
of
an
object
creation
of
an
object,
an
update
to
the
object
or
removal
of
the
objects.
A
There
are
three
functions
that
you
need
to
register
your
interest
in
and
basically,
if
you
look
at
every
single
controller
that
it
is
available
out
there
all
they
do
is
they
will
look?
What
up?
What
was
what
was
sent
in
the
eventlistener
and
they
will
put
the
the
information
about
that
object
into
the
queue.
A
A
There
was
a
pretty
good
question
about
two
days
ago
during
the
contributor
summit,
why
the
Lister's
interface
is
different
from
the
client
go,
and
why
would
you
use
a
listers
versus
regular
client
go
so
the
reasons
for
using
Lister's
is
they
are
designed
specifically
for
to
be
used
within
controllers.
They
have
the
access
to
the
cache
using
client.
A
Go,
you
always
hit
the
API
server
and
obviously,
if
you're,
working
with
I
don't
know
ten
thousand
deployments
or
even
more,
you
do
care
about
using
the
cache
as
much
as
possible,
rather
than
going
back
to
the
API
server.
This
is
the
problem
within
the
cron
job
contourer
cron,
job
controller,
currently
lists
which
is
very
expensive
operation
lists
all
the
cron
jobs
within
the
cluster
and
lists
all
the
jobs
within
the
cluster
to
be
able
to
identify
with
jobs
much
to
which
controller
and
what
to
do
with
them.
This
is
super
expensive
operation.
A
I
know
that
there
are
companies
that
are
using
cron
jobs
up
to
a
couple.
Thousands-
and
it
is
working
fine,
but
with
my
knowledge
of
the
controller
and
what
I
do
told
you,
it
will
be
harder
if
they
exceed
the
number
a
little
bit
higher.
So
the
Sheraton
former's
is
the
key
for
working
with
controllers.
So
you
add
the
informers
and
then
you
yeah,
you
register
the
event
handlers,
and
then
you
say
for
yourself
the
the
Lister's,
and
now
we
go
to
the
main
loop.
A
This
is
where
you
actually
implement
the
logic
of
your
controller,
let's
step
by,
let's
go
one
line
by
one.
So
the
cue
that
I've
mentioned
you
at
the
beginning
is
not
storing
the
actual
objects.
It
is
storing
something
that
we
called
a
meta
namespace
key,
which
is
basically
a
namespace
and
the
name
of
the
resource
bundled
together.
There
are
even
specific
helper
methods
that
are
designed
to
encode
and
decode
those
keys
back
and
forth,
to
namespace,
name
and
and
and
in
the
other
directions.
A
Depending
of
your
controller,
you
might
be
getting
more
than
just
one
object
in
my
particular
case,
I'm
just
getting
apart,
because
if
I
would
try
to
put
more
resources
here,
then
obviously
that
wouldn't
won't
fit
into
into
the
screen
and
I
was
very
I
was
very
inclined
to
fill
in
as
much
of
the
code,
but
not
too
much
into
the
slide.
Does
that
make
sense
so
far?
A
Okay-
and
here
goes
the
most
important
thing
of
all
right
after
the
Sheraton
formers,
because
you're
using
a
shared
cache,
all
the
resources
you're
getting
are
come
you're
not
getting
a
life
you're,
not
getting
a
a
an
object,
you're
getting
a
pointer
to
the
cash
object,
which
means,
if
you're
just
reading
the
object,
you're,
fine,
you
don't
have
to
do
anything.
If
you
want
to
modify
the
object,
you
need
to
invoke
deep
copy
on
the
object
that
was
specifically
that
was
purposefully
done
for
the
contour
for
the
controllers.
A
But
every
single
controller
writer
has
to
remember
that
we
had
tons
of
problems
when
we
switched
to
this
pattern.
Currently
there
were
some
weird
controllers
stuff
happening.
If
you
don't
do
that,
so
always
ensure
to
do
a
deep
copy,
but
be
careful
deep
copy
is
very
expensive
operation
as
well,
so
only
do
deep
copy
if
you
will
be
modifying
the
object.
A
That's
pretty
much
it.
You
should
be
now
focusing
on
your
logic,
your
entire
logic
of
your
deployment,
your
job
controller
replica,
said
whatever
the
controller
you're
working
on
is
actually,
where
you
put
that
the
job
controller
will
be
looking
through
the
spec,
ensuring
that
there's
this
many
pause
running
at
the
time.
Fine
I'm
done
I'm
going
to
update
the
status.
Yes,
it's
fine
if
I
need
to
the
deployment
controller,
will
ensure
that,
yes,
there
is
a
replica
set
created
that
it
matches
the
dispatch
that
the
user
provided.
A
The
diamond
set
controller
will
ensure
that,
yes,
there
is
a
pod
running
on
every
single
note,
etc,
etc,
etc.
This
is
how
simple
it
actually
is.
I
know
that
I've
heard
lots
of
people
saying
that.
Well,
writing
controllers
is
very
hard,
but
for
me,
personally,
controllers
are
probably
the
simplest
things
in
the
entire
cube.
There
are
Hardware
stuff
like
the
API
machinery,
and
that
is
a
pain
for
me.
Personally,
okay,
remember
the
picture
that
I
mention
you.
A
Yes,
this
object
of
mine
was
updated,
I'm
gonna
put
it
in
the
queue
and
then
I'm
gonna
take
it
from
the
queue
do
whatever
I
want
to
do.
Within
my
controller
update
the
resource
status
and
on
and
on
and
on
does
that
make
sense?
Okay,
so,
as
a
closing
remarks,
let
me
let
me
show
you
with
some
ground
rules
for
every
single
controller
within
the
kubernetes
ecosystem.
A
There
are.
There
are
there's
a
link
also
in
the
contributors
repo,
so
that
you
can
read
a
little
bit
more.
But
here
are
the
eleven
rules
always
operate
on
a
single
item.
At
a
time
by
by
single
I
mean
if
you're
working
with
replication
controllers,
it
will
be
one
replication
controller,
but
obviously
a
replication
controller
operates
on
that
replication
controller,
but
also
on
all
the
parts
that
it
is
managing.
So
it
one
resource
at
a
time
doesn't
mean
literally
just
one
resource,
because
it
won't
be
possible
and
feasible
to
do
it.
A
A
job
will
be
always
a
job
and
that
many
job
that
many
parts
that
every
single
job
instance
is
managing
at
a
time.
So
that's
very
important
number
two.
You
need
to
ensure
that
you
are
not
hard-coding
any
ordering
between
the
items
that
you
are
processing
in
the
queue
it
should
be
entirely
random.
There
should
not
be
any
interconnection
between
the
objects
that
you
are
working
on
remember.
This
is
highly
reliable,
super
scaled
big
system.
At
the
end
of
the
day
that
we
were
talking
about
you
shredded
Informer's.
A
This
is
highlighted
and
I've
talked
about
it
I've,
given
you
example
of
poorly
written
controller
or
rather
not
updated
yet,
but
I'm
working
on
it
I
promise.
You
there's
one
thing
that
I
haven't
mentioned
at
the
at
the
beginning
of
your
controllers.
When
you
are
setting
up
everything,
you
need
to
wait
for
the
caches
to
be
ready,
so
there's
a
nicely
function
that
you
just
need
to
wait
for
to
notify
your
controller
that,
yes,
the
caches
are
ready.
They
synced
it
might
take
some
time
at
the
beginning
of
the
controller.
A
A
Said
it
before,
but
I'm
gonna
repeat
it,
there
are
a
direct.
There
are
other
actors
in
the
system,
so
you're,
not
the
only
controller,
be
mindful
when
you're
writing
stuff
to
the
API.
You
need
to
be
prepared
that,
for
example,
when
you're
updating
your
resource,
there
might
be
a
conflict,
because
somebody
else
in
the
meantime
updated
your
resource.
Your
controller
has
to
be
prepared
for
the
conflicts.
You
need
to
resync
your
object
once
again
or
repeat
your
update
process.
A
Ensure
that
you
are
providing
to
users
a
reasonable
error
messages
for
your
controllers.
There
are
events
and
the
events
is
the
best
options
for
you
as
a
controller
author
to
provide
information.
If
there's
any
error
happening
with
your
resource,
that
you
cannot
update
it
properly
and
queue
the
same
object
once
again,
so
that
it
will
be
recent
in
a
while
again-
and
maybe
the
error
is
just
a
temporary
issue
that
will
be
will
be
gone
within
a
couple
minutes.
A
A
This
is
basically
if
your
resource
provides
something
that
is
called
a
generation,
which
means
you
are
encoding
within
the
object,
specification
and
information
that
yes,
there's
something
new
happening
to
the
object,
and
the
controller
should
pick
it
up.
You
need
to.
You
should
provide
that
information
in
the
observed
generation
field
in
the
metadata
of
an
object,
for
example,
currently
job
controller
I
know,
I'm
gonna
be
talking
about
both
job
and
cron
job
controller,
and
the
reason
is
that
I
know
all
about
the
box
that
they
have
because
I
wrote
both
of
them.
The
job
controller.
A
Has
this
problem
I'm
honest
with
all
of
that.
The
job
controller
has
this
following
problem:
it
currently
requires
all
the
parts
created
by
the
job
controller
to
exist
all
the
time.
It's
fine
if
your
job
is
creating
ten,
a
hundred,
maybe
a
thousand
parts,
but
it
will
be
a
problem
if
your
job
will
start
creating
ten
thousand
a
hundred
thousand
or
a
million
parts.
The
reason
for
that
is
the
current
calculation
that
we
have
for
the
status
of
the
job
is
based
on
the
pots
that
finished
or
are
currently
running
or
are
to
be
run.
A
A
This
is
a
topic
that
we
put
together
about
two
releases
two
three
releases
ago.
This
is
about
embedding
the
information
about
your
parents
and
the
objects
that
you
are
creating.
For
example,
a
job
controller
would
put
a
owner
a
reference
information
within
all
the
parts
that
it
is
creating
why
it
is
important.
This
is
important
when
you
are
removing
the
objects,
if
I'm
removing
my
job.
The
majority
of
time
I
want
all
the
parts
that
are
connected
that
are
created
by
the
job
to
be
gone
as
well.
A
The
current
garbage
collection
mechanism
that
we
have
within
the
kubernetes
is
looking
at
the
owner
references.
You
can
explicitly
say
I,
don't
care
about
the
kids
just
leave
them.
I
want
to
have
just
my
job
removed,
but
the
default
behavior
is
remove
job
and
all
the
parts
and
actually
the
Cronk.
The
garbage
collection
will
find
figure
it
out
all
the
parts
first
remove
the
parts
first
and
only
the
job
at
the
end.
A
Obviously,
for
for
the
job,
controller
and
pots,
that's
very
simple:
there's
only
one
level,
but
for
a
deployment
deployment
creates
replicas,
which
creates
parts.
If
you
would
be
manually
removing
that-
and
we
were
doing
that
in
a
cube,
CTL
command
delete
and
cube
CTL
delete
command.
Then
that
was
a
pain
because
we
had
to
remove
all
the
parts
and
we
had
to
remove
all
the
replicas
set,
and
only
after
that
we
were
able
to
remove
the
actual
deployment
with
the
garbage
collection.
All
you
all.
A
The
deployment
controller
has
to
do
is
set
the
owner
references
appropriately.
At
the
replication
controller,
the
replica
replica
set
controller
will
set
the
owner
references
at
the
pod
level.
Then,
when
I
removing
the
deployment,
all
the
way
down
is
happening
automatically,
because
the
garbage
collection
is
looking
at
the
owner
references,
that's
it.
That
is
actually
pretty
straightforward
idea.
The
only
reference
is
something
like
a
namespace
and
name
and
the
type
of
the
resource
that
owns
the
object.
That's
basically
it
and
that's
basically,
all
I
had
for
you.
A
A
Yeah
sure
the
question
was
how
it
relates
to
the
operator
framework.
Well,
the
operator
framework
is
it's
very
similar
to
the
controller.
You
might
say
it
is
more
to
help
you
operate.
Other
stuff
and
operator
will
be
more
targeted,
I
specific
at
a
specific
application,
like
in
the
case
that
we
have
a
TC
d
operator,
the
a
TC
d
operator
know
how
to
interact
with
TC
d.
Specifically,
it
has
the
knowledge
of
the
the
carrier
key
and
everything
how
dat
dat
C
D
should
work.
A
A
A
So,
unfortunately,
the
shirt
and
former's
are
not
available
in
any
other
language
then
go,
and
that
is
the
biggest
problem.
Client
go
I
mean
there
are
clients
for
kubernetes
api
in
other
languages,
and
python
was
actually
the
second
one
right
after
go.
There
are
java
and
something
else,
but
I
can't
remember
that
are
being
generated
from
the
API.
A
The
only
downside
currently
is
that
you
would
have
to
implement
your
own
shared
and
foremost
type
of
a
functionality,
or
you
are
dealing
basically
with
the
problems
of
interacting
with
the
with
the
kubernetes
api
directly
instead
of
through
the
cache,
which
is
obviously
the
benefit
of
having
and
not
having
a
cache,
is
a
significant,
a
significant
problem,
but
that
but
then
again
it's
fun
and
I
think
somebody
wrote
and
actually
a
controller
with
Python
for
yeah.
That's
that's
awesome.
So
I
would
like
to
see
that
happening.
A
I'm
not
sure
if
I
correctly
understood
you're
asking
about
running
controllers
in
a
pod,
your
CRT
controller
in
a
pod,
or
that's
something:
yes,
okay,
so
currently
in
the
coronaries
all
the
controllers.
Each
controller
is
a
separate
mechanism.
Each
is
running
in
their
own
guru
teen,
as
is
so.
If
we're
talking
about
the
CR
DS,
every
CR
D
will
have
their
own
controller,
because
controller
is
always
tied
to
a
specific
type
of
a
resource.
A
A
Obviously,
the
sink
handler
method
will
be
the
most
important
and
the
most
heavily
tested
element
in
all
the
controllers.
If
you
go
through
the
current
cork
Banaras
controllers,
the
sink
header
where
you
feed
the
sink
handler
with
some
data
in
and
expect
that
at
the
end
of
the
of
the
execution,
this
state
will
be
this
and
not
the
other
way
around.
So,
for
example,
for
both
job
and
cron
job.
A
The
tables
are
pretty
nifty
themselves,
but
it
takes
a
while
to
get
used
to
it,
because
we
we
created
lots
of
one
letter
variables
so
that
you
can
fit
in
in
one
line
the
entire
specification
for
a
test,
so
it
takes
a
while
every
even
for
me,
sometimes
when
I'm
returning
to
that
oh
yeah.
Ok,
this
is
this.
This
is
that,
but
it's
at
the
end
of
the
day
it
it
works
pretty
well
and
then
that
that
covers
the
main
functionality.
A
And
then
you
need
to
have
a
été
test
that
will
be
actually
stress
testing
your
your
particular
controller,
whether
it
is
interacting
the
way
you
expect
it
to
be,
and
that
includes
creating
deleting
scaling
up
down
removing
parts
from
the
controller
if
you're,
managing
parts
or
braking
and
ensuring
that
the
controller
will
pick
it
up.
The
way
you
design
it
to
be,
there's
no
patterns
by
itself,
but
it's
these
two
areas
that
you
will
be
focusing
on
for
your
tests.
Sure.
B
A
There
is
nothing
like
that
to
send
notifications
between
the
controllers.
You're,
always
none
of
the
current
controllers
is
running
or
actually
we
are.
If
we
are
running
the
same
controllers,
there
always
have
the
leader
that
is
actually
working
on
it
and
the
backup
one
as
you
know,
so
that
it
picks
it
up.
There's
only
one
controller
of
any
kind
running
once
at
a
time.
At
least
that's
the
that's
the
implementation
that
we
have
in
OpenShift.
A
We
always
try
to
keep
some
kind
of
a
leader
election
for
the
controllers.
There's
only
one
running
at
a
time.
I
like
all
of
them
are
running,
but
only
one
is
actually
picking
up
the
work.
That's
that's
that's
what
we
implemented
an
open
ship
and
if
you
look
at
the
kubernetes,
it
I
think
they
have
something
like
that.
Similarly,
two
different
contours
is
a
separate
thing
and,
like
I
said,
you
cannot
escape
from
the
fact
that
there
are
other
actors
in
the
system.
A
C
A
There
are
several,
it
also
depends
because
some
tests
are
specifically
for
performance
testing.
There
will
be
in
a
separate
suit,
but
generally
they
all
live
in
a
one
place.
It'll
be
only
labeled
differently,
whether
yeah
the
tags,
whether
you
want
to
have
these
run
as
a
confirm,
is
for
the
conformance
we're
trying
to
focus
only
on
the
car
functionality
of
every
single
controller
to
be
actually
run
within
the
conformers,
because
then
that
also
applies
to
all
the
kubernetes
distributions.
If
you
want
to
be
a
certified
kubernetes
distribution,
you
need
to
pass
all
the
conformance
testing.