►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Keep the Space Shuttle Flying: Writing Robust Operators - Illya Chekrygin, Upbound
As the operator pattern becomes the de-facto pattern for extending Kubernetes, the community is increasingly adding new declarative APIs for external resources that are managed by traditional imperative API.
In this talk, we will explore a set of lessons learned from building Operators that bridge Kubernetes’ declarative API with external imperative API. We’ll cover some of the challenges around resource identity, long-running changes, retries, progress, handling failures, validation, dry-runs, deletions, and finalizers. These lessons are based on the real-world experience of building multiple complex operators within the Rook (https://rook.io), Crossplane (https://crossplane.io).
https://sched.co/MPaN
A
Hello
and
welcome
to
my
talk,
writing
robust
operators.
My
name
is
Ilya
chick
Reagan
I'm,
a
founding
engineer
at
abound.
I'm
also
super
tired,
like
most
of
you,
probably
and
unlike
most
of
am
super-nervous
and
also
I'm
a
little
meeting
and
crossplane
open
source
project.
We
started
about
eight
to
nine
months
ago,
at
a
bound,
we
open
source
is
right
before
cube
con
in
Seattle
last
December
crossplane
is
an
open
source,
multi-cloud
control,
plane
and
without
going
into
any
more
details.
A
A
Okay,
that's
what's
terrible
joke
I
was
terrible
idea,
but
idle
didn't
work
out
for
shuttle
that
well
either
yeah
program
was
cancelled,
so
I
guess
there
is
lesson
for
there
somewhere.
Okay.
Joking
aside,
I
think
this
comment
captures
precisely
the
spirit
and
intent
of
this
presentation.
Writing
robust
kubernetes
operators.
A
A
It
took
some
time
for
me
to
realize
that
arguably
the
best
feature
of
kubernetes
is
kubernetes
api,
with
its
declarative
style
versus
traditional
imperative,
where
users
express
captures
once
versus
how
to
accomplish
things.
It's
level
based
versus
age
based
which
enables
robust
behavior
in
the
presence
of
mists
intermediate
steps,
changes
and
it's
complete
and
authoritative,
and
last
but
not
least,
it's
extensible,
and
this
is
what
the
rest
of
this
talk
about.
A
So
you
can
customize
grace
by
either
configuration
and
when
you
change,
let's
say
you
config
files
for
kubernetes,
api
controllers,
cubelet
or
extensions
which
involve
running
additional
programs.
Kubernetes
is
designed
to
be
automated
by
writing.
Client
programs
and
any
program
that
read
and
write
to
complete
API
key
and
provide
useful
automation.
Do
the
specific
pattern
for
writing?
A
To
do
that,
you
need
to
write
your
own
API
server
to
support
your
API
types
and
there's
also,
you
can
define
series
which
separate
objects
which
you
can
create
your
schema
and
type,
and
you
can
register
it
and
store
it
in
existing
companies
API.
This
frees
you
from
writing.
Api
server
to
handle
your
type,
but
generic
nature
of
the
implementation
means
that
you
have
less
flexibility
done
in
a
pea.
Aggregation.
A
A
It's
a
queue
builder
and
a
parade
SDK,
and
this
is
a
great
time
for
me
to
thank
all
contributors
to
company
six
controller
runtime,
especially
direct
x-men,
12,
Sally,
Ross
I,
don't
know
he's
an
audience
or
maybe
he'll
be
watching
this
presentation
later
on
for
super
amazing
and
super
useful
project,
obviously
build
that
to
bootstrap
multiple
project.
We
started
and
abound,
including
crossplane.
So
this
I'm,
the
most
familiar
with
meta
control,
is
a
different
take
on
extending
kubernetes
by
using
meta
controller
server
and
the
lambda
controls
and
lambda
hooks
in
their
own
words.
A
Meta
controller
is
an
add-on
for
kubernetes.
That
makes
it
easy
to
write
and
deploy
custom
controls
in
the
form
of
simple
scripts,
and
this
is
fresh
of
the
press.
The
I
read
today
the
shell
appeared
was
just
announced
and
I
didn't
look
too
much
into
it,
but
it
kind
of
very
cursory
look
from
reading
initial
blog
post.
It's
very
similar
conceptual
tomato
controller
in
terms
of
using
shell
script
hooks.
A
Then,
of
course,
there
is
a
low-level
apparatus
like
Luca,
predicate,
probably
one
of
the
oldest
frameworks
and
is
very
popular
within
a
root
community
and
outside
of
it.
It's
pretty
low
level,
as
I
mentioned,
it's
very
close
to
rock
lion,
go
and
also
simple
controller,
which
is
basically
example
how
to
write
your
own
operating
implementation
from
scratch.
Using
client,
though,.
A
So
big
question,
which
framework?
Would
you
like
to
use,
or
do
you
want
to
use,
to
write
your
operator
and
I
think
this
is
not
so
straightforward?
If
you
are
new
to
operators,
I
would
recommend
to
use
controller
runtime
with
this
you
build
the
base
tour
operators
decay.
Now,
if
you
feel
like
controller
runtime,
is
too
restrictive
because
introduces
too
much
magic,
which
Darrent
the
very
correctly
pointed
out.
It's
kind
of
sometimes
hides
details
from
you.
It's
harder
to
understand.
A
Okay,
so
we
have
pretty
good
number
of
people
so
I'll
quickly
skip
to
the
next
slides,
which
deals
with
the
description
of
type
and
I
jump
straight
into
controller
control.
Loop,
roughly,
can
be
described
by
follows.
A
controller
is
back
to
this
component,
which
you
write
code
for
in
deploy
into
kubernetes,
which
will
process
you
types
can
be
roughly
described
by
following
state,
and
this
is
the
diagram
I
bought
from
the
block
of
from
Couchbase.
They
have
a
it's
observed,
state
analyze
and
react
so
in
general,
in
controller
reconcile
loop.
A
So
when
you
write
operator,
which
manages
external
types,
let's
say
a
cross
plane,
for
example,
it
manages
types
using
cloud,
public
cloud,
API,
I'll,
say,
spin
up
me
a
RDS
database
or
create
GCS,
buckets
and
so
forth.
Then
steps
exactly
the
same,
except
the
diagram
slightly
different.
The
operator
will
make
external
calls.
However,
it
also
has
a
certification
on
the
keepin
state
of
those
objects
because
they
not
share
in
the
same
notification
mechanism,
so
you
need
to
do
a
little
bit
extra
work.
It
also
requires
a
little
extra
work
to
clean
them
up.
A
So,
just
as
you
write
mentioned,
internal
states
or
internal
objects
with
operators,
you
will
use
kubernetes
api
and
client
go
and,
as
we
increase
play,
for
example,
managing
external
objects,
we
use
cloud
providers
api's
and
the
respective
go
clients
and
of
course
there
could
be
other
external
objects
as
well.
How
could
not
you
can
also
use
external
manager
external
objects
with
the
kubernetes
api
if
you
manage
other
objects
in
other
clusters,.
A
As
I
have
mentioned,
the
controller
runtime
does
a
great
job
of
hiding
all
the
low-level
details
of
Watchers,
informers,
indexers
and
so
forth
and
present
throughout
that
clean
reconcile
interface,
reconcile
function
is
called
to
handle
object,
which
was
in
queued
into
work
you
for
practicing,
and
it's
happened
by
a
external
update
or
internal
in
this
context.
External
or
internal
means
external.
To
this
controller
example,
external
would
be
you
use,
keep
CTL
to
create
your
object,
or
you
keep
yourself
to
apply
changes
to
your
object
internal.
A
When
you
practice
your
beauty
controller,
you
want
to
update
some
states
what
is
metadata
with
its
spec
or
status.
You
can
sell
function,
returns
to
values,
result
always
and
error.
If
any
and
quickly
recap
and
riku
and
result
it
has
the
different
review
properties
when
you
set
it
to
true.
In
other
words,
you
say:
I
want
to
Riku
the
subject
for
next
processing.
Typically,
you
want
to
use
it
when
you
encourage
some
error
condition.
In
other
words,
that
is
saying
that
I
could
not
fulfill
your
cancellation
at
this
loop.
I
want
to
Riku
the
process.
A
Again,
it
be
cautious
with
Riku.
You
use
it
at
for
waiting
States,
for
example,
when
you
create
external
resources,
it
takes
time
for
them
to
get
provision,
so
you
say:
ok,
I,
issue,
requests
to
create
bucket
I
want
to
now
go
away
and
come
back
next
time
in
to
check
on
it
again.
So
if
you
do
Riku
true,
it
will
fulfill
that
functionality.
However,
there
is
the
ramification.
Typically
when
you
in
queue
object.
A
He
has
to
interface
in
control,
runtime,
it's
called
delayed
and
the
one
which
is
causes
exponential,
back-off,
now
name
escapes
me
using
those
two
interface,
let's
say,
for
example,
if
you
look
you
true,
it
will
cause
exponential
back-off.
So
if
you
use
it
frequently,
your
object
will
be
delayed
in
the
pew
and
further
and
further
and
further-
and
this
is
what
happened
initially
and
crossplane
when
we
let's
say
waiting
for
kubernetes
cluster,
to
come
up
as
external
resource,
we'll
be
reviewing
waiting
for
it
and
it
will
increase
exponential
back-off.
A
So
by
the
time
restriction
sauce
is
ready.
We
so
far
back
in
exponential
back-off
than
our
controller
simply
sits
idle
waiting
for
object
to
being
healed
again,
and
we
have
this
gap
of
minutes,
sometimes
up
to
5
minutes
where
our
state,
in
the
controller
on
object,
doesn't
match
the
state
of
external
resource,
so
be
careful
with
that.
It's
not
good
thing
to
use
for
waiting
instead
consider
using
the
riku
after
very
similar.
A
It's
using,
however,
delayed
interface
and
what
it
does
it
say:
hey
I'm
provision,
external
resource
which
takes
about
five
minutes
to
prevail,
so
I
will
go
ahead
and
I
will
Riku
after
five
minutes
after
five
minutes
is
getting
place
and
work
you
if
there's
no
external
changes,
and
then
you
perform
functionality
again
and
hopefully,
if
resource
is
ready,
you
update
your
status.
However,
it's
not
ready,
then
you
can
use
more
gradual
Sinicki
from
this
point
on
Riku
and
wait
for
10
seconds,
because
it's
almost
ready
I
want
to
have
a
more
frequent
checks.
A
Also
be
careful
again
review
after
would
be
a
granular
cause,
because
if
your
object
will
never
get
in
the
ready
state,
never
get
successful
provision
and
you
do
in
this
vehicle-
checked.
You're,
basically
going
to
be
fresh
from
the
operating
system
continuously.
Spinning
glued
Riku
falls
basically
means
I'm
done
with
it
forget
about
it.
I
need
to
process
it
anymore.
Don't
let
Y
me
anymore
with
that,
the
only
time
you
will
get
that
quite
back,
if
somebody
externally,
edit
would
say
somebody
do,
keep
still
update
or
keep
still
deleting
it.
A
And
finally,
there
is
a
synch
period,
Control
Manager
property,
which
basically
said
that
at
a
given
point,
I
want
everything
to
be
refreshed,
such
a
global
refresh
for
simple
projects.
Maybe
it's
not
a
bad
thing
to
use,
however,
by
default
value
of
synced
set
to
10
hours,
and
maybe
it's
deliberately
so
can
discourage
you
from
using
it
to
maintain
state
of
your
system.
A
However,
you
can
tweak
it,
and
you
said
too
short
a
period
of
time,
however,
be
careful
again
if
you
set
it
even
through
the
system
with
a
simple
project
like
has
one
type,
you
can
have
many
instance
of
the
same
type.
So
if
you
set
it
to
very
short
period
time,
let's
say:
we've
Riku
every
10
seconds,
you'll
find
yourself
a
speeding
and
refreshing
many
objects
very
frequently
and
you're
going
to
always
have
busy
loop
without
actual
external
change
or
state
change.
A
Yes,
quick:
what
about
status
in
early
days,
kubernetes
type
used
phase
to
track
the
object,
status
and
phase
was
deprecated,
because
it's
I
think
it's
adequately
captured
the
stars
of
the
object
it
more
of
a
state
machine
rather
than
actual
state
of
an
object.
Today,
it's
encouraged
to
use,
and
by
IP
a
convention
is
encouraged
to
use
conditions.
Conditions
represent
the
latest
available
observation
of
object.
A
State
conditions
should
be
added
to
explicitly
convey
properties
that
users
or
components
care,
but
rather
than
requiring
those
properties
to
be
inferred
from
the
other
observations
once
defined,
conditions
should
be
treated
just
like
an
API,
a
meaning
that
any
changes
cannot
be
applied
fairly
and
have
to
be
considered
for
backward
and
for
compatibility,
objects,
may
report,
multiple
states
conditions
and
new
types
conditions
may
be
added
by
third
party
controllers.
Therefore,
conditions
represented
by
a
slice
or
list
where
you
have
all
of
them
have
similar
structure.
A
All
conditions
have
a
type
and
those
types
should
be
specific
for
your
application.
All
of
them
have
a
status,
whether
it's
active
condition
are
not
a
true
or
false
and
unknown
reason.
It's
a
single
word
camelcase.
This
of
the
condition
and
message
is
more
expanded
message
why
this
condition
was
activated
and
finally
have
a
timestamps.
This
is
all
captured
in
the
API
conventions.
The
link
at
the
end
of
the
slide
deck
in
references
we
used
last
transition
time.
A
We
don't
use,
deduce
heartbeat
or
I
didn't
use
carbon,
so
you
typically
use
the
timestamp
to
set
one
condition
been
activated.
What
of
caution
about
timestamp
used
in
timestamp?
You
want
to
set
that
time
only
one
condition
being
set.
You
know
what's
being
changed.
Initially
we
had
controllers,
which
basically
say
okay,
condition,
unset
or
condition
when
I'm
processing
and
then
at
the
end,
check
the
condition
of
the
object
and
match
in
one
and
was
created
that
we
had
the
object
resource,
which
was
always
in
ready
condition,
ready
to
go.
How
IRS's
will
perform
this
check.
A
All
the
time
rules
always
update
the
last
transmission
time,
which,
in
turn
created
object
to
be
always
in
dirty
state,
meaning
that
we
always
perpetually
worry
queuing
same
object
inventor.
There
were
no
changes
to
state,
need
external,
no
spec,
so
be
careful
on
that.
You
only
want
to
activate
what
set
this
time
step,
well,
conditions
being
activated
or
deactivated
all
right.
This
one
is
concurrency.
Yes,
so
typically
controller
runtime
has
a
single
girl,
routine
per
controller
per
type.
So
if
your
project
have
only
single
type,
you
will
have
single
routine
controller.
A
If
you
have
multiple
types,
all
of
them
all
controls
will
be
run,
separate,
go
routines,
control,
runtime,
supports
functionality
to
run
multiple,
go
routines
for
controller.
We
haven't
used
it,
but
I
think
you
have
to
provide
the
own
bookkeeping.
So
you
don't
want
the
same
instance
being
processed
by
multiple
controllers.
A
Also,
another
consideration
is
to
make
if
you
make
in
the
call
which
let's
say
your
internal
object,
and
you
have
to
make
external
call
which
takes
a
long
time
to
fulfill.
Typically,
you
don't
want
to
block
your
controller,
because
all
the
other
instances
of
the
same
type
will
be
queued
and
will
be
waiting
to
be
processed.
So,
let's
say
if
I'm
spinning
up
resource
which
takes
five
minutes
to
provision
I,
don't
want
to
sit
and
wait
for
response
to
be
ready,
because
other
resources
will
be
sitting
in
Hue.
A
A
You
can
spawn
up
separate,
go
routine
and
saying
okay
return,
reconcile,
know
and
go
ahead
and
do
due
process
in
the
waiting
in
different
go
routine.
However,
again
you
need
to
be
mindful
about
the
concurrency,
so
you
need
to
do
keep
and
make
sure
that
you
don't
process
the
same
instance
by
multiple
operators
have
multiple
controllers,
so
don't
block
don't
branch
and
take
everything
here
in
the
slide
and
in
this
presentation,
grain
of
salt.
This
is
kind
of
my
experience.
Your
mileage
may
vary,
so
it
may
have
different
needs.
A
Okay
status.
Yes,
you
may
have
need
for
additional
fields
in
your
status.
Besides
the
conditions,
typically,
application
creates
other
things,
like
you
say,
create
service
or
create
secrets
and
external
resources
as
well.
So
sometimes
it's
nice
to
propagate
those
states
back
and
especially,
if
you're,
using
additional
printer
columns
in
your
definition,
it's
nice,
for
you
object
user
types,
they
get
my
type
and
it
gets
additional
columns
which
attach
to
it
via
status.
A
So
this
is
an
example
of
one
of
the
crossplane
which
I
think
it's
kubernetes
cluster,
which
shows
the
end
point
of
cluster
cluster
name
and
some
other
things
so
that
those
things
you
can
put
those
stars
as
well
and
of
course
now
there
is
other
things
like
running
operations.
Some
cloud
providers
instead
of
provision,
resource
and
saying,
okay,
I've
got
it
it's
pending,
come
back
and
when
you
ask
and
give
me
resources,
okay,
it's
there,
but
it's
still
in
panic
state.
Some
cloud
providers
will
actually
say:
here's
the
ID
of
operation.
A
So,
instead
of
kind
of
and
when
you
ask
for
resource
again,
it
says
up:
it's
not
fun,
because
it's
not
being
fully
flushed
out
yet
so,
instead
of
kind
of
inferring
about
resource,
you
need
to
infer
now
about
the
status
of
operation.
So
you
may
want
to
save
with
that
operation
ID
into
your
status
in
object.
So
this
way,
next
time,
when
you
reconcile
and
see
oh
yeah
I
have
pending
operation
going
in
against
this
cloud
provider,
go
ahead
and
use
that
oppression
ID
infer.
What
is
the
status
right
now?
A
Is
it
finished
operating
or
not?
It's
a
little
bit
a
little
bit
clunky,
but
I
think
that's
the
only
way
to
kind
of
keep
track
of
those
process,
because,
ironically,
the
same
cloud
provider
will
treat,
create
and
update
as
a
single
call.
So
what
happens
that
you
say
create?
Is
okay,
I'm
creating
it
it
exists
for,
but
if
you
turn
around
say
create
again,
it
will
actually
issue
update
command
because
it
will
see
that
there
is
create
oppression
pending
and
which
ultimately
would
lead
to
failure.
A
Okay.
This
is
very
interesting
topic
and
I.
Think
we've
got
it
from
or
I've
got
it
wrong
several
times.
So
how
do
you
delete
things
in
kubernetes
when
you
delete
an
object?
Kubernetes
api
marks
it
for
dilution
by
setting
dilution
timestamp
in
metadata,
and
then
it
updates
the
object
which
makes
dirty
and
then
object
triggers
to
be
put
in
the
queue
and
that's
why
you
reconcile
get
cold
and
you
try
to
get
a
route
through
that
object,
and
so
what
happened?
A
Next,
it's
kind
of
up
to
you
how
you
implement
your
delete
logic,
if
you
need
any
know
that
once
object
is
deleted,
it
cannot
be
undelivered,
so
it's
kind
of
any
future
updates
to
delete.
The
object
will
be
ignored
by
kubernetes
api
and
if
your
controller
does
not
create
any
children
resources,
internal/external,
then
cleanup
is
rather
straightforward
upon
the
average
of
deletion.
A
Object
gets
automatically
garbage
collected
by
api
and
remove
from
the
api,
Urich
and
cellular
will
call
one
more
last
time
with
key
and
you
attempt
to
retrieve
an
object
and
get
all
four
object
not
found,
which
is
okay
in
this
case,
because
it's
already
been
garbage
collected
now
when
you,
when
you
controller,
create
some
additional
company's
resources:
okay,
for
example,
the
deployment
of
secret
or
external
resources,
then
other
a
few
option.
You
need
to
consider.
A
Typically,
almost
always
you
wanna
sign
object
reference
for
kubernetes
resources,
published
reference
and
the
children
resources
which
point
on
a
reference
which
point
to
your
parent
object
or
parent
controller.
Thus,
when
you
delete
your
parent
again,
you
don't
have
to
do
anything
special
and
your
children
object
will
be
automatically
garbage
collected
as
well,
so
it
will
delete
a
parent
garbage
collected
and
cascade,
delete.
All
children
objects.
A
If
you
want
to
perform
additional
cleanup
steps,
then
you
must
use
finalized
errs.
In
fact,
this
is
the
only
way
to
handle
deletion
and
external
resources.
That's
kind
of
effect
of
what
we
do
in
communica
and
crossplane
object.
Metadata
contains
the
definition
for
list
the
finalized
errs.
They
could
be
more
than
one
and
as
long
as
finalize
the
list
is
not
empty,
the
object
will
not
be
removed
from
the
API.
You
can
set
finalizes
in
different
way.
For
example,
when
you
create
your
type
will
say,
keep
steel
trade.
A
A
When
you
delete
your
object,
reconcile
it
gets
notified
and
it
retrieves
an
object
because
it's
not
removed
from
the
API,
it
will
examine
if
it
has
dilution
timestamp
and
if
it
says,
okay,
yes,
dilution
time
stem
is
set,
then
I
need
to
perform
some
deletion
steps
and
once
it
perform
the
steps
and
cleanup
is
successful.
Then
controller
can
remove
finalizer
and
again
it
goes
through
the
standard
garbage
collection.
If
there
is
no
more
finalizes
in
list,
it's
get
garbage
collected
and
move
from
AP
automatically
and
finally,
quick
word
about
the
foreground.
A
Dilution
policy
chances
are
that
you
will
never
need
to
use
it.
The
key
difference
between
foreground
and
background
policy
is
that,
with
the
background
policy,
when
parent
object
is
marked
for
deletion,
its
removed
and
garbage
collected,
and
then
children
kind
of
suffer
the
same
fate
in
foreground
policy,
parent
object
marked
for
deletion,
but
it's
not
removed
from
API.
Instead,
all
children
then
get
marked
for
deletion
and
garbage
collected
and
then
once
all
children,
a
gun,
then
parent
gets
garbage
collected
as
well.
A
So
the
good
use
case
for
that
may
be
kind
of
contrived,
but
what
we
experience
sometimes
is
that
if
your
children
object
depend
on
some
properties
of
your
parent
object,
for
example,
it
needs
some
connection
reference
or
something
like
that.
If
you
delete
parent
first,
your
children
cannot
retrieve
anymore
and
they
become
stranded
and
orphaned,
not
by
intent.
But
as
a
matter
of
fact,
so
you
may
use
this
for
grant
policy
and
another
what
about
for
grant
policy
that
it
cannot
be
activated
from
keep
CTL.
A
So
if
you
try
to
say,
keep
CTL
delete,
you
cannot
set
foreground
policy
in
order
to
achieve
that
and
again
there
is
a
issue
github
issue
against
that
I
linked
it.
In
my
reference
slide,
if
you
want
to
achieve
foreground
policies
through
kind
of
from
client
side,
you
need
to
use
curl
command
where
you
will
say
curl.
This
API
request
with
the
options,
deletion,
options.
A
Okay,
let's
a
couple
of
words
about
the
issues
we've
encountered,
while
working
with
controller
run
time,
control
run
time,
not
so
much
control
run
time,
but
rather
controller
tools
which
kind
of
comes
with
it,
which
enables
you
to
bootstrap
and
set
up
the
additional
inventory
tasks
like
generating
series
and
so
forth,
they're,
very
particular
about
the
project,
structure
and
I.
Think
maybe
it's
a
good
thing.
A
It
basically
expects
your
types
to
reside
in
specific
directory
structure
like
you're
gonna.
Have
you
ap?
Is
you
have
a
group
version
and
you
type?
If
you
deviate
from
that
layout,
then,
when
you
try
to
generate
series,
it
may
not
work
through
control
tools
we
kind
of
in
cross
plane.
When
we
initially
bootstrap
the
project,
we
decided
to
go
mono
repo
and
we
like
lump
a
lot
of
cloud
providers,
all
claw
providers
and
all
types
and
groups
into
one
project
which
may
be
in
hindsight
not
really
a
good
thing.
A
We
kind
of
still
in
defense
about
that.
But
what
it
lets
you
we
realize
that
we
need
to
subgroup
further
down,
but
then
once
we
did
that
and
try
to
use
control
tools
to
generate
series
of
this
series,
we've
stuck
in
an
errors,
and
we
end
up
to
actually
patch
in
control
tools
to
handle
the
specific
condition.
I
initially
create
the
issue
in
the
control
tools,
but
then
I
realized
that
maybe
it's
a
good
reason
why
that
structure
in
the
first
place
you
want
to
keep
things
simple
and
whatnot
better
example.
A
How
to
keep
things
simple
is
testing
when
you
use
cube
builder,
I'm
not
sure
if
you
build
a
property,
so
controller
a
runtime,
it
will
automatically
generate
tests
for
you.
It
kind
of
scaffolding
for
your
tests
for
both
type
and
the
controller,
so
the
way
type
just
set
up
it
will
basically
every
time
you
run
tests,
it
will
spin
up
new
cube
API
and
you
sed.
A
So
you
have
this
pristine
API,
to
which
you
can
register
your
type
and
perform
some
tests,
which
is
great
because
you
always
have
this
reproducible
test.
One
thing
it
you
worry
about.
If
you
have
a
very
busy
project
with
a
lot
of
groups
with
a
lot
of
types,
the
way
go
run
tests
it.
Basically
it's
per
package,
so
it
many
packages
can
run
in
pair
long
ago.
A
So
imagine
if
you
have
many
many
groups,
let's
say
10
and
all
of
them
are
gonna
run
parallel,
so
you're
gonna
have
10
sets
of
cube,
API
and
H
CDs,
which
can
be
taxed
on
your
system
before.
If
we
don't
put
any
clamp
on
it,
how
many
we
can
run
a
parallel
if
I
try
to
build
or
run
test
for
control
cross
plane
in
my
laptop,
my
laptop
will
freeze
for
like
5
minutes.
A
I
cannot
do
anything
now,
so
we
have
to
actually
clamp
down
and
make
file
to
say,
ok,
use
this
ratio
CPU
cores
per
how
many
parallel
go
tests
you
can
run
and
that's
I
think
we'll
be
good
testament.
Why
you
want
to
keep
your
project
simple,
don't
kind
of
try
to
attempt
to
lump
many
different
types.
I
mean
different
controllers.
Rather
than
have
the
specific
may
be
dedicated
things.
A
What
else?
Yes-
and
there
are
other
issues
this
one
I
want
to
highlight-
you
need
to
be
mindful
about
the
still
routes
through
controller
runtime
when
you
update
your
object,
it
also
increments
the
revision
version
on
your
object
metadata.
However,
when
you
read
that
object
again
right
away
from
the
controller
runtime,
you
may
get
cached
version
which
actually
have
a
stale
revision
in
which
may
lead
to
this
error.
You,
probably
some
of
you,
saw
this
error
when
you
try
to
let's
say,
keep
his
jail
dump
in
the
ammo
and
then
apply
at
some
point
later.
A
But
when
revision
been
changed,
it
will
reject
because
said:
hey,
you
have
a
stale
copy
of
the
object.
So
this
ongoing
issue
has
a
link
in
reference
slide,
so
you
can
check
it
out.
Also
so
take
away
from
this
presentation.
Try
to
keep
your
operators
simple
if
it
is
possible
start
with
controller
runtime,
you
build
the
operators,
decay
and,
if
it's
too
much
magic
for
you,
refactor
and
use
either
rook
or
sample
controller.
Try
not
to
prematurely
optimize.
Your
types,
specs
or
statuses
in
others
do
not
try
to
share
them
right
away.
A
A
As
I
mentioned
earlier,
I
started
with
kubernetes
about
four
years
ago
and
in
my
previous
life
I
had
the
companies
all
dead,
who
breeds
migration
effort
at
each
bill
so
and
it
culminated
and
run
the
Game
of
Thrones
Season
seven.
It
was
big
milestone
for
us
for
production
readiness
and
the
big
factor
of
success
back
then,
and
still
true.
A
Now
is
this
active
and
vibrant
communities
communities
so
I
encourage
all
of
you
to
become
part
of
it
and
with
that
I
have
this
reference
list,
so
you
can
actually,
when
you
download
the
presentation
check
it
and
I
will
open
the
floor
for
questions.
This
is
end
of
my
presentation,
so
I
want
to
thank
you
all
of
you
coming.
C
A
So
how
do
you
gain
all
that?
That's
that's
a
good
question,
I'm,
not
sure,
fully
prepared
to
answer
it,
but
I
think
if
I
put
it
in
context
of
the
controller
run
time,
you
will
always
get
the
notification
with
the
key
and
what
you
will
do
you
basically
retrieve
the
object
from
the
animate
API
and
you
examine
the
state
of
the
object
against
the
desired
July
state
of
the
object
against
the
state
of
your
system.
A
Then
you
perform
reconciliation
when
you
do
external
resource
management,
let's
say
I'm
the
Truman,
State
and
say:
okay
I
supposed
to
have
this
database
with
this
many
replicas.
What
is
my
state
right
now?
If
the
state
identical
then
I
can
continue
rocket
science
and
I'm
done
otherwise
I'll
perform
steps
to
update
it.
D
Hi
new
question
operators
are
new.
For
me,
I
was
wondering
if
they
are
used
cases
you
would
discourage
use
operators
for
I,
think
I'm
thinking,
for
example,
there
are
many
things
that
you
could
do
either
in
a
CIC
the
pipeline
or
an
operator.
I
was
wondering
if
there
is
a
pattern
to
help
decide
when
some
functionality
should
be
implemented
in
a
place
or
another.
It's.
A
Hard
to
answer
this
question
without
known
specific
details
of
the
CIC
deep
steps
like
that,
I
would
say
that
you
can
use
a
period
of
almost
for
anything
effectively
whatever
its
kind
of
specific
logic
for
your
type.
You
can
manage
anything
outside
of
Cabrini's
to
good
use.
Case
example
we're
managing
provision
in
external
resources
in
a
public
cloud,
so
you
can
manage
it
to
do
that
and
you
can
manage
external
source
in
other
companies
clusters.
You
can
manage
things
in
your
notes,
so
it
depend
what
you
need.
A
B
A
A
That's
when
I
decided
to
go
with
the
cube
builder
first
now
since
then
they
changed
and
they
use
in
controller
runtime,
I,
didn't
eat
and
I
checked
briefly
about
two
weeks
ago
and
a
generate
new
project
and
I
could
have
briefed
to
look
at
it.
The
scaffolding
looks
very
similar,
so
it's
almost
I
would
say
you
can
use
either
or
what
are
we
more
comfortable
with
I
like
personally
cube,
build
up
because
I'm
more
familiar
with
it
for
once
so
this
I'm
been
using,
but
I
also
like
the
ability
to
control
tools.
A
They
provide
to
generate
this
irides
and
regenerate
the
schemas,
and
everything
like
that.
So
it's
really
nice.
It's
also
have
this
nice
comment
based
annotation,
where
you
can
say:
hey
Q
builder,
generate
this.
When
they
have
slide
with
custom
columns,
you
can
actually
don't
have
to
do
it
yourself.
You
can
put
these
comments,
you
build
the
comments
and
it
will
happen
automatically
so
there's
a
lot
of
nice
is
I'm
sure,
probably
a
critique
a
has
also
similar
things.
I
just
don't
know
them
so
I
can
you
can
use
either
or
you
equals
successful.
A
E
E
Question
is
one
problem
I
often
ran
into
while
writing
tests
for
a
controller
was
waiting
for
reconciliation.
It's
like
basically,
two
approaches
worked
like
waiting
for
some
time
till
there
are
no
more
reconciliation,
requests,
yeah
or
just
turning
off
caching
with
which
helped
in
some
cases,
but
the
second
approach,
which
maybe
is
not
very
correct,
and
the
first
one
makes
this
like
forever
to
run
an
advice
system.
Yes,.
A
So,
as
I
mentioned
earlier,
that
we
used
initially
the
default
test,
harness
when
it
straps
the
control
test
and
it
started
controller
to
start
API
and
HCD
and
then
basically
submit
object
a
and
your
controller
actually
running.
So
you
kind
of
run
the
concurrent
test.
You
have
a
controller
running
and
you
have
your
test
running
and
you
try
to
kind
of
synchronize
and
see
okay.
A
What's
no
conciliation
step
right
now,
it
was
a
little
bit
difficult
to
time
response,
especially
because
if
you
mutate
object,
state
automatically
get
reviewed,
so
it's
gonna
hurt
to
a
certain
result.
So
what
we
end
up
doing
we
completely
stop
doing
that
and
instead
we
just
use
basic
unit
tests
for
reconcile
function
where
you
can
actually
say
call
it
once
go
through
the
function,
flow
run
and
then
basically
examine
the
output,
and
you
can
repeat
that
with
the
different
input
parameters,
because
remember,
the
reconcile
function
makes
the
decision
based
on
the
object
state.