►
From YouTube: Managing Addons with Operators (Or How We Dropped Untested... - Jeff Johnson & Justin Santa Barbara
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Managing Addons with Operators (Or How We Dropped Untested bash/sed for Go) - Jeff Johnson & Justin Santa Barbara, Google
Kubernetes has long deployed core components like kube-dns by running kubectl apply every few minutes from a directory on a master node. This worked great with simple addons on a single master node. Addons are a rapidly growing avenue for delivering new features and this growth has demonstrated the shortcomings of the current approach (k/k#23233, eg: configuration is hard and opaque; changes require a cluster restart; addons must be developed in the main repo) Jeff and Justin will talk about their work in replacing thousands of lines of bash, sed and replacing it with operators. They’ll discuss managing dependencies between addons, rolling upgrades, configuration, and health. They’ll cover a few addons that have already been converted and open-sourced, and demonstrate building an operator with their open source toolkit which makes it easy to build a best-practice operator.
To learn more: https://sched.co/GrY1
A
All
right,
thank
you.
Everyone
I,
want
to
thank
everyone
for
sticking
around.
This
will
be
the
last
talk
you
see
at
cube
con.
So
it's
a
it's
fantastic
that
you're
here
and
you've
chosen
us
just
a
reminder.
Please
grab
your
chairs
on
the
way
out
and
last
one
to
turn
off
the
lights.
We
appreciate
that.
A
B
A
So
we've
got
a
pretty
typical
conference
agenda
here
and
we're
gonna
try
to
keep
it
demo
heavy
and
keep
things
exciting.
But
before
we
do,
we've
got
to
talk
a
little
bit
about
the
background.
So
what
is
an
add-on
and
to
think
about
add-ons
and
sort
of
how
we've
got
to
the
point
where
we
are
I
think
it's
good
to
look
at
what
are
add-ons
more
chronologically.
A
So
if
you
look
at
a
lot
of
the
early
add-ons
things
like
cube
proxy
cube
DNS,
these
are
very
simple
deployments,
something
that
every
cluster
has
and
it
made
all
have
a
sense
to
bake
them
directly
into
the
kubernetes
release.
So
if
you
go
to
cake
a
cluster
add-ons
you'll
see
a
bunch
of
gamal
for
these
over
time
as
kubernetes
got
more
complex
as
we
grew
a
bigger
ecosystem,
so
did
the
add-ons.
Now
we
have
things
like
CNI,
CSI
and
ingress
controllers,
those
are
sure
adding
cluster
functionality
but
they're
very
different.
A
You
you're
picking
a
different
CNI
depending
on
your
preference,
you're
picking
a
different
look
ingress
controller
based
on
where
you're
put
in
your
cluster
and
with
them.
You
also
have
a
lot
of
different
configuration
for
each
one
of
those,
and
if
we
go
a
year
or
less
I
mean
these
things
are
getting
quite
a
bit
weirder.
We
now
have
things
like
Ischia
sto
has
a
very
complex
addon,
for
those
of
you
are
familiar,
there's
a
lot
to
orchestrate
there
now
with
an
atom.
You
have
this
expectation
that
it
is
managed
by
the
cluster.
A
A
It
has
to
do
the
sort
of
crud
operations
that
you'd
have
to
do
with
any
kubernetes
deployment
and
to
be
really
nice
if
it
gave
us
a
stance
of
health
and
availability.
If
I'm
deploying
a
cluster
with
cubed
ENS
I'd
like
to
know
that
that
cube
dns
is
healthy,
if
it's
not
healthy,
my
cluster
is
not
really
ready
to
be
used.
A
So,
let's
talk
about
how
it's
done
today
we're
going
to
talk
about
Cuba,
but
this
applies
pretty
well
to
mini
cube
as
well.
So
today
we
package
all
these
add-ons
and
the
kubernetes
release.
So
if
you
look
at
these
green
boxes
here-
and
that's
like
in
the
kubernetes
tarball
on
the
startup
of
our
of
our
main
machine
of
our
master
vm
we're
going
to
extract
some
of
those
gamal
out
of
there
and
we
run
them
through
this
bash
script.
A
A
A
Only
these
are
environment
variables
that
over
time,
over
new
releases
of
new
plugins
were
adding
more
and
more
to
the
core
kubernetes
release,
and
it's
the
problem
with
this
sort
of
approach
is
that
we
really
have
no
clue
what's
going
to
come
out
the
other
end
until
it
does
so
until
we
create
a
cluster
until
it
picks
up
all
the
environment
values
until
it
decides
which
he
ambles
it
wants.
This
one
has
a
switch
based
on
some
flag.
We
don't
know,
what's
going
to
come
out
the
other
end.
A
This
is
our
cue
baton,
manager.
Well,
I
would
say
it's
some
of
the
best
bash
I've
ever
written
written
there,
sorry
red
redden,
some
of
the
best
bash
I've
ever
read.
It
is
very
hard
to
test.
We
really
don't
know
until
we
have
an
integration
test,
run
that
something
failed
and
just
really
known
ready
to
know
what
failed
until
you
dig
into
it.
A
A
We
have
that
loop
that
loop
I
mentioned
where
Cube
control
runs
every
minute,
or
so
that
is
to
prevent
user
modification
and
to
steal
a
line
from
Justin.
Here
it
runs
every
60
seconds,
which
is
just
enough
time
for
you
to
fix,
cube,
DNS
and
tell
your
boss
and
then
promptly
be
reverted.
When
you
go
to
show
them,
you
fixed
it.
A
A
Clearly,
Gamble
is
not
enough.
We
have
add-ons
that
need
more
we're
doing
templating
that
is
getting
increasingly
more
complex.
We
really
want
to
know
about
health
and
status,
and
we
start
to
have
scenarios
we're
ordering
becomes
important,
so
I
mentioned
K
native
on
their
Candida
won't
work
without
sto
ready
to
go.
A
It's
pretty
hard
to
express
that
in
a
loop
that
just
applies
gamal
every
60
seconds.
The
message
and
sort
of
why
this
is
coming
out
of
cluster
lifecycle
is
that
our
cluster
creation
logic
really
ought
to
stop
caring
about
these
things.
You
really
ought
to
simplify
that
and
enable
the
people
who
are
the
teams,
the
the
folks
are
really
concerned
with
how
those
things
deploy
and
how
those
things
upgrade
to
own
that
experience.
So
why
are
we
taking
on
this
work
now,
a
little
plug?
A
We're
working
on
I've
been
working
on
GK
on-prem
and
it's
a
brand
new
project.
It's
a
new
project
where
we're
concerned
about
things
running
in
customer
data
centers.
We
need
increasingly
a
larger
amount
of
visibility
into
the
configuration
and
the
health
and
status
of
those
clusters,
but
it's
also
a
great
time
to
just
really
eight
what
we've
been
doing
for
a
while
and
start
to
set
new
standards.
A
A
A
A
So
here
I've
got
that
instance.
It's
a
normal
kubernetes
api
like
I've,
come
to
be
comfortable
and
familiar
with.
It's
got
that
spec
that
I
specified
here
where
I
said
I
want
the
latest
stable,
but
now
it
has
a
status
associated
with
it
and
that
status
says
that
this
deployment
is
not
healthy,
healthy,
false
and
it
gives
me
a
human
readable
reason
why
it
says
that
SEO
system
is
tío,
sidecar
injectors
not
found.
So
we
can
take
a
look
at
that.
Namespace.
A
A
A
Let's
take
a
look
at
that
key
natives
object
again,
so
it's
still
unhealthy,
but
now
we
have
a
different
set
of
errors
so
now
we're
seeing
that
the
Canadian
built
component
is
not
healthy.
Now
we're
seeing
these
individual
pieces,
if
you
aren't
familiar
Canada,
have
has
several
components:
a
build
of
surveying
monitoring,
several
things
involved.
So
if
we
can
race,
maybe
we
can
see
how
that
object
is
doing.
A
A
A
Yeah,
a
lot
has
happened
so
great
day.
One
operation
is
flashy
and
it's
great
for
a
demo,
but
it's
really
something
that
you
could
just
put
on
your
github
and
you
could
say:
you've
gotta
apply
this
URL,
you
know,
will
curl
bash
any
script:
I,
don't
I,
don't
look
at
them,
but
what
about
day
two
and
day
two?
We
expect
things
to
fail
and
we
expect
things
new
features
to
come
along
and
want
to
upgrade.
A
So
I
expect
to
see
some
activity
here,
an
update
like
this.
It
was
just
a
pod
bump,
so
we
just
updated
some
pod
specs,
that's
something
that
will
let
kubernetes
handle
we're
just
gonna
edit
the
deployment
object
and
let
it
reconcile
and
roll
out
that
update,
but
because
we
have
a
controller
involved
in
this
because
we're
talking
about
operators,
we
have
the
ability
to
add
more
complex
updates.
So
let's
say
that
was
a
data
migration.
We
now
have
the
opportunity
to
assure
that
data
migration
and
make
that
happen.
A
A
So
we're
here
to
talk
to
you
about
is
exactly
that:
add-on
operators,
if
kubernetes
add-ons
are
just
normal
workloads.
Let's
use
modern
application
lifecycle
tools
to
manage
them.
In
practice.
This
means
a
CR
D
for
every
add-on.
We
have
an
instance
of
that
c,
rd
a
CR
to
activate
it,
and
each
add-on
has
its
own
controller
to
manage
it,
and
we
I'll
tell
you
that
that
is
easier
than
it
may
sound.
A
Let's
walk
through
what
happened
in
that
demo.
So
let's
say
we
have
a
cube
DNS
CRD
that
CR
D
is
typed.
It
has
a
spec
and
validation
associated
with
it.
I
want
to
install
that
add-on
and
I
can
either
specify
potentially
a
version
here
or
a
source,
and
maybe
there's
configuration
there
not
sure
I'm
gonna
have
this
long-running
add-ons
controller,
so
that
red
box
is
the
cube
DNS
controller
and
that
databases
our
kubernetes
api
and
me
the
smiley
face.
I
just
applied
that
CR
and
I
got
a
cube,
DNS
CR
in
the
center.
A
Our
controller
is
watching
for
that
cube,
DN
SCR.
When
it
sees
that
created,
it's
gonna
fetch
the
manifest
this
could
be
from
an
HTTP
endpoint.
It
could
be
from
another
kubernetes
api
object,
or
perhaps
it's
just
a
file
on
disk
I
can
then
specialize
that
manifest
and
then
deploy
it
to
kubernetes.
So
this
deployment
created
a
cube,
dns
deployment,
a
service
I'll
also
set
up
owner
refs
on
that
CR.
A
As
it's
doing
that,
setting
up
watches
for
all
these
objects
and
the
important
part
about
that,
and
the
reason
why
we
do
that
is
so
we
can
have
these
health
and
steadily
gonna
have
status
for
the
deployment.
So
we
can
watch
for
the
successfulness
of
those
objects.
We
can
watch
each
ever
one
of
those
and
create
an
aggregate
health.
My
goal
here
is
just
have
a
boolean.
A
Kubernetes
does
its
thing
over
time.
It
creates
the
pod
associated
with
that
deployment,
and
we
see
our
status
got
it
true
when
our
dns
starts,
crashed,
looping,
there's
a
number
of
reasons
why
things
like
this
can
happen.
Maybe
we
have
a
bug,
maybe
have
a
configuration
problem.
That
controller
is
continuing
to
watch
that
it's
continuing
to
see
alerts
for
that
specific
object.
So
we
can
then
surface
that
things
have
gone
awry.
Things
are
not
healthy.
A
A
B
B
I
think
that
behavior
is
is
great
and
I
think
we
can
all
agree
that
we
would
like
behavior
of
that
level
from
our
add-ons
I
think
the
challenge
is
obviously
alright.
How
hard
is
it
to
actually
create
those
those
operators
like?
Is
this
something
that
that
is
actually
possible
for
for
people
that
are
writing
components
to
actually
do
or
do
you
have
to
be
a
criminal's
expert
to
do
that?
B
If
we
want
to
grow
our
community,
we
have
to
ensure
that
it
is
everything
that
we
do
is
easy
and
we
have
actually
the
beginnings
of
an
SDK
in
kubernetes
to
do
that,
which
is
called
code
builder,
and
what
we've
actually
done
is
is
build
a
small
layer
on
top
of
qu
builder.
It's
we
call
it
a
common
set
of
ad
extensions
and
an
opinionated
controller
on
top
of
this,
and
there
are
sort
of
two
layers
up
to
that.
B
There's
there's
sort
of
a
declarative
layer
and
then
there's
an
additional
ad
on
there
on
top
and
we're
using
this
thing
called
pattern.
Soku
builders
is
this
framework
that
gives
you
where
it's
a
program
that
immediately
gives
you
a
bare-bones
controller,
and
you
see
we've
added
the
this
flag
pattern,
add-on
to
the
queue
builder
commands.
And
now,
when
you
pass
pattern
item,
you
get
a
little
bit
less
of
a
bare
bone
controller
and
we
will
show
you
how
you
grow,
that
into
a
full
add-on
operator
from
that.
B
So
here
we
go.
Let's
try
this
second
demo,
giving
a
demo
heavy
all
right,
that's
nice!
This
is
a
mean
cluster
which
I
should
have
just
cleaned
up
and
we
are
going
to
I
am
not
as
good
as
typing
as
at
typing,
as
Jeff
is
so
I'm
going
to
use
a
Auto
typer,
but
rest
assured
this
is
a
real
cluster.
The
last
couple
of
commands
will
be
real
just
for
you,
so
we
are
in
no
problem
detector.
We
are
going
to
build
an
operator
for
no
problems.
Detector,
no
prom
director
is
a
component.
B
It's
a
daemon
set
that
runs
on
every
node
and
looks
for
common
errors
in
syslog
or
other
things
that
have
gone
wrong
going
to
cater
your
node
might
not
be
healthy,
so
we
were
in
building
a
go
controller,
no
path.
We've
checked
out,
no
problems.
Detector,
remember
the
the
idea
here.
Is
that
really?
This
is
something
that
no
problem
detector
can
do
themselves.
So
they're
gonna
create
an
operator
in
their
in
their
repo.
They
call
who
builder
in
it.
They
remember
to
pass
the
magic
pattern
flag.
B
They
are
going
to
create
an
API
again
with
a
magic
pattern
flag.
You
can
see
they're
they're
specifying
a
kind
of
a
no
problem
detector.
We
get
an
error
which
just
demonstrates
that
it's
real
that's
because
I
skipped
DEP
for
a
reason
that
will
become
very
clear
in
a
second
and
but
we
have.
We
have
printer
skeleton,
API
kind
of
kind,
no
problem,
detector
and
we'll
instantiate
that
in
a
minute
we
want
to
manage
no
problem
detector
in
a
declarative
way.
B
Everyone
should
be
very
comfortable
with
doing
that
by
oh
manifests,
and
so
we
are
going
to
build
up
that
manifest
right.
Now.
We
have
this
channels
directory,
which
is
where
you
define
the
channels
that
you
know
define
your
stable
version
and
your
alpha
version.
You
could
invent
other
ones
beta
and
gamma
and
whatever
other
versions
you
want,
you
can
see
here
we
have
just
one
one
channel
which
is
stable
and
stable.
Currently,
the
skeleton
creates
version
0,
0
1.
B
B
We
will
so
it
is
it
now
0,
5,
0
and
now
we're
actually
have
to
go
and
create
that
manifest
or
define
what
0
5
0
really
represents.
So
there
0
1,
0,
0,
0,
1,
we'll
rename
it
and
we'll
go
in,
and
we
will
have
a
look
at
the
skeleton
manifest
and
it
is
a
empty
manifest.
We
are
the
know
where
I'm
wearing
my
hat
as
a
node
problem.
Detector
developer,
so
I
know
exactly
where
those
manifests
live
and
I'm
going
to
copy
them
in
here.
B
This
is
exactly
I
didn't
want
to
take
this.
There
we
go
and
we
now
have
a
manifest
gamal
which,
as
our
it's
actually
a
daemon
set
in
the
config
map,
it's
fairly
simple.
We
actually
want
to
remove
the
namespace
declaration
that
the
add-on
framework
basically
takes
care
of
namespaces
for
us,
so
we
don't
need
that
and
let's
so
that
was
that
wasn't
too
bad
I
hope
people
agree
with
that.
So
are
we
ready?
Basically,
yes,
we
are.
B
We
have
to
run
def
now
so
now,
I
have
them
a
couple
of
minutes
to
ad
lib
while
def
runs
so
the
I
think
the
idea
is,
we've
already
created
our
operator
now
and
I
think
it's
important
to
realize
that
whether
you
created
in
a
separate
repo
or
you
create
in
the
subdirectory
like
that
was
not
hard.
What
we've
achieved
so
far.
So
if
you,
if
you're,
currently
defining
an
add-on
via
manifest,
it
is
not
very
hard
to
create
an
operator
and
I
would
encourage
your
one
to
go.
Give
it
a
go.
B
If
we
have
a
couple
of
PRS
out
there
that
are,
hopefully
gonna
merge
soon
and
then
it
will
be
that
easy
to
create
your
operator.
One
need
one
of
the
things
we
really
strived
for
when
we
were
building.
This
was
to
make
sure
that
we
also
could
evolve
the
functionality
so
we're
gonna
look
at
the
code
in
a
second
that's
mental,
we're
gonna,
look
at
the
code
in
a
second
and
you'll,
see
that
we've
used
encapsulation
and
things
like
that,
rather
than
horrendous
code
generation.
B
B
B
B
And
we
will
now
want
to
run
that
operator
again.
The
operator
will
now
see
that
see
our
instance
and
take
the
manifest
that
we
defined
or
lookup
the
stable
version
go,
find
zero
five
zero,
take
the
manifest
and
coupet
apply
it,
and
there
we
are.
It
is
the
last
where
at
the
bottom
there
you
can
see
it
ran
a
coop
cutter
apply
with
two
objects
in
it,
which
are
our
diamond
set
in
config
map.
B
So
if
we
have
a
look,
we
should
see
our
diamond
set,
no
problem
detector
created
13
seconds
ago,
so
I
did
remembers
that
RIM
I
did
remember
to
delete
that
one
before
the
demo
and
our
config
map,
and
we
should
have
three
pods
there.
They
are
running
twenty
nine
seconds,
so
we
just
created
our
first
at
an
operator
live
in
a
couple
of
minutes.
Okay,.
B
I
went
to
I
want
to
just
quickly
talk
about
some
of
the
code
points,
so
this
is
our
use
of
encapsulation,
rather
than
like,
generating
lots
and
lots
of
code.
So
you
see
here
I
think
we
talked
about
the
idea
of
common
spec
fields
and
common
status
fields.
Those
common
spec
fields
include,
for
example,
the
channel
and
the
version.
The
common
status
includes
the
health
information
and
the
errors,
and
you
can
see
we
we
cleverly
named
them
common
spec
and
common
status,
and
the
nice
thing
there
is,
of
course,
is
that
evolves.
B
We
can
continue
to
to
enhance
that
behavior
or
sorry
add
more.
We
can
add
more
fields
into
common
spec
and
come
and
status,
and
you
can
just
recompile
them
and
pick
those
up
and
then
the
controller
itself.
We
basically
reconcile
no
problem.
Detector
struct
is
the
struct
that
cube
builder
builds
for
you
and
we've
added
this
declarative,
reconciler
type,
which
we
have
been
encapsulated,
I,
guess
we're
embedded
and
that
basically
defines
most
of
the
behavior.
So
we
just
call
init
and
you
get
basically
standard
behavior
for
a
manifest
driven
reconcile
there.
B
If
you
want
to
opt-in
to
health
and
status,
checking
I
think
we
currently
require
you
to
actively
opt
into
that.
So
you
have
a
couple
more
arguments
that
in
it
thing,
but
that's
essentially
how
easy
it
is,
it's
all
very,
very
much
automatic
behavior,
and
this
isn't.
This
is
something
we
think
everyone
every
pro
every
project
should
be
able
to
do
this.
B
So
how
can
you
help
and
what
are
we
so
got
to
do?
So?
What
are
the
big
things
on
the
roadmap?
Well,
one
of
the
things
we
didn't
talk
about
is
as
I'm
sure
you
know,
particularly
in
OSS.
We've
learned
that
every
field
in
every
manifest
for
every
field
in
every
manifest
there's
someone
that
wants
to
change
it.
You
know,
like
certainly
in
my
cops
experience
and
anyone's
ever
work
with
helm,
knows
that
every
field
becomes
a
parameter
and
we
find
that
that
is
a
difficult
thing
to
do.
B
We
don't
necessary
want
to
expose
everything
in
this
in
this
CRD.
We
want
to
keep
it
nice
and
simple
with
the
core
functionality.
That's
why
I
think
Jeff
was
talking
about
it's,
not
clear
what
is
required
and
what
we
can
punt
off
and
the
we
have
this
magic
tool
now
in
the
Kim
in
the
open-source
community
called
customize,
which
basically
enables
everyone,
it's
a
nice
trade-off,
you
can,
you
can
specify
a
patch,
we
can
support
those
patches
and
we
don't
necessarily
have
to
map
every
single
field,
so
it
becomes
maintainable
to
create
these
add-on
operators.
B
One
of
things
I
didn't
show
is
the
are
back
generation.
The
are
back.
Rules
are
a
little
bit
tricky
for
operators,
because
when
you,
if
you
want
to
create
an
are
back
permission,
you
need
to
have
green
to
bind
it.
You
want
to
bind
an
Arabic
emission,
you
need
to
have
all
the
permissions
that
you
are
binding
and
so
the
permission
sort
of
roll
up
and
it
becomes
a
little
bit
tedious.
We
definitely
to
make
that
better.
B
One
of
the
ways
we
might
be
able
to
do
that
is
by
bypassing
our
back
for
for
these
operators.
It's
not
sure
that
we
want
to
do
that,
but
that's
a
discussion,
we're
gonna,
be
having
in
this
in
the
Sigyn.
In
the
community,
so
your
input
there
is
very
welcome.
We
have
PRS
open
currently
to
upstream
all
of
this
code,
there's
a
portion
to
coup
builder
that
pattern
equals
add-on
flag,
there's
a
portion
into
controller
runtime,
the
implementation
of
the
declarative,
reconciler
and
the
opinionated
add-on
logic.
B
On
top
of
it,
I
don't
know
if
you
notice,
but
we
edited
files
locally
there
they
were
in
the
Channel's
directory.
It
would
be
great
and
the
the
if
currently
is
that
we
we
bake
them
into
the
docker
image.
We
really
don't
want
to
have
to
build
a
new
operator
image.
Every
time
you
release
a
new
version
of
the
package,
you're
you're,
building
or
deploying,
and
so
we,
when
we
need
to
get
those
manifest
sorts
through
either
HTTP
or
for
offline
or
air-gap
to
access
the
kubernetes
api.
B
There's
a
Google
project
by
Josh
Hoch
called
the
the
bundle
which
looks
very
promising,
but
another
thing
we're
gonna
be
discussing
in
the
sig
I
hope
Jeff
mentioned
the
I
emailed
my
boss,
and
now
it
reverted
we.
It
would
be
better
to
have
a
web
hook
that
actually
intercepted
any
changes
and
prevented
you
from
gaining
that
full
sense
of
euphoria
that
you
would
actually
solve
the
problem.
They
told
you
you,
you
haven't
solved
a
problem.
You
should
use
customize
I'm,
so
working
on
that
we
actually
need
to
then
get
this
integrated
into
the
tooling.
B
Oss
is
obviously
a
good
one,
and
particularly
the
cube
up.
Scripts
has
been
deprecated
for
I
think
a
number
of
years
now,
and
it
would
be
great
to
finally
get
rid
of
the
add-on
manager
out
of
Cuba
or
hopefully,
maybe
even
get
operative
cube
up
entirely,
but
like
integration
into
cops
into
cube
ATM
into
the
cluster
API
into
mini
cube
into
all
the
projects
that
are
out
there
and
we've
tagged
this
as
Help
Wanted.
If
this
is
something
you
want
to
get
involved
in,
please
please
do
and
you'll
be
very
welcome.
Sync.
B
We're
trying
to
see
what
are
the
use
cases
that
we
see
the
various
operators
in
the
real
world.
What
do
they
all
need
where
they
have
in
common
and
when
we
find
one
of
those
patterns,
we
promote
it
into
the
add-on
pattern
and
we've
covered
our
roadmap.
So
I
think
with
that
we
are.
If
there's
any
questions
would
be
great
to
take
them.
A
Lot
of
this
that
we're
building
is
all
in
the
controller
runtime,
and
so
we
think
that
this
is
in
the
whole
sort
of
operator
ecosystem
to
take
advantage
of
so
I.
Think
that
there's
a
lot
of
opportunity
for
for
sharing
their
one,
since
we're
sort
of
based
off
a
lot
of
the
same
tooling,
now
so
figuring
out.
What
is
the
use
case
for
each
of
them
is
something
we
should
do
sort
of
out
in
the
community.
I
think
and.
B
B
A
D
B
A
Cute
builder,
today
is
fantastic
at
creating
api's
and
creating
frameworks
for
you
to
build
controllers
and
the
the
pattern
add-on
gives
you
a
bit
more
opinionated
controller
to
get
started
with.
So
it
allows
you
to
get
a
much
faster
path
and
gives
you
some
of
the
features
that
we
think
core
cluster
add-ons
ought
to
all
share.
A
I
do
agree
that
there's
a
lot
of
overlap
and
that's
why
we
started
and
have
received
the
feedback
to
put
a
lot
of
these
into
this
declarative
sort
of
package
instead
of
just
add-on
so
separating
what
is
purely
an
add-on
concept
and
what
is
just
a
normal
workload
concept
and
so
I
think
there's
a
lot
of
opportunity
in
the
future
for
that
to
extend
the
controller
experience
but,
like
Justin,
said
we're
very
use
case
driven.
So
we
want
to
focus
on
this
very
specific
need
and
see
what
we
can
share.
You
know
long
term.
A
So
an
example
of
what
is
different
and
when
you
just
do
create
API,
you
get
a
function
that
says
reconcile
and
in
that
function
you
get
the
name
of
your
CR
object
and
that's
it.
So
when
you
pass,
when
you
specify
add
on
you're
going
to
get
this
template
of
code,
that
will
implement
reconcile
for
you
in
this
way
that
works
based
off
of
a
manifest.
B
Work,
you
can
use
that
separately
from
the
add-on
pattern.
So
if
you
have
some
other
operator
that
you
can
best
define
it,
how
you
reconcile
by
means
of
a
coop
cuddle,
apply
from
a
manifest
or
something
or
some
of
some
other
means
of
sourcing
objects.
You
should
absolutely
consider
using
that
layer.
My
suspicion
is
that's
why
they
had
to
split
it.
E
B
B
The
the
I
think
the
rule
that
we
propose
is
that
the
operators
themselves
must
be
coop
cuddle-up
liable
and
there
will
be
some,
in
other
words,
should
not
themselves
require
complex
logic
of
any
sort
and
that
and
that
there
will
be
some
piece
that
will
be
sort
of
part
of
your
tool.
That
is
able
to
do
that
is
able
to
install
that
root.
Bootstrap
turtle,
as
it
were,.
A
A
That
that
atom,
that
cigs
is
just
a
namespace.
So
as
part
of
working
on
this,
we
started
thinking
about
well,
do
we
just
want
an
add-on
type?
Do
you
want
a
pure
add-on,
C
or
D?
You
know
type
add-on,
name,
cube,
DNS
type,
add-on
name,
you
know
native
and
instead
of
we
thought
about
going
that
route,
but
it
feels
like
that
the
risk
is
sort
of
ending
up
with
all
these
configuration
configuration
options
that
are
unrelated,
so
each
add-on
has
its
own
unique
CRD
and
there
isn't
like
a
tie
back
to
a
central
add-on
CRD.
A
B
B
Yes,
so
that
there
are,
we
have
to
install
both
actually
an
instance
of
the
add-on
CRD
and
we
have
to
install
the
CRT
itself
and
we
have
to
install
typically
a
deployment
or
a
stateful
set
for
the
operator.
Those
three
things
we,
we
sort
of
by
rule
of
we
rule
that
they
must
be
easy
to
install
that
they
must
be
coop,
cuddle-up
liable.
We
say
that
you,
you
shouldn't,
build
an
operator
that
requires
complex
orchestration,
so
once
we've
said
that
we
hunt
the
problem
to
your
to
your
installation
tool
right,
so
we
think
worst
case.
B
B
Yes,
I
think
so
for
people
didn't
hear.
Alex
was
saying
it's
also
about
delegation
of
responsibility
and
it's
much
easier
to
install
those
three
components.
It's
much
easier
for
the
sig
cluster
lifecycle
or
the
add-on
tooling,
to
install
those
three
easily
managed
components
than
it
is
within
the
current
situation.
Where
effectively,
the
the
tooling
is
responsible
for
managing
the
one
hundred
and
fifty
odd
objects
that
make
up
in
this.
Do
Lucas
I
think
you
were
next
and
then.
A
A
So
we
do
expect
you
to
once
in
a
while
need
to
bump
the
cube
DNS
operator,
but
we
don't
expect
you
to
have
to
bump
it
very
often
really
just
when
the
semantics
of
cube
dns
change,
so
we
always
will
have
some
sort
of
orchestration
tool
that
is
bumping
the
API
server.
That's
pumping
the
cubelets
and
we'll
probably
do
that
at
the
same
time
as
the
operators.
A
So
we
expect
to
be
able
to
cube
cuddle
apply
all
this
stuff,
so
we
expect
to
have
sort
of
an
opaque
blob
of
really
simple,
really
simple
deployment
that
our
cluster
tooling
can
say.
Well,
our
upgrade
story
is
the
cube.
Dns
team
is
supplied
us
with
a
new
version
of
their
operator,
so
we
just
apply
it
and
that's
it
probably.
A
So
in
that,
in
the
thing
we
expect
the
cluster
tooling
to
apply,
we
expect
it
to
apply
the
deployment
of
the
operator,
but
for
those
really
core
add-ons
like
QD
NS,
we
would
expect
it
to
also
apply
an
instance
of
the
CR
so
that
CR
will
carry
with
it
the
version
that
we
expect
it
to
be
at
and
that
version
may
be
stable
and
that's
stable,
very
well
may
resolve
to
say.
Okay
I
see
that
my
kubernetes
is
at
version.
A
111
I'll
ask
the
you
know
the
the
Debian
app
you
know,
equivalent
of
the
kubernetes
manifest
service
or
whatever.
It
is
to
give
me
that
version,
I,
think
that'll,
probably
be
encoded
in
in
different
tooling,
and
we're
looking
I
think
it
bundles
to
solve
some
of
that
in
terms
of
what's
compatible
with
what
but
I
think,
there's
a
lot
of
open
questions
there
as
well,
a
lot
of
great
stuff
to
explore.
Yeah.
B
I
think
but
I
think
also
one
of
the
nice
things
is
because
we've
used
go
code.
We
don't
have
to
invent
some
language
to
get
started.
We
don't
have
to
define
a
DSL
to
sort
of
start,
the
ball,
rolling
and
sort
of
that's
how
we've
been
developing
this
so
far
like
we've
been
encouraging
teams
to
build
their
operators
and
seeing
what
problems
arise
and
what
use
cases
we
can
identify
in
common
across
them
and
I
hope.
We
can
do
that
in
the
sake.
D
B
B
The
yes
there
aren't
that
many
tools
left
that
use
it
to
be
honest,
but
yes,
the
this
is
the
the
intention
is
to
replace
the
the
both
that
particular
script,
but
also
to
produce
something
that
meets
the
needs
that
is
sufficiently
advanced
to
meet
the
needs
of
everyone,
all
the
two
members
out
there.
So
we
don't
have
the
situation
we
have
today
where
201
uses
this
approach
until
2
uses
another
approach.
B
No
more
it's
nowhere
at
Seattle's,
government,
I,
think
I
think.
Yes,
we,
that
is
definitely
a
Etsy
errand
scriveners
is
a
particular
problem.
The
Nets
yeah,
its
particular
them
for
a
che
clusters.
So,
yes,
we
definitely
like.
We
saw
the
the
idea
of
plumbing
from
HTTP
or
from
something
inside
the
cluster
instead,
but
yeah.
That's
it
that's
another
good
motivating
reason,
because
essentially,
a
file
on
the
local
disk
is
not
very
amenable
to
multi
master
operation.
Oh
nice.
A
That's
a
great
question
and
I've
seen
it
so
internally.
We
have
teams
working
on
this
and
you
and
have
been
using
this
for
a
while
and
and
we've
we've
tried
a
couple
things
out
and
it's
not
clear
exactly
what's
the
best
and
I
think
it
depends
on
what
you're
deploying
an
example
I've
seen,
it
was
a
situation
where
we
wanted
a
password
and
we
had
a
lot
of
approaches
or
where
we
could
stick
the
password
in
a
secret
ahead
of
time.
A
But
what
we
ended
up
doing
is
an
approach
where
the
operator,
when
it
runs
its
reconciliation
checks
to
see
if
there
is
a
password
as
a
password
secret
that
has
excess
to.
If
it's
there,
it
will
use
that
and
do
its
deployment
with
that
secret.
If
it's
not
there,
it'll
generate
it
and
save
it
in
that
secret,
and
so
it's
accessible
to
the
user.
There's
a
lot
of
questions
are
about
how
you
rotate
that
thing,
but
I
think
by
sticking
to
kubernetes
constructs
for
defining
those
we're
sort
of
able
to
solve
that.
B
I
think
maybe
the
reason
why
this
is
hard
is
because
different
applications
have
slightly
different
semantics
on,
like
how
often
do
I
rotate
my
seat
can
I
rotate
my
secret.
How
often
should
I
rotate
my
secret
and
is
my
secret,
a
random
string,
or
is
it
actually
a
like
a
certificate
there,
yeah
and
and
I
think
that
again
by
using
go
code,
we
are
able
to
not
restrict
you
in
any
way.
It's
it.
B
Maybe
that
feels
a
little
bit
like
you
know,
go,
do
it
yourself,
but
the
nice
thing
is
like
when
we
see
the
patterns
I
hope
we
can
pull
them
out
and
make
it
so
that
if
you're
the
first
one
you
have
to
do
it
yourself,
so
you'll
pro
a
fitter
yourself,
but
when
you're
the
person
behind
me
so
and
then
hopefully
we
can
just
like
say
point
to
that
pattern
and
say
this
is
a
simple
text.
Password
and
never
changes.
Go
use,
Mason's
pattern
for
that.
B
B
So
the
question
was:
how
do
we
draw
the
line
between
which
we
managed
as
an
add-on
and
what
to
be
managed
separately?
We,
that
is
a
excellent,
excellent
question.
We
we
try
to
define
things
which
we
think
are
clearly
we
things
which
are
clearly
add-ons
and
they
are
things
that
are
managed
as
part
of
the
cluster
lifecycle,
so
things
that
are
tightly
bound
to
the
kubernetes
version.
B
Think
one
of
the
things
we
we
are
aiming
to
make
easier
is
to
make
it
so
that
you
I
think
there
will
be.
There
will
be
an
opinion
ated
set
that
your
tooling
will
probably
do,
but
it
should
be
very
easy
to
turn
add-ons
on
and
off
so
we'll
likely
install.
We
hope
to
anyone
still
the
the
add-ons
for
a
larger
superset,
even
if
they
don't
necessarily
turn
them
on
we're,
not
that
make
sense
yeah,
not
there
any
other.
B
A
B
And
also
I
think
we
got
that
feeble
door
from
we
got
that
feedback
from
the
coop
builder
team
that
they
actually
were
seeing
people
that
wanted
to
build
more
more
patterns
and
one
of
their
one
of
the
reasons
this
hasn't
emerged
yet
is
because
COO
builder
is
trying
to
go.
Stable
and
patterns
are
probably
gonna,
be
on
a
different
life
or
a
different
stability
path.
B
They're
not
they're,
not
gonna,
be
stable
right
away,
and
so
the
idea
is
basically
to
have
the
ability
to
have
these
clearly
defined
sections
of
functionality,
and
there
are
other
people
that
are
also
reaching
out
to
the
coop
builder
team.
That
wants
to
do
that,
and
so,
if,
when
we
figure
out
the
repo
where
it
should
live,
then
yes,
if
you
want
to
build
a
pattern,
then
I
I
think
they
would
be
very
receptive
to
that.
B
So
the
question
was
kindof
needs
or
depends
on
sto
when
they
are
both
add-on
operators
should
operator
the
it
should
the
native
operator
call
or
invoke
the
ISTE
operator.
I
think
it's
that
affair.
It's
it's
another
one
of
the
TBD
we
we
certainly
can.
We
could
have
had
the
native
I
think
there
are
a
couple
of
patterns
we
could
have
had.
B
The
native
went
so
had
Jeff's
demo
when
he
created
the
native
CRD,
actually
created
instances
of
native,
build
native,
serving
and
native
metric
monitoring,
and
it
could
easily
have
created
an
instance
of
the
sto
CR,
which
I
think
is
what's
on
my
cluster
anyway,
if
it's
not
Jeff,
and
the
challenge
is
whether
whether
that
also
implies
an
ownership
relationship,
so
that's
the
first
one
and
it
probably
does
so.
That
would
be
a
little
weird
and
the
other
thing
is
sort
of.
B
How
are
you
enabling
it
so,
in
other
words,
how
much
value
is
there
and
automatically
turning
it
on?
And
we
sort
of
my
guess-
and
this
is
where
we
need
your
input-
is
that
if
you're
gonna
be
installing
it
anyway
by
coop,
cut,
applying
it
or
running
a
script
or
copying
something
one
command
versus
two
commands
is
not
a
huge
deal
versus
the
sort
of
cleaner
separation.
But
that
could
well
be
very
wrong.
And
if
you
have
opinions,
I
very
much
welcome
them.
B
B
Think
so,
Mason
I
think
we're
saying
that
what
if
the
sto
are
CR
already
exists
and
out
of
band
management
and
yes
and
I
think
I
think
it
goes
to
the
native
use
case.
Like
am
I
am
ia
user
that
is
wanting
to
use
native,
or
am
I
user?
That's
wanting
to
install
native
on
my
kubernetes
cluster,
and
so
that's
why
I
like
the
idea
of
the
parameter
and
that
it
enables
you
to
say
I
am
persona
a
or
persona
B.
So
that
might
be
a
great
answer,
but
I
think
we
can.