►
From YouTube: Essential Patterns For Designing And Implementing Your Operator - Michael Hrivnak & Austin Macdonald
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Essential Patterns For Designing And Implementing Your Operator - Michael Hrivnak & Austin Macdonald, Red Hat
It’s easy to get started developing operators with kubebuilder and operator-sdk to manage your workloads and infrastructure – but what challenges will you face as your operator matures? This presentation will share the most essential lessons learned across years of experience helping teams and organizations design and implement real-world operators for a wide variety of use cases. Coding topics will focus on Go-based operators. You will learn about: - API anti-patterns: Common API design choices that lead to future regret, and how to overcome them in the wild. - Bridging the gap between slow (and complex and buggy) imperative infrastructure management and the declarative Kubernetes API. - Taking control of the client’s cache to maximize its usefulness and avoid memory bloat. - Interacting with multiple clusters efficiently from a single operator instance. - Minimizing load imposed on the API server. Attendees will be ready to face key challenges as they enhance their operators with new features and evolving APIs.
A
A
Let's
get
right
to
it
getting
started
with
operators
in
operator.
Development
is
pretty
easy.
These
days
these
days,
we've
got
the
operator
SDK,
there's
other
tooling
out
there
there's
Cube
Builder,
a
variety
of
languages
are
supported
now
that
you
can
use
for
developing
some
kind
of
operator,
and
you
can
get
something
up
and
going
pretty
quick.
That
does
installation
on
installation,
maybe
a
little
bit
of
reconfiguration
and
be
pretty
satisfied
with
that.
A
But
we
want
you
to
have
more
advanced,
more
useful
operators
that
you
can
really
do
things
with
that
really
make
day
two
operations
successful
in
day
two
as
we've
learned,
is
full
of
messy
challenges.
These
are
just
you
know
the
first
20
or
so
that
came
to
mind
that
you
can
face
when
you're
orchestrating
operational
concerns
with
your
operator
really
solving
operational
problems.
What
kind
of
things
SRE
teams
worry
about.
A
So
we
have
a
lot
of
experience
managing
stateful
infrastructure,
and
that
comes
with
a
lot
of
those
day.
Two
challenges.
You
know
things
like
you
know
bare
metal,
server,
provisioning
and
these
kinds
of
things
we've
helped
make
operators
for
cluster
installation
cluster
upgrade
a
lot
of
infrastructure
related
stuff
that
can
take
time.
It
can
be
error
prone.
It
can
be
very
configuration
sensitive
when
you
have
a
cloud
API.
A
Of
course
you
know
what
you're
going
to
get
in
advance,
for
example,
when
you're
deploying
a
cluster
when
you're
doing
that
in
a
data
center,
there's
a
lot
more
opportunity
for
for
variation
of
environment
right.
So
we've
we've
done
a
lot
of
that
and
made
a
lot
of
these
kind
of
operators
in
that
space,
and
yes,
we've
even
made
an
operator
for
installing
more
operators,
if
only
it
was
so
easy.
As
as
this,
this
is
my
Fogo
I
hope
you'll
be
okay
with
some
of
that
along
the
way.
A
If
we
could
reconcile
and
just
say,
what's
the
current
state,
what's
the
desired
state
do
a
little
bit
of
work
to
move
the
one
a
little
closer
to
the
other
and
then
set
some
status
and
we're
out
of
there
success.
That
would
be
nice
right,
and
that
is
largely
what
we
want
to
do
in
our
operators,
but
the
implementation
of
that
can
get
a
little
more
complex.
So,
let's
start
we've
got
five
sections
for
you,
and
this
is
number
one
we're
going
to
start
with
some
API
anti-patterns
lessons.
We've
learned
about
API
design.
A
A
Why
would
you
do
this?
Well,
sometimes
you
need
a
temporary
Behavior,
it's
a
workaround
for
something
it's
not
going
to
be
here
for
long.
Let's
just
put
it
in
an
annotation.
You
know
we
know
what
the
reality
is
of
a
lot
of
temporary
solutions.
Sometimes
there's
a
deadline.
Pressure
we've
got
to
get
this
out
quick.
We
need
to
enable
this
new
thing.
Maybe
it's
a
feature
gate
whatever
it
is.
A
Don't
have
time
to
go
through
a
full
API
review
right
now
or
dealing
with
a
crd
upgrade
is
not
really
in
the
cards
right
now.
Let's
just
put
an
annotation
on
this
resource.
We
can
get
the
feature
out
there
and
and
we'll
we'll
deal
with
the
rest
of
it
later
or
sometimes
a
hidden
feature.
You
can
actually
use
this
as
something
of
a
feature
gate.
You
don't
want
to
advertise
it
in
your
in
your
resource,
but
you
still
want
it
to
be
available.
So
what's
what's
the
problem?
What's
the
downside
of
this?
A
Well,
you
don't
get
API
version
you're
not
plugged
into
the
versioning
scheme
of
your
crd
anymore,
there's
no
status
to
reflect
feedback
to
the
API
user
of
whatever
this
is.
Unless,
of
course,
you
put
another
annotation
on
your
resource
to
give
a
little
status.
I
have
seen
that.
Don't
recommend
that
there's
no
validation!
A
You
know
we
have
access
to,
like
amazing
validation,
now
right
out
of
the
box
enforced
by
the
kubernetes
API
server,
it's
real
easy
to
use,
but
when
you're
using
an
annotation,
of
course,
we
don't
have
access
to
that
kind
of
validation,
quite
so
easily.
It's
harder
to
document.
This
annotations
are
not
somewhere
that
people
normally
look
when
they're
trying
to
figure
out
how
can
they
accomplish
a
behavior
with
your
API
and
then,
of
course,
for
a
user
who's?
A
A
We've
done
quite
a
bit
of
that
in
the
openshift
world,
a
mix
of
documentation
on
the
field
itself,
combined
with
a
feature
flag.
That's
usually
a
configuration
property
of
the
operator
itself,
not
the
API
the
operator
implements,
but
the
operator
itself
that's
a
better
place
to
put
that
kind
of
a
feature
gate
that
can
then
make
a
field
either
be
useful
or
not,
so
that
that's
a
good
way
to
handle
that
great.
A
Now,
here's
another
anti-pattern
outgrowing
booleans
this
one
borrowed
the
description
right
from
the
kubernetes
API
conventions,
which
is
a
fantastic
document.
If
you
have
anything
to
do
with
designing
kubernetes
apis,
either
as
part
of
kubernetes
or
for
your
own
uses
and
operators,
highly
recommend
set
aside
some
time,
maybe
for
taking
a
flight
home
from
this
event,
read
through
this
API
conventions
document.
It
has
a
lot
of
insight
and
and
guidelines
that
are,
you
know,
written
in
the
ink
of
people
who
have
seen
the
problems
they've
been
there.
A
They've
lived
through
the
pain
and
they're
sharing
them
here,
and
this
is
a
favorite
outgrowing,
a
Boolean.
So
here's,
let's
imagine
we.
We
have
a
new
DNS
provider,
new
DNS
service
called
Fancy
DNS
and
we're
running
it
in
a
clusters.
We
made
an
operator
for
it.
It's
very
simple.
Now
you
may
know
that
DNS
queries
tend
to
happen
over
UDP,
but
also
can
happen
over
TCP.
A
A
Well,
a
quarter
goes
by
and
PM
comes
back
says
you
know,
got
a
growing
number
of
customers.
They've
got
some
Oddball
use
case.
They
actually
don't
want
to
use
UDP
at
all.
They
want
to
do
just
TCP
for
for
their
DNS
queries
kind
of
weird,
but
you
know
it
happens.
Can
we
make
that
also
configurable?
Can
we
be
able
to
turn
off
UDP
like
keep
TCP?
Okay,
we
had
another
field,
let's
be
able
to
enable
UDP
optionally.
A
So
now
we've
got
a
growing
list
of
booleans
and
we've
got
a
circumstance
where
one
point
in
our
four
square
Matrix
is
not
really
valid.
We
don't
want
both
of
these
to
be
false
right.
So
now
we've
got
a
validation
concern.
That's
gotten
more
complicated
everybody's
life
is
getting
more
complicated.
All
of
a
sudden.
A
What
should
we
have
done
something
more
like
this
reframe
the
question
reframe
the
API
itself
think
about
it
differently,
not
as
a
Boolean
on
or
off.
But
what
listen
protocol
do
you
want?
Is
it
UDP?
Is
it
TCP?
Is
it
both
mean
instead
of
both
I
should
have
said
all
in
case?
The
third
protocol
shows
up
someday,
but
in
any
case,
that
I
find
is
the
secret
to
having
a
better
API
field
than
using
a
Boolean
as
the
type
reframe.
How
you're
thinking
about
that
kind
of
setting?
A
A
You
know,
we've
seen
things
like
device
inventories
being
tracked.
I've
done
this
device
inventories
of
various
kinds
being
tracked.
We
we
do
kubernetes,
Native
infrastructures,
we're
very
deep
in
the
openshift
world,
at
least
philosophically,
into
managing
infrastructure
using
kubernetes
native
apis.
A
So
we
run
up
against
these
scale
concerns
periodically
and
when
you
think
about
something,
a
problem
space
like
advice
inventory,
a
good
metric
would
be
like
hundreds
of
devices
that
are
related
to
this
cluster,
like
the
nodes
in
that
cluster
and
the
machines
that
are
part
of
the
cluster
API,
these
kinds
of
things
bad
would
be
like
millions
of
edge
devices.
You
probably
should
not
be
trying
to
track
that
as
crds
in
your
kubernetes
cluster.
A
A
It
makes
a
nice
easy
story
and
it'll
work
for
a
while
until
it
doesn't
because
there's
a
size
limit
to
how
much
data
you
can
stuff
into
a
config
map
and
and
the
secret
is
the
same
and
TLS
certificates
can
be
another
one
that
occasionally
in
some
education
can
get
you
it's
easy
to.
Think.
Oh
certificates.
Aren't
that
large!
A
Well
until
they
are,
you
know
you
get
somebody
with
a
long
chain
of
trust,
trying
to
put
the
whole
chain
into
one
Secret,
and
things
can
get
rough,
so
don't
use
the
kubernetes
API
as
a
data
store,
is
a
best
practice.
What
she?
What
should
you
do?
Instead
use
a
different
data,
store,
find
a
database
I
love,
kubernetes
native
apis
as
much
as
about
anybody.
A
Kcp
is
an
up-and-coming
emerging
project
that
I
think
has
a
lot
of
promise
for
this
kind
of
scale,
for
managing
with
a
kubernetes
native
API,
a
very
large
scale
of
of
data
and
resources.
That
is
the
kind
of
project
you
might
be
interested
in
using
Sunday
for
something
like
millions
of
edge
devices,
but
stay
tuned,
highly,
recommend
I
know,
there's
probably
a
lot
of
kcp
talks
this
week.
A
What
are
examples
of
this,
so
cluster
installation
and
upgrade?
This
is
the
kind
of
thing
that
could
take.
20
minutes
could
take
an
hour,
might
take
a
couple
hours
upgrade
depending
on
what
kind
of
constraints
you
have?
How
big
is
your
cluster?
How
long
does
it
take
to
drain
a
node
then
upgrade
that
node
in
place,
for
example,
if
that's
the
upgrade
pattern
you
follow
and
and
get
workloads
back
on
and
then
go
on
to
the
next
node.
A
You
know:
we've
heard
stories
of
people
take
days
a
week
or
more,
depending
on
the
various
constraints
to
get
the
requester
upgrade,
especially
with
very
very
large
clusters.
So
these
kind
of
things,
especially
if
you
put
a
kubernetes
native
API
in
front
of
this,
it's
long
comparative
errors,
Could
Happen
a
long
time
after
somebody
declares
what
their
desired
State
provisioning
bare
metal.
A
Likewise,
it
might
take
10
or
15
minutes
for
a
server
that
you're
trying
to
provision
with
the
kubernetes
native
API,
just
a
boot,
and
then
you
write
some
stuff
to
disk,
and
then
you
reboot
it
okay.
Now
it's
another
10
or
15
minutes
to
see
what
happened,
backing
up
data
of
course,
backups
take
a
long
time.
So
these
are
the
kind
of
infrastructure
concerns
that
are
slow,
they're
pretty
imperative
and
are
ripe
for
some
kind
of
something
to
not
go
according
to
plan.
A
A
This
is
from
the
metal
cubed
project.
This
is
don't
squint
too
hard.
I
didn't
really
mean
for
you
to
actually
be
reading
this.
This
is
just
like,
for
you
know.
For
effect,
this
is
the
actual
State
machine
used
by
that
project.
This
is
a
project
that
interacts
with
bmc's
baseboard
management
controllers,
on
real
server
Hardware
to
control
it
from
kubernetes.
Apis
turn
saw,
you
know,
turn
Hardware
on
and
off
provision
it
these
kinds
of
things
and
they
absolutely
they
wrote
it.
A
They
have
an
operator,
the
bare
metal
operator
and
they
use
a
state
machine
and
it
works
well.
Many
resources
can
have
an
end
State.
You
know
a
job
is
a
good
example.
That's
one
that
many
of
us
are
familiar
with.
That's
built
in
it
starts
it
does
some
work,
it
ends.
It
has
a
finite
State.
You
can
Embrace
that
and
you
can
create
your
own
resource
in
that
same
kind
of
pattern
that
is
designed
to
have
a
final
State
and
that's
okay.
A
Returning
from
your
reconcile
function
promptly,
this
is
a
best
practice
in
general,
but
try
not
to
like
start
your
reconcile
function,
kick
off
and
install
wait,
wait,
wait,
wait,
wait,
wait,
wait!
An
hour
later
it
finishes,
and
then
you
return
from
your
reconcile.
Of
course
you
don't
want
to
do
that
you're
going
to
want
to
monitor
some
other
way,
return
from
your
reconcile
function
in
your
controller,
we'll
talk
in
a
minute
about
some
other
ways
to
monitor
that,
but
you'll
be
happier
in
the
long
run.
A
A
Like
a
BMC,
that's
rebooting,
a
server
that
that
behavior
is
implemented
in
The,
BMC
you're,
controlling
it
you're
telling
it
reboot,
and
then
you
wait
and
you
can
pull
or
you
know
through
some
mechanism,
observe
what
that
BMC
is
doing,
but
you
don't
really
want
to
implement
a
long
like
have
like
install
openshift,
install
Logic,
for
example
like
in
your
reconcile
function.
That
would
be
a
bad
point.
A
So
what
can
you
do?
You
could
spawn
a
job
and
actually
use
a
job
to
implement
some
of
that
logic
or
a
Cron
job
would
be
very
similar
and
your
controller
can
create
those
jobs
and
just
watch
them
until
they
can
they
complete
it's
a
very
nice
experience,
because
it's
a
kubernetes
API
itself.
It
has
it
generates
events.
A
You
can
watch
those
events
reconcile
whenever
that
job
emits
an
event
and
and
therefore
be
immediately
plugged
in
when,
when
that
operation
that
you're
interested
in
finishes
another
option,
you
could
use
some
kind
of
serverless
or
function
as
a
service,
or
maybe
some
config
management
like
ansible
awx,
is
an
example
of
that
some
external
system.
That's
not
part
of
your
operator
that
can
designed,
for
these
start
to
end
workflows
that
you
can
that
you
can
trigger
wait
on,
observe
the
state
of
and
and
let
them
complete.
A
But
this
is
an
important
Point.
Don't
access
the
kubernetes
API
from
those
external
processes?
If
you
can
avoid
it,
because
I
think
as
Austin's
going
to
talk
about
in
just
a
minute
using
a
caching
client
is
hugely
valuable.
We
want
to
avoid
querying
the
kubernetes
API
as
much
as
we
can
without
some
kind
of
caching
layer
in
between
controller
runtime,
Cube
Builder
operators,
decay.
All
these
use
a
caching
client
that
that
we
want
to
use
as
much
as
possible,
and
then
you
can
use
a
channel
source.
A
So
you
can
either
maybe
you're
using
a
go
routine
to
pull
some
external
system
and
whenever
you
get
that
system
gets
to
a
certain
State.
Now
you
stick
that
event
into
this.
Go
Channel
or
maybe
you've
got
an
HTTP
callback
that
you're
listening
for
and
that
sticks
out
of
engine
that
channel.
But
this
is
a
best
practice
and
a
really
nice
pattern
for
bridging
that
Gap
and
I'm
going
to
hand
it
off
to
Austin
for
our
third
topic.
B
Okay
so
we've
talked
about
good
patterns
for
developing
your
operator
so
that
you
don't
get
into
trouble.
But
now,
let's
talk
about
how
to
be
a
good
citizen
on
the
cluster.
We've
got
to
minimize
the
API
load,
because
all
those
operators
have
to
live
there
together
so
step,
one
you've
got
to
collect
metrics
and
it's
a
good
idea
to
start
collecting
metrics
early
in
the
development
cycle.
B
It's
so
easy
to
say:
oh
we'll
get
it
fully
functioning
and
then
we'll
start
measuring
it,
but
it's
much
better
to
know
how
your
performance
is
being
impacted
as
you're
adding
features,
controller
runtime,
which
is
underneath
the
operator
SDK,
will
expose
Prometheus
metrics
to
you
out
of
the
box.
So
if
you
have
Prometheus
installed
on
your
cluster,
there's
no
reason
not
to
be
consuming
those
metrics
you
can
com.
B
So
how
can
an
operator
be
a
bad
citizen?
Well,
the
first
thing
is
a
watch.
Watches
are
much
more
expensive
than
you
might
realize
and
when
you
have
lots
of
operators
on
there,
this
can
scale
up
very
badly.
So
just
remember,
watches
are
expensive,
Cube
API
server.
It
can
limit
the
number
of
watches
allowed
simultaneously
on
the
cluster.
I
know
that
openshift
does
this,
and
even
if
that
limit
isn't
reached,
those
resource
utilization
can
continue
to
scale
in
a
way
that
is
not
desirable.
B
So
how
do
you
avoid
this?
Well,
if
you've
got
some
very
tightly
coupled
operators,
maybe
they
don't
need
to
be
two
operators
if
you've
got
two
control
loops
that
are
watching
the
same
objects,
consider
putting
them
into
the
same
operator
so
that
they
can
share
a
cache
that
cash
will
emit
a
watch
to
the
API
server
and
any
of
the
extra
performance
is
going
to
be
between
your
operator
and
that
cache?
B
Second,
we
want
to
avoid
the
low
sync
periods.
So
if
you're
not
familiar
async
period,
is
the
maximum
amount
of
time
that
your
operator
will
go
without
syncing
again,
most
of
the
time
we
think
of
an
operator
as
reconciling
based
on
events
and
that's
the
way
it
should
be.
You
don't
want
your
operator
running
every
five
minutes,
because
it
is
expensive.
B
B
If
you
do
have
some
bugs
a
low
sink
period,
can
cover
them
up,
and
wouldn't
it
be
better
to
just
know
that
you've
got
a
bug
and,
if
absolutely
necessary,
if
you
need
to
be
triggering
reconciles
more
frequently,
then
events
are
coming
in
consider
using
re-queue
after
this
is
something
that
happens
as
your
operator
is
exiting.
You
can
say:
hey
I'm,
not
sure
about
the
state.
Can
you
recue
like
right
now
or
wait?
10
minutes,
try
again.
B
So
we
know
that
we
can
make
our
reconciliations
cost
less,
but
how
can
we
do
fewer
reconciliations?
Well,
you
do
not
need
to
reconcile
every
single
time
any
event
from
anything
that
you're
watching
comes
on
in
you
can
filter
those
events
with
predicates
controller
runtime
provides
a
handful
of
very
good
predicates,
just
right
out
of
the
box
that
you
can
use
just
like
I've
done
here
with
the
with
event
filter.
B
So
inside
of
your
reconcile
Loop
when
you're
hitting
the
kubernetes
API
know
when
you
can
get
away
with
a
patch,
a
patch
uses
many
fewer
resources
than
an
apply
and
it's
going
to
go
faster,
but
be
very
careful,
because
if
there
are
multiple
actors,
changing
these
resources
and
apply
is
safer,
particularly
a
server
side
apply,
which
is
the
new
way.
All
of
us
need
to
know
about
that.
So,
if
you're
going
to
start
playing
with
this,
please
be
sure
to
check
out
server
side
apply,
as
well
as
the
various
merge
strategies.
B
B
B
A
Well,
this
is
one
of
the
more
interesting
areas,
I
think
of
operator
design
right
now,
and
we
see
the
world
changing
to
be
far
more
multi-cluster
than
it's
been
in
the
past
more
clusters
that
are
smaller
clusters
that
have
more
tightly
scoped
purposes
and
this
this
cluster
does
this
some
other
cluster?
Does
that,
as
opposed
to
having
one
big
broad
cluster,
that
does
it
all
clusters
that
come
and
go
cluster
is
a
service.
A
A
So
this
topic
is
as
much
sharing
with
you
what
the
state
of
the
world
is
and
what
you
can
do
now,
as
it
is
a
an
invitation
to
help
an
invitation
to
get
involved,
even
if
it's
just
sharing
use
cases
or
come
help
build
the
next
generation
of
multi-cluster
clients,
certainly
in
control
of
runtime,
at
least
so.
Here's
some
use
cases
we
do
again
kubernetes
native
infrastructure
management,
so
we
do
it
with
an
openshift
policy
and
governance
through
kubernetes.
Apis
you've
got
a
hub
cluster.
A
That
itself
is
not
just
interacting
and
managing
resources
on
the
cluster
that
it's
running
on,
but
maybe
it's
One
controller,
managing
resources
and
and
operands
across
many
clusters.
Let's
say
you
have
10
12
clusters
running
all
in
the
same
facility.
Do
you
need
to
run
the
Pod
set
Operator
just
to
pick
on
our
own
fictional
experimental
operator?
Do
you
need
to
run
the
Pod
set
operator
on
every
single
cluster,
or
could
you
just
run
it
on
one
of
them
and
have
that
one
process
be
watching
and
interacting
with
all
of
with
all
of
those
clusters?
A
That
would
be
nice
right,
that's
not
quite
possible.
Yet
so
here's
what
we
do
have
this
is
in
control
runtime.
It
probably
could
use
some
better
or
more
prominent
documentation
if
anybody's
up
for
that.
But
basically
what
we're
looking
at
here
is
this
new
resource,
this
new
type
called
cluster.
So
what
you
can
do
here?
What
I'm
showing
here
is
imagining
you're
going
to
watch
config
maps
in
some
other
external
cluster
and
then
be
able
to
retrieve
the
config
maps
from
that
external
cluster
when
necessary.
So
in
this
first
function
we
get
a
cube.
A
Config
we
set
up
this
cluster
resource
with
that
Cube
config
and
then,
in
that
cluster
resource
we
can
get
a
client
and
that
client
now,
of
course,
can
be
used
to
access
those
things.
But
here's
an
important
aspect
there's
nothing
yet
in
control
runtime.
That
really
helps
us
keep
track
of
that
reference,
keep
track
of
either
that
client
or
keep
track
of
that
cluster
resource.
So
you
got
to
keep
just
store
that
somewhere
yourself.
So
here
I've
got
that
first,
red
arrow
I
just
picked
an
arbitrary
key
called
name.
A
A
We
want
to
read
it
off
that
external
cluster
you're
going
to
have
to
retrieve
that
client
and
so
here
again
you're
inventing
your
own
mechanism
for
getting
access
to
that
client,
the
right
client
at
the
right
time.
So
it's
workable
and
it's
a
step
in
the
right
direction,
but
it
clearly
it's
not
quite
a
complete
story
for
all
the
use
cases
that
we
see
in
front
of
us,
so
things
you
might
consider
for
your
multi-cluster
access.
There
are
different
patterns
based
on
different
needs.
A
Do
you
really
need
to
watch
events?
Do
you
need
to
do
this
list
watch
pattern,
or
not?
Maybe
you're
just
very
occasionally
doing
a
request
to
some
external
cluster,
in
which
case
don't
worry
so
much
about
a
caching
client
all
this
stuff.
You
can
just
stand
up
a
client
on
demand,
and
just
do
it.
Maybe
that's
just
okay
for
your
circumstance.
How
many
clusters
do
you
really
need
to
interact
with?
Is
it
just
a
handful?
Is
it
a
thousand?
A
You
know
about
these
in
advance
and
that's
the
job
you're
doing
this,
this
mirroring
back
and
forth,
but
in
these
other
Multicultural
management
kind
of
scenarios,
you
start
your
operator
and
then
after
it's
running
you're
telling
it
dynamically
okay,
there's
a
new
cluster
that
I'm
introducing
I
need
you
to
start
managing
this
cluster
is
has
gone
away.
Now
it's
been
de-provisioned
so
as
these
come
and
go
that
that
adds
a
significant
layer
of
complexity.
A
Right
now,
at
least
today,
watches
are
a
lot
easier
to
add
than
they
are
to
remove,
so
that
that's
a
significant
consideration.
Now,
where
is
this
going?
Kcp,
is
doing
some
excellent
client
work.
If
you're
interested
in
this,
we
have
at
least
one
I
thought
we
had
two,
maybe
still
around-
of
the
people
who
are
actually
doing
this
work
are
up
here
in
the
second
row.
A
You
come
talk
to
them,
but
there's
a
world
developing
in
a
fork
of
controller
runtime
by
the
kcp
project
to
to
move
this
forward
where
your
reconciled
function
will
get
a
request.
That
includes
now
not
just
a
name
and
a
namespace,
but
a
cluster
identifier
as
part
of
that
key,
and
then
you
can
create
your
a
context
that
has
that
cluster
attached
to
it
and
now,
as
long
as
you
use
that
context
with
future
client
requests
it.
A
Your
client
will
use
the
correct
cluster
the
correct
cache
and
give
you
that
sort
of
seamless
client
experience,
we've
come
to
to
Really,
Love
and
and
take
advantage
of
in
controller
runtime.
So
that's
I
think
where
things
are
going,
but
it's
clearly
still
in
development,
and
if
this
is
something
that
matters
to
you
get
involved
with,
probably
the
kcp
community
is
a
good
place
to
go
the
what's
the
channel
in
the
kubernetes.
Well,
I
think
it's
on
our
last
slide.
Well,
it
was
a
good
last
night.
A
No,
it's
not
on
our
last
slide,
kcp
Dev
in
the
kubernetes
slack
and
then
what's
the
operator,
one
kubernetes
operators
in
the
kubernetes
slack.
Both
those
places
are
good
places
to
ask
about
this.
All
right.
That's
the
end
of
our
content.
I
think
we
have
like
a
minute
or
two
for
questions
it,
especially
if
you
like
this,
give
us
a
review
at
this
QR
code.
A
C
C
B
Operator
has
not
been
a
good
citizen
and
is
consuming
lots
of
resources.
What
do
I
do?
Well,
it
depends
on
what
it
what
is
going
wrong
if
you're,
using
a
bunch
of
watches,
there's
some
nice
ways
to
trim
that
back
and
if
not
I
would
recommend
you
come
by
kubernetes
operators
in
kubernetes,
slack
and
you'll
find
that
the
operator
folks
are
very
helpful
and
just
nice
people.