►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Writing a Kubernetes Operator: the Hard Parts - Sebastien Guilloux, Elastic
Building a custom controller or operator to manage your Kubernetes applications is becoming easier, with the help of libraries and tools such as controller-runtime and Kubebuilder. Putting together an initial working prototype is fairly straightforward, but devil is in the details. This talk focuses on lessons learned while writing Kubernetes controllers for stateful workloads with the help of controller-runtime. It covers some of the "hard parts". The operator lives in the past: how to deal with resources cache inconsistencies? Why does idempotency matter? What can you do when StatefulSets are not good enough for the orchestration you need? How to empower advanced users but still provide good defaults? What namespace(s) should the operator have access to? How to test that monster you ended up building? These are questions engineers at Elastic had to answer.
https://sched.co/UaeV
A
Thank
you
Chris.
That
is
a
huge
room.
I
need
to
make
sure
I.
Take
pictures,
so
I
can
tell
my
wife
I'm
actually
doing
some
work
here.
I'm,
not
drinking
beers
in
San
Diego,
so
my
name
is
Sebastien
I'm
from
France
I
hope
you
don't
mind
the
French
accent
today,
I'm
going
to
talk
about
kubernetes
operators
and
what
I
personally
consider
the
hard
parts
of
writing
one.
A
So
the
past
year
I've
been
working
on
this
project.
We
call
it
e
ck
elastic
cloud
on
kubernetes,
it's
basically
a
kubernetes
operator
to
deploy
the
elastic
stack
products,
manage
your
elastic
search
cluster
like
smooth
migrations,
rolling
up
great
and
so
on,
cabana
and
APM
server.
You
can
check
it
out.
It's
on
github.
The
code
is
open.
You
can
try
it
for
free,
but
I'm
not
going
to
talk
about
egk
today,
I'm
just
going
to
focus
on
building
operators
in
general.
A
So
I
will
assume
most
of
you
already
know
what's
an
operator,
but
if
you
don't
I,
have
a
couple
slides
to
remind
you
of
what
it
is.
So
usually
an
operator
comes
with
its
own
ceoddi,
some
resource
definition,
that's
a
way
of
extending
the
kubernetes
api
with
your
own
set
of
api's.
So
here
an
example
with
elasticsearch
the
operator
will
come
with
its
own
schema.
That
would
allow
you
to
define
an
elastic
search
cluster.
So
here
we
want
to
deploy
elastic,
search
in
version
7,
which
we
master
nodes
and
two
data
nodes.
A
Then
the
operator
itself
is
basically
just
a
pod
running
in
the
communities
cluster
and
what
it
does
is
watching
resources
in
the
API
server.
So
here
it
would
most
probably
watch
all
resources
of
kind
elastic,
search
where
it
would
also
probably
be
interested
into
watching
pod
services
secrets
and
everything
we
need
to
deploy.
In
the
last
thing,
combinators,
the
operator
has
this
concept
of
a
reconciliation
loop.
A
So
whenever
something
happens
in
the
API
server
like
resource
we're,
watching
gets
created,
updated
deleted,
it's
going
to
trigger
a
reconciliation
internally
and
the
reconciliation
is
just
basically
a
securing
a
flow
of
sequential
steps
doesn't
have
to
be
sequential,
but
most
of
the
time
it
is
so
that
we
eventually
reconcile
the
expected
resources.
We
should
have
based
on
CLE
specification
so
that
we
have
all
these
resources
running
on
kubernetes.
So
this
reconciliation,
loop,
usually
can
really
return.
Early,
like
say,
for
example,
is
creating
a
set
of
pods.
A
A
The
one
I
use
that
I
find
really
good
is
crew
builder
maintained
by
the
API
imaginary
special
interest
group,
who
builder
basically
helps
you
scaffold
a
go
line
project
to
build
your
own
operator.
It
helps
you
automatically
convert
your
internal
go
model
to
the
ceoddi
channel
representation
and
bootstrap
everything
related
to
the
reconciliation
we
talked
about
behind
the
scenes.
Crew
builder
is
using
the
controller
runtime
library
which
sets
up
everything
related
to
watches.
A
A
A
So
of
course
the
operator
is
kind
of
working
in
real
time.
The
sense
that
whatever
happens
in
a
cluster,
it's
going
to
reconcile
things
over
and
over
again,
but
you
have
to
assume
that
every
time
that
code
gets
executed,
you
kind
of
one
step
behind
the
reality
of
things
in
the
state
of
the
world.
Let's
take
an
example
here
say
in
the
middle
of
my
reconciliation
at
least
I
need
to
list
all
the
pods
I
have
in
a
cluster
related
to
my
particular
application.
So
I
would
list
a
pulse
and
I
would
see.
A
Maybe
I
have
two
pods,
but
a
and
part
b
and
then
compared
what
I
expect,
which
would
be
having
three
pods.
I
realize
one
is
missing,
so
I
need
to
create
it.
So
I
would
create
Part
C
here
and
then,
if
later
on,
in
that
same
reconciliation
or
maybe
the
next
reconciliation,
this
part
again,
I
would
probably
expect
to
have
three
pods
now
but
I.
What
I
will
most
probably
see
here
is
that
I
have
only
two
parts,
because
pod
C
is
not
there,
yet
he
was
still
created.
A
A
The
reason
is
the
client
we
use
to
reach
the
API
server
uses
a
cache
reader,
which
means
we
are
not
sending
a
request
to
the
API
server
every
time
we
want
to
list
things
or
get
a
particular
resource,
because
that
would
be
pretty
expensive.
There
would
be
a
big
performance
hit.
So
what
we
do
instead
is
that
we
have
this
local
cache
to
the
shoulder
and
the
cash
gets
populated
from
Roach
events.
So
whenever
we
need
to
list
things,
we
just
look
at
what
we
have
in
the
cache.
A
This
can
lead
to
some
funny
situation
in
the
history
of
our
project.
We
had
several
backs
that
are
related
directly
to
this
usage
of
the
cache
things
like
creating
too
many
poles
like
you,
you
expect
rivers
to
exist,
but
because
every
single
reconciliation
creates
one
based
on
an
out-of-date
cache,
you
actually
end
up
with
25
poster
three
seconds
or
if
you
need
to
manage
stateful
workloads
like
elastic
search
or
any
complex
distributed
system.
A
Pretty
often
you
have
this
notion
of
Chrome
where
you
have
a
set
of
map
of
master
nodes,
and
you
need
to
make
sure
somehow
that
you
maintain
some
kind
of
consistency
here,
for
example,
in
the
case
of
elastic
search,
if
it's
running
five
master
nodes,
you
make,
you
need
to
make
sure
that
three
master
nodes
from
the
crew
of
the
Quran.
So
if
your
assumption
on
the
number
of
nodes
relies
on
the
number
of
pods
you're
having
a
cache,
you
may
have
a
problem
here,
because
your
cache
does
not
really
represent
reality.
A
A
When
you
do
a
creation,
the
resource
you
create
has
a
name
and
probably
in
the
in
space,
say
I'm,
creating
one
part
at
one
conciliation
and
at
the
next
reconciliation
I'm
crane
trying
to
create
the
exact
same
pod,
because
I
don't
see
it
in
the
cache.
The
API
server
is
probably
gonna
is
well
truly
actually
going
to
reject
that
creation,
because
another
resource
with
the
same
name
already
exists
so
in
the
reconciliation.
A
I
would
get
a
conflict
error
and
that's
just
fine
if
I
get
that
I
just
riku
like
returned
early
from
the
deviation
and
another
reconciliation
is
going
to
be
cute
again
whenever
there's
a
chance,
a
change
in
the
resources
I'd
wash.
So
whenever
that
you
thought
I
created
appears
in
the
cash,
the
new
reconciliation
is
going
to
be
triggered,
and
this
time
it
will
probably
not
fail.
It
conflicts
because
I'll
notice,
the
body
stares,
but
not
try
to
create
it
again.
A
We
had
the
same
kind
of
stuff,
with
updates.
Every
resource
has
a
resource
version,
so
whenever
you
do
an
update
of
a
resource,
what
you
can
do
is,
with
your
update,
call
you
ship
this
resource
version
that
you
have
from
the
cache
and
if
the
resource
in
the
API
server
does
not
match
that
version.
So
probably
it
has
been
updated
country
on
V,
with
what
you're
trying
to
do.
A
Deletions
basically
work
the
same
way
so
here
I'm
using
the
controller
runtime
client
to
talk
to
the
API
server
and
whenever
I
do
a
deletion.
I
can
just
pass
me
in
the
UID
of
the
resource.
I
want
to
delete
also
the
version
of
that
resource,
so
I,
don't,
for
example,
by
accidentally
the
resource
I
would
probably
not
have
deleted
if
I
had
a
proper
version
of
that
resource.
This
is
also
kind
of
useful
if
you
dealing
with
stateful
sets,
especially
if
you're
using
the
Anjali
strategy.
We'll
talk
about
that
later.
A
What
could
happen
is
that
you
delete
a
part.
You
expect
the
staples
head
controller
to
recreate
that
part,
but
at
the
next
reconciliation.
Maybe
you
would
do
that
deletion
again,
but
if
the
stateful
said
controller
already
like
recreated
the
part,
it
gets
a
newer
ID,
so
you
actually
deleting
again
the,
but
that
was
just
restarted,
which
is
not
what
you
want
so
shipping.
The
UID
here
in
this
call
helps
with
that.
A
So
this
is
all
fine,
but
in
some
situation
that's
not
enough.
I
said
earlier
that
whenever
you
manage
stateful
workloads,
you
have
this
additional
complexity
of
maintaining
some
kind
of
consistency
between
two
systems.
On
one
hand,
we
have
the
community's
resources.
On
the
other
hand,
we
have
this
external,
maybe
distributed
complex
system
to
operate,
and
we
need
to
make
sure
that
there's
some
kind
of
consistency
here,
the
API
server
optimistic
looking-
does
not
always
help
with
that.
A
So
the
way
it
works,
the
expectations
thing
is
that
whenever
we
do
a
creation
like
here
whenever
we
create
a
pod
at
the
same
time,
we
register
in
memory
the
fact
that
we
did
this
creation.
So
we
expect
that
creation
to
happen,
and
we
expect
that
at
some
point
this
pod
will
appear
in
our
cache,
then
at
the
next
reconciliation
when
this
gets
executed
again
before
we
do
any
other
creation.
We
look
at
this
expectation
we
registered.
A
A
A
Have
a
few
set
of
best
practices
so
that
whatever
you
write
in
the
reconciliation
actually
helps
you
like
saves
you
from
doing
anything
wrong,
and
the
first
thing
is
to
use
deterministic
naming.
We
saw
earlier
this
optimistic
concurrency
on
creation,
which
works
fine
as
long
as
the
resource
you
create.
Has
this
table
name,
so
you
created
the
first
time
you
show
you're
not
going
to
create
it
a
second
time,
because
the
second
creation
will
be
rejected.
A
But
if
the
resources
you
create
have
a
random
name,
then
this
doesn't
work
right.
So
you
need
some
kind
of
stable
deterministic
naming,
for
example,
if
you
stateful
sets
you'll
notice
that
every
bud
and
the
line
pod
that
gets
created
has
the
name
of
the
stateful
set
dress
and
all
in
all,
so
that's
very
deterministic,
stateful
set
controller
knows
exactly
what's
the
name
of
the
body
is
going
to
create,
then
at
any
point
of
the
reconciliation
you
have
to
assume
that
probably
your
cache
is
scale.
A
You
have
to
think
about
it
like
whenever
you
do
something
try
to
think
what,
if
the
resources
is
in
the
cache
are
not
up-to-date.
Is
everything
going
to
be
okay,
most
probably?
Yes?
Sometimes
it
isn't,
and
finally,
the
entire
reconciliation
should
be
idempotent,
which
means
it's
going
to
run
over
and
over
and
over
again,
you
want
the
output
of
this
reconciliation
to
be
the
exact
same
state
of
resources
in
kubernetes.
It's
really
fine.
We
trigger
the
same
reconciliation
and,
at
the
same
time,
it's
if
anything
goes
wrong
like
if
you
bought.
A
But
here
we
have
a
problem
in
this
piece
of
code.
What,
if
the
operator
crashes
in
between
those
two
creation,
we
create
the
stateful
set,
there's
a
crash:
the
operator
gets
restarted
and
we
don't
go
through
that
code
path
again,
so
we're
actually
never
going
to
create
the
Headless
service,
or
maybe
the
code
will
be
held
at
service
creation,
returns
an
error
and
whenever
we're
going
to
retry
that
we're
actually
not
going
to
retry
that
because
the
stateful
set
is
already
there,
we
created
it.
So
we
don't
enter
the
live
condition
again.
A
So
one
way
to
fix
that,
of
course,
is
to
reverse
the
ordering,
since
our
condition
is
based
on
the
presence
of
the
stateful
set.
If
we
create
the
stateful
set
last,
then
it's
okay
because
say
we
create
the
Headless
service.
Then
the
operator
crashes,
then,
when
it's
gonna
start
again,
State
one
cell
is
still
not
there.
So
we'll
do
that
again,
but
still
probably
this
is
how
you
really
want
to
do
it.
A
You
want
to
decouple
the
creation
of
every
single
resource,
because
the
problem
with
that
second
version
is
that
if
someone
accidentally
deletes
the
headless
service,
it's
never
going
to
be
created
again
because
will
never
enter
that
code
path.
And
here,
if
we
try
to
keep
things
flat,
then
we're
pretty
sure
whatever
happens.
If
the
statement
said
is
removed,
the
health
service
is
removed,
then
the
next
reconciliation
is
going
to
fix
things
again.
A
A
So
I
talked
about
reconciling
resources,
handling
creations
and
deletions
actually
fairly
easy,
but
handling
updates,
sometimes
very
hard
say
whatever
you
have
in
the
new
CEO
of
the
internally
maps
to
the
creation
of
a
stateful
set.
Maybe
that's
yadi
was
resource,
but
it
dated
so
that
now
you
know
you
should
update
the
underlying
stateful
set
from
three
replicas
to
five
replicas
there's
an
update
operation.
A
Here
you
need
to
do,
and
usually
what
happens
is
that
internally
in
the
code
you
build
this
expected
resource
that
you'd
like
to
have
in
urban
areas,
and
you
compare
that
with
the
actual
resource
you
have
that
you
fetch
from
the
cache
and
then
based
on
this
comparison.
You
decide
whether
you
need
to
update
the
resource
or
not.
A
A
Say
on
the
Left
I'm
creating
a
pot,
it's
like
minimalistic,
but
I
created
this
way.
Then,
if
I
retrieve
it,
that's
that's
what
I
get
back.
That's
not
exactly
the
product
created
right,
there's
a
whole
bunch
of
metadata
I
was
I
did
by
kubernetes
also
that
pod
creation
went
through
a
whole
lot
of
mutating
web
hooks
and,
for
example,
here
things
like
image
pool
policy
was
added.
Maybe
I
have
a
limit
range
in
the
cluster.
A
So
now,
suddenly
I
have
resources,
requests
in
the
pods
back
and
also
maybe
I
have
a
bunch
of
environment
variables
that
we
are
in
here.
That's
what
happens,
for
example,
if
you,
if
you
do
that
on
a
sure
kubernetes,
so
obviously
you
understand,
we
cannot
just
like
reflect
that.
Dp
call
these
two
resources.
It
doesn't
work.
What
can
you
do,
then?
A
Then
you
can
try
to
go
one
step
further,
like
you
would
build
like
custom
comparison
function
for
everything,
a
single
field
in
the
spec,
and
then
you
end
up
with
this
super
big
list
of
comparison
functions
that
you
execute.
There
are
very
hard
to
maintain
because
there's
always
something
you
didn't
plan
like
you.
Didn't
plan
this
new
label
to
be
parried,
you
didn't
plan
this
new
environment
variables,
so
you
need
to
touch
the
code
again
and
it's
never-ending.
A
A
A
So
the
world's
map
thing
here
is
that
the
actual
hash
is
built
at
creation
time,
so
it
does
not
care
at
all
about
any
field
that
was
patched
like
kubernetes,
like
any
metadata
that
was
added
any
any
new
environment
variables
or
whatever
and
mutating
webhook
patch.
So
really
we
compare
hashes
of
the
resource
at
the
time.
It
was
built
not
at
the
time
afterwards
that
it
was
patched
and
it's
pretty
interesting
to
use
whenever
you
need
to
reconcile
some
complex
resources.
A
Usually,
when
we
build
here
these,
we
try
to
build
an
abstraction
of
reality.
So
users
don't
have
to
care
a
lot
about
what
happens
on
the
underlying
Banaras
cluster.
You
want
to
deploy
a
3-node
elasticsearch
cluster.
You
don't
care
about
the
underlying
dots,
so
we
can
build
this
really
minimalistic
simplistic
version
of
the
CLD,
with
only
the
fields
you
care
about,
but
in
reality
what
happens
is
that
most
users
want
to
do
more.
They
ask
us,
hey.
A
Can
you
can
you
add
this
feature
so
that
whatever
pod
created
by
the
operator
actually
has
this
particular
label
or
maybe
give
me
the
power
to
change
the
affinity
rule
of
the
pulse?
So
I
can
do
zone
awareness
in
in
the
deployment
of
control
and
turns
out
best
way
to
do.
That
is
actually
to
empower
the
user
completely
to
do
whatever
they
want
in
the
co
D
and
that's
what
happens
when
you
create
a
stable,
set
or
deployment.
You
have
this
blood
template
that
you
can
specify
entirely
so
here,
for
example,
in
the
operator.
A
What
we
do
is
that
we
have
this
optional,
pod
template
section:
users
can
do
whatever
they
want
here
and
in
a
reconciliation
when
we
build
the
expected,
but
we
want.
We
take
that
template
from
the
user
and
we
build
on
top
of
it.
We
add
our
own
fields
in
there
and,
if
they're
already
specified
by
the
user,
we
don't
touch
it
that's
kind
of
the
user
responsibility,
so
they
can
shoot
themselves
in
the
food
if
they
want
to.
A
Finally,
I'd
like
to
end
this
presentation
with
a
few
things
about
stateful
sets
that
I
learned
system
volumes
that,
maybe
you
don't
know
about-
and
they
are
really
interesting
whenever
you
want
to
build
operators
that
are
managing
stateful
workloads
which,
by
the
way,
are
pretty
good
reason
to
use
an
operator
like
stateful
sets
by
themselves
on
ice.
But
if
the
workload
you
manage
is
a
bit
more
complex
than
what
the
state
full
sets
allows
you
to
do,
operators
a
pretty
good
way
to
solve
that.
A
One
first
thing
is
that
you
cannot
resize
volumes
in
stateful
sets
it's
not
supported
by
the
state
set
controller.
Actually,
if
you
create
a
stateful
set,
and
you
give
each
replica
ten
gigs
of
storage,
you
try
to
update
it.
You
cannot
because
that
section
in
the
stateful
sets
back
like
the
volume
clean
template
is
immutable.
You
cannot
change
that
locally.
There's
a
cap
in
progress
to
fix
that
I'm,
not
sure
it's
going
to
learn
in
the
next
version
of
kubernetes
that
there's
an
issue
for
rate,
at
least
so
it's
good.
So
what
can
you
do?
A
A
And
other
important
things
about
volumes
is
the
way
storage
classes
are
specified.
So
you
probably
know
that
whenever
you
use
persistent
volume,
what
you
usually
do
is
first
create
a
persistent
volume
claim
specifying
which
storage
class
you
want
and
then
the
controller
is
going
to
bind
that
claim
to
an
actual
persistent
volume.
A
That's
actually
what
the
state
fools
had
controller
does,
for
example,
it
just
created
system
volume
claim
and
then
creates
a
pod.
That
is
mapped
this
way
here
in
this
in
the
storage
class.
This
is
important
field
called
the
volume
binding
mode
and,
if
you
not
set
it
to
wait
for
first
consumer
here,
for
example,
what's
going
to
happen
is
that
you
create
a
stateful
set.
A
The
stateful
said
controller
whenever
it
needs
to
create
a
pod,
is
going
to
create
a
persistent
volume
claim
first,
that
persistent
volume
frame
is
going
to
be
bound
to
persistent
volume,
but
maybe
that
persistent
volume
has
some
affinity
constraints
so
that
the
volume
is
only
available
in
a
given
region,
or
maybe
it's
a
local
volume.
So
it's
only
available
in
a
particular
host
and
then
the
stateful
side
console
creates
the
part
mapping.
A
The
persistent
volume
claim-
and
what
could
happen
is
that
the
pod
itself
is
scheduled
with
its
own
Trainz,
so
maybe
on
a
particular
region
or
particular
host,
and
it's
possible
that
the
place
where
the
body
schedule
cannot
actually
access
the
volume
and
that's
a
big
problem
and
solve
that.
There's.
This
wait
for
first
consumer
option
so
that
the
body
schedule
first
and
then
the
volume
is
created
based
on
the
constraints
on
the
public
pot
and
I'm,
actually
surprised
that
most
persistent
volume
available
out
there
in
many
cloud
providers,
don't
have
this
enabled
by
default.
A
Don't
have
this
option
set
in
many
cases.
You
can
just
do
it
yourself.
You
can
touch
the
existing
system,
volume,
storage,
class
available
and
use
this
option,
or
maybe
just
clone
it
to
a
different
storage
class
with
a
different
name
which
have
hashed
this
option,
because
it's
really
important,
especially
when
dealing
with
local
volumes.
A
Now
what
happens
when
you
need
to
a
great
a
stateful
set,
because
the
spec
changed
normally,
what
will
happen
is
that
processing,
but
by
pod
the
stateful
set
controller
would
delete
a
pod.
Then
we
create
it
in
the
new
version
and
then
it's
going
to
reattach
automatically
the
same
system
volume
all
thanks
to
deterministic
naming
by
the
way
because
turns
out.
If
we
take
the
name
of
the
pod
and
we
use
the
same
name
for
the
volume,
then
it's
pretty
easy
to
match
them.
A
But
here
between
operation,
one
and
two,
what
could
happen
in
your
kubernetes
cluster
is
that
another
pod
get
scheduled
concurrently
and
it's
actually
taking
the
resources
that
were
necessary
to
recreate
that
pod.
That
was
deleted.
So
you
expect
stable,
set
upgrade
to
grow
through
like
pause
activity.
Then
we
created
one
by
one,
but
if,
in
the
meantime
another
pod
gets
allocated
through
a
particular
host
where
your
pod
needs
to
be
recreated,
then
you
have
a
problem,
because
that
pot
will
then
stay
pending
forever.
A
Then,
whenever
you
did,
will
we
do
with
stateful
sets
in
and
operate,
or
you
really
have
to
pay
attention
to
the
update
strategy.
You
pick
so
by
default.
Stateful
set
works
in
such
a
way
that
each
pod
gets
rotated
one
by
one
when
you
do
an
upgrade,
so
it's
based
on
the
willingness
of
other
pods,
if
all
three
replicas
already
I
kind
of
grayed,
the
first
one,
then
when
it's
ready
again
I
can
up
create
the
second
one
and
so
on
and
so
forth.
A
And
if
that's
fine
with
you
chances
are,
you
may
actually
not
need
an
operator
at
all
because
you
can
just
create
stateful
sets
and
it
just
works
right.
But
if
you
really
need
an
operator
of
something
else,
because
you
need
some
kind
of
control,
whether
we're
rolling
upgrades
are
done.
For
example,
then
probably
this
default
strategy
of
upgrading
will
not
be
good
for
you,
and
luckily
we
have
all
the
choices
here.
We
can
rely
on
a
rolling
or
gate
partition.
A
This
is
an
index
in
the
stateful
set
spec
that
says
every
replicas
having
an
ordinal
higher
than
that
particular
index
can
be
safely
rotated
and
upgraded
by
the
state
set
controller.
For
example,
if
I
have
three
replicas
and
in
my
operator,
I
I
set
this
rolling
update
petition
to
it
means
stable,
set
controller
can
safely
upgrade
the
replicas
on
the
three,
because,
probably
on
my
side
of
things
in
the
operator,
I
did
whatever
is
necessary
so
that
your
grid
of
that
bodies
isn't
is
fine.
A
Another
way
to
deal
with
that
is
to
use
the
undelete
strategy,
and
the
way
it
works
here
is
that
in
the
operator
yourself
to
have
the
responsibility
to
delete
the
part
you
want
to
upgrade,
because
you
know
they
can
be
safely
deleted.
Maybe
I,
don't
know
you
migrated
data
away
from
them
or
you
can
have
prepared
your
workload
for
that
restart
and
then,
as
soon
as
the
ability,
the
stateful
side,
controller
she's
in
a
way,
also,
an
operator
sort
of
controller
will
reconcile
the
stable
set.
Resources
again
is
going
to
recreate
the
pods
itself.
A
A
Finally,
what
I
wanted
to
say
about
stateful
sell,
is
you
don't
have
to
use
stateful
sets
to
manage
a
full
workload.
What
a
stateful
sets
really
is
just
a
way
of
creating
a
pod
and
binding
that
pod
to
persistent
volume
thanks
to
deterministic
naming.
You
know
that
if
that
pod
gets
killed,
I
need
to
recreate
it
and
the
volume
actually
already
exists
with
the
same
name.
You
can
just
match
and
bind
that
volume
mounted
to
that
boat
so
that
that's
actually
possible.
You
don't
need
the
state
to
set.