►
From YouTube: Leveraging State Machines to Build Operators in Rust - Kevin Flansburg, Moose Consulting
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Leveraging State Machines to Build Operators in Rust - Kevin Flansburg, Moose Consulting
This talk will introduce and demonstrate Krator, a crate for building state-machine-based Operators in Rust. Last fall, the Krustlet team developed a state machine API for describing the Pod lifecycle for their Rust implementation of Kubelet. Based on interest from the community, this API was split out into a new crate (Krator) to allow it to be used to implement arbitrary Operators. First, the talk will walk through the process of developing a simple Operator with Krator, including a live demo. Next, it will explore the API in more detail, including design decisions that were made and how it meshes well with many Kubernetes design philosophies. Finally, it will outline some new and interesting Operator functionality that has been introduced, beyond what was needed to support Krustlet.
A
A
A
So
that's
how
crater
was
started.
I
split
that
out
in
december
and
january
and
then
crater
was
officially
launched
in
a
blog
post
on
february.
1St
and
crater
stands
for
kubernetes
rust
state
machine
operator.
Next,
a
little
bit
of
prior
art.
There's
lots
of
frameworks
out
there
for
developing
operators.
A
A
The
the
helm
variant
has
a
limited
subset
of
functionality,
but
it
allows
you
to
get
deploying
oper
your
operator
very
quickly
by
simply
supplying
a
helm
chart
and
then
the
custom
resource
definition
spec
contains
helm,
chart
values
which
are
populated
and
then
applied,
and
then
the
operator
ensures
that
the
helm
chart
stays
consistent
with
cluster
state,
other
languages,
such
as
python
and
and
ruby,
etc.
All
have
controller
frameworks
which
tend
to
sit
on
top
of
the
main
kubernetes
client
library.
For
that
language.
A
A
You
you
have
to
sort
of
explicitly
define
that
watching
logic,
and
then
you
have
methods
that
you
populate
for
create,
update
and
delete
events
for
for
your
managed
resource,
and
these
methods
can
be
end
up
being
quite
large
and
monolithic,
and
it
doesn't
provide
a
great
structure
for
you
to
define
your
operator's
logic
in
so
more
sophisticated
frameworks
like
the
go
operator.
Sdk
do
a
lot
of
code
generation
for
you.
They
wrap
the
logic
of
your
controller
with
a
lot
of
boilerplate.
A
So
the
state
machine
api
was
an
attempt
to
increase
the
reliability
of
of
implementing
this
kind
of
application
logic
and
break
up
these
monolithic
functions.
So
the
general
approach
is
that
for
each
sort
of
area
of
concern
that
you
might
have
in
your
application's
logic,
so
an
example
would
be
pulling
images
for
pods
or
mounting
volumes.
These
would
be
separate
areas
of
concern.
You
would
define
a
node
in
your
state
machine
graph
and
you
can
define
an
arbitrary
state
machine
graph
for
your
application.
A
These
nodes
are
intended
to
be
infallible
and
if
they
do
encounter
an
error,
they
should
transition
to
an
explicit
error.
Node
in
the
state
machine
graph,
which
allows
better
error,
handling
and
and
retrying
and
things
like
that,
and
then
the
state
handlers
I
use
the
term
I
use
as
handlers
and
that's
the
actual
body
of
code,
that's
executed
within
each
node
in
the
state
machine
graph.
A
A
So
you
can
ensure
that
the
state
of
your
object
only
ever
transitions
between
states
that
you've
considered
and
it
only
transitions
to
well-defined
states
and
by
using
infallible
state
handlers,
we're
able
to
sort
of
avoid
just
completely
dropping
the
management
of
objects
and
better
encode.
The
retry
logic
that
you
may
want
to
use
with
your
your
application
and
then
some
other
things
that
you
get
out
of
the
box.
A
Next
is
the
state
trait.
This
is
to
be
implemented
for
each
state
in
your
state
machine
graph,
so
you'll
implement
this
a
number
of
times,
and
this
has
methods
for
defining
the
status
that
should
be
reported
to
the
kubernetes
api
when
entering
this
state
and
then,
of
course,
the
state
handler
is
the
the
next
function
which
returns
a
indication
of
which
state
should
be
transitioned
to
or
whether
the
state
machine
should
exit
the
remaining
two
traits
are
used
to
define
types
that
will
be
provided
to
your
your
state
machine.
A
A
I
use
cube
the
cubecrates
custom
resource
derived
macro
to
specify
the
type
for
my
custom
resource,
and
this
is
great
because
I
just
have
to
specify
the
contents
of
the
spec
field
and
it
derives
the
rest
of
the
type.
But
one
thing
that
I'd
like
to
call
out
is
that
crater
requires
that
the
status
field
is
used.
That's
optional,
on
custom
resources,
but
it's
required
by
crater.
A
A
There's
a
trait
implementation
for
this
object,
which
specifies
the
custom
resource
type
that
we
defined
above
the
status
type
for
that
resource,
and
then
there
is
a
type
for
sharing
state
between
objects
which
is
defined
below
and
then
finally,
there's
an
async
drop
handler.
If
you
run
need
to
run
asynchronous
code
when
the
object
is
deregistered.
A
A
It
has
access
to
the
state
for
this
specific
object
which
it
has
a
mutable
reference
to,
because
that
is
owned
by
this
object,
and
then
it
has
access
to
this
manifest
type,
and
this
manifest
type
implements
streams.
So
you
can
watch
for
changes
to
the
the
object
and
it
also
has
a
method
called
latest
which
supplies
the
latest
copy
of
the
object
immediately.
A
So
you
can
see
here,
I
run
some
code
to
to
create
register
this
object
in
the
shared
state,
and
then
I
call
transition
next
and
tell
crater
which
state
I'd
like
to
transition
to
now
in
order
to
transition
to
a
state.
I
have
to
implement
transition
to
and
explicitly
tag
that
transition
as
as
valid
with
the
compiler,
and
this
is
to
improve
the
the
rigor
of
the
state
machine
implementation
and
ensure
that
invalid
state
transitions
are
not
taken.
A
A
The
moose
will
do
that
until
it
gets
hungry
and
with
some
probability,
it'll
make
a
friend
and
update
that
shared
state
map
when
it
gets
hungry
it
transitions
to
the
eating
state.
This
simply
waits
for
some
duration
and
then
replenishes
the
moose's
food
and
then
transitions
to
the
sleeping
state
from
the
sleeping
state.
We
simply
wait
20
seconds
and
then
transition
back
to
roaming
and
then
the
final
state
is
a
deleted
or
or
deregistration
state.
A
I
have
a
simple
hash
map
of
relationships
between
mooses
and
then
I
implement
moose
tracker.
This
is
the
type
that
will
implement
our
operator
trait,
and
these
are
created
essentially
as
singletons
so
scrolling
down.
This
is
the
implementation
of
the
operator
trait.
I
again
reference
the
status
and
resource
type
definitions
from
above.
I
indicate
that
the
tag
state
is
the
first
state
that
should
be
entered
when
an
object
state
machine
is
created.
A
A
That's
that's
trying
to
be
changed
or
deleted
or
created,
and
of
course
you
can
reference
any
shared
state
on
the
operator.
If
you
need
to
validate
within
sort
of
the
context
of
all
of
the
objects
that
the
operator
is
managing
and
then
here
I'm
simply
validating
that
the
moose's
name
starts
with
the
letter
m
and
then
I
can
allow
the
object
or
the
the
change
to
to
occur.
And
then
I
can
optionally
mutate
the
object
here.
A
I'm
I'm
leaving
it
unmutated
or
I
can
deny
the
object
and
return
kind
of
a
standard,
kubernetes
status,
which
indicates
why
this
has
been
denied
and
that
will
be
returned
by
the
kubernetes
kubernetes
api
to
the
client
so
scrolling
down
a
little
bit.
This
final
piece
of
code
is
our
main
function
and
it's
all
you
need
to
do
to
actually
start
an
operator
with
crater.
So
I
create
a
cube
config,
which
I
simply
infer.
A
I
create
my
operator
singleton
and
then
I
call
operator
runtime
new
and
pass
it
that
cubeconfig
and
that
operator
singleton,
and
then
I
can
optionally
supply
list
parameters
to
filter
the
objects
that
I'm
managing
and
then.
Finally,
I
call
runtimestart.weight
and
that'll
block
forever
and
it
will
spawn
asynchronous
tasks
for
the
execution
of
the
state
machines
for
each
object.
A
So
if
I
run
this
it'll,
I'm
in
the
wrong
directory
there,
we
go
it'll
print
out
the
crd
here,
but
it
doesn't
actually
apply
it.
The
crd
is
located
in
the
assets
directory
within
the
examples
folder.
So
you
can
apply
this
yourself
if
you
want
to
play
around,
but
you
can
see
here
that
there
was
already
a
moose
registered
with
the
api,
so
we
got
a
resync
event
when
we
started
watching
for
mooses,
we
created
an
event
handler
here.
That's
creating
a
state
machine
for
this.
This
moose.
A
A
Excuse
me:
there's
a
status
update,
that's
patched
to
kubernetes
when
it
enters
this
state
and
then
next
it
transitions
to
the
roaming
state
and
there's
a
status
update
for
that,
and
then
the
moose
will
continue
to
move
through
its
states
as
the
operator
progresses.
So
here
in
canines
you
can
see.
I
can
list
mooses
and
I
have
set
certain
fields
on
the
status
to
be
printed
out
when
you
use
cube
cuddle,
and
so
we
can
see
the
moose
transitioning
through
the
states.
A
So
here
I
generate
25
new
mooses
and
the
amount
of
time
that
the
moose
spends
in
the
roaming
state
is
determined
by
its
body
weight
that
that
determines
how
long
it
takes
to
get
hungry
and
and
the
body
weight
is
sampled
from
a
normal
distribution.
So
the
mooses
should
go
through
the
state
machine
at
different
rates,
which,
which
makes
it
a
little
bit
more
interesting
and
then
finally,
I
can
demonstrate
the
the
web
hook
capability.
So
I
have
a
moose
here
named
george,
which
is
an
invalid
name.
A
According
to
my
rule-
and
so
I
can
apply
this
manifest
and
you
can
see
in
the
log
that
the
request
was
denied
that
went
by
quickly,
but
you
can
also
see
that
the
response
returned
to
the
client
is
that
this
is
a
invalid
moose
name.
So
hopefully
this
has
been
a
a
nice
overview
of
the
api
and
the
functionality
that
you
can
implement
fairly
concisely.
Using
the
crater
api.
A
Finally,
I'd
like
to
expand
the
templating
and
sort
of
boilerplate
generation
surrounding
creating
and
deploying
operators.
This
is
something
that's
offered
by
the
go
sdk
and
I
think,
there's
a
lot
of
room
for
improvement
here
with
crater,
and
then
I'd
like
to
look
at
the
operator
sdk's
capabilities
roadmap
for
ways
that
I
can
offer
very
simple
apis
to
implement
some
of
the
more
advanced
features
such
as
auto
scaling
and
abnormal
state
detection.