►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubelet Deep Dive: Writing a Kubelet in Rust - Kevin Flansburg, Moose Consulting
Kubelet is a critical part of the Kubernetes project. Kubernetes deployments can vary a great deal in terms of container runtime, self-hosted or static control plane, CNI provider, etc., but they must all have Kubelet running on each node host. Many intermediate Kubernetes users could benefit from a deeper understanding of Kubelet behavior. This talk discusses the development of a Kubelet in Rust, and offers a deep dive into the expected behavior and implementation of Kubelet. The talk will begin with a discussion of how Kubelet fits into a Kubernetes deployment, and its relationship with Kubernetes Operators and the Container Runtime Interface (CRI). Next, the talk will cover important crates for Kubernetes development in Rust, as well as the development of Operators and use of gRPC. Finally, the talk will end with pros and cons of using Rust today for Kubernetes development.
https://sched.co/ekDy
A
Hello,
I'm
kevin
flansberg
and
welcome
to
cubelet
deep
dive
writing
a
cubelet
in
rust.
This
talk
covers
cubelet
architecture
and
behavior
in
detail,
which
was
learned
whilst
rewriting
cubelet
in
rust.
It
also
covers
the
benefits
of
rust
for
writing,
distributed
applications
such
as
kubernetes
components.
A
First,
a
little
bit
about
myself.
I
have
been
a
rust
developer
for
around
three
years.
During
this
time,
I've
been
responsible
for
developing
and
deploying
globally
distributed
microservices
as
such,
I'm
pretty
familiar
with
the
rust
software
development
lifecycle
from
running
git,
init
to
ci,
cd
and
finally,
monitoring.
My
devops
journey
began
with
docker
swarm.
However,
around
a
year
ago,
I
migrated
to
kubernetes,
I
have
found
it
to
be
a
very
powerful
tool,
even
in
a
single
developer
shop.
A
A
A
I
have
also
begun
work
on
a
cubelet
implementation,
which
targets
traditional
linux
containers
and
makes
use
of
the
container
runtime
interface
in
the
rest
of
the
talk.
I
will
focus
primarily
on
cubelet
in
the
abstract
or
as
it
pertains
to
linux
containers,
but
I
think
the
use
of
wasm
within
kubernetes
is
very
exciting,
and
I
encourage
you
to
check
out
these
projects.
A
A
These
can
run
directly
on
the
host
in
control,
plane,
nodes
or
in
a
self-hosted
cluster
in
pods
in
the
cluster
itself
or
in
the
case
of
many
cloud.
Offerings
are
managed
by
the
vendor
and
only
the
api
server
endpoint
is
made
available
to
the
customer
within
the
control
plane.
You
have
etcd,
which
is
a
distributed.
Key
value
store
that
kubernetes
uses
to
persist.
Cluster
state.
A
The
scheduler
is
responsible
for
assigning
pods
to
nodes,
and
the
controller
manager
runs
cluster
control
loops,
such
as
adding
endpoints
to
services,
provisioning
service
accounts
for
new
namespaces
and
pruning
dead
nodes.
These
services
leverage
an
extremely
central
concept
to
kubernetes
the
controller
pattern.
A
A
large
number
of
kubernetes
components
use
this
pattern.
It
begins
with
declarative
manifests,
which
represent
instances
of
the
various
api
resources
that
the
cluster
supports.
Most
users
will
be
familiar
with.
These
manifests
in
the
form
of
yaml
files,
which
are
submitted
to
the
cluster
to
create
resources
such
as
deployments
and
services.
A
These
manifests
are
mostly
immutable,
but
you
can
find
some
exceptions
controllers
within
the
cluster
watch
for
changes
to
the
types
of
resources
they
manage.
These
changes
are
typically
the
creation
of
a
new
instance
of
the
resource,
modifications
or
deletion
watching
a
resource
for
these
types
of
changes
is
typically
referred
to
as
the
informer
pattern.
A
When
changes
are
detected,
the
controller
will
then
drive
cluster
state
to
match
the
desired
state
specified
in
the
manifest.
In
this
way,
different
components
with
different
responsibilities
ensure
that
the
cluster
is
eventually
consistent
with
its
desired
state.
As
an
aside,
there
is
also
the
operator
pattern
which
you
will
see
pretty
often
when
working
with
kubernetes
the
distinction
between
an
operator
and
a
controller
is
fairly
subtle,
but
an
operator
is
essentially
a
controller
that
defines
a
custom
api
resource
and
is
application
or
domain
specific.
A
There
is
a
grpc
endpoint
for
this
communication
called
the
container
runtime
interface,
which
was
introduced
in
2016..
It
is
not
100
adopted,
however,
and
you
can
still
find
some
cubelets
configured
to
use
docker
with
its
daemon
socket.
Additionally,
kubota
exposes
an
http
endpoint
for
streaming
logs
and
exec
sessions
to
clients.
A
First
class
support
for
this
via
the
api.
Server
and
cube
cuddle
is
one
of
the
special
things
about
cubelet
for
configuring
storage,
such
as
block
volumes.
There
is
the
container
storage
interface,
another
grpc
protocol,
endpoint,
which
cuba
interacts
with
legacy
storage
drivers
were
originally
included
in
the
cubelet
source
code,
so
you
may
find
some
in
the
wild
that
do
not
make
use
of
csi.
A
A
The
fact
is:
pods
are
very
complex
if
the
ops
community
has
learned
one
thing
over
the
last
few
decades,
it
is
that
hosting
applications
involves
a
lot
of
moving
parts.
Cubelet
must
interact
with
many
other
components
and
does
not
just
act
as
a
shim
to
the
container
run
time,
for
instance,
at
some
point
it
must
fetch
and
configure
secrets
and
config
maps,
respect
the
image,
pull
policy
and
mount
service
account
tokens.
A
The
pod
spec
allows
for
a
lot
of
customization
of
behavior,
which
requires
a
lot
of
kubernetes
specific
decisions
to
be
made
by
cubelet
before
the
runtime
can
start.
The
container
I'd
like
to
spend
some
time
discussing
why
I
think
rust
is
a
great
language
for
developing
distributed.
Applications
like
cubelet
first
rust
can
produce
very
high
performance
software
and
frequently
matches
c
plus
in
performance
benchmarks.
A
The
result
of
this
focus
on
performance
is
not
only
the
ability
to
scale
but
also
efficiency,
which
can
have
a
big
impact
in
the
data
center
and
at
the
edge
next
rust
employs
a
strong
type
system,
as
well
as
a
borrow
checker,
which
enforces
memory
safety.
At
compile
time,
many
first-time
users
of
rust
quickly
grow
irritated
with
the
compiler
and
borrow
checker.
A
A
A
A
The
developers
of
rust
have
also
done
a
lot
of
work
and
continue
to
improve
the
error
messages
produced
by
the
compiler.
While
not
all
error
messages
are
the
most
informative.
There
is
a
great
framework
in
place
for
extremely
descriptive
messages
which
underline
the
exact
code.
The
error
is
referring
to
and
suggest
exactly
what
needs
to
be
changed.
A
Rust
also
has
an
excellent
ecosystem,
I'd
like
to
mention
some
crates,
which
are
extremely
high
quality.
The
first
is
sergei,
which
implements
serialization
and
deserialization.
I
genuinely
miss
serde
in
every
other
language.
I
use
many
offer
similar
options,
but
they
are
simply
not
as
useful
or
feature
complete
sure
automatically
derives
the
ability
to
convert
rust
types
to
serialized
strings
such
as
json,
yaml
or
avro,
and
vice
versa.
A
A
A
A
This
makes
it
incredibly
easy
to
develop
rust
code
to
interact
with
grpc
endpoints
like
cri
and
csi.
Rust
has
also
made
great
documentation
a
major
focus.
It
has
first
class
language
support
for
doc,
strings
and
rust
dock,
which
ships
with
rust
can
be
used
to
generate
great
documentation
pages,
including
runnable
examples.
A
Rust
also
has
strong
support
for
dependency
management
through
its
package
manager,
cargo,
when
working
with
crestlet,
we
are
balancing
a
number
of
rapidly
evolving
dependencies.
Changing
kubernetes
api
versions
and
complex
combinations
of
libraries
and
standalone
binaries
in
a
single
project.
Cargo
makes
it
a
breeze
to
manage
all
of
this
and
is
a
very
well
thought
out
tool.
A
A
I
would
also
like
to
mention
some
useful
crates
specifically
for
kubernetes
development
in
rust.
Kate's
open
api
contains
automatically
generated
types
for
the
kubernetes
open
api
spec.
It
is
very
useful
for
manipulating
kubernetes
manifests
and
its
documents
are
actually
the
main
docs
I
use
for
referencing
kubernetes
api
resources.
A
Second
is
cube,
which
is
the
primary
kubernetes
client
for
rust.
It
is
what
crestlet
uses
to
parse
cube
configs
connect
to
the
api,
server,
patch
resources
and
watch
for
pod
changes.
Finally,
kate,
cri
and
kate
csi
are
crates
that
I
have
published,
which
provide
automatically
generated
grpc
clients
for
cri
and
csi
respectively.
A
Let's
take
a
closer
look
at
the
control
loop
used
by
cubelet
to
run
a
pod.
This
is
the
loop
that
was
developed
for
the
crestlet
project
based
on
observed
behavior
in
kubernetes.
When
a
pot
is
added,
some
validation
happens
and
then
the
image
pull
policy
is
evaluated
to
determine
if
images
need
to
be
pulled.
A
A
A
This
is
a
brief
overview
and
there
are
a
lot
of
details
surrounding
the
behavior
of
cubelet,
but
I
think
this
provides
a
good
working
outline
for
debugging
pods.
Note
that
inherent
to
this
is
our
control
loop.
While
sapod
exists,
the
graph
tries
to
get
us
into
the
running
state.
The
only
way
to
get
to
states
that
actually
exit
the
loop
are
under
specific
conditions,
such
as
pod
deletion
or
a
restart
policy
of
never
for
crestlet.
A
We
spent
quite
a
bit
of
time
exploring
how
we
could
best
implement
this
control
loop,
while
still
allowing
downstream
developers
to
write
highly
specialized
cubelets.
What
we
developed
is
thoroughly
documented
in
a
blog
post.
I
wrote
earlier
this
year
to
summarize:
we
released
a
rust
api
for
building
a
state
machine
which
captures
the
logic
of
the
control
loop
and
leverages
rust's
type
system
to
ensure
correctness.
A
This
state
machine
is
fully
customizable
by
the
developer.
The
state
machine
has
a
number
of
constraints
which
we
believe
improve
the
reliability
of
the
application,
and
these
are
enforced
at
compile
time.
We
ensure
that
only
valid
states
are
used
and
only
valid
transitions
between
states
are
taken.
A
A
A
A
The
pod
driver
will
run
the
handler
associated
with
this
initial
state,
which
will
return
the
next
state.
The
pod
driver
will
iteratively
execute
these
state
handlers
until
either
a
state
returns
saying
that
it
is
an
end
state
such
as
failed
or
succeeded,
or
the
pod
is
deleted
when
a
pod
is
deleted
or
modified.
The
pod
event
dispatcher
will
notify
the
appropriate
pod
driver
in
step
3
after
a
pod
is
deleted.
A
A
Rust
gives
our
small
team
the
confidence
and
ability
to
iterate
on
apis
and
rapidly
add
features
while
avoiding
entire
classes
of
errors
and
maintaining
high
code
quality.
Rust's
ecosystem
provides
many
high
quality
crates.
A
helpful
community,
straightforward
dependency
management
and
automatic
high
quality
documentation.
A
I
feel
that
complex
and
highly
concurrent
distributed
applications
like
this
are
especially
suited
for
rust,
and
I
hope
you
will
check
it
out
if
you
haven't
already
I'd
like
to
wrap
up
with
some
key
takeaways
from
this
talk.
First,
we
covered
cubelet
architecture
and
communication
patterns,
including
the
components
that
it
interacts
with
and
how
it
fits
into
a
kubernetes
cluster.
I
hope
this
information
will
be
useful
to
you
in
the
future
for
debugging
and
administration
tasks
related
to
cubelet.
A
A
I'd
also
like
to
shout
out
the
core
maintainers
of
the
crestlip
project,
taylor,
matt
and
ivan,
who
have
been
great
to
work
with,
as
we
try
to
roll
out
some
very
ambitious
features.
If
you'd
like
to
contribute
to
crustlet,
you
can
find
us
on
github
and
it
would
be
a
great
way
to
start
working
with
rust.