►
From YouTube: Automated, Distributed Systems Testing for Kubernetes Controllers - Lalith Suresh & Xudong Sun
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Automated, Distributed Systems Testing for Kubernetes Controllers - Lalith Suresh, VMware & Xudong Sun, University of Illinois at Urbana-Champaign
The Kubernetes ecosystem has thousands of controller implementations for different applications and platform capabilities. A controller’s correctness is therefore critical, and yet, can be compromised by myriad factors, such as asynchrony in the overall distributed system, unexpected failures, networking issues, and controller restarts. This in turn can lead to severe safety violations, such as incorrectly deleting StatefulSets and PVCs. Controller-developers unfortunately lack automated testing tools to harden their code against these conditions. In this talk, Xudong Sun and Lalith Suresh will describe common bug patterns in Kubernetes controllers. They will also present an automated testing tool called Sieve, which systematically tests Kubernetes controllers to harden them against the aforementioned scenarios. Sieve has already discovered (and led to fixes for) several safety-critical bugs in popular Kubernetes controllers for Zookeeper, Cassandra, RabbitMQ, MongoDB, XtraDB, etc.
A
So
for
the
purpose
of
illustration,
in
this
talk,
I'm
just
going
to
say:
hey,
let's
have
a
controller
manage
something
new
that
I'm
writing
called
my
app
whatever
it
is.
So
the
standard
way
to
go
about
this
is
to
register
a
new
kind
of
resource
in
the
kubernetes
api
that
describes
an
instance
of
my
app
and
this
resource
will
have
a
spec
field
which
says
what
the
desired
state
of
my
app
should
be.
A
Now
the
controller
monitors
the
current
state
of
my
app
and
it
issues
different
kinds
of
side
effects
in
order
to
reconcile
the
desired
state
and
current
state
of
this
myapp
instance
right,
and
in
order
to
do
so,
it
might
issue
more
commands
back
to
the
kubernetes
api
or
it
might
use
some
kind
of
out-of-band
mechanisms
to
reconfigure
my
app
right
so
now.
The
cache
here
is
that
this
controller
is
simply
just
one
component
in
a
fairly
complex
distributed
system,
so
the
cool-
and
why
is
that?
A
Because
the
kubernetes
api
itself
is
actually
a
bunch
of
api
servers
in
a
highly
available
setup,
backed
by
a
bunch
of
lcd
instances,
which
is
really
the
persistent
store
for
the
actual
cluster
state.
Now
the
controller
in
order
to
do
its
job
might
lean
on
some
built-in
controllers
and
kubernetes.
For
example,
it
might
be
managing
my
app
as
a
stateful
set
and
that
will
require
an
interaction
with
the
staple
set
controller
in
kubernetes,
or
it
might
even
rely
on
third-party
controllers
right
and
my
app
itself
might
be
this.
A
There
might
be
a
bunch
of
processors
running
in
different
parts
representing
something
like.
Let's
say
a
distributed
database
or
something
right,
so
the
controller
is
just
one
entity
in
this
fairly
complex
distributed
system,
and
that
means
the
controller
is
not
immune
from
any
of
the
distributed
systems.
He
challenges
that
any
system
has
to
deal
with
right.
Like
a
controller
has
to
deal
with
all
kinds
of
crashes.
A
It
has
to
still
do
the
right
thing
when
some
components
are
slow,
it
still
has
to
deal
with
things
like
missed
events
or
message
losses
because
of
again
crashes,
partition
bugs
and
whatnot
right
now.
The
tricky
part
is
it's
very
hard
to
make
the
controller
code
robust
to
any
of
these
kinds
of
fonts
right,
like
it's
hard
to
anticipate
how
things
can
go
wrong
and
when
things
do
go
wrong,
the
consequences
can
be
fairly
dire.
A
A
So
we've
used
sieve
to
study
several
kubernetes
controllers
already
and
across
the
board.
We've
never
had
to
make
any
source
modifications
and
all
the
bugs
we
found
in
our
process
of
hardening.
These
controllers
have
been
reported
on
that
url
there.
You
can
reproduce
any
of
those
bugs
at
your
own
convenience
right
and
we
invite
the
community
to
join
us,
help
develop
steve
and
make
it
a
tool
that
we
hope
kubernetes
controllers
will
increasingly
adopt.
B
A
A
System
we
typically
categorize
correctness
along
two
axis,
so
this
safety,
which
means
nothing
bad
ever
happens,
and
bad
of
course,
depends
on
the
controller
that
you're
trying
to
test
right.
So,
for
example,
accidentally
deleting
volumes
that
make
you
lose
data
is
a
very
bad
outcome
and
you
should
try
to
avoid
it.
The
other
kind
of
axis
is
liveness,
which
is
that
eventually,
the
system
does
what
it's
supposed
to
do
like
something
good
happens
eventually
and
in
the
context
of
controller.
A
This
means
that,
given
a
desired
state
under
control
and
a
current
state,
the
controller
should
eventually
drive
the
current
state
to
match
the
current
desired
state,
even
though
there
have
been
some
faults
injected
right
within
reason.
Obviously,
so
how
can
we
test
controllers
for
both
safety
and
liveness
in
a
systematic
fashion?
So
let
me
give
you
an
overview
of
how
c
works
in
this
regard.
A
A
So
now
the
first
thing
steve
does,
with
your
controller,
is
that
it
tries
to
go
into
what's
called
learning
mode
where
siem
is
trying
to
find
out
like
how
can
I
like?
When?
Should
I
restart
this
particular
controller
in
the
middle
of
an
execution
to
coax
it
into
a
certain
buggy
corner
right
or
when
do
I
have
to
lag
an
api
server
to
confuse
the
controller?
A
But
if,
let's
say
your
controller
is
a
cassandra
operator,
your
test
workload
might
be
something
like
create
an
application,
delete
it
and
create
it
again
right
and
so
now
what
sieve
does
is
that
it
takes
this
controller.
It
runs
its
side
and
it
runs
it
inside
this
kind
cluster
and
it
replace
this
test
workload
and,
on
the
way,
see
make
sure
make
sure
to
automatically
instrument.
A
The
controller,
runtime
and
client
go
libraries
that
this
controller
is
using
in
order
to
again
trace
the
controller's
execution
and
make
sure
we
have
a
hook
into
its
timings
right
so
now
see
it
does
all
of
that
it
runs.
The
controller
runs
the
test
workload.
It
collects
a
bunch
of
traces,
it
analyzes
the
traces
and
then
it
outputs
a
bunch
of
configuration
files.
Each
of
these
configuration
files
represents
a
suspected
event
sequence.
A
Basically,
it's
an
instance
where
sieve
saw
some
event
sequence
happening.
I
thought
when
this
event
happens.
If
I
inject
this
particular
fault,
something's
going
to
go
wrong
right,
it's
basically
a
hunch
that
c
has
and
a
triggering
event
might
be
something
like
hey.
A
stateful
set
with
a
particular
description
was
created
and
a
fault
might
be
crash.
The
controller
when
that
happens.
A
It's
basically
going
to
compare
the
traces
that
were
produced
in
the
learning
run
and
the
testing
run
or
put
differently.
It's
going
to
compare
how
the
system
behaved
with
and
without
the
injected
faults,
and
once
that's
done,
sieve
can
compare
these
two
runs
using
a
collection
of
general
purpose
checks
that
we
provide
and
in
the
future.
A
This
volume
with
this
particular
name,
had
a
15
gigabytes
of
capacity
at
the
end
of
the
learning
run,
but
in
the
testing
run
it
had
20
gigabytes
of
capacity.
So
what
happened
right
so,
when
steve
detects
these
kinds
of
inconsistencies,
it
flags
it
and
then
it
complains-
and
this
is
usually
the
pointer
to
the
existence
of
a
bug
and
c-
will
tell
you
exactly
why
it
thinks.
There's
a
bug
there
and
he'll
give
you
some
hints
on
how
you
might
want
to
go
about
debugging
it.
A
But
the
catch
is
that
this
file,
the
self-contained
file
that
describes
the
suspected
timing
sequence-
you
can
rerun
this
over
and
over
again,
you
can
rerun
the
the
testing
phase
over
and
over
again
and
it's
entirely
reproducible-
and
you
can
pass
this
on
to
your
colleagues
if
they
want
to
reproduce
this
bug
as
well.
So
with
that
I'm
going
to
pass
it
on
to
sudan,
who
will
show
you
an
actual
set
of
demos
that
reflect
this
workflow
in
the
context
of
certain
real
world
use
cases
over
to
you
should
on.
B
Let's
go
to
the
terminal,
as
mentioned
before,
the
user
to
uc
need
to
first
run
series
a
learning
mode,
so
that
c
we
can
learn
about
the
controller's
behavior
and
generate
some
even
sequences
which
will
guide
the
later
fault
injection
in
the
testing
mode.
So
here
to
uc,
let's
just
specify
to
make
the
tool
to
run
in
the
learning
modes
and
also
the
user
needs
to
specify
the
controller
to
run
here.
B
B
The
user
also
needs
to
specify
a
test
of
workload,
which
is
a
scale
down
here
to
run.
The
scada
is
a
test
workload
written
by
us.
It
simply
creates
a
customer
cluster
with
two
replicas
and
later
scale
down
to
one
rubric
and
of
course
the
user
can
also
use
the
existing
end
time
test
for
sale
to
run
and
after
typing.
These
commands
c
will
run
the
learning
modes
and
generate
some
even
sequence,
which
encodes
where
to
inject
the
work
handle
faults
to
save
some
time.
For
this
demo
we
have
already
run
the
command
before.
B
The
suspect,
even
sequence,
is
about
automatically
generated
in
this
folder.
You
can
see
how
c
generates,
in
total,
21
even
sequence
fail
after
running
the
learning
modes
and
each
event
second
field
here,
encodes
a
timing
which
still
found
promising
to
inject
the
faults,
and
since
we
get
the
event
second
file
from
the
learning
mode,
we
can
directly
run
the
testing
modes
to
the
text
box
and
similarly,
we
just
specify
c
w
to
run
the
testing
modes.
B
B
B
Basically,
during
the
testing
run,
still
will
perform
the
photo
injection
testing.
According
to
this,
even
sequence,
fill
and
the
test
workload
is
still
running
and
you
can
see
that
the
serial
is
trying
to
detect
absorbing
gaps
back.
So
there
is
a
running.
Let
me
first
explain
to
you
what
is
observable
gap
start
and
we
will
go
back
to
the
terminal
later
to
see
the
results.
B
B
We
know
that
the
controller
maintains
a
local
copy
of
the
cluster
state,
which
is
denoted
as
x1
here
and
besides.
The
controller
also
runs
the
reconciler
internally.
The
reconciler
is
run
like
a
loop.
It
basically
checks
the
cluster
state
and
decides
how
to
choose
the
desired
state
in
the
next
step.
B
So,
for
example,
let's
say
if
the
return
cylinder
at
this
moment
is
running
slow
and
has
not
finished
the
previous
reconcile
the
red
concealer.
So
it
has
not
seen
s2
yet
and
later.
The
controller
receives
a
second
event
which
updates
the
state
from
s2
to
s3,
and
the
controller
only
maintains
one
copy
of
the
current
cluster
state.
So
after
updating
s2
to
s3,
the
reconciler
will
never
be
able
to
cs2.
It
has
completely
missed
s2
and,
of
course,
some
other
figures
like
network
issues.
Node
crashes
can
also
make
the
reconciliation
meet.
B
B
So
the
important
question
simply
needs
to
answer
during
the
learning
mode.
Is
that
for
given
state
s
we'll
reconcile
the
goal
run?
If
the
cluster
speed
s
is
missed
by
the
reconciler,
so
basically
collect
the
trees
of
the
controller
analyze
the
trees
and
conduct
some
custodial
analysis
and
apply
some
heuristics
to
answer
the
question,
and
one
of
the
heuristics
we
use
here
is
that
sing
will
pick
a
state
as
if
it
triggers
a
controller
set
effect,
and
here
a
side
effect
represents
a
resource
equation.
B
So
you
basically
run
test
the
workloads
and
make
the
reconciliator
miss
success
by
injecting
delay
at
the
end
of
the
testing
c
will
compare
the
resource
status
during
the
learning
run
and
the
testing
run,
and
if
there
is
any
inconsistency
between
the
two
runs,
you
will
basically
send
about
reports
and
there
must
be
something
wrong
which
mix
causes.
This
inconsistency.
B
B
Which
is
the
fault
free
run?
There
is
only
one
pvc,
but
after
the
testing
run
still
found
that
there
are
two
pvc
left
and
we
can
try
to
get
to
the
post
and
to
see
what's
going
on
so
by
getting
the
post.
You
will
find
that
except
the
operator
port,
there's
only
one
customer
post
running
here
and
recall
that
we
scale
down
the
personal
cluster
side
from
two
to
one,
so
one
consumer
port
is
actually
correct.
B
B
So
there
must
have
been
something
wrong
which
causes
this
inconsistency
and
makes
this
like
off
from
pvc
happening.
But
we
don't
know
the
root
cause
at
the
moment
to
help
us
debug.
The
problem
c
will
also
provide
some
debugging
suggestion.
So
here
sim
tells
us
what
happened
during
the
testing
role.
It
makes
the
reconcile
miss
the
state
and
the
state
is
basically
simplified
by
c
sim
highlighters
as
a
state
inside
the
state.
B
B
It
has
not
seen
addition
timestamp
in
s2
yet
and
since
the
poll
does
not
have
any
finalizer,
the
quantities
will
directly
go
ahead
and
detail
the
post.
So
very
soon
the
controller
will
receive
a
second
event
which
deleted
the
port
from
s2
and
now
in
a
new
state
as
three
of
votes
as
well
as
decision
timestamp
is
gone
and
now
the
reconciler
has
completely
missed
addition.
Timestamp
in
the
s2
and
the
reconciler
will
never
be
able
to
delete
the
pvc
used
by
the
hidden
port.
B
B
B
Civil
is
architected
as
follows.
First,
we
have
a
seal
tester
driver
which
accepts
the
user
command,
set
up
the
current
cluster
and
submit
as
a
workload.
The
tesla
driver
is
implemented
as
python
script
as
you'll
see
in
the
demo
and
in
the
client
cluster
every
eps
server
and
the
controller
runs
a
civ
client
internally.
B
The
sim
client
collects
the
runtime
information
from
the
eps
server
and
controller
site.
Then
report
the
information
to
the
shift
server
during
learning
modes.
Save
server
is
the
process
running
in
the
commodities
cluster
and
the
safe
server
will
gather
all
the
information
and
formal
controller
trees
and
report
the
controller
trees
to
the
same
analyzer.
B
B
B
B
B
Basically,
the
analyzer
enforces
the
event
received
by
the
controller
later
due
to
the
effect
issued
by
better
controller
and
depending
on
which
pattern
we
want
to
test
c,
will
perform
different
processing
and
generate
the
suspend
given
segments
or
different
bug
patterns,
for
example,
for
testing
observability
caps
bag
say
we
are
trying
to
make
the
reconciler
miss
the
state
updated
by
a
particular
event.
If
the
event
can
lead
to
some
static
factor
here,.
B
Sieve
is
easy
to
use
as
it
requires
no
source
modification
from
the
developer
and
it
can
reliably
reproduce
the
bugs
it
found
before
see
with
open
source
and
github
since
runs
in
two
modes.
A
learning
mode
will
still
learn
about
the
controller's
behavior
and
generated
even
segments
and
a
testing
mode.
There
is
an
injector
fault
according
to
the
event
sequence
and
trigger
box.