►
From YouTube: KBE Insider (E14) - Lalith Suresh and Michael Gasch
Description
KBE Insider interviews Lalith Suresh, researcher at VMware and Michael Gasch, formerly a staff engineer in the Office of the CTO at VMware. They both contribute to the Sieve open source project, automatic reliability testing for Kubernetes controllers. We’ll dive into how the Sieve project got started, what’s next with the project, as well as how you can contribute.
A
A
Sorry,
I'm
actually
hearing
myself
on
a
like
a
five
second
delay.
Hopefully
that
will
be
better
now
all
right
so,
like
I
said
a
few
minutes
ago,
if
we
don't
have
a
couple
of
technical
difficulties,
we're
afraid
the
show
is
gonna
go
terrible,
so
so
that
works
out
well
yeah.
So
I
think
I
had
a
twitch
stream
running
in
the
background
that
was
off
mute,
which
is
always
annoying,
but
all
right.
B
Sure
langdon
thanks,
I'm
josh
wood,
I'm
currently
a
developer
advocate
at
red
hat.
My
focus
is
on
openshift
and
particularly
interesting
for
today's
guests
and
the
project
that
we'll
be
talking
with
them
about
kubernetes
operators
and
how
they
work
at
the
core
of
openshift
to
deliver
auto
updates
and
management
of
foundation
software
on
the
platform
and
where
we
see
a
lot
of
people,
building
cool
new
stuff
sort
of
in
the
space
and
on
the
scaffolding
that
the
operator
pattern
or
concept
and
the
toolkits
around
it
represent.
A
Nice,
it's
always
nice
when
you
can
have
a
tool
that
does
its
own
like
installation
and
kind
of
updating
right,
because
then
you
don't
have
to
figure
out
how
to
get
it
right.
You
know
this.
We've
we've
celebrated
this
in
the
linux
world
right
with
package
managers
for
a
long
long
time-
and
you
know
I
don't
it's
so
painful-
to
use
something
like
windows
where
you
have
to
go
manually,
update
things
all
the
time,
although
it's
getting
much
much
better.
C
C
A
D
Sure,
hey
everyone
thanks
for
for
having
me
on
the
show,
I'm
michael
garch,
I'm
based
out
of
germany.
I
hope
my
audio
is
good.
Now
we
had
some
technical
issues,
as
you
said
earlier,
and
as
of
today,
I'm
still
with
vmware,
but
on
september
1st
I'll
be
rolling
into
a
new
adventure
as
part
of
my
career
I'll,
announce
on
twitter,
I
think
maybe
tomorrow
or
so
so
stay
tuned
and
follow
me
on
twitter.
If
you're
curious,
where
I'm
headed
to.
A
Cool
yeah,
and
so,
if
you,
if
you
want
to
follow
both
these
folks
on
twitter,
you
can
find
their
twitter
handles
as
well
as
ours,
on
the
notes
on
the
show
on
cube
by
example.
But
why
don't
we
get
right
into
it?
And
so
the
first
thing
we
like
to
ask
our
guests
is
you
know,
kind
of
what
got
you
into
open
source?
C
Yeah,
I
can
start
so
yeah.
This
goes
back
to
my
undergraduate
days
when
I
was
first
introduced
to
things
like
linux
and
foss,
and
I
used
to
attend
a
lot
of
these
open
source
conferences
back
back
home
during
that
time
and
then
for
me,
like
sort
of
the
sort
of
the
first
time
that
I
started
becoming
a
contributor
was
with
the
ns3
project.
C
B
C
C
Of
the
wheel
in
cluster
management,
right
problems
like
software
yeah
or,
like
I
don't
know,
say
rolling
upgrades
all
of
these
things.
You
know
the
concepts
are
very
similar,
except
reinventing
that
wheel
in
every
new
system
that
we
come
across
and
in
one
particular
project
where
I
was
looking
at
you
know,
how
do
you
write
things
using
declarative,
programming
and
stuff?
You
were
like
hey.
Let's
we
have
this
cool
technique,
we're
sure
it
makes
it
easier
to
build
certain
things,
it's
the
second
and
third
times.
C
D
A
A
B
D
Yeah
exactly
because
I
also
had
like
two
entry
points
into
open
source.
The
first
one
was
2002,
where
I
was
part
of
a
training
program
at
the
research
institute
and
there
they
were
all
doing
linux
stuff,
and
I
was
tasked
to
migrate.
The
windows,
nte
domain
system
and
file
services
towards
samba,
some
of
the
file
services
and
by
the
time
samba
was
moving
from
version
two
to
version
three,
and
it
was
an
alpha
one.
So
I
was
like
there
was
a
lot
of
bugs
and
and
issues.
D
D
I
built
a
lot
of
systems
on
linux,
a
lot
of
file
services,
parallel
file
services
systems,
and
then
years
many
years
later,
in
2015
I
joined
vmware
and
I
was
going
through
an
onboarding
phase
at
the
ember
doing
all
that
the
hiring
stuff-
and
I
saw
a
talk
from
somewhere
from
some
some
of
our
colleagues
talking
about
messers
and
kubernetes,
and
I
got
somehow
intrigued,
even
though
my
role
did
not
require
that
knowledge.
By
the
time.
D
D
So
I
was
working
on
this
on
this
blog
post
about
kubernetes
and
lcd,
and
you
know
how
the
informals
work
and
the
controllers
and
all
these
patterns-
and
I
was
writing
this
article
and
lalis
called
me
and
said
you
know,
michael,
we
have
a
question
on
etcd.
Maybe
you
can
help
us
like.
I
was
like
this
is
great
because
I
was
just
literally
typing
the
answer
to
your
question.
In
you
know
the
unreleased
blog
post
back
back
in
the
I
think
it
was
two
years
ago,
maybe
yeah,
that.
A
C
C
Coming
from
sort
of
like
an
act
like
the
academic
half
of
me
right,
I
I
think
it's
one
of
the
best
ways
to
learn
how
real
world
software
works.
It's
right
out
there
right,
like
you,
don't
have
to
I.
I
really
credit
a
lot
of
my
education
in
computer
science
to
open
source
right.
I
never
had
to
wonder
what
these
concepts
in
the
books
meant
when
I
could
just
look
at
real
world
software
and
see
how
it's
actually
implemented
right,
like
whether
it's
the
jvm
or
python
or
whatever,
like
hadoop.
C
All
of
these
things
are
right
out
there
right.
So
there's
always
that
aspect
of
open
source
that
appeals
to
me
right.
There's
all
these
widely
used
software
that
we
can
actually
see
how
exactly
how
it's
coded
right.
You
don't
have
to
guess
the
other
aspect
of
it
is
just
it's
more.
It's
just
easy
to
get
people
on
board
and
collaborate.
A
Yeah
totally
yeah,
it's
interesting.
My
my
my
experience
in
university,
though,
is
that
they
they
a
lot
of
students,
are
unfamiliar
with
open
source
and
I
find
it
kind
of
mind-blowing.
You
know
because,
like
when
I
was
a
college
student,
I
was
always
looking
for
things
that
were
cheap
or
free
right
and
you
know
if
nothing
else
right
you
get
that
advantage
with
open
source
and
so
one
of
the
things
one
of
my
kind
of
quote-unquote
missions.
A
Since
I
joined
academy
you
know
academia
is
to
try
to
make
open
source
more
available
and
more
prevalent
amongst
university
students.
So,
let's
move
on
and
talk
about
steve.
You
know
this
is
one
of
those
words
that
I
learned
from
reading.
So
I
pronounced
it
civ
for
a
long
time.
But
can
you
tell
us
a
little
bit
more
about
the
project
like?
Where
does
it
come
from?
You
know:
what
does
it
do.
C
Yeah,
so
the
the
genesis
of
this
project
was
another
project,
like
I
said
where
I
was
trying
to
build
a
kubernetes
scheduler,
and
I
was
running
into
all
these
little
quirks
with
you
know,
building
just
one
piece
of
like
one
piece
that
runs
in
a
larger
distributed
environment
right.
So
every
time
I
get
some
kind
of
stale
input
from
the
api
server
or
there's
a
sort
of
some
kind
of
jitter
in
the
network
or
lag
or
something
like
this.
C
It
turns
out
that
these
very
small
interactions
with
the
environment
would
basically
break
my
code
right
and
at
some
point
and
every
time
I
remember
I
would
keep
going
to
michael's
like.
Why
is
the
kick?
Why
is
this
happening
right?
What
am
I
supposed
to
do
and
then
he'll
tell
me
about
some
pattern
I
was
supposed
to
use
and
I'm
like,
okay,
fine
and
so
after
that
project
ended.
I
was
like
okay,
this.
If
it's
you
know,
I
don't
know
if
it
takes
two
of
us
to
write
this
right.
I
can't
imagine
this.
C
There
right
write
these
correct
controllers
that
you
know
no
matter
what
kind
of
external
input
they
get.
They'll
still
do
the
right
thing
and
not
do
anything
goofy
and
and
he's
an
expert
in
reliability
and
testing,
and
so
on,
and
that's
kind
of
how
we
you
know,
we
clicked
right
there
and
we
teamed
up
and
his
student
would
intern
with
us
the
next
summer.
That's
judong
he's
the
lead
phd
student
behind
this
project.
C
Pretty
much
the
start
of
the
show,
I
would
say,
and
yeah
that
eventually
led
to
c
basically
right
click.
It
was
a
very
organic
process
and,
at
the
time
like
michael,
was
also
looking
into
things
like
the
list
watch
and
all
of
those
the
details
around
that-
and
I
don't
know
that's
how
the
whole
team
got
formed
was
very
organic.
I
think
yeah.
D
D
I
also
spent
a
lot
of
time
with
him
and
we
were
like
just
having
lunch
and
sharing
ideas,
and
so
that's
why,
whenever
you
know
there
were
like
kubernetes
questions
or
debates
around
some
of
the
the
stuff
that
how
kubernetes
works,
as
lilith
already
said,
we
just
hopped
on
a
zoom
call
and
we
were
chatting
about
it.
And,
coincidentally,
there
was
this
time
where
jodong
and
lilith
were
having
questions
about
how
hcd
the
some
of
the
semantics
that
city
provides
to
kubernetes.
D
You
know
like
for
consistency,
semantics
and
also
how
the
informers
then
work
in
all
these
controllers
and,
as
I
was,
writing
those
blog
posts
about
lcd.
I
was
very
familiar
with
the
ad
city
codebase
and
consistency,
semantics
and
lalith.
Had
this
one
questions
about
the
linear,
realizability
guarantees
in
scd
and
whether
it
provides
like
a
logically
and
monotonically
growing
order
of
events
or
changes
in
the
system.
D
I
was
just
literally
writing
that
paragraph
in
the
blog
post-
and
so
I
I
said
yes,
it's
there,
it's
a
number,
it's
just
a
counter
which
goes
up
the
revision
in
that
city,
and
he
said
this
is
great,
because
we
need
that,
in
order
to
you
know,
make
some
assumptions
about
the
work
that
civ
is
doing
and
so
and
then
I
was
asking
okay.
So
what
is
this
project
all
about?
And
we
started
working
working
on
this
and
I
gave
more
input
and
some
of
the
reviews
in
in
the
paper.
C
A
And
right
like
it
can
be,
it
gives
you
a
different
perspective.
You
know
whatever,
so
that's
pretty
cool.
So
so
let
me
just
kind
of
ask
a
little
bit
more
of
a
background
question
so,
like
I,
I
come
across
the
project.
What
do
I
want
to
use
it
for?
What
is
it?
What
does
it
actually
do
for
me,
as
kind
of
a
you
know,
a
deployer
or
an
application
author.
C
Yeah,
so
this
the
main
target
audience
is
controller
or
operator
developers.
Right,
let's
say
you,
you
want
to
write,
say
a
cassandra
operator
and
you'd
go
through.
I
don't
know
you
would
use
the
operator
sdk
or
you
use
one
of
the.
You
might
build
on
things
like
controller
runtime
and
you
you're
going
to
write
some
code
that
maybe
it
monitors
some
kind
of
custom
resource
in
kubernetes,
and
then
it
takes
some.
It
does.
C
So
you
give
us
some
tests.
We
currently
have
our
own
little
way
of
writing
these
tests,
but
it
could
very
well
be
the
e2e
framework
or
something
like
this
right,
but
you
give
us
some
tests
and
what
c
will
do
is
it
will
explore
many
executions
where
it
introduces
failures
in
these
tests
and
if
and
what
steve
can
do
is
that
it
will
automatically
be
able
to
flag
when
an
execution
where
it
introduced
a
failure,
actually
produced
a
different
result
than
the
failure-free
run.
C
This
is
one
of
the
other
innovations
in
the
in
c.
Basically,
it
can
do
that
diffing
and
then
it
will
tell
you
that
hey
look.
I
found
some
executions
where
I
introduced
a
failure
and
it
turns
out
that
the
cluster
looks
a
bit
different
than
it
did
before,
or
it
went
through
some
steps
that
it
wasn't
visible
in
the
failure
free
run
right.
A
B
Yeah
is
actually
a
question
I
wanted
to
to
kind
of
at
least
run
through
quickly
as
we
get
into
to
see.
Even
what
it
is
is
if
I
were
telling
someone
else
about
this
project
at
a
high
level,
what
what
kind
of
testing
does
it
do
like?
Would
I
refer
to
this
as
chaos,
testing
or
you
know
like?
Can
I
put
it
into
one
of
those
pigeon
holes
that
we
kind
of
use
as
a
shorthand
to
talk
about
like
what
surfaces,
we're
testing
and
what
techniques
we're
using.
C
Yeah,
so
chaos
testing
tends
to
be
something
that
you
would
run
in
production
live.
Usually
this
is
not
how
your
you
would
typically
run
receive.
It's
meant
to
be
a
development
time
testing
tool
right,
so
you
can
think
of
c
more
as
another
layer
of
tests
that
you
would
run
say
in
your
ci
platform
or
something
like
this
right
on
a
cluster
of
machines
where
this
just
runs.
I
don't
know
once
a
week
or
something
like
this
right.
C
C
If
c
finds
a
bug
based
on
a
fault
it
injected
you
will
you
basically
have
a
what's
called
a
test
plan.
It's
a
self-contained
file
that
save
runs
where
it
knows
that.
Okay,
it's
waiting
for
exactly
this
event
to
occur
in
the
api,
for
it
to
pause
the
controller
and
inject
a
particular
type
of
failure
like
it
knows.
All
of
that
is
an
encoding
file
with
that.
With
just
that
file,
you
can
reproduce
exactly
that
execution
over
and
over
again
right.
So
it's
that's
how
it
really
differs
from
chaos.
C
A
Right
right
so
that
you
can
kind
of
go
back
and
and
find
the
bug
and
fix
the
bug,
and
then
you
know
so
in
in
a
lot
of
ways.
It
sounds
it's
a
it's
a
bit
more
like
unit
or
integration
testing,
in
the
sense
that
you
know
it's,
it's
a
well-prescribed
set
of
tests
that
you
know,
then
you
basically
you
know
you
know
when
you
have
introduced
a
new
problem
when
you
go
to
the
next
revision
or
whatever,
which
you
know,
I
think
is.
A
Is
software
development
component
that
is
gets
a
lot
of
short
shrift.
You
know
like
we
always
like
to
write
the
first
one
you
know,
but
having
to
maintain
it
over
time
is
always
much
much
harder
and
nowhere
near
as
much
fun
cool.
So
so
have
you
seen
it's
kind
of
the
operators
using
it
in
the
wild
yet
is
it
is
it
you
know?
Has
it
been
around
long
enough?
Have
people
been
adopting
the
usage.
C
We
we've
been
getting
like
a
trickle
of
reports
from
people
who
are
not
us
trying
to
run
the
tool,
so
that's
always
promising,
there's
still
more
work
that
we
need
to
do
to
make
it
easier
to
consume
as
well.
Right
like
ergonomics
is
one
place
again,
like
we've
been
talking
with
michael
about
this
on.
C
What's
the
right
way
to
you
know,
make
it
easy
for
anyone
say
using
the
operator,
sdk
or
controller
runtime
to
just
you
know,
come
generate
a
project
batteries
attached
to
start
testing
with
c
right
like
so
there's
a
bit
of
leg
work
there.
So
it's
the
least
automated
part
using
c,
which
is
onboarding
c1
to
your
project
right.
B
Do
do
you
think
there's
potential
for
for
building
support
for
for
receiver,
for
tools
like
it
into
things
like
the
operator
sdk
or
some
of
the
other
tool
kits
that
have
grown
up
around
building
operators
and
then
second
sidebar
kind
of
to
both
of
you
and
like,
I
hope,
and
it's
something
that'll
be
useful
for
the
audience.
Is
it
we
have
so
far
in
this
in
this
discussion,
used
two
words
interchangeably
or
even
kind
of
neglected?
The
word
controller
and,
like
I'm,
I'd
like
to
kind
of
draw
out
like
what's
a
controller?
B
C
D
So
the
good
thing
is
that,
whatever,
like
type
of
software
you're
building,
whether
it's
a
pure
or
plain
controller,
or
it's
a
more
advanced
operator,
sieve
works
in
both
places
because
at
the
end
of
the
day,
sieve
instruments,
kubernetes
api
and
client,
libraries
and
calls,
and
since
both
of
these
controller
patterns,
if
I
want
to
call
them
this
way,
they
all
use.
You
know
client
go
or
controller
runtime
in
some
form,
whether
it's
through
cubel
or
the
operator.
D
Sdk
cf
would
apply
there
as
well,
and
the
test
plans
obviously
might
be
more
complex
in
the
operator
world,
because
you
know
if,
if
you,
if
you
have
a
system
that
does
like
backups
snapshots,
scaling
platforms
or
like
storage
volumes,
etc,
obviously
the
test
plans
and
that
maybe
even
the
time
running
these
tests
are
would
be
longer
than
in
a
very
simple
controller,
which
just
spins
up
parts
right
but
yeah.
I
would
not
throw
any
distinction
there,
where
c
would
apply.
C
C
D
A
good
one
to
test
and
see
like
how
good
the
the
operator
is
written
or
how
well
it
is
written,
and
one
of
the
bugs
we
found
was
an
interesting
one
where,
under
some
circumstances,
the
the
operator,
the
rabbitmq
operator,
would
delete
the
wrong
stateful
set,
which
you
know
would
have
been
attached
to
a
rabbitmq
cluster.
That
someone
deploys
through
a
custom
resource
definition,
and
we
found
that
back
through
sieve
and
the
fix
was
an
easy
one.
C
There
was
another
one
with
volume
management
again
with
the
rabbitmq
like
that
was
one
of
the
most
fruitful
engagements
with
the
team
we've
had
like
they
so
c
found
a
bug
and
then
in
response
they've,
they
kind
of
redid
how
they
cook
about
volume
management,
because
there
were
some
bugs
around
incorrectly
managing
volumes
there
as
well,
and
then
they
actually
added
a
corpus
of
tests
to
exercise
things.
The
way
sieve
would
have
done
it
right
like
they.
That
was
pretty
cool
to
see
as
well,
so
yeah.
B
And
so
I
wonder
if,
if
there's
some
way
like
I'm
searching
or
trying
to
come
up
with
a
question,
that'll
kind
of
draw
out
there's
a
lot
of
ways.
I
might
test
an
operator
that
I
built
to
to
like
even
to
directly
adopting
the
example,
we've
kind
of
been
working
with
so
far
with
rabbitmq,
like
I'm,
going
to
manage
some
volumes
and
the
the
bug
that
you
were
just
bringing
up
lily,
then
what
there's
a
lot
of
ways
I
might
test
like
that
code
base?
If
that's
what
I'm
writing
yeah,
what
does
sieve
do.
B
That's
different
that
that
is
the
real
key
advantage
here
from
like,
say
any
other,
like
a
different
test
framework
that
I
might
look
at
now
in
like
in
my
in
my
understanding
and
a
little
bit
I
played
with
it
I
kind
of
understand,
or
at
least
have
a
clue
about
like
I'm,
I
can.
I
have
something
that
can
kind
of
predictively
look
at
a
bunch
of
points
along
execution
and
then
instrument
those
points
with
different
things
that
might
happen
in
the
underlying
underlying
api.
B
I'm
addressing
so,
I
I
think
that's
somewhere
close
to
the
heart
of
steve
and
I'm
wondering
if
for
me
and
for
the
audience,
you
could
kind
of
close
that
circle
and
tell
me
exactly
why
is
it
like
different
from
using
a
more
general
purpose
to
test
cool
toolkit
or
something
that
likes
me
like?
I
might
have
heard
of
in
the
past,
or
be
familiar
with,
of
course,.
C
Yeah
sure
so,
let's
I'll
take
a
let's
take
this
example
of
this
any
operator.
It
has
a
reconcile
loop
right
and
now,
when
this
loop
runs
typically
what
you
have
is
you'll
have
like.
I
know
this
can
be.
You
know,
scattered
across
many
files,
thousands
of
lines
of
code,
whatever
right-
and
there
are
many
points
in
that
reconcile
loop,
where
it's
actually
interacting
with
say
the
kubernetes
api
or
something
external
to
the
controller,
and
this
all
actions
of
that
nature
are
asynchronous.
C
That
is,
you
might
create
a
stateful
set,
but
then
the
pods
corresponding
to
that
state
will
set
there's
no
guarantee
that'll
arrive
by
the
time
this
blob
of
code
finishes
or
something
right.
So
everything
related
to
how
the
controller
interacts
with
the
environment
is,
what
c
will
perturb
right
and
so
see?
Perturbs
and
execution
like
you,
you
give
it
a
test.
Workload
to
run
c
will
run
the
test
workload
and
then,
when
it
it's
running
the
version
with
the
failures,
it
will
perturb
the
execution
in
many
ways.
C
So
one
of
the
patterns
we
introduce
is,
we
call
we
we're
basically
checking
if
your
code
is
making
an
assumption
that
your
reconcile
loop
will
always
run
to
completion
right,
it
will
check
for
autonomous
debugs.
So
in
this
case,
what
c
will
do
is
it
will
generate
a
bunch
of
plans
where
at
any
point
that
it
can
do
it?
It
will
basically
inject
a
fault
where
it
will
crash
the
controller
exactly
after
it
executes
one
interaction
with
the
kubernetes
api,
so
every
get
or
every
delete
or
every
update
it
will
introduce
a
crash
right.
C
So
this
is
one
thing
that,
like
I,
you
won't
find
any
off
the
box
testing
framework.
That
will
do
this
for
you
and
in
fact
I
would
say,
like
most
test
suites
that
you
can
find
in
open
source
controller
operator
projects
will
not
do
this
type
of
crash
testing
right
now
crashing
is
crashing
after
every
sort
of
client
client
go.
Interaction
is
one
thing,
but
we
also
do
more
right.
For
example,
we'll
also
do
this
thing
where
we
observe,
during
the
fault
free
run,
which
which
messages
are
the
control?
C
It
will
actually
reason
and
say:
okay,
it
looks
like
this
particular
message
is
only
meaningful
within
this
time
frame
and
that's
exactly
the
time
frame
where
it
will
give
you
a
stale
message
or
something
like
that
right.
It'll
actually
do
so.
We
do
three
patterns
right
now:
autonomously,
testing,
stale
states
and
the
sort
of
level
triggering
research
triggering
kind
of
states.
So.
D
D
A
So,
can
I
ask
a
little
bit
more
of
a
background
question
which
is
you
know,
you've
put
all
this
effort
into
sieve
right,
which
you
know
has
a
very
na
rel
like
narrow
use
case
quote
unquote.
Why
is
this
investment
in
kind
of
supporting
the
development
of
operators
such
a
good
idea
like
what
is
it
about
operators,
kind
of
or
controllers
in
general?
A
That
you
think
is
so
important
that
you
know
I
completely
agree
with
you,
michael,
like
testing
distributed
systems,
is
a
non-trivial
exercise
like
of
any
kind,
but
obviously
this
particular
space.
You
found
particularly
compelling
that
you
wanted
to
go
and
support
this.
So
what
is
it
about
that
space
that
you
think
is
so
interesting.
C
I
think
I
don't
know,
I
think
kubernetes
is
a
fantastic
platform
and
that's
why
the
excitement
really
comes
for
me
from
kubernetes
the
platform
itself,
but
there's
this
nice
thing
that
kubernetes
enables
and
that
most
things
running
on
a
kubernetes
cluster
is
probably
third
party
right.
It's
not
right,
like
most
operators,
controllers
and
control
plane
functions
that
come
into
any
kubernetes.
Cluster
is
third
party
and
they
all
work
pretty
much
off
of
the
same
core.
C
Libraries
right
like
client
go
controller
runtime
whatever
right,
and
that
just
means
that,
if
you
can,
if
you
have
that
nice
vantage
point
to
do
this
type
of
testing
and
that's
what
we
were
able
to
find
that
there
is
such
a
clean
vantage
point
from
where
you
can
do
this
type
of
automatic
testing.
That
means
all
these
operators
and
controllers
can
benefit
from
one
testing
tool
right.
C
B
B
C
From
steve's
point
of
view,
or
from
the
testing
vantage
point
right,
like
the
kubernetes
core,
the
api
servers
and
xcd
is
really
just
a
database
of
objects
right,
so
you
run
some
workload.
Let's
say
you
run
your
you
run
some
you
bring
up
a
cassandra
cluster,
you
tear
it
down
whatever,
like
you'll,
basically
see
this
at
any
point
in
time.
If
you
look
at
the
kubernetes
api
or
this
database
right,
you'll
see
some
set
of
objects
and
every
little
control
plane.
C
Action
where
you
create,
modify
or
delete
an
object,
will
give
you
a
slightly
different
database
right.
So
there's
this
history
or
this
trace
of
database
states
that
you
can
always
observe,
and
the
ordering
of
this
thing
is
important,
because
it
can't
be
the
case
that
every
controller
is
in
a
different
order
right.
It
has
to
be
one
order
shared
across
the
entire
system
and
that's
why
cds
guarantees
are
important
there,
but
all
we
need
to
trace
is
basically
at
any
given
point
in
time
when
the
controller
did
x.
C
Were
in
the
api
before
or
after
each
event,
right,
and
so
this
point
where
the
controllers
are
interacting
with
the
kubernetes
api-
is
also
very
simple:
semantically
right,
you
read,
modify
or
delete
or
create
objects
right.
It's
you
and
it's
literally,
just
a
simple
key
value
store
from
that
point
of
view,
which
has
very
simple
semantics
right,
so
we
can
actually
it
makes
it
easy
for
us
to
reason
about
things
like
the
lifetime
of
objects.
I
know
if
a
delete
went
for
a
key
x.
C
A
So
I
kind
of
was
curious
a
little
bit
more
about
kind
of
the
operation
general.
So
I
mean
I
guess
what
I'm
also
hearing
you
say
right
is
that
you
think
a
major
portion
of
the
ecosystem
for
kubernetes
is
and
should
be,
operators
whatever,
and
so
the
anything
we
can
do
to
kind
of
enable
their
creation
and
and
continued
flow
to
exist.
Right
is
better.
You
know
for
the
for
the
kubernetes
platform,
because
you
know
that's
a
that's
a
good
delivery
model
for
for
these
systems.
D
Maybe
I
would
add
to
this
is
that
if
you
look
at
how
kubernetes
has
grown
and
been
adopted
and
across
the
industry,
like
every
cloud
provider,
almost
every
software
company
is
using
or
offering
kubernetes
it
has
come
somehow
this
defacto
api
that
is,
is
offered,
and
so
for
software,
vendors
or
isvs.
It's
a
nice.
You
know
nice
platform
to
to
use
because
you
can
just
assume
it's
it's
ubiquitous.
It's
always
there
right
and
one
of
the
interesting
projects
that
I've
also
been
following
is
the
kcp
project.
D
From
from
red
hat-
and
I
was
I
was
having
early
discussions
with
stefan
schumanski
he's
one
of
the
lead
engineers
there
as
well
and
asking
because
kcp
takes
kubernetes.
You
know
to
the
next
level
if
you
will
like
taking
the
core
api
principles
and
you
know
making
it
more
than
just
called
a
a
container
orchestrator
and
there
the
same
semantics
apply.
You
know
you
use
the
same
libraries
or
the
same
concepts.
The
same
patterns
are
there,
but
you
might
do
user
management
or
workspace
management.
D
Not
every
not
everyone
right,
but
but
an
average,
but
imagine
I
would
just
go
out
and
start
building
a
bridge
and
call
it
a
day.
That's
probably
not
a
good
way
to
do
stuff
right,
there's
way
more
regulations
way
more.
You
know,
descriptions
and-
and
you
know
stuff
that
you
have
to
put
in
place
before
you
can
actually
start
building
the
bridge
and
then
in
computer
science.
D
People
like
leslie
lampert,
they
advocated
for
you,
know,
writing
specifications
first,
something
like
a
tla
plus
write
the
specifications
in
a
non-code
like
language
like
use,
mathematics
to
write
the
algorithm
and
then
prove
it
and
then
write
the
actual
code.
But
often
we
just
start
writing
the
code
and
then,
if
it
works
unit
test
passes.
So
it
sounds
like
it's
good
right
and
then
these
these
bugs
just
uncover
some
are
more
complex
and
subtle.
D
A
So,
interestingly
enough,
you
know
plugged
our
own
show
when
our
first
episode
when
we
interviewed
clayton
coleman,
we
talked
about
kcp
and
I'll,
throw
that
link
in
the
chat
like
I've
been
doing
a
bunch
of
others,
but
I
was
kind
of
curious
about.
Like
you
know,
I
think
that
kind
of
control,
plane
idea
right
is
that
you
know
kind
of
having
a
uniform
api
for
all
the
things
really
does
have
you
know
some
serious
advantages.
A
A
So
this
was
a
long
time
ago
right
and
we
actually
wrote
a
way
to
be
able
to
watch
for
distributed
system
kind
of
communication
challenges,
as
the
you
know,
as
the
application
was
running,
so
we
also
had
a
dev
time
tool,
but
we
also
kind
of
ran,
wanted
to
run
at
production.
It's
the
hardest
part
about
distributed
systems.
Is
you
put
it
in
production
and
all
of
a
sudden
you're
like?
A
C
So
far,
I
think
we're
doubling
down
on
the
development
side
thing
right,
like
I
think
as
you,
I
think,
you've
hit
the
nail
on
the
head
there
right.
Like
you,
don't
know
what
path
you
took
to
hit
a
bug
in
production,
and
I
think
our
philosophy
here
at
least
has
been
don't
even
give
a
chance
for
a
bug
to
happen
in
production.
C
If
you
can
avoid
it
right,
and
so
we
are
even
looking
at
things
like
verification
right
now,
because
against
eve
as
a
testing
tool,
it's
its
effectiveness
is
only
as
good
as
the
initial
happy
path
tests
that
you
provide
and
see.
Will
you
know
explore
all
its
conditions
based
on
those
initial
tests,
but
it's
not
going
to
guarantee
the
absence
of
bugs,
for
which
you
need
things
like
verification.
C
On
the
you
know,
doing
things
live
side,
I've
I'm
involved
in
some
projects
where
we
are
looking
again,
this
principle
of
like
if
all
of
your
state
is
in
one
globally
ordered
database,
and
you
don't
have
any
state
off
of
it.
You
know
say
all
state
outside
of
it
is
ephemeral
you
it
might
help
with
doing
some
things
like
debugging
there's
this
project,
I'm
involved
in
it's
called
like
dbos,
it's
with
collaborators
from
stanford
and
mit,
and
here
we
are
looking
into
some
of
these
type
of
questions
on.
C
How
can
you
attach
a
debugger
to
a
live
system
and
find
out
exactly
what
what
was
the
state
as
of
a
bug
happening?
And
you
can,
if
you
can
trace
all
of
the
information
until
then
like
you,
can
do
things
like
record
and
replay
debugging
life,
but
it
assumes
a
very
different
like
a
very
restricted
programming
environment.
C
A
No,
I
totally
understand
yeah,
I
mean
so
you
know,
but
I
guess
kind
of
what
we
should
be
looking
for
is
kind
of
the
you
know
breath
you
know
increased
breath
of
steve's
coverage
around
kind
of
dealing
with
these
tool
chains
with
you
know
the
hope
that
there
will
be
fewer
bugs
in
production.
I
don't
know-
maybe
maybe
I'm
less
optimistic
than
than
you
are
that
you
know.
C
B
C
A
Yeah,
I
I
still
remember
in
college
I
had
a
professor
who
told
me
that,
as
I
as
I
became
more
experienced,
I
would
stop
ever
having
syntax
errors
and
I
was
like-
and
you
know
x,
number
of
years
later,
like
25-ish,
I'm
still
waiting
for
that
day.
You
know,
for
example,
so
you
know
I
yeah
bugs
bugs
are
a
way
of
life
and
that
it's
more
about
being
able
to
identify
kind
of
why
they
happened,
how
they
keep
you
know,
keeping
them
from
happening
again.
C
There
are
also
security
holes
that
show
up
where
there
were.
There
were
some
operators
where
we
found
that,
like
a
crash
at
the
wrong
time,
would
cost
them
to
silently
not
configure
dls
things
like
this
right
like
so.
There
is
quite
a
price
for
these
type
of
bugs
and
therefore
it's
okay
to
spend
a
bit
more.
You
know
resources
and
development
time
trying
to
find
them
for
other
kinds
of
bugs
where
you
don't
have
this
much
of
a
price.
Perhaps
it's
not
yeah.
B
So,
if
I'm
somebody
who
wants
to
push
on
that
quality
curve
and
I'm
really
interested
in
sieve
and
I'm
so,
this
is
a
two-part
question
one
if
I
want
to
start
using
it
tomorrow
to
make
my
operators
better.
What's
the
first
thing,
I
should
do
two
if
I
want
to
start
working
on
sieve
tomorrow,
to
make
everyone's
operators
and
controllers
better.
What
is
the
first
thing
I
should
do.
C
So
the
first
part
is:
what
do
you?
What
should
you
do
if
you
want
to
use
it?
You
know
for
your
operator
today.
If
you
go
to
the
sieve
project,
you'll
basically
find
the
sporting
guide
on
how
to
you
know
brings
even
to
your
project,
and
it
also
will
answer
your
second
question,
because
you'll
see
a
lot
of
the
work
that
you're
doing
by
hand
that
could
be.
You
know,
defaults
that
you
get
from,
let's
say
using
an
sdk
directly
right.
C
So
here,
for
example,
you'll
see
that
you're
going
to
specify
a
bunch
of
version
info
like
what
kind
of
what
go
version.
Are
you
using
what
kubernetes
version
are
you
using?
What
commit
id?
Are
you
using
for
your
project
to
test
and
so
on?
A
lot
of
those
defaults,
I
think,
can
come
straight
out
of
like
if
you
integrated
this
with
say
the
operator
sdk,
and
there
were
some
batteries
attached
way
where
you
just
say,
you
know,
generate
this
project
to
be
save
enabled.
C
A
lot
of
those
defaults
could
just
come
off
of
the
project
itself
right
and
then,
for
example,
steve
needs.
Some
information
like
oh,
where
is
the
docker
file
for
this
project
or
where
like
it,
also
needs
some
information
on
the
cr
itself
so
that
it
can
put
in
some
annotations.
That
c
will
use
during
testing
to
find
the
pod
corresponding
to
the
controller.
Like
a
lot
of
these
things
can
just
really
be
it's
mechanical
work
that
I
think
we
can
automate
away.
A
C
A
A
B
No,
I
think
those
are
those
exactly
kind
of
the
the
two
things
I'm
I'm.
I
was
looking
for
and
especially
delighted
by
the
observation
that,
in
a
crowd
full
of
people
looking
for
sre
surfaces
to
automate
as
soon
as
you
encounter
the
first
order,
problems
of
manual
work
you'll
go
looking
to
how
to
how
to
make
them
not
manual.
The
second
time
you
go
through.
A
A
B
B
What's
my
hook
to
walk
out
with
here's
the
next
great
thing
about
it,
would
it
be
integrations
with
build
tool
kits
and
sdks
like
who
builder
and
operator
sdk,
and
some
of
those
things
we've
talked
about
is
any
of
that
work
active
like?
Is
there
some
of
that
going
on
that
we
could
take
a
look
at
now
or.
C
I
think
the
one
big
thing
that
we're
working
on
now
is
even
not
needing
like
not
needing
the
assumption
that,
for
example,
right
now,
we
assume
controller
on
time.
I
think
we're
working
to
remove
that
assumption
and
work
with
just
base
client
go
and
once
we
do
that
it
gets
easier
to
sort
of
go
upwards
as
needed
for
the
different
other
frameworks
that
show
up
right.
So
this
is
one
big
thing
we're
working
on
now,
I
would
say
going
forward
if
the
community
would
like
to
help
us
out.
C
C
The
long-term
thing
to
get
excited
about
is
that
we're
looking
at
sieve
as
not
just
something
to
test
controllers
or
operators,
but
also
like
how
what
part
of
this
can
also
extend
all
the
way
to
the
applications
that
are
being
managed
by
these
operators
right?
So
we
don't
have
any
visibility
into.
Let's
say
what
the
cassandra
cluster
itself
is
doing
today,
right
that
the
operator
might
manage.
C
D
D
Kubernetes
controller
related
talks
lately,
but
the
ones
that
you
know
were
presented.
They
were
highly
attended
by
a
lot
of
folks
out
there
and
does
not,
even
though
there's
two
good
books
that
I
know
of
josh,
you
wrote
one
and
stephan
giumanski
and
michael
housenblast
were
the
other
one,
but
I
think
more
on
this,
I
I
called
it
the
the
mental,
the
like
the
mode
that
you
had
to
have
in
your
head
or
the
mindset.
The
mindset
of
writing
these
controllers
there's
a
lot
of
things.
D
You
know,
like
maybe
the
five
or
ten
golden
rules
that
you
have
to
keep
in
mind
and
relating
it
to
like
best
practices
and
then
writing
these
operators
and
stuff,
like
that,
I
think,
sharing
the
knowledge.
The
learnings
from
the
safe
projects
was
also
going
to
be
very
helpful
and
then
obviously
driving
adoption
of
steve,
because
people
are
getting
aware
received.
A
So
you
should
definitely
join
us
as
well
as
we'll
be
doing
a
kind
of
a
meet
and
greet
event
at
the
openshift
commons.
A
lot
you
know
a
reception
the
day
before
so
definitely
join
us
there
and
watch
for
the
video
output
of
and
I'll
give
you,
the
teaser
of
you
know
maybe
riding
around
in
cars,
interviewing
people
with
motor
city
and
then
next
time
on
the
show
at
the
end
of
september.
A
Yeah-
and
you
know
obviously
I'm
I'm
a
little
biased
towards
it
anyway,
because
I
think
serverless
and
event-driven
architectures
are
really
the
way
all
software
development
should
be
done.
Yes,
I'm
a
little
biased
so,
but
thank
you
so
much
for
being
on
the
show.
We
really
appreciate
it
and
yeah.
We
look
forward
to
seeing
you
again
thanks.
Everybody.