►
From YouTube: Testing Kubernetes CRDs - Christie Wilson, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Testing Kubernetes CRDs - Christie Wilson, Google
Custom Resource Definitions (CRDs) open Kubernetes up for extensions in ways never before possible, but they can add new complications for testing. Achieving good test coverage for distributed systems can be a challenge. With so many components and moving pieces, it can be hard for unit tests to catch everything, and system tests become more important. So if you are relying on CRDs to implement your sweet new features, how can you write system tests for them? Drawing on examples and experiences from testing Kubernetes itself we will see how you can write effective, stable tests for your CRDs, controllers and webhooks. We’ll look at what Go libraries you can use, how you can combine them, what to avoid, and what tools you can use to make running these tests ints CI/CD a breeze.
To learn more click here: https://sched.co/FuJa
A
Oh,
can
everybody
hear
me:
okay,
cool
hi,
everybody,
I'm
Christy,
Wilson
I'm,
a
software
engineer
at
Google
and
I
work
on
the
container
tools.
Team
I've
worked
in
a
lot
of
different
industries.
I've
worked
in
mobile
foreign
currency
exchange
Triple
A
games.
This
talk
is
based
on
my
work,
most
recently
with
conformance
testing,
4k
native,
serving
and
also
from
leading
the
K
native
pipeline
project.
A
A
A
A
But
even
if
you
haven't
worked
with
see
IDs
before
you
still
might
find
some
of
this
content
interesting.
So
here's
a
quick
intro
to
what
they
are
CID
stands
for
custom
resource
definition.
So
this
is
something
that
lets
us
add
our
own
types
to
kubernetes
in
addition
to
the
built-in
ones
like
deployments
and
pods
on
their
own,
see
IDs.
Just
let
us
create
structured
data,
but
we
want
to
operate
on
that
data.
So
we
add
controllers,
which
are
binaries
that
we
run
and
together
we
can
make
declarative.
A
A
It's
usually
because
you're
creating
a
platform,
a
platform
is
software
that
we
build
other
software
on
top
of,
and
usually
the
point
of
the
platform.
Isn't
the
platform
itself,
it's
the
things
that
we
want
to
build
on
top
of
it.
That's
where
the
real
business
value
is
a
platform
is
an
abstraction
of
all
the
stuff
that
you
have
to
do
to
get
your
software
running.
A
A
You
may
have
seen
this
pyramid
before
this
is
the
classic
testing
pyramid.
This
tells
us
the
kinds
of
tests
that
we
should
be
writing
and
how
many
we
should
be
writing
of
each
type.
We
start
with
unit
tests
which
are
our
cheapest
and
fastest
tests,
and
we
wrote
we
write
mostly
unit
tests
and
a
little
bit
of
the
other
kind
which,
as
we
move
up
the
pyramid,
become
slower
and
more
expensive,
specifically
for
CR
DS
we're
going
to
be
talking
about
this
pyramid.
A
The
key
characteristic
of
these
tests
is
that
we
will
be
connecting
to
an
actual
kubernetes
cluster,
as
you
can
see
from
the
top
of
the
pyramid.
The
box
is
the
smallest,
so
this
means
we
should
be
writing
the
fewest
number
of
these
tests.
We
don't
necessarily
want
to
get
coverage
here.
What
we
want
is
to
make
sure
that
we've
actually
integrated
everything
properly
and
it
all
works
together.
A
So,
in
order
to
run
these
tests,
we
need
a
kubernetes
cluster
and
it
has
to
be
running
your
controllers.
You
could
deploy
your
controllers
as
part
of
your
test
setup,
but
I
recommend
that
you
offload
that
to
a
tool.
That's
specifically
for
deployment,
for
example,
something
like
Co.
Most
of
these
tools
are
pretty
easy
to
call
before
you
invoke
your
tests,
so
I
recommend
doing
that.
That
way,
you
can
iterate
on
the
tests
themselves
more
quickly.
A
Let's
look
at
seven
specific
things
that
you
can
do
to
get
your
end
to
hit
and
to
end
pests
up
and
running.
Also,
the
code
samples
aren't
going
to
be
showing
you
here
are
available
in
the
github
repo
at
the
bottom
of
the
slide.
If
you
want
to
take
a
look
later,
if
you
use
these
seven
things,
it's
actually
pretty
easy
to
get
your
end-to-end
tests
up
and
running.
A
So
the
first
thing
that
I
recommend
is
about
how
your
tests
are
invoked.
As
you
start,
adding
these
tests
you'll
notice
that
when
you
run
go
tests,
it
runs
all
of
your
tests
and
since
end-to-end
tests
have
to
connect
to
a
cluster
and
they're
slower
that
can
be
frustrating,
so
I
recommend
using
go
build
tags.
In
this
example,
we
use
the
tag
e
to
e
to
identify
the
test
that
we
don't
want
to
run
by
default.
A
The
next
recommendation
is
to
declare
your
test
inputs
and
go
instead
of
Gamal.
Gamal
is
definitely
more
readable
and
it
makes
sense
to
provide
it
as
examples
for
your
users,
but
your
test
will
probably
need
to
manipulate
your
CRTs
and
change
some
fields
like,
for
example,
the
namespace,
and
in
order
for
them
to
do
that
with
the
yellow
version,
you
have
to
load
the
llamo
and
parse
it
and
becomes
messy,
and
it's
kind
of
hard
to
understand
what
the
actual
test
input
is.
So
I
recommend
just
declaring
it
and
go.
A
A
Whichever
approach
you
use,
I
recommend
telling
your
tests
how
to
connect
to
your
cluster
using
your
coop
config.
Your
coop
config
file
already
has
your
endpoint
information
and
it
says
how
to
authenticate
with
the
cluster,
and
if
you
want
to
invoke
the
test
locally,
you
can
just
use
the
one
you
already
have
and
if
you
want
to
run
them
on
CI,
you
can
provide
a
static,
coop
config
file.
A
I
mentioned
at
the
beginning
of
the
talk
that
we
use
see
IDs
to
create
declarative
api's.
This
means
that
our
api's
are
eventually
consistent,
which
means
that
operations
that
we
make
are
asynchronous
so
just
because
we
made
a
create
call
and
it
returned
that
doesn't
mean
the
resource
was
actually
created.
So
this
means
our
tests
need
to
do
a
certain
amount
of
polling
to
wait
for
the
system
to
actually
realize
the
state
that
we
declared
and
I
find
the
poll
functions
in
the
API
machinery.
A
Library
really
helpful
for
this
I
use
this
poll
immediate
function
all
the
time.
This
is
an
example
of
it
in
action
where
it's
wrapped
in
a
function
called
wait
for
pod
state,
which
is
called
by
our
test.
The
interface
here
is
that
we
provide
a
function
which
is
called
immediately
and
it
continues
to
be
called
every
time
interval
passes
until
either
it
returns
true
or
it
signs
out.
A
Now,
if
you're
doing
all
of
this
polling,
it
means
your
tests
might
hang
for
a
bit.
Might
look
like
they're
hanging
because
they're
not
doing
anything
so
I
recommend
regularly
outputting
information
about
what
your
test
is
doing
so
before
you're
going
to
do
something
that
takes
a
while
log,
a
statement
explaining
what's
happening.
It's
important
to
note,
though,
that
if
you
use
the
log
functions
in
the
testing
library,
your
output
will
be
buffered.
A
This
means
that
you
won't
be
able
to
see
it
until
the
test
completes
which
defeats
the
purpose
so
use
an
unbuffered
library
even
better.
If
you
use
a
named
library,
it
makes
it
so
that
you
can
run
these
tests
in
parallel,
because
the
log
output
will
be
interleaved,
but
you
can
actually
see
which
test
each
line
came
from.
A
A
Okay,
so
let's
say
that
we
wrote
our
system
tests.
Where
do
we
go
from
here?
As
we
move
down
the
pyramid
we
get
to
integration
tests,
it's
really
hard
to
define
integration
tests,
but
it's
fair
to
say
that
there's
somewhere
between
end-to-end
tests
and
unit
tests,
they're
not
unit
tests
because
they
put
pieces
of
the
system
together
and
they
aren't
end-to-end
tests
because
they
don't
use
the
whole
system.
A
A
If
you
use
KU
builder
to
initialize
your
project,
it
actually
generates
integration
tests
for
you.
You
still
have
to
fill
in
all
the
assertions,
but
it
creates
all
the
setup
and
the
teardown
and
what
these
do
for
you
is
they
run
processes
for
you
locally.
They
start
an
instance
of
each
of
your
controllers.
They
start
an
instance
of
the
API
server
and
they
start
an
EDD
instance.
A
This
is
an
example
of
what
some
of
the
generated
code
looks
like
if
this
looks
kind
of
unfamiliar
to
you.
That's
because
it's
using
a
testing
library
for
BDD
called
ginkgo,
but
what's
happening
in
this
code-
is
that
we're
first
creating
an
instance
of
our
CID,
which
is
of
type
feline
and
then
we're
waiting
for
the
controller
to
create
a
corresponding
deployment
and
the
cool
thing
is.
We
can
do
this
without
actually
requiring
a
kubernetes
cluster,
but
because
there's
no
kubernetes
cluster
we're
a
little
bit
limited
in
what
we
can
test.
A
A
Instantiating
them
is
very
easy.
This
is
how
you
do
it,
and
then
you
use
this
to
instantiate
your
controller.
The
cool
part
comes
in
when
you
see
data
into
the
clients,
so
this
is
an
example
of
seating.
The
informer
inside
of
my
fake
client,
with
an
object
and
what
this
does
is
later
on.
If
any
code
tries
to
retrieve
objects,
it's
going
to
get
the
one
that
we
seated.
A
This
starts
to
look
a
little
bit
more
verbose
when
we
have
more
complicated
scenarios.
This
is
the
code
from
the
pipeline
CRD
project
that
we
use
to
instantiate
our
controller.
It
has
to
instantiate
six
different
types
of
C
IDs,
it
puts
them
into
the
clients
and
it
puts
them
into
the
Informer's
cache.
A
If
you're
used
to
unit
testing,
you
might
be
wondering
why
I'm
referring
to
these
as
integration
tests,
since
they
use
test
doubles.
There
are
a
few
reasons
for
this.
One
is
if
you're
testing
at
the
level
of
your
controller,
the
inputs
and
outputs
that
you
have
to
deal
with
are
actually
pretty
similar
to
what
you're
dealing
with
in
the
end-to-end
test
case.
A
Also,
you
aren't
isolating
the
code
that
you're
testing
you're
testing
the
controller
as
a
whole
and,
lastly,
you
still
are
combining
pieces
of
the
system
together,
you're
just
faking
out
the
data,
as
far
as
which
approach
to
use
again,
you're
kind
of
stuck
with
the
one
that
comes
with
the
system
that
you
use,
but
there
are
different
advantages.
The
advantage
of
the
qu
builder
approach
is
that
you
can
run
the
actual
processes.
A
A
This
brings
us
to
the
bottom
of
the
pyramid,
the
unit
tests,
so
this
is
where
we
want
to
have
the
most
coverage.
We
should
have
mostly
unit
tests
with
just
a
smattering
of
the
other
types
and
the
reason
there's
some
there's
several
reasons
for
that.
One
reason
is
because
they
are
cheap
and
they
are
fast,
but
there's
something
else
besides
that
this
is
because
all
the
types
of
tests
that
we've
been
talking
about
up
until
this
point
integrate
multiple
components
together.
A
A
To
understand
why
that
is,
we
have
to
look
at
what
good
design
is
good
design
is
something
that
we
often
refer
to
as
testable
code,
but
the
thing
is,
we
don't
write
code
to
be
testable
just
because
we
want
to
test
it.
We
write
testable
code
because
it
has
these
attributes.
It's
loosely
coupled
it's
highly
cohesive
and
the
interfaces
are
well
defined.
A
Integrated
tests,
however,
hide
all
of
these
design
problems.
For
example,
you
can't
see
the
details
here,
but
this
is
a
reconcile
loop,
even
just
glancing
at
it.
You
can
see
that
it's
very
long.
It
is
not
highly
cohesive.
It
does
a
lot
of
different
things.
It's
not
loosely
coupled
it
calls
in
to
a
lot
of
different
objects,
but
integrated
tests
don't
care.
You
can
still
write
an
integration
test
for
this
just
fine
unit
testing.
A
A
So
how
do
we
do
this?
Specifically
for
CR
DS
I
recommend
that
you
follow
the
functional
core
and
imperative
shell
approach.
The
idea
is
that
your
business
logic
is
in
something
called
the
functional
core
and
then
all
the
messy
stuff
that
communicates
with
the
outside
world
is
in
the
imperative
shell,
which
calls
into
the
functional
Court.
A
Now,
when
I
say
functional
here,
I'm
not
recommending
that
you
go
and
rewrite
all
of
your
code
to
be
purely
functional
or
that
you
start
using
monads
I
mean
specifically
that
you
can
write
code
that
avoids
mutating
state
that
doesn't
do
any
IO
and
doesn't
have
any
side
effects.
So
that's
what
I
mean
when
I
talk
about
the
functional
core.
You
still
need
the
other
stuff,
but
you
isolate
it
in
this
imperative.
Show.
A
What
we
want
to
do
is
treat
the
controller
and
all
of
its
related
objects
as
the
imperative
shell,
it's
the
glue
that
holds
our
controllers
together.
So
we
want
to
keep
that
isolated
from
the
business
logic
which
we
can
put
in.
These
really
well
factored
functions,
and
these
well
factored
functions
can
actually
be
unit
tested
without
requiring
any
test
doubles
at
all.
A
Specifically,
the
way
to
do
this
is
to
avoid
passing
around
Lister's
and
informers.
If
you
find
yourself
doing
that,
try
to
rewrite
the
functions
you're
writing
instead,
so
they
just
take
exactly
the
objects
they
need
to
operate
on.
This
separates
the
concern
of
retrieving
the
objects
from
what
you
actually
want
to
do
with
them.
A
So
what
can
you
do
to
make
this
happen?
There's
two
things.
The
first
thing
is
to
measure
your
coverage
and
see
where
you're
at
the
second
thing
is
to
take
a
look
at
your
reconcile
loop,
so
go
to
your
project
and
run,
go
test
with
the
cover
flag
and
see
where
you're
at
you
shouldn't
aim
for
a
hundred
percent
coverage,
because
that's
not
worth
the
effort
but
about
80
percent
is
pretty
good
and
then
take
a
look
at
your
reconcile
loop.
A
A
A
B
I
was
wondering
if
you
could
talk
about
we,
we
actually
have
quite
a
few
conversations
at
my
own
firm
about
how
what
the
right
level
of
CO
coverage
is
and
I
noticed.
There
you
put
that
magic,
80%
number
I'm,
just
curious,
because
I
know
it's
there's
a
degree
to
which
is
a
finger
in
the
wind.
But
how
did
you
reach
that?
Because
we
we?
We
also
came
to
the
conclusion
that
a
hundred
percent
is
kind
of
like
not
efficient,
not
realistic,
but
where
is
the
sweet
talk?
Why
is
80%
I
think.
A
That
80%
is
mostly
an
arbitrary
number,
but
from
all
the
projects
that
I've
worked
on
the
ones
that
are
the
best
tested,
usually
end
up
being
around
80%
and
I.
Think
if
you
follow
the
recommendation,
where
you
have
all
of
your
business
logic,
and
these
really
well
factored
functions
and
you
have
like
the
ugly
glue
stuff
outside
then,
what
ends
up
happening
is
that
you
provide
coverage
for
the
for
the
the
business
logic
in
those
functions
and
then,
like.
A
Don't
really
worry
so
much
about
the
ugly
glue
and
I
think
that
the
trade-off
in
like
how
much
code
you
have
it's
ugly
glue
is
like
20
percent.
It's
like
20
percent
gluing,
everything
together
at
80
percent,
the
actual
thing
you're
trying
to
do.
But
it's
pretty
arbitrary,
like
if
you're
at
like
69
percent,
and
you
look
at
your
code
and
there's
like
it's
all
the
stuff.
That's
not
covered
I!
Think
it's
important
to
know
what
you're
covering
and
what
you're,
not
if
the
stuff
you're
not
covering.
C
So,
on
your
point
about
using
avoiding
llamo
and
instead
using
deeply
nested
struck,
something
I
have
seen
kubernetes
test
tests.
Is
they
often
have
those
littered
directly
in
the
test
bodies,
so
to
read
what
a
single
test
is
doing
can
span
a
hundred
to
500
line.
It
feels
like
at
some
point
you
want
to
refactor
those
into
fixtures.
I've
also
seen
some
other
tests
that
use
something
akin
to
a
builder
pattern.
I
was
curious
in
the
same
sort
of
feeling
it
out
way.
A
I
think
the
only
recommendation
that
I
have
there
is
the
one
about
the
the
wrong
abstraction
is
worse
than
to
plication.
So
I
think
that
it's
okay
to
duplicate
that
I
know
it's
like
super
long
and
hard
to
read,
but
I
do
I
I
do
find
that
I
often
end
up
pulling
those
out
and
reusing
it.
It's
usually
because
another
test
needs
like
exactly
the
same
input
or
it's
like
pretty
much
the
same
thing,
but
the
name
is
different.
Something
like
that.
D
D
Hi
Christy
I
have
a
couple
questions,
so
the
first
one
is
when
you
mentioned
building
you
brought
your
own
unit.
Tests
I,
sometimes
find
that
I
had
to
use
different
libraries
like
API
is
clang,
go
API
missionary.
So
how
can
I
make
sure
that
the
code
or
the
unit
tests
I
write
will
come
compatible
with
the
like
the
latest
app
stream
in
those
projects,
because
they
those
code
change
as
well?
So
how?
How
do
I
make
sure
that
something
changes
on
the
other
repositories?
D
I'm
my
unit
tests
won't
get
affected
or
like
I'll
connect
controlled
affect.
Another
question
is
for
the
end-to-end
test
that
you
just
mentioned.
That
I
wondered
that
if
all
we
have
to
do
is
do
when
we
create
the
controller
we
just
have
to
monitor
at
the
at
the
add,
update
and
delete.
If
those
things
work
correctly,
I
can
assume
that
the
Antonine
test
it
works,
fine
and
last
question.
D
If
I
may
I
think
you
mentioned
that,
for
the
system
test
that
we
always
the
functions
is
more
asynchronous,
so
resource
is
created,
it
doesn't
mean
they
actually
exist,
so
you
have
to
hold
them.
So
I
wanted
to
get
your
feedback
on
like
polling
versus
pushing
using
informers,
because
for
informants
you
actually
just
like
wait
to
to
get
the
information
rather
than
you
for
someone
to
give
you
the
information
rather
than
you
go
to
get
it
fetch.
The
information
so
just
like
I
work,
opposite
directions
such
as
1k
or
your
opinions
on
all
the
questions.
A
So
I
think
the
only
thing
you
can
really
do
is
have
all
these
unit
tests
and
then
have
like
a
few
system
tests
and
integration
tests
and
then
because
you're
probably
been
during
in
the
code.
You
can,
you
know,
update
it
when
you're
ready
and
then
like
run
all
your
system,
tests
and
integration
tests
and
then
those
pass
you
should
be
good.
D
A
I
think
that
that's
pretty
good
in
an
older
version
of
this
presentation,
I
recommended
like
one
per
type
like
one
test
per
controller
and
Ciardi
sort
of
combo,
which
I
think
is
pretty
much
fine
and
like
mostly
happy
paths,
because
you
want
to
rely
on
your
other,
more
fine-grained
tests
to
test
all
of
the
edge
cases
that
you
might
encounter
and
I
think.
Maybe
for
the
third
one
come
up
to
me
after
we
can
keep
keep.
A
E
You
thank
you
very
much,
so
the
thing
I
wanted
to
know
a
little
bit
more.
Maybe
you
can
provide
more
of
an
example.
Is
that
the
idea
of
the
imperative
shell
in
the
functional
core
and
what
sort
of
what
it
looks
like
when
you're
you
know
how
you
decide
what
code
goes
into,
what
layer
there
and
then
also
with
the
imperative
shell?
Do
you
do
you
still
get
any
testing
coverage
of
that,
or
are
you
okay
with
not
testing
that
glue,
so.
A
E
A
Think
that's
pretty
hard
to
do.
If
you
take
a
look
at
the
example
repo
there
I
have
two
versions
of
one
of
the
same
controller,
where
I've
tried
to
factor
out
the
code
into
like
kind
of
kind
of
a
functional
core.
The
controller
is
super
simple
though
so
I
don't
know
if
it's
a
super
compelling
example,
but
you
can
kind
of
see
like
I
think
pretty
much
just
move
everything
out
that
you
can
just
like
keep
moving
stuff
out
until
you
can't
really
do
it
anymore.
It's
just
making
the
code
worse.
That's.
F
F
D
F
A
Is
a
great
question:
I
really
like
that
question,
because
I
found
over
my
career
that
has
definitely
shifted
like
at
the
beginning
of
my
career
I
spent
most
of
my
time
writing
the
logic
itself.
But
what
I
do
now
mostly
is
I,
try
to
actually
start
with
the
documentation
for
what
I'm
writing
so
I
start
with
the
documentation
and
then
from
there
I
go
into
tests
and
then
by
the
time,
I'm.
Actually,
writing
the
thing
itself.
A
It's
really
easy
and
fast
to
do,
and
the
reason
is
that,
having
to
write
the
tests
and
having
to
write
the
documentation
has
forced
me
to
think
a
lot
about
it.
So
I
end
up
thinking
a
lot
about
like
what
is
this
interface
that
I'm
writing?
What
should
it
actually
take?
What
should
it
actually
return?
A
What
is
it
going
to
be
like
to
call
it
and
then,
by
the
time,
I
get
around
to
actually
implementing
the
thing
I've
like
thought,
through
a
lot
of
it,
I've
just
like
and
I've
done
and
I
move
on
to
the
next
thing.
So
I
would
say
it's
actually
quite
a
large
percentage
of
my
time.
Writing
these
tests
and
then
like
sometimes
rewriting
them
because
I
started,
turns
out
that
it
I
designed
it
poorly
yeah.
A
F
G
G
I
think
you're
great
stay
forever.
Almighty
top
I
have
my
questions.
How
do
you
play
a
structured
or
test
like
we
have
three
contests
like
end
to
end
integration
and
unit
test?
So
how
are
you
and
what's
your
recommendation
on
the
way
we
structure
the
test
and,
for
example,
inside
unit
test?
We
have
some
code
and
can
we
reuse
Doc's
code
for
integration
testing
end
to
end
as
well,
so.
A
The
question
is
about
how
you
actually
structure
the
code.
I
think
the
test
code
is
just
like
any
other
code.
So
if
you
feel
like
it's
useful
to
factor
some
of
it
out
and
reuse,
it
then
totally
do
that
and
if
you
share
it
between
all
the
test
types,
that's
totally
fine.
The
other
thing
is
the
unit
tests
I
often
find
it
useful
to
put
them
with
the
code
they're
testing.
A
So
there's
like
the
you
know
the
file
that
has
the
code
and
then
right
next
to
that,
there's
the
underscore
test
version
that
has
the
test
in
it.
But
integration
testing
system
tests
usually
end
up
living
somewhere
else
because
they're
more
broad,
so
there's
like
a
test
directory
or
something
that
has
the
integration
tests,
the
system
tests
and
maybe
the
libraries
that
are
being
used
by
your
unit
tests
as
well.