►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Testing SIG - Benjamin Elder, Google & James Munnelly, Jetstack
The kind (Kubernetes-In-Docker) project virtualizes entire multi-node Kubernetes clusters in containers to allow for a lightweight deployment topology in order to facilitate testing. This presentation will explore how the kind project is used to support testing for the Kubernetes project, how it works, and how developers can make use of kind to facilitate their own testing needs.
https://sched.co/MPkC
A
A
Sweet
so
yeah,
here's
the
logo,
I'm
a
pretty
big
fan
of
that
I
think
this
is
one
of
you
idea
news
you
put
this
together
in
the
end,
Liuba
mere
bloom.
It
right
like
big,
big
kudos
to
him,
because
that's
great
so
yeah
what
does
kind
stand
for,
though
so
Cuba
Nettie's
in
Dhaka,
like
di't
dimed,
but
with
a
k'.
A
So
the
idea
here
is
to
use
docker
containers
to
simulate
nodes,
so
almost
like
the
machines
that
you're
used
to
with
the
kubernetes
cluster,
we
just
use
a
container
instead,
so
they're
kind
of
similar
enough
we'll
go
into
a
bit
more
detail,
obviously,
today
as
to
where
they're
not
quite
so
similar
but
yeah,
that's
the
general
idea.
The
goal
here
is
to
make
it
easy
to
run
clusters
anywhere
and
just
make
it
nice
and
portable.
A
One
of
the
big
sales
of
docker
is
obviously
that
you
can
run
your
same
code
here
or
there
or
anywhere
we're
trying
to
do
the
same
with
kubernetes,
so
it
supports
multi,
node
out
the
box
and
higher
h
a
control
plane.
So
you
can
have
multiple
kind
of
nodes
right
like
running
your
workloads
as
well
as
running.
A
D
A
Over
past
mm
six
to
nine
months
or
so
the
actual
cluster
boot
time
is
now
down
to
thirty
seconds.
So
we
can
go
from
kind
of
zero
to
an
actual
working
running
cluster
that
you
can
run
pods
on
and
yeah
thirty
seconds.
So
it's
great
for
doing
things
in
your
laptop
or
anywhere
else
so
yeah.
Why
would
you
use
kind,
there's
a
few
different
use
cases.
We
are
today
focus
on
testing
kubernetes
itself,
so
that's
kind
of
running
the
end-to-end
and
conformance
suite
you
could
be
using
this
I
personally
do
for
testing
applications.
A
You
build
on
cube-like
again
q
Bonetti,
so
your
own
controllers,
extensions,
even
things
that
don't
necessarily
extend
kubernetes
if
you
just
want
a
kubernetes
environment
to
run
them
in.
It's
really
useful
for
that.
It's
also
now
being
used
for
bootstrapping
clusters
that
are
built
using
the
cluster
API.
So
for
anyone
that
doesn't
know,
cluster
API
is
kind
of
creating
clusters
using
clusters
and
using
the
API
server.
A
We
need
a
bit
of
cuba
Nettie's
first
to
do
that
and
kind
is
perfect
for
that
with
its
slope
plates
quick
times
so
yeah
some
of
the
requirements
for
testing
kubernetes.
So
from
the
start
the
whole
thing
needs
to
be
able
to
run
in
a
container.
So
for
anyone
that
doesn't
know,
prowl
is
the
kubernetes
testing
CI
platform
and
it
all
runs
on
kubernetes.
So
every
single
thing
that
you
see,
then
we
do
today
has
to
be
able
to
run
inside
of
a
pod
as
well
so
yeah.
A
We
also
need
to
run
kubernetes
itself
from
source,
as
we
said
before,
so
take
a
take,
a
source
tarball
or
just
like
a
git,
commit
check
out
and
turn
that
into
a
running
cluster
so
that
we
can
go
and
run
the
likes
of
conformance
tests
on
and,
of
course,
it
should
pass
those
conformance
tests.
So
it's
not
as
simple
as
just
booting
the
cluster.
A
We
need
to
actually
show
that
the
cluster
behaves
as
it
should
and
yeah,
and
that's
what
conformant
sweet
for
it's
for
so
what
we
end
out
with
there
is
darker
and
darker
and
darker
and
darker
and-
and
you
can
kind
of
keep
going,
yeah
I
think
I
put
a
diagram
together.
A
while
ago,
when
we
first
were
working
on
this,
it
was
entitled
kin,
din
dink,
which
is
kubernetes
and
docker,
and
docker
in
cuba,
Nettie's
so
yeah.
It
kind
of
keeps
going
all
the
way
down.
A
A
What
it
does
is
part
of
its
kind
create
cluster
command.
It
will
actually
go
and
pull
the
kind
image
that
we've
built
and
that
image
contains
a
Kuebler
so
built
from
some
commit
ref
like
we
go
and
tag
so
much
like
for
each
cuba,
Nettie's
version
will
go
and
build
person,
publish
these
images,
pre-built
containing
a
Kuebler
cube
atom
with
the
correct
version,
docker
configured
correctly
systemd
as
well
and
I,
don't
know
if
anyone's
tried
to
run
systemd
in
a
container.
There
are
a
few
extra
things
you
have
to
do.
A
It's
not
too
happy
about
it
same
with
docker
and
same
with
the
cube
lot,
and
we
also
package
all
of
the
core
images.
Now.
The
reason
for
that
is.
We
want
to
be
able
to
run
kind
without
an
internet
connection
without
any
external
dependencies.
In
fact,
on
the
way
over
here,
I
had
to
quickly
copy
and
paste
and
cube.
Ctl
output
and
I
just
did
a
kind
create
cluster
on
the
plane.
30
seconds
later,
you've
got
a
cluster
running,
so
that
includes
the
likes
of
Etsy
D
Cora,
a
cored
DNS.
A
The
pause
container
cube
letters,
API
server.
All
of
these
other
images
that
have
to
be
built
they're
all
packaged
into
one
image,
so
you
can
just
run
that
offline
without
any
attendances
and
that's
great
for
testing
too,
because
you
don't
want
something
outside
the
cluster
to
change
or
something
outside
of
your
test
environment
to
change
and
cause
your
test
to
start
failing
one
day,
because
someone
broke
docker
hub.
Something
like
that.
A
So
this
is
roughly
what
it
looked
like.
This
is
a
deep
dive,
so
we're
kind
of
following
on
from
a
similar
talk
that
we
did
back
in
Seattle,
so
back
in
December
2018,
you
can
see.
We've
got
docker
running
on
the
host
workstation
host
workstations.
This
is
your
laptop
or
on
our
actual
CI
nodes
or
whatever
else.
The
kind
CLI
tells
docker
to
go
and
start
an
image,
and
we
do
a
few
extra
things
in
there
to
actually
get
it
going
and
then,
within
the
node
container.
A
As
you
can
see,
these
components
that
we
spoke
about
before
so
we've
got
system,
D
system
D
then
starts
docker
and
from
there
it's
kind
of
a
typical
kubernetes
architecture
with
the
Kuebler
user
pods
and
all
of
these
other
all
the
extra
bits.
So
we've
got
our
CNI
running
in
there
as
well.
As
I
said,
this
puts
multi
node.
So
there
might
be
multiple
of
these
no
containers,
but
the
illustrative
illustrations
sake.
We
ran
it
kind
of
chain.
One.
This
way
looks
like
so:
we've
done
a
fair
bit
of
work.
A
A
few
things
have
changed
since
then,
yeah
we've
taken
out
docker
itself,
that's
been
replaced
with
container
D.
Now
that
was
yeah.
That
brings
a
whole
load
of
speed
improvements
for
us
with
uploading
those
images
into
the
container.
So
that
is
one
of
the
key
things
has
allowed
us
to
reduce
the
start
time,
the
boot
time
down
to
30
seconds
when
that
comes
to
locally
developing,
that's
really
significant,
because
well,
no
one
likes
waiting
when
it
comes
to
CI
testing.
A
A
We've
also
replaced
the
CNI
with
a
much
a
much
lighter,
stripped
back
version
called
kind
net
D
that
gives
us
a
cross-platform
IP.
Well,
the
CNI
itself
gets
cross-platform
ipv6
support
and
a
number
of
other
things,
but
it's
generally
just
a
lot
simpler,
so
I'll
hand
over
to
Ben.
For
this,
our
darker
and
darker
and.
B
Darker
changes
too
now,
in
the
same
time
since
then,
our
GK
cluster,
that
we
run
this
all
on
as
switch
to
container
D
nodes.
Now
we
actually
have
container
D
and
docker
in
container
D.
This
is
fine
and
then
sit
out
to
work
pretty
well,
but
it
gets
a
little
confusing.
Sometimes
so,
we've
been
working
on
a
lot
to
make
this
better.
As
James
talked
about,
we
fully
pre-loaded
the
images
so
at
runtime.
What
we
used
to
have
to
do
is
we
had
a
bunch
of
docker
archives,
packed
in
the
image.
B
We'd
wait
for
docker
to
start
inside
the
nodes
and
then
once
docker
was
ready.
We
would
load
all
of
those
tar
balls
into
docker
and
that's
pretty
slow.
It's
copying
around
a
lot
of
data.
These
kubernetes
images
are
pretty
big,
so
with
container
D,
there's
a
separate
content
store
and
what
we've
done
is,
instead
of
having
them
actually
unpacked.
B
It's
just
in
the
content
store
with
D
duped
image,
layers
stored
in
a
Content
address
store,
and
we
are
able
to
do
that
at
Build
time
so
that
when
you
boot
the
image
they're
just
already
there
and
it
just
unpacks
them
to
disk
and
starts
running,
and
we
don't
unpack
them
to
disk
ahead
of
time,
because
the
performance
trade-off.
There
is
not
worth
it
and
it
also
allows
you
to
potentially
use
different
snapshots
which
matters
depending
on
which
file
system
you're
running
on.
B
So
this
may
have
to
start
up
a
lot
faster
and
with
these
images
being
fully
pre-loaded
as
James
mentioned,
offline
development
works.
We
also
built
a
tool
for
side
loading
images
so
that
what
you
can
do
instead
of
pushing
your
like
your
application,
that
you've
built
I,
said
I'm
going
to
like
push
that
to
a
registry
and
I've
Cabrini's,
pull
it
down
when
you're
running
CI
since
kind
is
already
local.
You
can
just
build
the
image
in
the
CI
and
then
you
can
run
these
side.
B
Loading
commands
to
copy
that
image
into
the
node
containers
and
into
their
runtime,
and
then
you
never
push
anything.
There's
no
internet
everything's
local,
we
got
it
certified
conformance,
so
we
didn't
just
you
know,
make
sure
the
test
passed.
We
actually
went
through
the
process
with
the
seen
CF
to
make
sure
that
you
know.
We've
got
this
certified
we're
in
the
landscape.
We
got
some
limited
support
for
arm
64
and
PPC
64
le.
B
If
anyone
is
interested
that
we'd
love
some
more
help
with
us
a
little
bit
of
CI,
but
we
haven't
quite
fully
like
you
need
to
build
your
own
images
right
now,
but
it
should
work.
We
got
a
documentation
side
up
and
that
was
actually
a
lot
more
than
you'd.
Think
we've
been
setting
up
a
process
so
that
other
kubernetes
sub
projects
can
request
a
site
like
this
and
you
know,
set
up
their
docks
and
whatnot
the
independent
of
the
main
site.
B
Not
everything
belongs
on
the
main
kubernetes
website,
but
we
really
wanted
to
make
sure
that
we
can.
You
know,
show
you
how
you
do
everything
the
lightweight
minimal
networking
with
James
talked
about
was
actually
a
pretty
big
deal.
It
turns
out
most
of
scene
eyes
are
pretty
expensive
and
maybe
they
make
sense
in
your
environment,
but
for
kind
we
have
everything
on
one
machine
on
a
docker
bridge.
We
know
all
they
peas.
B
We
have
pretty
good
control
over
all
of
it,
so
we're
just
installing
routes
between
the
nodes
for
the
pod
ciders
and
we're
doing
a
basic
bridge
and
host
local
IPM.
It's
very
light
very
simple,
easy
to
follow.
We've
had
a
lot
more
configuration
support,
so
you
can
do
things
like
add
extra
mounts
to
your
nodes,
so
you
can
pass
their
data
and
so
you
can
set
like
the
API
server,
listen
address
and
port
and
just
basic
things
like
that.
We've
got
proxy
support
because
we've
found
a
lot
of
environments
have
issues
with
this.
B
We
got
a
lot
more
going
on
so,
as
we
talked
about,
we've
been
running
this
in
kubernetes
CI,
so
this
is
prowl
we're
running
here.
What
we're
doing
is
we're
running
the
conformance
test,
but
not
the
serial
ones,
so
we
can
run
them
really
fast
and
that's
been
great.
We
have
like
a
whole
grid
of
we
run
this
against,
like
every
kubernetes
version
under
development,
and
we
get
a
much
faster
signal
than
we
otherwise
would
normally.
This
will
take
like
an
hour
and
a
half
plus
with
the
parallel
run.
B
B
We've
got
a
test
here
that
you
know
doesn't
run
as
many
tests
yet,
but
because
it's
so
light
and
local
and
we
save
so
much
time
bringing
up
clusters
and
tearing
them
down
and
managing
all
the
resources
and
whatnot
we're
able
to
get
results,
sometimes
in
this
case
as
low
as
about
15
minutes,
as
opposed
to
well
over
an
hour.
So
it
should
be
a
much
better
experience.
The
other
big
thing
here,
which
we're
considering
doing
talking
with
cig
testing,
is
actually
just
having
the
pre
submits.
That
do.
B
Clusters
use
this
because
anyone
can
run
this
on
their
laptop
and
there's
no
cloud
provider
specific
anything.
Then
we
take
the
cloud
provider
test
and
we
run
them
in
post
submit
and
we
run
across
a
bunch
of
cloud
providers
and
everyone's
on
a
level
playing
field.
Whereas
right
now
because
of
how
the
resources
are,
it's
just
Google
cloud
so
now
we're
gonna
demo,
hopefully
see
how
this
goes.
I,
don't
know
how
many
people
like
booting
clusters
on
stage,
but
so
I've
got
this
patch
here.
B
I
wrote
this
this
morning
trying
to
come
up
with
one
for
a
while
and
eventually
just
said.
Okay,
so
we're
gonna,
make
everything
say:
hello
cube
con.
So
this
is
the
logging
library
in
kubernetes
Kellog,
because
of
course
we
have
K
everything
and
so
I've,
just
added
a
little
bit
in
the
init
function
here
to
make
sure
that
it
will
just
keep
printing
hello,
cube
con,
say
hi
to
everyone
so
because
boot,
building
kubernetes
is
still
slow
and
boring.
B
I've
already
built
a
kind
image,
so
we're
just
gonna
boot
that
image
now
and
skip
the
build
part
take
about
30
seconds.
So
what
it's
doing
is
it's
just
created
the
node
container.
After
making
sure
the
image
is
present,
we
generated
a
cube,
adium
config,
based
on
all
the
inputs,
which
in
this
case
are
defaults
and
now
we're
starting
the
API
server.
That's
what's
gonna
take
the
most
time,
but
it
should
be
about
another
like
10
or
20
seconds
here,.
A
B
B
C
B
B
So,
what's
next
network
management
we've
gotta
nail
that
down?
We
found
a
lot
of
people
have
like
weird
dns
on
their
house
or
like.
In
our
case,
people
like
to
run
in
kubernetes
cluster
is
just
like.
We
do
and
you'll
wind
up
in
a
situation
where
it's
trying
to
talk
to
coordinates
from
like
the
outer
cluster,
and
it's
talking
to
an
IP
I
can't
see
so
we're
gonna
work
some
more
on
that
and
try
to
really
nail
down
all
of
that
make
sure
it
works
great.
B
Every
environment
we're
getting
close
to
having
ipv6
testing
and
kubernetes
CI,
which
will
be
a
first
finally,
and
we
can
start
working
on
making
sure
the
tests
actually
work
shout
out
to
Antonio
who's
actually
sitting
right.
There,
who's
done
a
lot
of
awesome
work
on
this.
Thank
you.
Thank
you
very
much.
It's
gonna
be
exciting.
There's
already
work
on
a
dual
stack
kept,
but
we
don't
have
anything
to
make
sure
that
single
stack
even
works
with
ipv6,
so
we're
gonna
we're
gonna
get
there.
B
So
we're
working
on
some
tooling
trying
some
different
experiments
for
ways
that
you
can
be
able
to
contact
those
things
from
your
hosts
and
maybe
install
about
something
like
that.
We
also
want
to
make
building
cheaper
for
everyone.
So
right
now,
you
actually
need
to
build
kubernetes,
that's
kind
of
unnecessary
because
we're
constantly
building
urban
days
in
CI
and
uploading
in
GCS.
B
So
what
we
want
to
do
is
add
support
for
grabbing
one
of
those
builds,
so
you
can
just
specify
a
kubernetes
version
and
get
an
image
that
we
may
not
have
published
yet,
and
we
just
want
to
make
things
faster
and
we're
trying
to
get
from
alpha
to
beta,
make
things
stable,
but
we're
probably
in
a
break
a
few
more
things
along
the
way,
using
a
fix
a
network
stuff.
So
please,
please,
please
use
a
release
version
and
pin
your
node
images.
B
By
default,
we
pinned
the
image
name
and
the
hash
just
to
make
sure
nothing
changes
or
you
know
if
docker
hub
gets
hacked
that
you're
not
running
a
privileged
container
with
who
knows
what
code
in
it.
So
if
you
want
to
learn
more
about
some
of
that,
we
have
a
different
talk
which
actually
Liz
Frost
from
hep
do
now.
Vmware
submitted
and
on.for
so
wasn't
able
to
make,
but
I
pulled
James
in
here
and
we're
going
to
talk
about
testing
your
kubernetes
absolute
kind
tomorrow
morning,
and
that
will
get
more
into
that
topic.
A
A
B
I
actually
saved
the
output,
but
I
didn't
think
anybody
beat
super
interest
in
this.
So
you
run
kind
billed
note
image
and
if
you
want
you
can
specify
the
image
name
that
you
want
it
to
produce
by
default.
It
will
just
call
it
kinda,
snowed
latest
and
in
this
case
it's
using
the
kubernetes
this
upstream
make
build
for
most
of
the
process.
B
You
will
always
see
don't
use
doctrine
docker,
ok,
if
you're
worried
about
security,
probably
not
put
it
on
a
different
machine
or
something,
but
if
you're
running
a
kubernetes
cluster
you're
already
expecting
it
and
control
your
machine
so
sure,
and
they
also
will
tell
you
don't
use
docker
commit.
You
know,
use
a
use,
a
docker
file
well
yeah,
probably,
but
in
this
case
we've
got
some
a
go
program,
that's
being
very
careful
about
it,
and
what
we're
actually
doing
is
we're
booting
the
image
to
a
sleep
entry
point.
B
So
it's
doing
nothing
on
the
way,
our
exacting
in
and
taking
care
of
things,
and
when
we
know
the
execs
done,
we
know
nothing
is
modifying
anything
and
the
reason
they
tell
you
not
who's
docker
commits
besides.
Reproducibility
is
just
that
something
could
be
modifying
the
file
system
while
it's
trying
to
save
and
then
it
can
get
corrupted.
But
we
know
that's
not
happening
because
we
shut
everything
down
and
we
know
that
alt
the
only
process
running
keeping
the
container
open
is
a
sleep
call
and
it
works
great.
A
Cool
so
yeah
as
Ben
was
saying:
we've
got
another
talk
tomorrow
morning,
which
is
kind
of
focusing
on
more
of
their
testing
kubernetes
applications
use
case.
That's
at
11:05
a.m.
and
I
actually
don't
know
the
room
number.
So
I
probably
should
go.
Look
that
one
up,
but
yeah
do
come
along
if
you're
interested
seeing
any
more
around
this.
Thank
you
very
much.