►
From YouTube: SIG Testing Intro - Sean Chase & Erick Fejta, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
SIG Testing Intro - Sean Chase & Erick Fejta, Google
This session will provide an overview of the testing infrastructure and automation used by the kubernetes project. We manage over 180 GitHub repos, generate test results from over 10,000 jobs per day. We'll walk through some of the improvements we've made to enable contributor self-service since last KubeCon.
https://sched.co/UajU
A
All
right,
it
is
10:55.
So,
let's
begin,
can
everyone
hear
me?
Okay,
all
right.
This
is
on
then
hello
and
welcome
to
the
SIG's
testing
intro.
My
name
is
sean
chase.
This
is
eric.
Fada
I
am
missing
testing
newbie
and
he
is
one
of
our
sync
testing
chairs
and
we
both
work
at
Google
and
I'm.
Gonna.
Tell
you
all
about
SIG's
testing,
so
this
is
kubernetes.
A
Kubernetes
has
over
a
thousand
active
contributors
over
35,000
contributors
to
date
and
69
very
nice
repositories
that
are
constantly
pushing
code.
There's
a
lot
of
code
changes
all
the
time
and
it
would
be
really
nice
if
kubernetes
worked
and
if
we
had
tests.
So
that
way
we
can
keep
the
kubernetes
jungle
juice
flowing.
A
So
sig
testings
ultimate
goal
is
to
keep
the
current
release
of
kubernetes
functioning
and
make
sure
that
it
gets
tested
and
that
we
have
all
the
infrastructure
and
the
support
we
need
to
do
it.
We
do
this
by
maintaining
a
lot
of
different
tools.
Here's
an
overview
of
everything
we
do.
I
am
going
I'm
going
to
go
into
one
of
them
prowl
a
little
bit
and
then
near
the
end.
A
Eric
is
going
to
tell
you
about
all
of
the
rest,
because
there's
a
lot
to
go
over
prowl
is
the
CI
CD
system
that
runs
on
kubernetes,
that
the
kubernetes
community
also
uses
it's
complicated.
It's
got
a
lot
of
pieces
and
a
lot
of
the
pieces
do
a
lot
of
different
things
and
if
you're
new
like
I
was
it
can
be
a
little
intimidating
to
figure
out
what
this
means.
I'm
not
going
to
go
over
this.
What
I'm,
gonna
do
is
I'm
gonna
tell
you.
A
Instead
what
it
was
like
for
the
first
time
I
submitted
PR
to
the
kubernetes
community,
so
I
started
out.
I
pulled
up
a
good
first
issue:
I
copied
a
bunch
of
stuff
from
stack.
Oh
I
did
some
coding
and
then
I
got
it
to
test
and
is
pretty
happy
with
it,
and
I
ran
tap
I
submitted
to
PR
on
github
after
I.
Did
that
two
things
happened
at
once.
One
was
that
a
very
nice
robot
came
by
and
tell
told
me
what
I
did
wrong.
A
All
of
the
tests
that
the
robot
did
are
called
pre
submits
we'll
get
to
that
in
a
minute.
But
I
go
back.
I
change,
my
PR.
This
goes
back
and
forth
for
a
little
bit
until
eventually
I
make
some
human
happy
and
they
say
all
right.
This
change
is
good
to
go
once
they
say
the
change
is
good
to
go.
The
robot
runs
a
bunch
more
other
code
that
we'll
get
into
we'll
call
those
post
submits,
and
then
my
code
is
part
of
kubernetes
forever.
No
take-backs.
A
Pre
submits
are
when
somebody
wants
to
put
the
code
in
post
submits,
are
after
somebody's
put
the
code
in
and
periodic
happen,
every
X
arbitrary
units
of
time
or
every
Tuesday
at
1
a.m.
or
whatever
and
Y
is
any
image.
If
you
can
put
it
in
an
image
and
pass
it
some
commands,
then
prowl
can
run
it
as
a
proud
job,
which
makes
it
incredibly
flexible
and
a
little
hard
to
understand
sometimes.
A
But
this
is
what
a
proud
job
does.
Prow
jobs
also
have
something's
packaged
with
them.
Most
of
the
time.
These
things
are
called
pod
utilities.
When
you
run
an
image,
it
has
a
number
of
containers
that
kind
of
go
on
the
side
with
it
that
run
with
it
that
do
useful
things.
They
mostly
do
two
very
useful
things.
One
is
that
they
pull
your
code
from
github,
so
you
don't
have
to
write,
get
clone
every
time,
and
the
second
is
that
it
exports
the
logs
to
external
tools
like
test
grid
and
Griffin
ax.
A
So
that
way,
when
the
tests
fail,
we
can
see
that
the
tests
failed.
These
kind
of
go
with
it,
because
we
want
prowl
jobs
to
generally
do
these
things
and
they
are
activated
with
a
single
animal
line.
Decorate
equals
true
right
there
that
adds
four
pods
and
does
all
of
these
very
nice
things
for
you
and
it's
very
convenient
once
you
know
what's
happening
so
I've
talked
about
X
the
trigger
and
Y
the
payload
a
little
bit,
but
I
kind
of
glossed
over
exactly
what
do
means.
A
What
Prout,
specifically
a
component
of
prowl
called
plank,
is
doing
is
that
it
takes
your
image.
It
bundles
it
up
with
a
bunch
of
very
nice
neat
stuff,
and
then
it
launches
it
off
to
some
kubernetes
cluster
and
then
you're,
proud
job
runs
and
then
it
dies
and
when
it
runs
it
runs
either
a
pre
submit
test
or
it
deploys
some
code
or
whatever
that
arbitrary
image
is
build.
Clusters
can
be
other
kubernetes
instances
or
the
same
one.
A
A
A
Pg
unkind.
Does
that
process
that
I
just
described
only
instead
of
building
an
entire
prow
to
test
your
proud
job?
You
run
a
script
so
instead
of
a
ship
you're
a
little
man
with
a
cannon
and
it
creates
a
client,
a
kind
cluster
to
actually
run
your
proud
job
in.
So
it's
still
launching
it
to
a
kubernetes
cluster
somewhere.
That
somewhere
just
happens
to
be
in
an
image
on
your
local
machine,
and
this
is
great
for
testing
prowl
jobs.
A
A
So
pre
submits
are
generally
used
for
running
code
and
making
sure
nobody
puts
in
anything,
really
crazy.
If
you
put
in
go
code
that
doesn't
compile
some
pre
submit
should
run
and
say,
this
doesn't
actually
like
make
sense.
Please
don't
merge
it.
Post
submits,
are
a
bit
more
powerful
because,
with
a
post
submit
after
you
change
code,
say
a
configuration
file,
you
could
then
change
the
state
of
the
world.
You
could
have
it
run.
A
The
way
that
the
kubernetes
community
manages
this
is
through
ownership
and
the
concept
of
owners
files
throughout
not
only
testing
for
but
any
of
the
kubernetes
repositories.
You
will
see
owners
files
and
we
use
these
basically
to
determine
who
needs
to
say
yes
to
your
PR
to
let
it
merge
remember
earlier,
when
I
made
actually
made
a
human
happy
and
then
my
code
started
to
merge.
A
A
So
if
you
want
to
see
this
in
action,
you
can
actually
see
that
in
our
prowl
instance
that
the
kubernetes
community
uses,
which
is
a
the
kubernetes
testing
for
github
together,
it's
great
you
can
see
that
there
is
a
periodic
job
that
looks
for
a
new
version
and
creates
a
PR.
A
robot
wrote
this
and
that
PR
updates
some
cluster
file
from
the
old
version
to
the
new
version
and
then
a
human
approves
it.
A
And
then
the
cluster
file
changes
and
then
a
post
submit
launches
to
actually
update
prowl
itself,
and
this
concept
can
kind
of
be
extended
much
further.
So
if
you
want
to
use
prowl
yourself
as
a
CI
CD
solution,
you
can
write
prowl
jobs
to
do
actually
anything
you
want,
because
it
takes
an
arbitrary
image.
B
Alright
great
Thank,
You,
Sean
and
yeah,
so
we
this,
you
know
we
were
already.
We
talked
a
little
bit
about
prowl
and
this
is
one
of
the
important
things
that
sig
testing
is
involved
with
and
I.
Think
that
you
know
hopefully
there.
Maybe
this
is
getting
you
excited
about
SiC
testing,
so
I
wanted
to,
or
maybe
there's
other
things
that
are
interesting
to
you.
If
you
want
to
learn
more
about
prowl,
actually,
Steve
and
Alvaro
are
talking
in
a
little
in
room,
6,
&,
7,
I.
B
Think
it's
seven,
so
yeah
so
I
would
encourage
you
to
go
there
to
learn
more
about
it,
and
so
I'm
gonna
talk
about
some
of
the
other
things
that
we
work
on
and
I.
Think
one
of
the
cool
things
about
the
justa
kubernetes
in
general
and
sig
testing
is
no
different.
Is
that
you
know
we
tend
to
be
pretty
friendly
and
you
know
so.
B
We
have
a
bunch
of
people
like
Sean
or
others
who
you
know,
have
fairly
rapidly
gone
from
being
unfamiliar
with
kubernetes
unfamiliar
with
sig
testing
and
proud
to
being
able
to
meaningfully
contribute
and
have
cool
things
that
they
can.
You
know
talk
about
and
show
off
here,
so
I
think
that
is
cool,
and
maybe
that
can
be
one
of
you
for
the
next
coupon,
so
test
grid
is
I.
B
It
sort
of
pushed
them
into
a
visualized
grid
where
you
have
runs
on
the
columns
and
tests
on
rows,
and
then
it
has
all
sorts
of
exciting
features
to
let
you
filter
and
sorts
and
group
you
can
click
and
drag
and
sort
of
figure
out
like
which
commits
might
have
caused
the
thing
to
go
red.
It's
pretty
cool,
a
new
thing
that
Sean
is
actually
pretty
instrumental
in
writing
recently
is
the
transfigure
tool
that
allows
you
to
automatically
manage
the.
B
If
you
have
proud
jobs
in
your
org
and
you
want
to
publish
them
on
a
regular
basis
to
the
kubernetes
test
grid,
their
various
companies
that
publish
their
public
results
that
you
can
use
transfigure
to
automatically
manage
that
so
using
the
same
pattern
as
in
the
previous
slide.
It's
will.
You
know,
determine
what
your
test
grid
should
look
like,
based
on
what
you're,
proud
jobs
are
and
then
automatically
create
a
pull
request
to
update
the
kubernetes
test
grid
with
those
results,
and
so
we're
in
the
process
of
open
sourcing.
B
More
of
it,
we're
going
to
continue
on
that.
We
have
ideas
for
automatic
issue
management
to
where
like,
if
something
goes,
red,
it'll
open
an
issue
and
when
it
goes
back,
green
it'll
automatically
close
it
and
right
now
it
is
largely
a
App
Engine
Python
app,
and
we
want
to
eventually
turn
it
into
a
go.
App
that
runs
on
kubernetes,
so
kind
is
super
cool.
There
is
a
deep
dive
about
that,
as
well
as
a
workshop
tomorrow.
B
B
So
so
kind
is
super
cool
because
it
allows,
as
he
mentioned
earlier,
it
allows
you
to
schedule
whole
pod,
our
whole
cluster,
into
a
single
container.
There's
the
command.
It's
really
easy
to
use.
There's
you
know
a
lot
of
traction
behind
it.
We
want
to
add
multi
node
persistent
volume,
support,
kubernetes
is
a
large
community,
and
so
we
sort
of
test
the
boundaries
of
what
you
can
accomplish
with
slack,
and
so
we
have
some
tooling
to
help
us
manage
that.
B
So
something
that's
unusual,
I
feel,
like
slack,
is
usually
used
for
sort
of
more
closed
corporate
communities.
Kubernetes
is
the
open
public
community
and
that
sort
of
comes
with
challenges
around
size,
also
abuse,
and
so
we
sort
of
have
some
tooling
to
help
with
that,
and
then
also
we
really
like
this
pattern
of
to
make
things
open
sort
of
declaring
what
channels
we
want
to
exist,
which
is
the
tempest
tool
that
allows
us
to
sort
of.
If
you
want
to
create
a
new
channel,
you
merge
a
PR
and
then
it'll
get
created.
We
want
to.
B
You
know
we're
sort
of
working
with
adding
more
alerting,
and
we
want
to
sort
of
make.
The
UI
of
our
extensions
better
Bosco's
is
how
we
do.
We
sort
is
how
we
sort
of
do
into
end
testing.
It
allows
you
to
rent
resources,
so
it
has
a
list
of
resources
in
state
X
and
it
allows
you
to
atomically
move
them
to
state
Y
and
then
three,
furthermore,
from
Y
to
Z
or
Y
back
to
X.
B
So
we
use
this
it's
extensible,
but
we
primarily
use
it
to
give
tests
a
dynamic
project
that
they
can
use
exclusively
for
testing,
and
then
we
have
a
bunch
of
janitors
which
will
clean
up
resources
after
the
tests
are
over.
So
we
want
to
win
this
fails.
It's
really
sad
because
all
of
our
testing
dies,
and
so
we
need
better
monitoring
around
to
that
yes,
go
fridge
is
a
utility
to
sort
of
help
you
with
code
coverage,
and
so
we
try
to
do
some
into
end
code
coverage
and
that
with
Gophers
allows
us
to
merge.
B
B
Cube
test
is
the
interface
that
we
have
for
Indian
testing,
so
it
sort
of
gives
you
a
single
command
that
can
do
a
bunch
of
stuff
like
deploy
the
cluster,
maybe
do
an
upgrade
or
run
performance
tests
or
whatever
other
test.
You
want,
then
tear
it
down
at
the
end
and
it's
aware
of
like
where
it
needs
to
download
the
kubernetes
version
or
if
it
needs
to
build
it
and
whether
it
should
be
deploying
to
this
or
that
cloud
provider
and
yeah.
B
So
it's
pretty
powerful,
but
it's
also
extremely
complicated,
they're
sort
of
like
in
order
to
run
it.
You
have
to
send
it
a
bajillion
flags
and
an
ability,
lien
environment
variables,
so
we've
been
sort
of
in
the
process
of
moving
in
the
environment
variables
to
flags,
and
then
we
also
generally
just
want
to
make
it
reduce
the
minimum
number
of
flags
you
need
to
run.
B
B
The
PR
dashboard,
I
love
it
tracks
PR
attention.
So
if
you
have
a
ton
of
PRS
that
you
need
to
review
or
you
have
sent
out
a
ton
of
PRS,
its
github
doesn't
provide
a
great
way
to
see
which
ones
you
know.
I
have
actually
replied
to
and
which
ones
are
waiting
for
me
to
do
something
on
and
so
good
the
goober
Nader
PR
dashboard.
Does
that
it'll
sort
of
sorts?
Your
list
about
like
here
are
PRS
that
need
your
attention
and
here
are
PRS
that
you
need
to
whatever
go
bug.
B
Someone
and
yeah,
so
I
think
it's
great.
We
want
to
also
convert
this
to
go
and
kubernetes
from
Python
in
App
Engine.
This
was
kind
of
an
interesting
experiment.
We
sort
of
created
prowl
and
goober
nater
at
the
same
time,
and
prowl
has
a
bunch
of
excited
people
working
on
it
now,
maybe
it's
just
I'm
not
sure
why
it
is.
B
But
potentially
one
thing
is
that
it's
in
go
which
the
kubernetes
community
seems
to
love
and
not
in
Python,
which
has
you
know
whatever
it's
it's
stuff
and
then
there's
the
testing
Commons
project,
which
is
a
tool
to
or
whatever
is
a
frame,
is
trying
to
give
people
a
consistent
set
of
frameworks
to
do
Indian
testing.
They
have
the
testing
Commons
channel
on
slack,
and
then
they
have
meetings,
I
think
every
Monday
you
can
find
more
about
it,
yeah
there
and
yes,
I,
guess
that
is
it
we
are
friendly.
B
B
B
Right,
that's
it
yeah!
Okay!
So
essentially
you
know
the
I
guess
the
K
Native
community
likes
to
do
everything
with
go
test
and
maybe
be
nice
to
sort
of
import
that
library
and
do
things
like
deploy
the
cluster
and
whatnot.
That
is
a
really
good
idea.
I
think
you
know.
Originally
it
was
kind
of
just
a
way
to
lock.
Kubernetes
has
lots
of
bash,
and
so
this
was
originally
a
way
to
not
have
it
in
bash
and
yeah.
That's
a
really
great
point
and
yeah.
C
B
A
I,
guess
paraphrase
your
question:
the
let
me
know
if
I'm
perjuring
you
here,
the
question
is
about
the
scope
of
the
work
that
we
do.
Does
it
apply
to
just
kubernetes?
Does
it
apply
to
everyone
and
have
we
reached
out
to
others,
I'm
gonna,
try
and
answer
that
with
two
parts.
One
is
that
within
the
kubernetes
community
we
have
a
lot
of
outreach
to
other
SIG's
to
help
them
get
their
testing
working.
A
There's
a
big
push,
I
believe
by
sig
architecture
in
icing,
conformance
conformance
tests
are
a
big
thing
that
we
are
working
on
and
we've
been
working
closely
with
them
to
make
that
happen,
and
as
for
things
that
are
not
kubernetes.
A
lot
of
these
tools
are
not
only
open
source
but
regularly
see
use
on
things
that
are
not
kubernetes
prowl,
for
example,
has
on
its
front
page
in
the
github
repo.
It
has
a
list
of
all
of
these
companies
that
we
know
of
that
are
using
prowl.
A
There
could
be
more
because
it's
open
source
and
you
can
just
take
it
and
we'd-
never
know,
but
there's
there's
a
lot
and
so
a
lot
of
these
tools
do
you
have
testing
use
in
general
beyond
just
what
kubernetes
uses
in
needs,
they're
kind
of
built
for
kubernetes,
but
they're,
very
extensible
as
a
whole
and
I
think
that
goes
for
all
of
our
tools.
I
can't
think
of
an
exception,
but
there's
a
lot
of
them.
So
if
I'm
wrong,
send
me
a
postcard
or
something.
B
Yeah,
so
the
question
is
decorating:
will
export
your
logs
to
GCS?
Is
there
you
know
what
plans
are
there
for
other
destinations
and
there
have
been
some
various
every
now
and
again,
someone
you
know
comes
up
with
a
proposal
for
that
and
yeah.
So
I
think
it's.
You
know
that
I
think
that's
a
potentially
a
great
way
to
contribute,
and
so
yeah
I
mean
I,
don't
think,
there's
anything
I,
don't
think.
There's
any
opposition
to
that.
But
just
sort
of
you
know
I
guess
open-source
works
by
what
people
are
interested
in
willing
to
work.
B
B
B
B
Okay,
yeah
III,
guess
I
mean
the
the
core
design
of
decoration
is
that
it
will
add
in
it
containers
to
your
pod.
So
if
you
that's,
how
we
essentially
add
an
init
container
to
check
out
the
repo
and
init
container
to
upload
some
metadata
to
cloud
storage
and
yeah
if
your
cluster
does
not
allow
init
containers,
that
obviously
is
not
going
to
work
and
you'll
have
to
manually.
Do
it.
B
Oh,
oh
I
see
I,
see
you
haven't
it
contained.
You
want
to
write
a
proud
job
that
hasn't
in
it
container.
Oh
right,
yeah,
that's
that
that's
the
same
thing
it's
kind
of
like
no
one
has
used
it.
No
none!
None
of
the
maintainer
x'
have
needed
to
use
that.
Yet
that's
a
solvable
problem,
but
it
hasn't
been
solved
yet
so
another
great
way
to
contribute
to
pro
is
to
make
support.
A
We've
had
some
mumblings
about
wanting
more
than
one
non
in
it
container,
but
as
it
is
with
open
source
in
general,
there
are
a
lot
of
great
ideas
and
a
lot
of
really
wonderful
ways
to
contribute.
So
please
help
us
pyaare
is
welcome,
we're
all
very
friendly
we're
on
slack
and
yeah.
There's
a
there's,
a
bunch
of
things
like
that.
A
So
the
question
was:
is
this
the
platform
and
the
tools
that
everyone
in
the
kubernetes
uses
across
all
of
the
kubernetes
org,
or
are
there
other
tools
out
there
to
my
knowledge?
I,
don't
believe
there
are
a
lot
of.
There
are
other
tools
beyond
this.
The
kubernetes
org
as
a
whole
uses
prowl
and
uses
test
grid,
and
all
of
these
other
things,
what
does
tend
to
diverge
between
organizations
is
that
we
don't
really
enforce
what
actually
runs
your
tests.
We
say
here
is
an
image
use
that
image
to
run
your
tests.
A
Yeah,
sometimes
there's
a
directory
called
hack.
I
know
that
ginkgo
is
used,
though
I
don't
actually
understand
what
that
is,
but
there
are
a
lot
of
different
ways
to
run
your
tests
depending
on
what
kind
of
tests
you
need,
so
that
will
vary
from
org
to
organ
person
to
person,
but
by
and
large,
to
my
knowledge,
they're
all
being
run
by
prowl.
They
all
generally
expose
stuff
on
test
grid
and
graph
fauna.
They
all
could
use.
Go
fridge,
though
of
my
understanding
is
a
lot
of
people.
Just
don't
does
that
answer
your
question.
D
B
I
would
say
that
the
change
is
now
prowl
is
available
in
all
repos,
but
you
know
it's
kind
of
I
mean
like
one
thing
you
it
yeah.
It
is
what
we
would
like
people
to
use,
but
they're
like
right
now,
there's
not
a
great
solution
for
if
you
want
to
do
testing
on
Mac
prowess
design
for
creating
a
pod,
and
you
can't
create
a
pod
on
a
Mac.
So
that
might
be
a
really
great.
You
know
good
reason
to
use
a
different
CI
system
to
run
your
tests,
but
yeah
we
try
it.
B
You
know,
because
you
get
all
the
other
goodies
like
test
grade
and
whatnot
if
you
use
prowl,
and
so
we
try
to
get
people.
But
you
know
it's
a
large
community
and
so
there's
going
to
be
lots
of
different
ways
that
people
do
things,
but
we
try
to
standardize
on
that
all
right
and
I.
Think
with
that
we
are
out
of
time,
but
we
again
we
do
have
the
stickers
and
do
not
be
shy.