►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Using Prow for Testing Outside of K8s - Matt Landis, Amazon Web Services
Prow plays an important role in every kubernetes PR, but it doesn't need to stop there. We will discuss how Amazon uses prow to test EKS and why you should consider using prow as part of your test setup. We will briefly cover the components that prow, how to use them, how to create your own jobs and how to extend prow to meet your testing needs.
To learn more: https://sched.co/GrSz
A
All
right,
hi
everybody
you're
at
using
prowl
for
testing
outside
of
Kate's,
if
you're
not
here
for
that
you're
in
the
wrong
room,
but
you
feel
free
to
stay.
My
name
is
Matt
Landis
I'm,
an
engineer
at
AWS
on
eks
I've,
been
at
Amazon
for
about
four
years,
just
over
four
years
now
and
I've
been
on
eks
for
approaching
a
year
and
a
half,
so
we're
gonna
talk
about
what
prowl
is
and
how
you
can
look
at
using
it
for
your
your
testing
needs
if
it
fits
so.
A
A
A
The
core
of
it
is,
it
will
respond
to
events
that
can
come
from
multiple
places
and
it
will
run
jobs
and
then
report
the
status
of
those
jobs
to
other
services
end.
It
say
you
know
in
our
kubernetes
way
it
is
a
micro
service
architecture,
which
means
we
can
replace
pieces
that
don't
do
what
we
need,
or
we
need
to
do
something
else
and
we'll
talk
about
that.
A
little
bit
more
as
we
go
in,
but
the
core
is
it
gives
you
what
you
need
to
have
a
CIO
DCCCD
system
right.
A
You
have
an
event
that
happens,
you've
got
a
pull
request
or
a
commit
that
happens
emerge
and
you
need
to
test
it,
and
then
it
runs
the
job
and
reports
our
results.
So
if
you
jump
into
the
getting
started
guide
your
and
run
the
animal
on
your
cluster
you're
going
to
get
something
that
looks
a
little
bit
like
this
yeah.
A
That's
gonna
trip
me
up
is
how
to
correctly
say
this
word
horologium
is
that
I
got
it
maybe
close
enough,
so
horologium
is
responsible
for
running
schedule,
periodic
jobs
within
a
proud
cluster,
and
so,
when
you
start
up
your
test,
cluster
you're
going
to
have
a
deployment
for
corolla
gam
I'm,
going
to
stop
saying
that
word
and
it's
gonna
be
sitting
there
running,
and
it
reads
it's
configure
out
of
the
config
map
and
there's
a
ml
in
there.
That
has
a
periodic
job.
A
That's
going
to
run
every
10
minutes
and
use
a
alpine
container
to
print
out
the
date
it's
fairly
simple,
and
when
you're
coming
from
this,
you
can
put
any
container.
You
want
there
and
run
anything
you
want.
So
if
you
have
a
container,
that
is
a
test
that
can
be
put
into
a
container,
you
can
now
run
it
on
a
periodic
schedule,
and
so
all
Herodium
does
is
create
a
proud
job
entry
and
stick
it
in
the
CRT.
A
So
what
do
we
do
from
here
from
here?
We're
going
to
have
plank,
which
is
kind
of
the
core
of
running
the
jobs
within
a
proud
cluster
and
plank,
is
going
to
pick
it's
sitting
there
watching
on
entry
changes
to
the
CRT,
and
it's
gonna
pick
this
up
and
execute
the
job
and
manage
the
life
cycle
of
it.
It's
looking
for
jobs
that
were
created
by
prowl.
A
So
if
it
a
Giunta
burn
it
is.
That
means
it
is
a
community
job
and
can
be
run
on
the
kubernetes
cluster.
It's
going
to
start
the
job.
It's
going
to
watch
it
and
update
the
state
and
the
other
thing
that
we'll
do
if
you're
running
pre-summit
job
is
it
to
text.
If
there
is
been
a
change
right,
so
it
kicked
off
a
presubmit
job
for
my
PR
I
noticed
that
I
had
a
bug
in
it.
I
go
and
update.
A
A
It
updates
the
state
depending
of
the
job.
So
when
it's
triggered
by
her
Oldham,
it's
has
a
state
of
triggered.
It
gets
updated
depending
it
set
some
flags
and
then
it
creates
a
pod
that
it's
gonna
sit
and
monitor
for
completion
or
termination,
hopefully,
completion
with
a
exit
status
of
1
or
0,
not
one
once
bad
yeah.
So
it's
also
going
to
insert
some
environmental
variables
for
us
there,
like
the
build
number
of
the
job
name.
A
A
We
get
something
that
looks
a
little
bit
like
this
right.
We've
executed
a
lot
recently,
you'll
notice
that
the
the
time
there's
a
little
bit
tighter
than
10
minutes,
because
I
wanted
a
few
more
entries
on
my
dashboard
and
I.
Don't
want
to
wait
40
minutes
so
yeah.
You
can
see
what's
past,
you
get
a
little
red
mark
there
if
you've
got
jobs
failing,
and
so
you
can
get
a
quick
view
of
you
know
what
is
my
recent
history?
A
You
can
filter
by
different
job
names
and
what
is
the
state
of
the
job,
and
so
now
we've
been
running
this
and
we've
been
running
it
for
a
long
time
and
if
you
let
it
run
forever.
You're
gonna
end
up
with
a
whole
bunch
of
CR
DS
floating
around
and
potentially
some
pods
that
haven't
been
deleted
and
that's
where
sinker
comes
in
sinker
is
kind
of
the
cleanup,
so
it
deletes
any
completed,
proud
jobs
from
the
the
CR
D
after
whatever
the
designated
time
frame
is
right.
You
want
to
keep
them
around
for
a
while.
A
You
know
four
hundred
thousand
entries,
but
it
does
what
one
key
thing
it
does
is
keep
around
the
most
recent
periodic
job
because
it
needs
to
know
when
the
last
time
it
ran
was
to
kick
off
the
next
one
right
if
I
end
up
restarting
her
old
game,
it's
gonna
lose
that
state,
and
you
know
if
you've
got
a
job
supposed
to
run
every
ten
minutes
and
you've
deleted
it.
It
doesn't
know
that
it
was
triggered
so
it'll
run
it
again
immediately.
A
These
are
the
components
that
we've
we've
made
the
most
use
of,
and
then
you
get
into
hook
hook
in
my
opinion,
is
probably
one
of
the
coolest
things
about
prowl
and
that's
its
ability
to
respond
to
these
github
events
and
how
modular
it
is
with
plugins,
there's
tons
of
plugins
and
we're
gonna
go
over
a
few
of
those
here
in
a
little
bit,
and
it
also
supports
external
plugins.
So
it's
not
just
the
stuff,
that's
been
baked
in
and
checked
into
the
main
upstream
repository
that
you
can
use.
A
If
you
look
at
the
definition
file-
and
you
know,
there's
some
really
important
stuff
in
there
like
CLA,
you
know
everybody
has
to
sign
the
CLA
before
you
can
do
anything
to
any
of
the
kubernetes
repositories,
lgt
m
and
then
our
favorites
like
dog
and
cat.
So
those
need
more
usage
by
the
way.
I
haven't
seen
a
lot
of
dogs
on
read
at
least
pr's
I've
looked
at
recently,
and
then
you
can
also
apply
so
you
get
this
hierarchical
structure
and
how
you
can
define
plugins
right.
A
So
we've
decided
that
everybody
needs
the
CLA,
so
it
needs
to
be
on
all
kubernetes
repositories,
but
you
may
have
want
other
plugins
to
only
apply
to
your
sub
repository.
So
if
your
kubernetes
cloud
provider
AWS,
you
know
we
have
things
like
the
welcome
message
and
the
trigger,
which
is
on
almost
most
repositories.
You'll
see
that
one
on
which
is
responsible
for
triggering
the
jobs.
A
A
Thank
you
to
all
those
people
who
who
put
that
out.
There
I've
heard
some
impressive
numbers
on
the
last
when
the
last
release
was
unblocked
on
how
quickly
it
was
able
to
clear
the
PR
backlog,
yeah.
So
retests
everything
that's
been
done,
so
it
batches
up
a
bunch
of
PRS
and
retests
it
against
the
current
head,
and
if
everything
passes,
it's
going
to
go
ahead
and
merge
our
changes
in,
so
we
don't
have
to
do
that
by
hand
yay,
so
that
is
kind
of
where
you're
at
with
you
know
what
is
prowl.
A
Some
things
that
I've
seen
recently
that
I
did
not
put
on
the
slides
and
kind
of
an
extension
point
is
you
know
we
talked
about
hook
and
that
only
does
github
style
hooks
or
github
hook,
style
messages.
I
know
recently
Garret
was
checked
in
so
we
have
support
for
Garrett
and
I'd
like
to
see
a
lot
of
there's
been
some
talk
about
doing
work
where
we
make
these
interfaces
a
little
bit
more
generic.
So
we
can
reuse
some
of
these
plugins
and
other
features
to
apply
to
more
source
control
systems.
A
I
know
I,
see,
cently
saw
somebody
on
slack
a
spur.
You
know.
Can
we
do
bitbucket,
or
can
we
do
get
lab
and
I
think
that
the
core
of
prowl
is
there
that
it'd
be
great?
If
we
could
extend
this
beyond
this,
so
now
we
have
a
little
bit
of
background
on
what
prowl
is
I'm
going
to
talk
about
kind
of
where
we
started
with
it
any
KS
and
what
we're
using
it
for
and
where
I'd
like
to
go
from
here.
So
in
the
beginning,
so
bob
comes
up
to
me
and
says:
hey
Matt.
A
So
I
did
a
one-off
in
said:
okay,
yeah
Bob
would
pass
it
he's
like
okay,
we
need
to
make
sure
we
don't
break
okay.
How
do
we
want
to
run
this?
What
is
the
community
using
for
continuous
jobs?
Like
this
and
that's
when
I
found
proud
so
I
started
looking
at
prowl
more
and
said
it
seems
like
it
does.
What
we
want
to
do,
plus
there's
a
lot
of
other
tests
that
are
already
baked
into
this,
that
we
can
look
at
leveraging
as
we
go
forward.
You
know
not
reinvent
the
wheel.
Kubernetes
is
using
this.
A
Let's,
let's
build
on
top
of
it
and
see
where
we
can
go
so
we
started
out.
We
had
one
prowl
job.
There
was
a
periodic
that
ran
every
two
hours.
That
said,
does
our
beta
environment,
past
conformance,
and
so
the
conformance
test?
Take
you
know
an
hour
so
to
run
so
to
our
interval
seems
like
a
good
idea
to
kind
of
get
that
cycle
going,
except
that
now
you're
two
hours
behind.
A
A
You
know
we
stripped
out
the
serial
stuff
said
we
can
find
out
if
that
breaks
later.
If
we
run
this
stuff
in
parallel
the
tests,
you
can
run
them
in
like
five
to
ten
minutes,
much
much
better
iteration,
and
so
that's
what
we
went
with
for
a
while
and
we're
getting
ready
to
launch
I
said:
oh
now
we
should
run
tests
and
our
production
environments.
You
know
we
need.
We
know
beta,
is
working
well
everything's
together,
but
it'd
be
nice
to
have
a
deeper
signal
on
our
production
environments.
A
So
we
started
running
conformance
on
the
two
regions
we
launched
with
us
two
and
US
East
one,
and
so
now
we're
at
six
jobs,
and
so
this
config
file
is
starting
to
grow
and
then
we
verified
it
announced
we
had
GPU
support.
So
now
we
have
nine
jobs
that
are
running
periodically
and
the
files
growing
and
every
time
we
you
know,
want
to
change
an
omni
ID
that
we
launched
these
jobs
with
now
I
have
to
edit
nine
places.
A
Instead
of
you
know
one
per
region,
and
then
we
have
HP
a
support
now
we're
at
twelve
jobs-
and
this
is
just
gonna-
continue
to
grow.
As
we
launched
us,
one
I
got
sixteen
jobs,
and
so
what
we
looked
at
doing
was
using
how
to
render
these
templates
right.
We
want
to
run
a
lot
of
these
jobs
across
all
of
our
regions
at
some
interval,
just
to
have
some
signal
to
make
sure
stuffs
not
going
wrong.
You
know
we
test
very
heavily
in
beta
and
the
first
regions
you
push
out.
A
You
tend
to
test
heavily,
but
it's
nice
to
have
that
check
mark.
That
stuff
is
going
well
across
the
board,
and
so
we
wrote
a
template
for
home,
we're
not
using
Helen
to
deploy
we're
just
using
it
to
render
this
the
the
ya
know
file
and
then
apply
that
to
prowl
and
update
the
config,
so
that
made
launching
us
ease
to
much
much
simpler.
We
added
a
u.s.
A
East
two
to
the
list
of
regions,
and
now
we
have
jobs
running
in
us
ease
to
for
all
of
our
jobs,
and
so
there
is
one
little
thing
that
I
kind
of
lied
about
going
through.
This
is
when
we
were
running
our
fast
jobs.
We
wanted
to
run
them
more
frequently,
and
so
that
means
we
created
four
entries
for
the
fast
jobs
jobs,
one
two
three
and
four,
because
prowl
currently
star
PR
incoming,
doesn't
support
concurrent
periodic
jobs
right.
A
So
if
you
have
a
periodic
job-
and
you
say,
I
want
to
run
this
every
five
minutes
and
it
takes
an
hour
to
run
it's
actually
going
to
run
every
hour
every
hour
and
we'll
wait
for
the
previous
job
to
complete.
So
that's
why
we
have
four
jobs
running
and
so
I
did
a
little
bit
of
digging
and
figured
out
that
we
can.
It
doesn't
take
too
much
work
to
a
while,
proud
to
run
periodic
jobs.
Every
10
minutes
prowl
currently
in.
A
Corolla
geom
ignores
max
concurrency,
it's
it
just
says:
if
there
is
a
job
running,
don't
run
anything
so
we
have
a
PR
open,
that's
kind
of
debating
what
the
default
behavior
should
be
for
presubmit
jobs,
max
concurrency
defaults
to
zero,
which
means
infinite
for
periodic
jobs.
Currently,
the
default
is
ignore
this
thing
and
treat
it
as
one
so
we're
trying
to
figure
out
what
the
right
use
case
is
there.
A
If
you
have
any
opinions,
come
on
and
jump
on
the
PR
but
I
think
we're
getting
close
to
a
resolution
on
that
and
how
to
handle
that.
But
now
we
can
just
say
every
10
minutes
run
it,
and
if
the
job
takes
an
hour,
we'll
still
have
that
signal
of
this
started
every
10
minutes.
So
we
get
a
much
better.
It
works
better
with
our
monitoring
too,
so
that
it
made
monitors
easier
to
reason
about
right.
We
know
we
have
something
happening
every
10
minutes.
A
But
where
are
we
going
from
here?
One
thing
I'd
like
to
see
is
code
commit
integration.
We
have
some
of
our
code
that
we
build
in
code,
commit
and
I
would
love
to
be
able
to
trigger
off.
Of
that
so
there's
been
as
I
mentioned
earlier,
there's
been
Garrett
recently
has
support
and
there's
been
some
discussion
of
how
to
write
the
right
interfaces
to
allow
things
to
be
reused
across
plugins.
A
So
there's
a
few
things
that
I'd
like
to
integrate
and
then
test
grid
is
currently
from
my
understanding
kind
of
being
ripped
out
of
its
dependency
on
Google
infrastructure,
but
that's
kind
of
a
up
in
the
air
they're
asking
for
use
cases
and
and
justification
and
I'm
saying
I
want
test
grid
so
that
I
don't
have
to
write
something
to
display
all
of
this
history.
So
I'd
like
to
talk
to
the
test
grid
group
after
this
other
places
were
going
so
I
was
hoping
to
have
this
done
and
demoed
before.
A
A
Currently,
working
on
setting
up
route,
cluster
2
to
run,
builds
and
integration
tests
and
end-to-end
for
our
CNI
plugin,
so
standing
up
clusters
both
on
cook
using
cops
and
using
eks,
applying
the
new
version
of
the
plugin
at
pre
oak,
at
as
a
pre
submit
running
the
tests
and
verifying
that
you
know
we
have
a
good
commit
because
right
now
it
takes
a
little
bit
of
work
to
verify
all
of
this.
Without
brow
we
also
have
the
Authenticator
plugin,
which
is
actually
in
cig
testing.
A
So
this
one's
a
little
bit
easier,
I,
don't
have
to
stand
up,
is
in
kubernetes
IG's.
I
don't
have
to
stand
up
my
own
prop
cluster
I
can
just
move
it
into
the
the
kubernetes
cluster,
but
when
we
started
off,
we
were
not
in
kubernetes
SIG's
we
were
in,
it
was
in
hefty
owes
account,
and
so
we
were
using
Travis
for
the
integration
there.
So
I'd
like
to
see
that
moved.
A
A
You
started
out
of
the
box
which
we
build
using
Packer
and
that's
all
open
sourced,
so
I
and
it's
on
github,
so
people
are
sending
us
full
requests
and
I
would
like
to
see
us
try
and
do
to
build
and
test
the
ami
as
a
pre
submit
from
using
prowl,
and
so
that's
a
little
bit
more
of
a
I.
Think
a
new
use
case
I
mean.
Is
anybody
here,
building
Ami's
using
prowl
and
testing
them?
So
yes,
the
first
one
to
do
it
yeah,
so
that's
kind
of
the
end
of
where
we're
at
with
prowl.
A
This
is
a
few
other
things
I
wanted
to
hit
on
I
just
found
out
yesterday.
That
eks
has
put
the
the
AWS
has
put
up
the
container
roadmap,
so
I'm
hoping
some
of
these
things
that
we'd
like
to
build
out,
can
kind
of
make
it
on
to
the
roadmap,
but
other
things
that
you
may
be
looking
for
from
eks
are
showing
up
there.
As
you
know,
coming
soon
things
like
111
and
reagent
launches.
A
We're
hiring
I
think
they
make
me
put
that
up
there,
or
else
I
don't
get
to
come
and
talk,
and,
lastly,
I
just
like
to
say
thank
you
to
everybody,
who's
kind
of
put
the
work
into
getting
proud
to
where
it's
at
and
helping
me
get
started
on
it.
I
think
there's
been
a
lot
of
updates
to
the
documentation
in
the
last
six
months
when
I
started
looking
at
it.
A
A
Sorry
I'm
looking
for
no
peace
on
here
so
yeah.
That's
the
end
of
my
presentation.
I'm
not
gonna,
take
questions
up
here,
but
I
will
be
available
outside.
If
anybody
has
any
questions
feel
free
to
come
by
and
talk
to
me
I'll
also
be
over
at
the
AWS
booth
tomorrow
morning.
I
think
I
have
the
early
shift.
So
thank
you.