►
From YouTube: Prow! Leveraging Developer-Centric CI for Your OSS Project! - Nabarun Pal & Arsh Sharma
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Prow! Leveraging Developer-Centric CI for Your OSS Project! - Nabarun Pal, VMware & Arsh Sharma, Okteto
Prow is a CI system maintained by Kubernetes SIG Testing to test Kubernetes on Kubernetes. Prow is designed as a pluggable system of components and it can be used as a generic CI system. The robust architecture of Prow can lead to challenges in deploying it. In the talk, we will navigate the challenges faced when deploying and using Prow, including setting up the Prow control plane components, configuring access for GitHub repos, and enabling Prow plugins. Prow is used by large projects in the CNCF landscape like Kubernetes, Knative, cert-manager, Falco, to name a few. Even though a lot of these projects have deployed Prow successfully, it is a challenge to set up Prow. The talk will highlight the common pitfalls and gotchas that one will run into when deploying Prow. The talk would cover * A roundup of Prow Architecture * Cloud resources required for Prow and setting them up * Capabilities of Prow like running tests, using GitHub comments for interaction, auto merging pull requests.
A
A
They
have
seen
it
comment
on
their
pr
or
their
issue,
but
they've
liked
it,
but
when
it
comes
to
installing
it
for
their
own
repos
or
setting
it
up
or
just
understanding
what
is
happening
behind
the
scenes
of
prow
how
it
is
working.
That
is
not
clear,
and
that
is
what
we
aim
to
solve
in
this
talk.
We
aim
that
once
you
go
through
our
session
after
this,
you
feel
confident
enough
in
trying
out
prow
on
your
own,
installing
it
on
your
repositories
and
basically
having
fun
with
it.
A
So
before
we
begin
a
bit
of
introductions
are
in
order
who
are
we?
I
am
sharma,
I
work
as
a
developer
and
experience
engineer
at
octato.
I
also
actively
contribute
to
the
kubernetes
project.
I
will
be
leading
the
ci
signal
team
in
the
125
release
and
I'm
a
new
contributor
ambassador
for
kubernetes
sig
docs,
where
I
help
people
contribute
to
the
project
I'll
hand
over
to
nabarron.
Now,
hey.
B
I
am
also
an
elected
member
of
the
kubernetes
code
of
conduct
committee.
I
have
been
in
the
committee
for
the
past
eight
or
so
months.
Besides,
that
I
helped
maintaining
some
other
areas
in
communities
like
github
administration
and
contributing
to
other
areas
of
the
project.
Now,
why
are
we
here?
What
is
prop
is
a
kubernetes
based
cicd
system
being
based
on
top
of
kubernetes.
It
brings
all
the
pros
of
running
any
application
of
kubernetes.
B
Like
you
get
replication,
you
get
scaling,
you
get
rolling
updates.
You
get
highly
available
nature
of
any
stateless
workload
that
you
usually
run
on.
Kubernetes
brow
can
manage
a
life
cycle
of
pull
requests
like
review
approvals,
merging
them
based
on
some
criteria
through
a
component
called
tide,
which
we
will
take
a
brief
look
at
later.
B
B
There
are
pros
and
cons
of
both
the
approaches
I
shall
be
covering
them
later
on,
in
a
future
section
pro
plus
pro
ships,
with
a
very,
very
highly
extensible
plugin
system,
which
helps
you
to
infinitely
extend
prow
and
change
its
and
modify
its
behavior
to.
However,
you
need,
we
also
set
a
we
also
ship
with
pro
a
set
of
inbuilt
plugins,
which
help
you
manage
several
parts
of
a
developer's
experience
in
that
system,
while
doing
continuous
integration
or
deployment
of
your
code.
B
Pro
also
includes
something
called
chat.
Ops.
We
might
have
heard
this
a
lot
of
times
and
how
we
do.
It
is
through
slash
something
like
commands
like
slash
honk,
which
will
essentially
give
you
an
image
of
a
duck
doing
some
some
naughty
things
around
the
wild
ash
is
now
going
to
show
you
around
the
chat,
ops
system.
A
So
chad,
ops
is
probably
my
favorite
bits
about
brow
because
I
feel
like
when
I
was
starting
out
as
a
contributor
to
kubernetes
and
you
look
at
all
these
pr's
and
all
these
maintainers
super
experienced
folks
talking
in
languages,
you
don't
understand,
brow
is
the
only
simple
thing
there.
It's
showing
me
images
of
cats
and
dogs,
that's
what
they're,
what
I'm
there
for?
No
chat.
Ops
basically
allows
you
to
have
a
forward,
slash
and
then
a
command
which
allows
you
to
interact
with
github
issues
and
prs.
A
So
yolo
is
not
a
command.
Please
don't
try
that
you
can
assign
you
can
assign
issues
to
contributors.
So
let's
say
if
you're
a
new
contributor
getting
started,
and
you
want
to
work
on
an
issue
you
can
just
do
forward,
slash
assign
and
that
would
actually
assign
the
issue
and
let
everyone
know
that
you're
working
on
it.
Similarly,
you
can
label
issues
and
pull
requests.
A
So
don't
worry,
basically
go
18
introduced
something
and
we
had
a
fix
for
that,
but
then
18.1
fixed
that
and
we
wanted
to
remove
our
work
around.
So
that's
what
this
pr
was
for
and
let's
see
how
proud
has
been
helping
with
this
pr.
So
first,
what
happened
is
that
kubernetes
has
a
bunch
of
six
for
those
who
are
not
familiar,
and
every
part
of
some
code
belongs
to
a
six.
A
So
this
belong
to
sig
testing
and
using
forward
slash
sig
testing
mother
was
able
to
apply
this
label
to
the
pull
request
and
then
there's
some
conversation
which
we
are
definitely
not
going
into,
and
then
you
can
apply.
Labels
like
this
and
this
label
is
applied
to
the
pr
which
is
a
priority,
important
soon
label
and
let's
see
what
else
you
can
edit
milestones
for
pull,
requests
and
tests.
A
So
when
you
make
a
pr
tests
are
run,
and
sometimes
some
of
those
tests
are
flaky
and
you
feel
that
a
lot
of
times
what
happens
is
that
you
need
to
rerun
those
tests
and
for
that
you
can
simply
run
the
forward.
Slash
test
all
command,
and
once
you
see
that
ci
turning
green
as
soon
as
it
happens,
please
get
maintainers
to
approve
an
lgtm,
because
that
ci
might
turn
purple
anytime
and
you
will
not
have
a
good
time.
A
B
B
So
first,
we
do
have
a
user
who
interacts
on
github
right
on
the
left.
I
wish
I
had
a
pointer
but
I'll
try
to
explain
as
well.
I
can
and
when
you
create
a
pull
request
on
github
github
essentially
generates
an
event
and
that
event
goes
through
a
github
app
or
a
bot
account
to
a
component
called
hook,
which
is
in
our
brow
cluster,
and
that
component
keeps
on
processing
those
github
events
and
packages
them
into
internal
structures
which
have
some
more
metadata.
B
B
Crow
job
is
a
customer
resource
definition
that
is
a
central
core
concept
in
browse
landscape
and
whenever
any
job
has
to
run
in
prague,
you
basically
create
a
brow,
job
custom
resource
notice.
I
told
you
about
the
trigger
plugin.
If
you
don't
enable
the
trigger
plugin.
This
thing
won't
be
done.
Fortunately,
by
default,
the
starter
templates
that
we
have
the
plugin
the
trigger
plugin
is
enabled
so
any
web
hook
coming
through
a
hook
will
process
an
event
will
trigger
an
event.
Essentially
now
we
have
a
custom
resource.
What
do
we
do?
B
B
You
can
have
multiple
job
clusters
for
isolation.
For
example,
you
might
have
some
kind
of
trusted
job,
some
kind
of
untrusted
jobs
or
let's
say
you
are
working
on
a
private
repository
and
you
don't
want
to
mix
code
and
you
want
to
have
proper
isolation.
So
you
can
run
on
any
class
any
job
cluster
you
want
once
that
is
done.
B
Now
how?
How
do
we
get
that
status
back
to
github,
which
is
where
our
user
is
user,
won't
see
a
custom
resource
for
a
status
right?
We
have
a
component
called
cryer,
which
essentially
cries
all
the
output
in
the
status
sub-resource
to
github.
It
knows
how
and
what
to
infer
from
that
status
sub-resource,
and
it
sends
back
the
data
to
github
when
motherweight
created
that
pr
at
the
end
of
the
pr,
at
the
end
of
a
gita
pull
request,
you
essentially
see
a
list
of
checks.
B
There
are
some
other
components
which
not
necessarily
come
in
the
life
cycle,
but
life
cycle
of
a
user,
creating
a
pull
request
and
getting
an
output
of
a
c
test,
but
they
are
very
important.
We
have
one
component
called
horologame,
which
essentially
looks
at
your
configuration
for
any
periodic
job
that
you
may
have
created.
For
example,
you
just
want
to
check
your
code
sanity,
like
you,
want
to
check
the
code
coverage
every
day
and
post
them
somewhere.
B
It
will
create
that
project
cr
from
your
configuration
so
that
planck
can
reconcile
it
and
run
that
job
and
then
report
the
status
back.
Syncer
is
basically
your
recycle
bin.
What
it
does
is
it
looks
at
pods
which
have
passed
a
certain
retention
policy.
For
example.
Let's
say
there
are
job
pods
which
have
been
lying
around
for
more
than
24
hours.
Just
an
example.
B
We
might
set
like
more
strict
or
less
stricter
configs
and
when
it
sees
those
jobs,
it
basically
deletes
the
parts
as
well
as
the
project
crs,
so
that
the
system
remains
same
now,
I'm
talking
about
like
statuses
logs.
We
we
have
a
lot
of
artifacts
that
are
generated
in
this
process.
Where
do
you
see
them?
So
we
have
a
friend
called
deck.
B
If
you
ever
go
to
prow.ks.io,
what
you
see
is
essentially
deck
deck
is
basically
a
dashboard
where
you
see
what
jobs
we
have
in
the
community
around
all
repositories.
What
is
their
status?
What
was
the
last
pr
that
they
run
for
if
it
was
a
pre-submit
job
which
is
basically
a
pre-merge
job
or
any
periodic
job
status
or
post
submits?
For
example,
we
have
a
process
in
kubernetes
community
called
promoting
images.
B
B
This
was
about
the
core
components
of
pro.
We
will
talk
about
two
components
which
are
needed
when
you
want,
like
full
life
cycle
of
a
cicd
system
in
brow.
First,
one
is
tied,
so
we
have
talked
about
testing
of
your
changes
and
labeling
of
your
changes
or
whatever
chat
ups.
You
want
to
do
what
about
merging
now.
If
I
go
about
in
the
kubernetes
repository,
we
see
probably
like
100
to
500
pr's
every
15
days,
or
something
like
that.
B
B
It
collects
a
list
of
pr's
from
the
same
repository,
for
example,
we
are
talking
about
the
core
commodities
repository
which
is
kubernetes,
slash,
queries,
it,
let's
say:
collect,
collects,
10,
pull
requests
and
then
merges
like
merges
them
onto
the
base
branch
and
tries
to
test
it.
In
this
way,
you
can
test
like
a
lot
of
prs
in
short
amount
of
time.
B
B
After
doing
that,
it
satisfied
all
of
the
criterias,
but
still
after
that,
we
do
a
re-test
of
that
pr.
Once
the
retest
passes,
then
only
we
merge
it
and
you
don't
require
an
user
intervention.
After
all,
of
these
checks
have
passed
and
another
retest
has
gone
through.
We
don't
need
any
more
user
intervention
automatically.
The
pr
would
be
immersed
if
it
satisfies
all
the
criteria.
B
B
B
For
example,
we're
showing
the
sig
release
test
grid
and
where
signal
is
mastered,
blocking
is
a
group
of
jobs
and
verify
master
is
a
job
where
this
certain
set
of
tests
are
run,
and
we
can
see
for
the
previous
runs,
what
what
was
the
status-
and
this
is
how
we
can
decipher
what
is
failing
in
a
very
verb.
We
don't
need
to
churn
through
all
the
logs
of
previous
runs,
and
how
do
we
find
them?
What
are
the
refs?
They
ran
against.
What
was
the
test
intraref
all
those
kind
of
things.
A
So
when
it
comes
to
defining
proud
jobs,
there
are
two
main
approaches
you
can
choose
from
this
choice.
It
might
seem
that
this
choice
is
very
trivial
at
first
and
you
could
just
go
ahead
with
whatever
you
want,
but
if
you
plan
to
scale
your
project
to
the
size
of
kubernetes,
this
will
come
and
bite
you
later.
If
you
do
not
give
it
thought.
A
So
the
two
approaches
we
have
is
one
and
in
repo
configuration
and
in
this
configuration
you
basically
define
the
jobs,
the
brow
jobs
in
the
same
repository
where
your
source
code
is
living,
so
you
would
have
a
dot,
proud
directory
or
just
a
dot
brow.ml
file
at
the
root
of
the
repository
where
you
would
just
list
down
all
of
your
jobs.
It
is
important
to
note
that
in
this
approach,
your
jobs
are
not
defined
centrally.
They
would
be
present
for
each
repository
in
that
particular
repository.
A
The
other
approach
is
centralizing.
Your
job
configuration.
What
this
means
is
that
you
have
a
separate
repository,
and
the
purpose
for
that
repository
is
to
have
all
the
configuration
for
all
the
jobs.
You
might
be
running
across
many
different
repositories.
You
have
in
this
approach,
jobs
get
defined
centrally.
A
A
A
You
tell
a
name
for
the
job
and
you
tell
decorate
to
what
decorate
2
does
is
add,
sidecar
containers
which
help
in
source
code,
checkout
and
other
related
stuff,
and
then
you
mention
what
repository
you
want
this
job
to
be
run
for
in
this
case,
we
wanted
to
run
for
the
main
kubernetes
repository,
so
we
specified
the
org
and
the
repository
and
the
branch
two
path.
Alias
is
basically
when
this
job
checks
out
the
source
code
which
which
we
need
for
the
job
to
do
its
action.
What
directly
do
we
want
it?
A
For,
after
that,
the
spec
is
pretty
simple:
we
just
used
a
golang
based
image
and
what
we
did
is
essentially
install
a
tool
we
wanted
to
run
called
defstat
and
then
run
the
command
in
it,
and
that's
it
that's
this
is
how
simple
it
is.
It
is
nothing
but
a
container
where
you're
running
some
commands
and
that's
how
simple
prom
makes
your
life,
and
that
is
why
we
love
it
towards
the
end.
You
see
we
have
annotations,
and
this
is
where
the
test
grid
navarone
just
talked
about
comes
into
picture.
A
A
That's
it
that's
where
it
is,
and
this
is
the
periodic
job
look
flaky
how
nice,
but
we
go
to
here,
and
this
is
every
each
run.
It
did
for
the
job
and
you
can,
if
you
click
on
any
one
of
these
green
cells.
It'll
take
you
to
spyglass,
where
you
can
see
the
logs
for
the
job,
and
it
is
these
logs
that
we
wanted
for
this
particular
job.
So
if
we
just
look
here,
this
is
the
information
we
were
interested
in,
and
this
is
what
proud
it
for
us.
A
A
A
Next,
when
you're
creating
the
github
app,
this
is
something
you
need
to
be
careful
of.
Is
that
it'll
ask
you
for
a
webhook
url
and
when
you're
doing
this
step,
it
doesn't
actually
matter
what
url
you
add
right
now,
because
we
are
going
to
change
that
later,
but
remember
to
keep
a
note
of
this,
that
you
need
to
change
this
later
and
we'll
provide
a
secret
here,
but
we'll
get
to
that
later.
The
third
thing
you
need
to
take
care
of
is
adding
a
bunch
of
permissions,
because
these
permissions
are
needed
for
pro.
A
Otherwise,
it
will
not
be
able
to
interact
with
your
github
repositories,
and
this
is
just
a
snapshot.
There
are
more,
there
are
more
permissions
you
need
to
add,
but
we
didn't
want
to
add
a
url
right
now
in
the
slides
we'll
do
that
later
we
promise.
So
the
final
thing
you
need
to
take
care
of
is
in
there's
a
section
called
subscribe
to
events,
and
you
need
to
subscribe
to
all
events,
because
brown
needs
to
know
what's
happening
to
the
repository
in
order
to
react
to
it.
A
This
is
where
the
hook
system
navarone
talked
about,
comes
into
picture,
because
if
it
is
not
getting
notified
of
the
events,
it
really
can't
do
anything
for
you
once
you
are
done.
Creating
the
app
make
sure
to
note
the
app
id
and
to
generate
a
private
key,
keep
them
very
secure
because
we
are
going
to
be
using
them
later
now.
I
don't
take
it
from
him.
B
Awesome
so
after
you
set
up
your
github
app
upstream
kubernetes
maintains
a
binary
called
tackle.
Recently
we
made
some
changes
to
tackle,
which
made
it
a
little
bit
trivial
to
use
or
easier
to
use.
We
fixed
some
bugs
a
few
months
back
so
since
we
have
already
set
up
our
github
app
and
we
are
not
relying
on
a
bot
account,
what
you
can
do
is
just
run
tackle,
skip
github,
and
how
do
you
get
tackle
is
if
you
go
to
the
test,
infra
repo,
that
are
short.
B
If
you
go
to
cmd
the
command
directory,
you'll
see
the
tackle
subdirectory,
and
you
just
need
to
go,
install
check
out
the
source
code
and
go
install
that
should
do
it.
We
don't
ship
a
tackle
binary
right
now,
but
maybe
in
the
future,
when
it
becomes
more
stable
and
supports
more
providers,
we
may
start
like
shipping
it.
B
B
Yes,
the
documentation
also
tells
you
about
like
manual
deploy
steps,
but
in
the
interest
of
sanity
we
are
demonstrating
using
tackle.
So
you
let
tackle
run
you
tell
tackle
if
you
already
have
a
gcp
project
and
a
gcp
gke
cluster
running.
If
you
have
one
you
can
just
mention
the
name,
and
it
will
install
everything
for
you
in
that
cluster.
B
One
thing
we
need
is:
we
talked
about
a
lot
of
things:
artifacts
like
logs,
did
the
test
run
successfully
or
what
was
the
exit
code?
All
of
that
is
currently
stored
in
a
gcs
bucket.
Now
you
might
see
this
commands
are
very
scary,
but
I'll
decipher
it
for
you.
It
just
creates
a
service
account.
It
creates
a
bucket.
It
binds
it
to
the
server
it
binds.
The
service
account
to
the
bucket
gives
it
appropriate
permissions,
and
then
you
can
use
the
credentials
in
your
starter.ml.
B
So
you
get
the
starter
template
from
the
url
and
then
you
basically
grip
through
all
of
the
templated
strings
and
replace
them
with
the
appropriate
values
that
we
created
before
now.
Once
you
did,
that
you
can
just
tell
tackle,
because
tackle
would
be
waiting
for
you
to
give
a
location
of
a
starter
ml.
You
can
just
give
it
the
location,
it
will
download
or
read
it
from
local
disk
and
then
apply
it
once
it
is
done.
B
B
Final
step
is
to
install
the
app
on
a
repo.
Whatever
you
want,
the
org
that
you
put
in
there
the
or
can
be
user
account
itself
if
you
just
want
to
try
out
for
now,
but
if
you
are
doing
it
in
production
also,
the
starter
template
doesn't
is
not
suggested
for
production,
because
the
starter
template
creates
proud
clusters
and
job
class
brow
clusters
and
job
clusters
on
the
same
kubernetes
cluster.
Ideally,
they
should
be
separate.
That
is
what
we
do
in
upstream
kubernetes
and
once
you
have
all
of
this
set
up.
You
just
enjoy.
A
A
So
if
you
have
like
a
lot
of
repositories
right
and
you
set
a
pro
for
all
of
them,
what
this
does
is
ensure
that
developers
get
a
similar
experience
across
all
repositories
for
your
organization.
So
they
don't
have
to
learn
the
processes
again
and
learn
new
commands
or
figure
out
or
ask
what's
going
to
happen,
and
this
is
why
I
love
pro
because
developer
experience
is
something
which
is
very
close
to
my
heart,
and
I
feel
that
brow
provides
an
excellent
developer
experience.
A
The
chat,
ops
based
interaction,
ensures
that
you
do
not
have
to
be
a
github
wizard
like
if,
even
if,
like
someone
has
not
gone
through
the
nuts
and
bolts
of
github,
understanding
commands
like
forward
slash,
assign,
are
pretty
intuitive
because
you
get
the
idea
that
you
want
to
assign
someone.
And
lastly,
I
feel
that
prow
is
new
contributor
friendly,
because
if
I'm
a
new
contributor,
I
sometimes
feel
lost.
What
are
the
next
steps?
I've
created
a
pr
now,
where
do
I
go
from
here
and
with
brow?
A
You
can
set
up
automated
messages
and
friendly
stuff
which
provide
next
steps
that
hey
you
have
your
pr
here.
These
are
the
persons
you
need
to
ping
for
review.
Now.
We
agree
that
all
like
it's
not
it
does
not
always
get
them
correct,
but
as
a
new
contributor,
I
would
at
least
have
a
first
point
of
contact,
even
if
that
is
not
the
correct
point
of
contact
and
I
would
not
feel
lost
so
I
feel
like
these
are
the
benefits
of
brow,
which
is
why
we
love
it
in
upstream
kubernetes.
B
Now,
what's
the
point
of
this
talk,
we
want
you
to
use
pro.
We
want
you
to
use
prav
for
production
and
we
want
you
to
come
back
to
us
as
in
sick
test
kubernetes
testing
and
tell
us
what
more
we
can
do
and
not
just
that.
We
want
your
help
when
you
use
prow.
You
report
us
back
issues
and
even
you
can
contribute
to
that
yourself.
Be
those
good
samaritans
of
the
open
source
community
and
just
have
fun.