►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Knative Productivity BoF - Jessie Zhu & Adriano Cunha, Google
We will take a deep dive into Knative productivity infrastructure which covers test, perf, release and metrics. We will walk you through the life of a change, what type of tests it has to go through, what type of checks a PR has to go through, what automation is in placed, how different infrastructure pieces are wired together, and how everything contributes to the end result which is a successful release. There will be live demos as well.
To learn more: https://sched.co/Grdv
A
A
So
the
agenda
is
pretty
simple:
go
briefly
on,
what's
K
native
what
the
productivity
working
group
does,
then
I'll
go
deep
into
the
architecture
and
details
of
how
we
do
stuff
on
K
nave
to
achieve
the
goals
I
just
mentioned,
I'll
go
through
the
life
of
a
PR
and
a
simple
demo,
or
what
it
means
in
the
day
to
day
work
on
K
native
and
we'll
have
some
time
for
Q&A,
but
before
I
continue.
Let
me
be
pretty
clear
about
what
this
presentation
is.
So
this
is
not
a
presentation
about
K
native
itself.
A
We
had
several
wonderful
presentations
about
K
native
and
the
previous
days,
so
you
have
to
refer
to
those.
If
you
want
to
know
more
about
K
native,
so
I'm
not
going
to
be
talking
about
what
it's
serving,
what
it
does
or
CR
DS
or
what
the
beauty
I'p
line
does
or
K
would
be
used
on
the
productivity
not
now
maybe
in
a
future
conference.
This
is
also
not
about
prowl
I'm,
going
to
talk
a
lot
about
prowl
because
we
use
that
system.
A
But
if
you
want
to
know
details
about
prowl,
there
was
another
session
about
it
and
you
can
refer
to
that
later.
If
you
want
more
details
on
how
kubernetes
uses
kubernetes
to
run
the
kubernetes
flows
and
finally,
there
was
an
intersession
and
Tuesday.
My
colleague
shrinivas
went
through
a
lot
of
stuff,
we
do
and
the
productivity
working
group-
and
my
intention
is
to
go
deeper
into
what
we
do
on
the
technical
side
in
the
productivity
working
group,
all
right.
A
A
Okay,
we
know
what
Kay
native
is
pretty
clear
explanation.
What
the
productivity
working
group
does.
Our
mission
is
to
develop
engineering
guidelines,
choose
an
infrastructure
for
K
native
and
all
those
wonderful
things
that
every
software
developer
or
company
wants
high
quality
in
high
velocity
deliverables,
predictable
releases,
high
collaboration
or
high
productivity,
but
why
we
need
that
for
K
native,
because
K
native
is
not
a
toy,
we
have
12
ripples
in
the
organization
we
have
more
than
400
users,
creating
more
than
160
PRS
per
week
and
filing
more
than
60
new
issues
per
week.
A
A
System
used
to
test
and
release
K
native
artifacts
in
the
center
of
everything
is
proud
and
together
with
github
and
other
tools
like
test
grades,
testing
environment,
just
like
uber,
Nader
and
storage
systems,
we
creates
the
entire
framework
used
for
tests
and
release
I'm
going
to
go
through
all
the
components
one
by
one
and
what
we
do
and
how
they
apply
to
the
overall
case.
So,
starting
with
the
basics,
our
CI
CD
system
is
proud.
A
It
basically
provides
all
the
the
basic
things
we
want
on
our
CI
CD
system.
We
want
pre,
submit
tests
and
those
are
required
for
merging
our
PRS.
We
want
automatic
merging,
so
oh,
the
pr
was
was
approved.
The
pr
didn't
break
anything.
So
why
not
merging
it?
We
want
continuous
integration
flows.
Even
you
know
less
than
one
hour
or
maybe
two
hours,
it
depends
on
the
needs.
We
want
nightly
release.
We
want
code
coverage
reports
and
even
going
beyond
code
coverage.
A
We
want
playground,
so
we
can
test
the
latest
and
greatest
releases
of
K
native
without
having
to
bring
up
clusters
and
setting
up
a
lot
of
stuff.
We
want
to
just
a
single
command
line,
to
start
playing
with
that,
we
want
to
run
performance
tests
to
ensure
that
things
are
running
smoothly
as
we
wish.
A
Set
one
job
for
each
presubmit
test,
so
we
can
run
the
pre
submit
jobs
in
parallel.
We
want
those
pre
submit
jobs
to
be
required,
checks
and
github,
so
you
cannot
submit.
We
cannot
merge
a
PR
if
the
tests
are
not
passing,
we
want
periodic
jobs
because
we
want
daily
runs
of
something
or
nightly
runs
or
weekly,
but
we
might
want
to
collect
data.
We
might
want
to
run
tests
or
you
might
want
to
do
releases
in
a
periodic
fashion.
A
We
want
to
define
approvers
and
reviewers,
and
that
is
done
through
the
owners
files
in
the
repos,
so
we
set
responsibilities
and
we
set
permissions
to
approve
or
deny
PRS
to
be
merged
or
to
even
start
tests
on
our
system
and
to
achieve
that,
we
have
some
two
important
services
in
prowl.
One
is
tied
the
handles
the
automatic
merging.
A
It
basically
looks
for
reviews
and
approvals
from
repository
owners
that,
together
with
passing
tests,
means
ok.
This
PR
is
good,
it
can
be
merged
done
without
manual
intervention,
and
the
other
service
is
Bosco's,
which
is
a
pool
of
GCP
projects
that
are
used
for
our
end-to-end
tests,
ensuring
that
we
have
isolated
environments
and
not
only
isolated
but
also
fresh.
So
we
can
run
tests
without
interference,
so
get
a
project,
an
exclusive
project
to
run
your
tests
brings
up
a
cluster
run.
Your
tests
shut
it
down
done
next
time.
A
A
It
has
a
simple
state
machine
that
takes
care
of
PRS
that
were
where
you
were
assigned,
or
you
were
called
attention
to,
and
then
it
makes
your
life
easier
to
manage
PR
when
working
on
multiple
repos
on
github
comes
for
free,
okay,
and
where
does
all
this
configuration
and
stuff
live
in
a
separate
repo?
We
call
it
testing
for
and
we
keep
the
configurations
there.
We
keep
the
tools
we
use
for
the
test
infrastructure
there.
We
have
shared
code
for
the
death
started.
A
In
our
case,
the
frameworks
in
go
are
in
the
different
repo,
so
usually
tests
use
shared
code
from
two
different
triples,
but
that
was
project
Detroit's
in
this
repo.
We
also
maintain
test
related
tools
that
are
kind
of
invisible
to
whatever
repo
is
using
our
processes
and
tools,
and
this
repo
is
a
vendor
in
each
Kay
native
repo.
A
So
the
good
thing
about
it
is
that
it's
vendor,
so
you
avoid
breakages
when
you
make
changes
to
the
repo,
but
the
bad
thing
is
that
each
time
you
had
a
new
update,
you
have
to
replicate
that
to
each
other
repo.
In
this
case,
one
change
in
the
testing
for
repo
means
replicating
that
to
eleven
other
repos
and
what
about
our
tests?
We
have
at
least
three
types
of
tests
built
as
unit
tests
and
integration
tests
no
big
deal
here.
A
We
have
the
starter
or
the
runners,
as
helpers
in
testing,
for
so
they
are
vendored
and
basically
invisible
to
the
other
ripples.
They
do
all
the
magic.
Why
or
the
other
repos
just
worry
about
specific
testing
code,
those
tests
they
have
to
run
both
locally
or
on
prowl
the
same
way,
so
you
don't
have
different
way
of
running
in
one
environment
or
the
other.
So
it's
easy
to
replicate
failures.
A
Finally,
the
integration
test
because
they
are
the
most
complicated
ones
we
use
cube
tests
to
provide
the
kubernetes
cluster
cube
test,
was
developed
by
the
kubernetes
team
to
test
kubernetes,
but
it
can
be
used
to
test
other
stuff
that
runs
on
kubernetes,
so
we
are
using
it
to
easily
bring
up
a
test
cluster
and
then
shut
it
down
after
the
test
finishes:
4k
native,
we
also
have
conformance
tests.
These
are
a
special
class
of
end-to-end
tests
because
they
basically
validate.
A
Another
important
information
we
want
is
coverage
for
code
coverage,
there's
no
support
and
prowl.
Yet
there
is
work
in
progress,
but
there
is
not
there's
no
ready
to
use
solution,
so
we
developed
a
custom
tool
for
that
that
use
separate
jobs
in
prowl
to
report
the
code
coverage
either
on
a
PR
or
on
a
daily
basis.
So
that's
an
example
of
code
coverage
when
you
add
a
commit
to
a
pull
request,
make
sure
more
about
that
later
for
a
runtime
coverage
support
and,
namely
we're
talking
about
latency
and
API
coverage.
A
In
our
case,
we
have
special
logs
in
the
continuous
integration
flow
instead
of
having
another
job
to
run
the
test
just
to
collect
metrics,
we
piggyback
on
the
continuous
integration
flow
and
fetch
the
metrics
to
display
them.
We
use
task
grid
as
the
one-stop
shop
for
displaying
all
the
coverage
and
health
metrics
for
ke-.
A
About
the
pre
submits
I
mentioned
that
we
have
separate
jobs
for
running
the
test
and
that
the
required
steps
check
on
github
again,
meaning
that
you
can't
merge
a
PR
if
the
tests
are
not
passing
the
results
are
displayed
on
goober
nadir,
the
screenshot
I
mentioned
earlier.
However,
prow
has
no
support
for
raw
go
tests
or
go
tools,
so
we
built
a
custom
test
result
parser,
so
governador
can
display
nice
results
when
a
test
fails.
For
example,
it's
way
easier
than
going
down
logs
and
finding
out
what
failed
again
in
the
middle
of
300
tests.
A
And
I'm
not
releases,
those
are
periodic
brow
jobs
and
we
have
queue
right
now.
Two
different
kinds
of
release
we
have
nightly
they
are
built
against,
had
basically
run
out
tests.
They
past
built
a
native,
publish
tag,
the
image
with
a
date
and
commit
hash,
so
you
know
where
they
came
from
and
also
tag
them
latest.
So
it's
pretty
easy
for
anyone
wanting
to
install
or
use
the
latest
k
native
to
know
what
is
the
latest
image?
A
We
also
have
the
playground,
that
is
environment,
that
is
deployed
every
Saturday
using
the
latest
nightly.
So,
although
every
Saturday,
your
deployments
will
be
destroyed
and
vanish
at
least
you
have
the
whole
week
to
test
them
with
Canada.
If
the
idea
right
now
is
that
you
have
a
stable
and
a
clean
environment
to
make
tests
during
the
week
and
to
keep
track
of
all
that,
we
use
test
grid,
so
everything
is
tracked
on
test
with,
so
we
know
if
releases
failed
to
build
or
not,
and
if
the
playground
is
up
to
date
or
not.
A
So
yes,
let's
talk
about
displaying
results.
So
another
screenshot
of
uber
Nader
here
shown
more
than
just
the
error
message.
So
if
you
click
a
failing
test
on
a
PR
that
what
you
get
a
summary
of
your
test
run,
what
failures
happened
and
then
your
build
your
log.
It's
called
build
log,
but
it's
whatever
you
did
in
the
job.
If
you
need
dive
into
it
and
find
what
happens-
and
this
is
test
grid
like
the
name-
implies
it's
a
grid
with
tests
and
runs.
A
So
in
this
case
we're
seeing
the
coverage
code
coverage
grid
with
daily
runs
and
they
contain
the
code
coverage
percent
or
green,
meaning
that
you,
the
code
coverage,
is
above
given
threshold
and
read
the
opposite.
The
white
blocks
mean
that
the
test
wasn't
run
at
that
particular
moment.
It
could
be
because
the
tests
didn't
exist
at
that
point,
or
the
test
was
deleted
before
that
point
in
time
or
because
the
run
failed.
Somehow,
in
this
case
there
is
no
failed
run
to
fit
us
reads.
A
We
have
basically
two
ways
if
you
are
talking
about
pass/fail
test,
that's
the
parser.
The
custom
go
test.
Parser
I
mentioned
earlier.
That
also
feeds
results
to
Grenadier.
So
that's
grid
is
able
to
understand
that
as
well
and
display
red
and
green
blocks
for
all
the
tests.
You
ran
and
form
more
specific
results
in
the
code
coverage
or
the
test
latency.
Where
you
have
numbers
or
thresholds,
then
we
build
custom.
Xml
artifacts
continue
the
results
we
want
to
present
in
this
case.
That
will
be
the
code
coverage
numbers.
A
A
So,
let's
start
with
the
first
step
that
is
creating
a
PR,
nothing
new.
Here
you
just
go
to
github,
create
your
pool
crest
as
usual,
then
the
automatic
things
start
happening,
so
you
create
your
PR
and
automatically
prow
will
check
for
the
review
with
for
your
PR,
based
on
the
owners
file
and
it's
gonna
add
two
reviewers
to
your
PR.
A
If
you
don't
want
those
people
to
be
assigned,
because
another
person
is
more
suitable
to
understand
what
review
what
you
wrote,
you
can
do
it
as
usual
and
the
github
uy
no
problem
here,
but
usually
it
saves
you.
A
few
clicks.
Selecting
people
to
review
your
PR
and
also
the
pre
submit
test
will
start
and
when
they
start
at
some
point,
you're
going
to
get
results,
they
can
either
pass
and
you're
going
to
get
all
the
green
checkmarks
or
something
will
fail
and
you're
going
to
get
a
message.
A
A
post
saying:
oh,
this
job
failed
link
the
job
details
that
will
take
you
to
goober
Nader.
So
again,
you
can
Specter
log
see
what
happened.
Work
on
that
code
until
the
tests
are
passing
and
for
for
any
reason
you
want
to
rerun
your
tests.
You
can
do
it
either
each
separate
test
or
you
want
to
rerun
the
tests,
for
whatever
reason
comes
to
your
mind
and
of
course
there
is
the
usual
review
flow.
Oh,
your
reviewers
will
add
comments
you're
going
to
address
those
every
time
you
add
a
new
commit
to
your
PR.
A
The
test
will
run
again
and
they
have
to
be
green,
so
nothing
new
here
as
well
and
as
I
mentioned
about
the
code
coverage
job
every
time
you
add
a
new
commit
to
your
PR.
The
code
coverage
job
will
run
and
report.
What
is
the
current
status,
so
you
might
want
to
work
on
your
tests
to
increase
the
code
coverage
to
make
sure
that
everything
is
looking
good
in
your
PR
again,
the
usual
flow,
with
more
input
to
make
you
more
productive
and
finally,
at
some
point
your
reviewers
will
be
happy
with
your
code.
A
You'll
be
happy
with
your
code.
They
will
approve
it
either
by
using
the
commands
understood
by
prowl,
like
/lg,
TM,
+,
/
approve
or
clicking
the
approval
button
and
the
github
uy.
So
once
you
have
our
GTM
and
approval
and
your
tests
passed,
your
PR
is
ready
to
be
merged
and
prowl
will
notice
that
and
it
will
happily
merge
your
pure
and
it's
done.
A
That's
correct
the
question:
is
you
start
submitting
PRS?
Does
it
need
any
signal?
So
yours,
GI,
CD
system,
start
testing
the
pr
yes
I
didn't
mention
that
it's
part
of
proud,
but
that
is
a
safety
mechanism
and
prop.
So
only
members
of
the
organization
are
allowed
to
run
the
test.
Why?
Because
you
might
create
a
PR
with
bad
code
in
it
to
exploit
the
CI
CD
system
or
do
something
bad.
A
So
if
you
are
collaborator
that
is
outside
the
organization,
you
can
still
submit
your
pull
request,
but
no
pre
submit
test
will
run
until
a
member
of
the
organization
issues
the
command.
Ok
to
test
in
the
comments
basically
/
or
a
comment
saying:
/,
ok,
test.
It
means
hey
someone
who
is
a
member
of
the
organization
who
is
a
trusted
person
check
out
that
code
made
sure
that
it's
not
harmful
and
allowed
the
CI
CD
system
to
pick
up
the
changes
and
run
tests
with
them.
A
A
A
A
So
the
question
is,
and
please
correct
me:
if
I
misunderstood
your
question:
do
we
have
plans
to
build
que
native
as
independent
modules?
The
answer
is
yes,
but
right
now
we
are
building
as
a
single
thing.
We
have
two
plans
to
do
that,
but
just
like
making
key
native
build
a
native
we're
not
there
yet.
B
A
A
No
that's
pretty
pretty
much
dependent
on
the
team,
mostly
involved
in
the
component
or
the
process.
We
don't
have
general
guidelines
about
big
PRS,
but
usually
people
frown
at
two
big
PRS.
They
are
hard
to
review
and
it's
pretty
easy
to
get
lost
on
big
PRS.
So
probably
most
senior
reviewers
will
ask
the
collaborator
to
break
that
PR
into
smaller
pieces.
A
C
A
Okay,
so
the
question
is
instead
of
reinventing
the
wheel.
Every
time
is
there
no
I
understand,
that's
totally
valid
question
and
every
time
you
start
a
new
project.
You
are
tempted,
though
oh
I,
don't
like
the
previous
process.
I
work
with
I
want
to
do
something
better,
so
I'm
gonna
start
from
scratch.
In
this
case,
prowl
is
a
good
example.
We
decided
to
use
something
that
is
proven
to
work
that
is
well
tested.