►
From YouTube: Automation and the Kubernetes Contributor Experience - Aaron Crickenberger, Samsung SDS
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Machines Can Do The Work: Automation and the Kubernetes Contributor Experience - Aaron Crickenberger, Samsung SDS (Beginner Skill Level)
The decision to host a project of Kubernetes' scale on GitHub has led us to confront a number of unique challenges over the past three years, both technical and cultural. We were the most active project on GitHub in 2017 as measured by contributions beyond just code. This talk will walk you through a number of tools and processes we have developed to support this growth, including but not limited to: gubernator, mungegithub, prow, and testgrid By the end of the talk, contributors in the Kubernetes community will know more about how to use our existing automation to make their lives easier. Maintainers of other projects large or small will know how they can try out our automation on their project today.
About Aaron
Aaron has been involved in open source projects since 2007, cloud related projects since 2009, and Kubernetes since 2015. He was recently elected to the Kubernetes Steering Committee in 2017. He co-founded the Kubernetes Testing SIG, and actively contributes in the Architecture, Contributor Experience, Release, and Scalability SIGs. If you attend the weekly Kubernetes Community meetings, chances are you've seen him (or at least his beard.) He has participated in every Kubernetes release since v1.4, holding roles involving: release notes, issue triage, and CI signal. In previous lives he's implemented a database as a service offering in a public cloud, managed a fleet of cloud foundry deployments on cruise ships, and worked on a blackhawk flight simulator built inside of a shipping container.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
good
morning,
everybody
thank
you
for
coming
over
here,
so
quickly
over
the
keynotes
actually
cut
out,
for
it
ended.
So
I
wanted
to
welcome
you
to
my
talk.
Called
machines
can
do
the
work,
automation
and
the
kubernetes
contributor
experience
now
when
I
practice.
This
talk
the
first
time
I
practiced
it
I
ran
75
minutes.
A
A
My
name
is
Aaron
Berger
wathah.
Why
I
co-founded
sig
testing
I
participated
in
a
number
of
kubernetes
releases,
I'm
a
member
of
the
kubernetes
steering
committee
and
I'm
an
active
participant
in
the
contributor
experience,
but
all
these
people
up
here
are
responsible
for
contributing
to
or
writing
a
lot
of
the
infrastructure
and
tooling
that
I'm
going
to
talk
about
today
and
if
you
don't
recognize
their
github
avatars
off
the
top
of
your
head.
You
know
here
the
user
names
for
you
to
go
stock
them
later.
A
A
So
I
wanted
to
tell
you
the
story
of
kubernetes
as
a
project
and
how
sort
of
early
on
we
decided
that
if
this
was
going
to
be
a
successful
project
we
needed
to
go
to
where
the
developers
are
which
is
github,
and
so
we
figured
that
if
we
did
things
the
github
way,
we
would
succeed
beyond
our
wildest
dreams,
and
unfortunately
this
came
true,
but
we
found
that
over
time
we've
come
to
to
adapt
to
you
know
this.
This.
This
is
fine.
A
Dog
is
basically
our
our
spirit:
animal
in
the
testing
world
for
kubernetes
there's
a
lot
of
stuff
that
goes
on
in
the
project,
but
things
are
actually
okay.
If
we
can
figure
out
the
right
problems
to
solve
and
deal
with,
it
figure
out
ways
to
get
automation
to
help
us
do
the
boring
work
and
we
can
get
to
do
the
interesting
work
and
the
reason
that
things
are
fine
today
is
because
we
live
in
this
nice
tropical
abyss
of
using
a
thing
called
prowl.
This
is
a
diagram
put
together
by
Ben
the
elder.
A
He
posted
on
his
blog
back
in
December
and
I'm,
just
showing
it
up
here
real
quickly,
not
to
walk
you
through
it,
but
to
show
you
that
this
diagram
exists.
If
you
would
like
somebody
to
walk
you
through,
it
I
encourage
you
to
check
out
sig
testing,
but
basically
this
is
a
bunch
of
little
micro
services
that
help
us
sort
of
do
testing
in
a
more
kubernetes
native
way.
A
A
The
main
way
of
interacting
with
prowl
as
an
end
user
is
via
these
slash
commands
and
we
have
most
of
them
are
updated
on
the
fly
and
Prowse
UI,
so
you
get
to
see
which
command
corresponds
to
which
plug-in
what
it
allows
you
to
do
and
who
gets
to
use
it,
and
if
you
want
to
find
out
more
about
exactly
what
prowl
is
made
of
and
how
it
works.
I
encourage
you
to
come
to
the
sig
testing,
deep
dive
Friday
at
two
o'clock.
A
So
when
I
talk
about
machines,
doing
the
work,
what
do
I
mean
by
the
work?
You
know
if
you
break
it
in
town
into
four
categories?
It's
basically,
you
know
implementing
your
typical
PR
workflow
of
testing,
pull
requests,
making
sure
they
look
good
and
then
merging
them.
It's
helping
out
with
issue
triaging
and
pull
request
triage.
You
make
sure
that
they,
you
know
they're
categorized
appropriately
and
they
get
to
the
right.
A
A
So
I
wanted
to
walk
you
through
our
pull
request,
workflow
sort
of
as
the
lens
for
the
bulk
of
the
automation-
and
this
looks
relatively
non-controversial,
I.
Think
as
with
any
large
open-source
project.
You
know
people
need
to
sign
a
CLA
to
make
sure
that
their
code
gets
contributed.
The
pull
request
needs
to
pass
all
of
the
tests.
It's
got
to
be
approved
by
human
beings.
It's
got
to
be
mirja
below
there
can't
be
any
merge
conflicts
and
we've
got
to
make
sure
that
it
passes
tests
right
before
we
push
the
merge
commit.
A
So
when
it
comes
to
signing
the
CLA,
there's
really
not
much
for
me
to
say
here
other
than
the
Linux
Foundation
provides
an
awesome.
Cla
bot
and
all
we
do
is
talk
to
it
as
like
a
web
hook.
So
as
long
as
we
see
a
checkmark
come
back
from
that,
we're
satisfied
that
every
author
of
every
commit
in
a
given
pull
request
to
sign
the
CLA,
and
we
can
move
on
from
that
step.
A
So
when
it
comes
to
running
tests,
it
sure
looks
like
there
are
a
lot
of
tests
here
now
when
we
started
this
project
off,
we
knew
it
was
a
distributed
system,
and
so
it
was
going
to
involve
a
lot
of
moving
parts.
You
know
you
just
saw
Dan's
keynote
where
he
talked
about
continuous
integration
is,
is
really
super
fundamentally
important
to
the
cloud
native
way
of
doing
things.
A
So
you
know
we
definitely
wanted
to
test
all
of
the
things,
and
so
we
got
super
enthusiastic
about
making
sure
that
we're
not
just
testing
individual
components
of
kubernetes
in
isolation.
We're
not
just
writing
unit
tests,
we're
not
just
writing
integration
tests.
We're
writing
end-to-end
tests,
because
this
is
a
distributed
system.
So
we're
gonna
stand
up
entire
clusters.
A
We're
gonna
exercise
everything,
we're
gonna,
not
actually
trust
that
anything
works
until
we've
tested
the
whole
thing
and
and
all
the
way
through,
the
problem
is
pretty
much
everybody
felt
like
they
couldn't
confirm
that
their
code
was
feature
complete
or
that
it
logically
worked
until
they
did
and
and
tests,
and
the
problem
with
n
n
tests
is
at
least
back
in
the
bad
old
days.
There's
just
thousands
and
thousands
of
lines
of
bash
everywhere.
There's
bash
used
to
stand
up
a
cluster.
A
There
was
like
a
thousand
line
bash
file
to
define
how
to
actually
run
a
test.
Job
and
worst
of
all,
all
of
this
stuff
was
configured
with
environment
variables,
and
so
you
could
be
something
like
six
layers
deep
of
inclusion
in
bash
scripts
and
you
wouldn't
know
where
I
give
an
environment
variable
came
from
you
wouldn't
know
what
an
environment
variable
here
did
over
there
and
because
it's
bash
I
mean
you
can
write
unit
tests
for
bash,
but
it's
by
no
means
type
safe.
A
Jenkins
job
configuration
is
really
atrocious
to
manage.
We
went
through
a
period
of
time
with
Jenkins
job
builder,
where
we
define
jobs
as
yamo
files
that
then
got
rendered
to
XML
that
we
then
used
bash
to
diff
to
see
if
the
XML
actually
looked
saying,
because
of
course
we
didn't
have
a
linter
for
valid
XML
configurations
until
we
loaded
it
into
Jenkins
and
it
blew
up
and
yeah.
A
So
the
the
other
thing
I
wanted
to
hit
home,
though,
was
that
this
Jenkins
instance
that
was
responsible
for
driving
most
of
the
testing
for
the
project
was
visible
primarily
to
people
inside
the
walls
of
Google.
So
if
you
were
a
contributor
to
the
project
who
worked
anywhere
else,
you
would
hear
people
talk
about
the
blocking
jobs
or
the
Bill
of
blocking
jobs
or
people
who
could
click
through
and
see
historical,
artifacts
and
results
and
of
you
know,
a
large
community
of
people
couldn't
see
this
so
over
time
we
kind
of
got
to
do.
A
We
really
need
to
test
all
the
things.
Isn't
there
some
way
we
can
like
spend
less
energy
running
tests,
and
so
I
wanted
to
take
you
briefly
back
to
around
September
of
2016
when,
instead
of
that
pull
request,
shot
I
showed
you
earlier.
We
were
only
running
like
six
to
seven
tests
for
every
pull
request
and
they
were
still
you
know.
Does
it
run
end
to
end
on
Google?
Does
the
note
look
good
and-
and
you
know
we're
doing
smoke
testing
like
Dan,
said
we're
doing
unit
and
integration
testing.
That's
great.
A
The
problem
was
your
code
wasn't
allowed
to
merge
until
all
of
these
other
tests
were
passing
on
what
we
called
periodic,
or
continuous
and
continuous
or
post,
submit
blocking
jobs.
These
were
sets
of
jobs
or
tests.
That
would
run
every
hour
every
15
minutes
as
often
as
possible
on
the
latest
and
greatest
commits,
and
if
any
of
these
were
failing,
your
code
wouldn't
be
allowed
to
merge
and
it
got
into
this
unfortunate
situation
where
you
can
see
here,
like
all
these
tests
are
passing,
but
these
tests
are
failing.
Well,
that's
just
the
build
cops
problem.
A
That's
not
my
problem.
As
the
pull
request
author.
It's
somebody
else's
job
to
go
chase
down.
Why
those
you
know
20
25,
other
jobs
are
failing
and
then
to
figure
out
who
to
chase
down,
to
ask
them
very
kindly
to
revert
their
code
or
troubleshoot.
Why
the
test
wasn't
working
and
it
ultimately
ended
up
in
this
really
frustrating
where
I
could
check
code
in
that
would
break
the
build
and
I
would
have
no
idea
that
I
had
done
it
and
nobody
would
really
notice
until
the
build
cop
got
paged
and
came
looking
for
you.
A
So
fast
forward
today
to
today,
where
we're
using
prowl
and
Prowse
kind
of
helped
us
get
out
of
this
situation,
if
nothing
for
nothing
else
than
the
fact
that
it's
significantly
better
at
running
tests
concurrently
and
it's
much
easier
to
manage
than
Jenkins.
So
we
really
are
testing
all
the
things
so
we're
making
sure,
first
and
foremost
that
your
code,
you
know,
passes
our
coding
conventions.
Are
you
following
the
right
API
conventions?
Is
it
formatted
cleanly
things
like
that?
A
We're
also
making
sure
that
it
builds
and
we're
doing
a
quick
type
check
to
make
sure
that
it
kind
of
pseudo
compiles
across
all
different
architectures
that
we
support,
so
most
of
our
testing
is
done
for
the
linux
amd64
architecture.
You
know
we
also
release
artifacts
for
things
like
arm
and
we
want
to
make
sure
we're
not
breaking
those
builds
unbeknownst
to
us.
Then
you
know
you've
got
your
pretty
standard,
we
want
to
make
sure
it
passes
unit
and
integration
tests.
A
We
want
to
make
sure
that
it
passes
end-to-end
tests
on
two
different
cloud
platforms,
so
if
it
doesn't
work
in
either
Google
or
AWS,
we
want
to
prevent
this
code
from
merging.
We
want
to
make
sure
that
we
can
sort
of
do
specific
end-to-end
tests
just
on
the
note
level,
because
a
cluster
is
a
distributed
system
and
it
could
fail
for
a
variety
of
reasons.
A
We
really
want
to
look
at
specific
things
like
does
the
container
runtime
interface
work
well
for
this
engine
versus
this
other
engine,
and
then,
of
course,
we
also
want
to
make
sure
that
we
can.
We
can
stick
within
the
realm
of
our
performance
boundaries
by
using
simulated
performance
tests.
Called
cube.
Mark
stands
up
like
a
real
kubernetes
cluster,
but
with
fake
nodes.
A
So
we
can
pretend
like
we're
scheduling
to
a
hundred
node
or
500
node
cluster
without
expending
all
of
those
resources
and
then
finally,
we
also
allow
the
ability
to
run
optional
tests
for
every
pull
request,
but
these
don't
necessarily
prevent
it
from
merging.
Usually
these
are
tests
that
get
skipped
because
they
don't
have
to
run
on
every
poll
request.
A
A
From
there
we
could
scrape
the
data
into
bigquery
tables,
where
I
could
start
to
put
together
graphs
like
this.
For
you
so
around
October
of
2016,
we
started
using
prowl
as
the
primary
like
front
to
kick-off
jobs.
We
initially
had
prowl
than
talking
to
Jenkins,
but
because
prowl
was
the
thing
receiving
these
hundreds
and
thousands
of
requests,
we
could
throttle
and
not
kill
our
Jenkins.
A
So,
if
you
were
pull
requests
past
its
its
pull
request
tests,
that's
the
only
thing
it
needed
to
do
to
merge
and
then
six
months
after
that,
we
retired
the
build
cop
role,
because
it
was
just
punishment
and
pain
on
whoever
was
unfortunate
enough
to
come
up
in
the
rotation
for
that
and
people
have
been
a
lot
happier
for
it.
So
when
I
say
prowl
has
gotten
has
made
things
a
lot
better
for
us.
What
do
I
mean
by
that?
A
A
This
graph
here
shows
you
that
about
600,000
of
those
jobs
were
for
pull
requests.
The
blue
blue
line
is
for
kubernetes
kubernetes
related
jobs,
and
the
green
line
is
for
any
other
project
that
could
be
in
the
kubernetes
or
incubator
or
kubernetes,
SIG's,
orgs
or
other
projects
such
as
cube
flow.
You
can
see
that
the
other
3.7
million
jobs
were
some
of
those
continuously
running
after
the
fact
jobs.
A
You
can
also
see
just
by
the
number
of
unique
jobs,
not
necessarily
the
number
of
runs
that
have
happened,
that
many
more
projects
beyond
kubernetes
have
started
to
use
our
infrastructure.
It's
gotten
to
the
point
where
our
infrastructure
is
friendly
enough
and
reusable
enough
that
it
works
for
projects
beyond
just
kubernetes
kubernetes,
so
I
know
you
know,
cops
and
coop
flow
and
the
community
repo
all
use
this
stuff
today,
and
you
can
see-
that's
not
really
the
case
for
jobs
that
run
after
pull
requests
have
been
submitted
because
again
we
find
that
wow.
A
This
is
a
pattern,
that's
useful
for
a
project
as
large
as
kubernetes.
It's
maybe
not
necessary
for
your
project
today.
So
that's
a
little
bit
about
testing.
Next,
let's
talk
about
our
code
review
workflow.
So
you
know
what
is
code
review?
You
want
to
make
sure
that
you
find
somebody
to
review
UPR.
You
got
to
make
sure
that
you
find
the
right
person
to
find
your
Pete
to
review
your
PR.
A
But
how
do
you
know
who
that
right
person
is,
and
then
how
do
you
notify
them
that
they
should
take
a
look
at
your
PR
and
then
how
do
you
know
that
they're
all
of
the
right
necessary
people
have
taken
a
look
at
your
PR
and
so
things
that
we've
tried
in
the
past?
Is
why
don't
you
just
open
your
PR
and
wait
and
see
if
company?
If
somebody
comes
and
reviews
it
that
doesn't
work
on
a
project
that
the
size
of
kubernetes
there
are
just
too
many
people
and
the
codebase
is
too
large.
A
Let's
try
using
github
notifications.
You've
you've
figured
out
the
person
who's
most
responsible
for
this
area
of
code.
Why
don't
you
add
them
that
doesn't
work,
because
github
notifications
tend
to
just
go
into
the
void.
My
notification
inbox
collects
about
a
hundred
to
three
hundred
notifications
on
a
daily
basis.
It's
as
good
as
useless.
For
me.
We
could
assign
PRS
to
people
to
ask
them
to
review
it
that
way
they
could
use
a
github
issue.
Query
to
figure
out.
A
You
know
all
the
things
that
are
assigned
to
me,
but
how
do
you
know
who
to
assign?
Well,
maybe
you
could
use
github
feature
of
like
the
recommended
who's
committed
to
this
recently
you
could
stock
them
through
commit
logs
or
something
or
maybe
you
could
try
using
github
zko
donors
feature
where
you
can
specify
through
regular
expressions.
You
know
like
which
files
to
use,
unfortunately,
github
Skoda
owners
thing
if
you,
if
you
lay
out
like,
say,
there's
ten
possible
people
who
could
know
be
knowledgeable
about
code
in
the
couplet
subdirectory.
A
If
you
assign
a
review
to,
if
you
change
a
file
in
there
and
use
github
code
owners
feature,
it
will
assign
to
all
ten
of
those
people
at
once.
It
doesn't
just
randomly
choose
a
subset
further
if
you
were
to
assign
to
github
teams
instead
of
individual
people,
that's
great,
if
you're
a
member
of
the
org
and
know
who
is
on
that
team,
so
you
know
who
to
go
bug,
but
if
you're
a
new
contributor,
you
don't
get
any
visibility
into
who's
on
a
given
team.
A
So
you
have
no
idea
who
to
ask
or
talk
to
next.
If
you
need
some
follow-up,
finally,
what
if
we
just
like
use
the
fact
that
somebody
has
write
access
to
the
repo
to
figure
it
to
say
that
this
is
a
person
that
you
should
pay
attention
to.
You
know:
do
they
have
the
commitment
fliped?
Can
they
manually
set
labels?
We
found
that
none
of
these
really
helped
us
scale
as
effectively
as
using
owners
files.
A
So
here's
an
example
of
an
owner's
file
at
the
very
top
of
this
directory
for
the
contributors
guide
in
the
community
repo.
Let's
say
I
wanted
to
add
something
to
any
of
these
documents.
I
would
open
up
a
pull
request
and
our
our
automation
would
look
at
the
owners
file
closest
to
the
code
that
I
am
changing.
So
in
this
case
the
owners
file
specifies
two
groups
of
reviewers.
There
are
reviewers
and
approvers,
and
then
there
are
labels
down
here
for
triage.
So
reviewers
are
people
who
are
looking
at
your
code
qualitatively.
Is
it
written?
A
Well,
you
know,
is
it
checking
for
nil?
Does
it
catch
all
the
corner
cases
I
should
know
enough
about
go
lying,
but
I,
don't
necessarily
have
to
know
about
every
single
internal
of
all
parts
of
kubernetes
to
review
an
arbitrary
pull
request.
On
the
other
hand,
if
I'm
an
approver
I
should
know
exactly
how
this
area
of
code
relates
to
the
rest
of
the
project,
is
it
doing
the
right
thing?
Is
it
maintaining
backwards
compatibility?
Is
it
forward-looking
enough
that
sort
of
thing
and
then
finally,
you
know
we
can
use
these
labels
for
triage?
A
A
So
while
we
request
review
from
the
most
knowledgeable
people,
as
spelled
out
in
the
owners
files,
we
want
to
grow
our
reviewer
pool,
so
anybody
who
is
knowledgeable
enough
about
go
could
theoretically
review
any
pull
request.
However,
when
it
comes
to
somebody
who
wants
to
holistically
look
at
whether
or
not
the
PR
is
doing
the
right
thing,
we
want
to
make
sure
we
get
buy-in
from
all
the
right
approvers.
So
approvers
who
are
recommended
have
to
say
whether
or
not
it's
good
or
not.
You
can
find
out
more
at
that
link
there.
A
So
I
talked
about
applying
late
labels
to
pull
requests.
You
know.
Here's
one
of
my
favorite
pull
requests.
I
managed
to
find
this
set
of
labels
on
it
and
I
had
no
idea
of
what
these
labels
meant
for
the
longest
time.
I
finally
put
together
some
documentation
here,
but
honestly,
my
perspective
on
the
label
set
like
this
is
that
you
really
should
like.
A
You
only
need
the
labels
that
you
absolutely
need
to
survive
and
I've
come
to
realize
on
the
kubernetes
project
that,
like
these
are
my
industrial-strength
github
labels
and
the
project
can't
live
without
them
right.
If
you
break
it
down,
here's
the
description
of
the
labels,
that
sort
of
shows
you
know
what
are
they,
what
do
they
do?
What's
the
Proud
plugin
that
handles
them
and
who
can
apply
or
use
these
labels?
A
So
all
the
green
labels
are
the
labels
that
need
to
show
up
on
your
pull
request
in
order
for
it
to
be
mirja
pool,
you
know,
have
the
reviewers
looked
at
it,
there's
the
URL
GTM
label,
how
the
approvers
looked
at
it.
There's
your
approved
label.
Did
you
sign
the
CLA
great?
All
of
these
labels
are
things
that
are
applied
for
a
triage
purpose.
You
know,
is
it
a
bug?
What
kind
of
priority
is
this
issue
or
pull
requests?
What
area
does
it
apply
to?
A
This
helps
the
right
people
find
issues
that
they
care
about
or
are
interested
in
more
easily
and
further.
Well,
these
are
labels
that
you
normally
have
to
be
like.
You
have
to
have
right
access
to
a
repo
to
apply
labels
through
prowl
we've
allowed
anybody.
You
don't
even
have
to
be
a
member
of
the
organization
to
help
sort
of
triage
these
things
more
appropriately,
so
anybody
can
apply
a
priority.
Anybody
can,
you
know,
apply
an
area
or
a
cig
to
help
us
more
effectively
find
things
and
further.
A
A
If
you
happen
to
have
some
spare
time,
there's
not
much
to
say
about
these
labels
other
than
to
say
that
they
sort
of
helped
the
kubernetes
release
team
heard
the
the
cats
that
are
the
organized
chaos
of
open-source
into
something
resembling
a
top-down
hierarchy
and
a
consistent
release
process,
because
these
labels
are
so
important
for
us.
We
want
to
make
sure
that
the
same
labels
are
present
in
every
single
repo
in
the
kubernetes
project.
A
So
that's
all
sixty
repos
in
the
kubernetes
org,
all
twenty
repos
and
the
kubernetes
incubator
org,
and
the
number
of
repos
that
are
showing
up
in
the
kubernetes
SIG's
org
is
something
like
eight
today.
So
we
use
a
syncing
tool
that
allows
us
to
specify
you
know
new
labels
and
what
they
should
be
to
migrate
old
labels
or
rename
old
labels
to
new
labels
and
to
also
deprecated
labels
and
delete
them
after
some
amount
of
time.
A
This
is,
what's
allowed
us
to
migrate,
the
kubernetes
repo
from
two
hundred
and
twenty
odd
labels
down
to
about
a
hundred
and
seventy
labels.
Today,
labels
are
also
great
because
they
empower
the
community
to
you
know,
write
some
of
their
own
triage
tools,
so
here's
an
example
of
an
issue
query
that
a
community
member
dims
often
uses
to
find
first-time
contributors
who
might
want
to
be
shepherded
through
pull
requests.
If
you're,
not
a
member
of
the
project,
we
want
to
make
sure
that
your
pull
request
isn't
turning
kubernetes
into
a
Bitcoin
miner.
A
So
we
need
a
human
to
just
take
a
brief
look
at
your
poll
requests
and
apply
a
slash
okay
to
test
command
to
it,
to
remove
that
needs,
okay
to
test
label,
so
dims
mentioned
he'd
been
using
that
query
just
on
kind
of
a
daily
basis
when
he
had
time
we
wanted
more
people
in
the
community
to
use
it.
So
JC
Mars
bought
a
domain
that
just
redirects
to
that
query,
and
now
anybody
who
wants
to
help
out
or
identify
first-time
contributors
to
help
Shepherd
through
the
project
can
click
through
that.
A
A
hundred
percent
of
the
time
and
some
things
can
be
susceptible
to
environmental
conditions
like
the
relative
humidity
and
the
alignment
of
the
stars.
If
your
code
is
supposed
to
pass,
there's
a
75%
chance
that
it
actually
will
pass
when
we're
having
our
worst
times
that
what
that
roughly
means
is,
if
I
must
serially
passed,
the
pull
request
each
and
every
pull
request
before
I
merge
it
that
our
ideal
merge
velocity
is
somewhere
around
twenty
to
forty
pull
requests
a
day.
So
how
are
we
actually
doing
compared
to
that
theoretical
ideal?
A
A
If
it
fails
we're
not
going
to
know
for
sure
which
one
of
the
five
failed
the
test,
so,
unfortunately,
we're
gonna
have
to
take
a
loss
on
that,
but
by
and
large
you
can
see
that
most
of
our
pull
requests
are
now
tested
in
batches.
In
fact,
by
relative
amounts.
Over
50%
of
our
poll
requests
are
tested
in
batch,
there's
literally
no
way
our
project
could
maintain
the
velocity
it
does.
If
we
didn't
implement
batch
testing,
which
we
did
around
November
of
2017
as
I
recall,.
A
You
know
just
as
a
sanity
check
is
our
infrastructure
doing
well.
This
is
mostly
a
monday
to
friday
project.
Not
too
many
people
are
contributing
large
amounts
of
code
on
the
weekend.
So
I
would
expect
the
majority
of
our
merges
to
be
happening.
Monday
to
Friday,
assuming
everything
is
going
well
with
their
merge
infrastructure
and
for
the
most
part
we
are
merging
less
and
less
over
the
weekend.
Those
red
spikes
there.
A
So
the
other
part
of
the
work
issued
triage,
some
of
you,
who've,
worked
on
the
project
may
be
familiar
with
fada
bot,
faina
BOTS,
actually
doing
stuff
that
humans
could
100%
do
on
their
own
fada
BOTS
issuing
comments.
That's
all
he's
doing
he's
he's
issuing
a
comment
on
issues
or
pull
requests
that
haven't
had
any
activity
by
a
human
in
90
days
and
then,
if
nobody
does
anything
for
another
30
days,
he
applies
another
label
and
then,
if
nobody
does
anything
after
another
30
days,
he
closes
the
issue.
A
These
are
all
things
that
so
the
life
cycle
things
anybody
can
do
closing
the
issue.
Any
member
of
the
kubernetes
org
can
do
you
can
also
as
a
human
respond
and
say
no,
no
wait.
I
actually
think
this
is
a
really
important
issue
and
should
be
considered
for
more
than
120
days.
Without
any
comment,
so
I
can
apply
the
frozen
lifecycle
label
to
it
say
you
know,
the
lifecycle
of
this
issue
should
be
frozen
until
the
heat
death
of
the
universe,
because
I
know
it's
important
enough
for
somebody
to
come.
Take
a
look
at
it.
A
As
a
result,
the
number
of
open
issues
on
the
kubernetes
project,
at
least
in
the
kubernetes
kubernetes
repo,
has
gone
from
about
4,500
down
to
2,500
and,
interestingly,
over
the
past
couple
of
weeks.
It's
kind
of
hit
this
nice
steady
state
where
issues
are
getting
open
but
nobody's
really
paying
attention
to
them.
So
they
end
up
aging
out
now.
I
recognize
this
isn't
ideal.
A
This
is
the
this
is
another
graph
from
dev
stats.
That
shows
the
number
of
times
that
people
issued
the
slash
lifecycle
command,
probably
to
freeze
an
issue,
and
so
this
probably
tells
you
when
we
turned
on
Veda
bot
and
asked
him
to
start
commenting
on
issues
that
hadn't
had
any
activity.
It
forced
people
to
start
doing,
planning
down
the
line
that
they
hadn't
done
up
front.
A
This
is
the
prow
job
that
does
it
and
I'm
only
pointing
this
out
just
to
show
that
it's
just
a
simple
label.
Query:
it's
something
we
were
you
could
do
by
hand,
but
we
have
a
bot
to
do
this
for
us.
Similarly,
we
also
offer
the
ability
for
somebody
to
issue
a
slash,
retest
command
and
any
test
that
failed
will
automatically
be
retried
and
rerun.
This
was
one
of
the
most
useful
commands.
A
This
is
one
of
the
most
useful
commands
on
the
project.
You
can
see
its
usage
spiked
hugely
once
we
introduced
it,
but
the
reason
it's
gone
down
since
then
is
because
we
have
fada
bot
automatically
issue
this
command
to
any
pull
request.
That's
had
an
LG
TM
applied
and
it's
approved,
so
it's
good
to
go
it
just
the
tests
are
flaking
for,
for
whatever
reason
and
so
it'll
come
through
and
automatically
just
apply
the
retest
command
until
things
actually
work
and
it
gets
pushed
through
again,
it's
just
yeah.
A
So
as
far
as
the
rest
of
the
work
that
I
laid
out
I,
don't
think
I
have
any
time
to
get
that.
That's
okay!
This
is
fine
because
if
you
go
to
the
cig
testing
deep
dive
on
Friday
at
2:00
o'clock,
you'll
get
to
hear
a
lot
more
about
how
we
actually
configure
manage
and
run
our
jobs
and
how
you
can
use
this
on
your
project.
So
thank
you
very
much.