►
From YouTube: Intro: Anatomy of a Kubernetes Release: Success Through Team and... - Sascha Grunert & Daniel Mangum
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Intro: Anatomy of a Kubernetes Release: Success Through Team and Tools - Sascha Grunert, SUSE & Daniel Mangum, Upbound
As Kubernetes is one of the biggest open-source projects worldwide, the creation of a new release can be an overwhelming topic. The major burden is not only to release a new version every three months, but also to provide stability and security fixes over the whole support period of a release. In this talk, we will cover the details behind the lifecycle of a Kubernetes release. The presentation will contain everything from assembling the Release Team before the actual cycle begins up to maintaining the release branches, which provide continuous patch releases to the community. The talk will outline the technical process of branch management by covering all of the necessary stages during the release cycle. We will also demonstrate our Release Engineering tooling and dive into the design decisions we’ve made as a group that shape the overall process.
https://sched.co/cpAD
A
B
All
right,
hello,
folks,
my
name
is
daniel
mangum,
I'm
presenting
today
with
sasha
grennert
sasha
is
a
tech
lead
for
sig
release,
and
I
am
the
ci
signal
lead
which
we'll
get
into
what
that
means
later
on,
for
the
1.19
kubernetes
release,
we're
going
to
give
a
talk
today
about
how
we
do
kubernetes
releases.
This
is,
as
you
see
as
you'll,
see
on
the
slides
in
just
a
moment.
B
This
is
part
one
of
a
talk
and
we
are
going
to
be
giving
part
two
at
kubecon
eu,
which
is
also
virtual-
and
I
guess
just
a
few
weeks
after
this
airs,
so
we're
excited
to
kind
of
give
a
nice
introduction
to
this
and
then
go
into
some
more
depth.
So
we
hope
you
tune
in
for
that
one
as
well,
so
I'm
gonna
go
ahead
and
share
my
screen
here
and
we
can
get
right
into
it.
B
All
right
so,
like
I
said
today,
we're
talking
about
anatomy,
of
a
kubernetes
release,
success
through
team
and
tools.
So
a
big
part
of
releasing
kubernetes
is
a
lot
of
folks
that
come
together
to
have
individual
tasks
that
then
roll
up
to
this
kind
of
big
operation
that
results
in
new
kubernetes
releases
being
available
for
the
community,
and
we
try
to
automate
a
lot
of
that
through
tools.
But
there's
also
a
lot
of
individuals
spread
out
all
over
the
world.
Who
do
a
lot
of
work
to
make
this
happen.
B
So
to
kind
of
give
you
an
overview
of
what
we're
going
to
talk
about
today,
we're
going
to
first
just
look
at
how
kubernetes
releases
are
made
in
general.
Just
logistically,
then
we'll
provide
an
overview
about
the
siegeless
special
interest
group,
which
sasha
and
I
are
both
a
part
of
and
then
we'll
deep
dive
into
the
actual
1.19
release
cycle.
B
I'm
excited
to
talk
about
this
today
and
sasha
is
going
to
give
us
a
lot
of
detail
on
it,
because
1.19
is
a
little
bit
unique
because
of
the
global
pandemic
pandemic
situation
and
how
some
of
the
timeline
has
to
be
adjusted
and
how
that's
affected
subsequent
releases
and
that
sort
of
thing.
So
I'm
excited
to
kind
of
go
through
how
the
the
group
work
together
to
accommodate
this
changing
times
that
we're
living
in
right
now.
So
this
will
be
a
especially
unique
cycle
to
look
at
all
right.
B
So
looking
at
just
an
overview
of
the
kubernetes
release
cycle
at
the
start
of
the
release
cycle,
the
first
thing
you
have
to
do
is
is
build
the
release
team.
So,
as
we'll
get
into
a
moment,
one
of
the
I
think,
most
thriving
parts
of
the
kubernetes
community
is
the
sig
release
shadow
program.
So
essentially,
there
are
different
teams
as
part
of
sig
release,
and
they
all
do
different
things
to
make
a
release
happen
and
there's
an
application
process
for
you
to
become
a
shadow
of
a
given
team.
B
So
each
team
has
a
lead
and
then
there's
usually
three
or
four
shadows.
So
at
the
beginning,
the
first
thing
you
want
to
do
is
establish
the
leads
which
are
typically
shadows
or
previous
leads
from
previous
releases
and
then
you'll
look
through
the
applications
for
all
of
the
shadow
applicants
and
select
them
for
the
different
teams.
B
And
then,
after
that,
you
go
ahead
and
start
tracking
the
things
that
have
to
be
done
by
each
of
those
deadlines
and
that's
where
the
sig
release
team
definitely
comes
into
play
quite
a
bit
and
then
throughout
the
cycle.
We
have
continuous
updates
starting
on
a
weekly
basis
and
then
getting
into
more
granular,
and
then
there's
also
intermediate
releases.
So
alpha
beta
and
rc
releases,
that's
done
by
the
release
engineering
subgroup.
B
Another
group
that
sasha-
and
I
are
both
a
part
of
and
he's
going
to
talk
about
that
a
little
bit
more
all
right.
So
looking
at
some
of
those
deadlines
that
I
mentioned,
the
first
thing
is
enhancements
freeze.
So
if
you're
not
familiar
with
how
new
features
get
added
to
kubernetes,
you
have
to
create
a
kepp,
so
a
kubernetes
enhancement
proposal.
So
basically,
this
is
just
a
write-up.
B
So
that
typically
looks
like
three
times
a
week
and
then
eventually
right
up
close
to
the
release
time
potentially
every
day
and
those
that's
just
where
leads
and
shadows
from
the
different
teams
within
sig
release
will
give
updates
on
their
status,
which
help
inform
how
we're
moving
towards
a
release
next
after
that
is
the
docs
deadline,
so
you
have
to
imagine
for
new
features.
We
have
to
document
them
at
least
most
of
them,
especially
if
they're
user
facing
and
for
the
docs
deadline.
B
The
docs
team,
which
is
one
of
the
sig
release
kind
of
sub
teams,
make
sure
that
all
of
the
prs
for
docs
are
in
place
and
that
they
can
be
reviewed
so
that
they
can
quickly
be
merged
later
on
code
freeze
is
probably
the
biggest
thing
that
you've
heard
about.
If
you
follow
any
kubernetes
folks
on
twitter
or
anything
like
that,
so
code
freeze
is
exactly
what
it
sounds
like.
You
can't
merge
any
more
code
into
the
release
branch,
so
you
have
to
have
code
complete.
B
You
have
to
have
your
pr's
open,
they
have
to
be
approved
and
then
merged
this
this
cycle.
In
particular,
we
had
some
issues
with
ci,
so
we
always
work
around
that
to
make
sure
that
things
that
are
complete
are
able
to
be
included
in
the
release.
But
generally
you
have
to
have
things
merged
by
that
date.
B
Cherry
pick
deadline
basically
means
that
no
more
changes
can
go
into
the
release
branch,
so
you
can
imagine
after
code
freeze
cherry
picks
have
to
happen
from
the
main
branch
onto
the
release
branch,
but
we
also
have
a
deadline
for
for
when
that
can
be
done,
and
those
are
usually
for
fixing
critical,
bugs
or
fixing
a
test.
That's
failing
or
something
like
that,
and
then.
Finally,
we
culminate
in
release
day.
B
So
this
is
exactly
what
it
sounds
like
it's
just
the
day
that
we
make
the
release
and
there's
a
lot
of
celebration
for
all
the
hard
work
that
folks
have
done.
I
want
to
mention
here
at
the
bottom.
We
have
some
links
to
the
sig
release.
Repo
sig
release
is
where
you
have
handbooks
for
each
of
the
roles
that
folks
play
in
the
release,
as
well
as
the
calendar
for
all
previous
and
current
releases.
B
B
B
They
can
be
really
helpful
in
that
regard.
This
this
release
cycle
for
1.19,
taylor,
dolezal,
is
our
release,
lead
and
he's
done.
An
excellent
job
and
you'll
see
a
release
coming
out
in
august
because
of
the
the
work
that
he's
done
in
leading
these
teams,
the
emeritus
advisor
is
typically
someone
who
was
a
release
team
lead
in
a
previous
release.
B
So
this
this
release,
it's
tim
pepper
and
they
really
just
helped
their
release
team
lead
whenever
they
have
questions
about
the
way
things
should
work
or
just
kind
of
help
shoulder
some
of
the
burden.
If
there's
a
lot
of
stuff
going
on,
so
they
can
be
especially
helpful
and
they're.
Usually
someone
who
is
very
very
familiar
with
the
kubernetes
community
release
engineering
is
a
little
bit
different
than
all
the
other
teams,
because
it's
its
own
sub
project.
B
So,
as
I
mentioned
sasha
and
I
both
work
on
release
engineering,
so
these
are
the
actual
tools
like
the
command
line
interface,
which
is
called
krell.
That
allows
us
to
actually
do
the
operations
to
physically
make
the
release,
so
that
includes
you
know,
pushing
new
builds,
generating
release,
notes
all
things
like
that.
So
that's
kind
of
the
engineering
side
of
producing
a
release.
There's
some
really
interesting
stuff
that
happens
there.
B
There's
always
plenty
of
work
around
that
tooling
for
folks
that
are
looking
to
get
involved
and
that
actually
has
a
cycle
in
terms
of
membership
outside
of
just
a
release
cycle.
So
typically,
if
you're
part
of
release
engineering
you're
part
of
that
sub
project
for
multiple
cycles,
it's
not
really
the
typical
shadow
program,
all
right,
looking
at
bug,
triage
ci
signal
on
the
rest,
these
are
all
critical
teams
for
making
sure
different
parts
of
the
release
are
in
place.
You
can
kind
of
guess
by
their
name
what
they
do.
B
Each
of
these
teams
typically
has
three
to
four
shadows
and
then
a
lead,
who
is
usually
a
shadow
before
so
just
to
give
you
some
context
on
my
involvement,
this
release
cycle
for
1.19-
I
am
the
ci
signal
lead
for
the
past
two
releases,
1.17
and
1.18.
I
was
a
shadow
on
that
team,
so
it
really
is
a
good
process
for
helping
you
to
build
up
kind
of
acumen
in
your
given
area
and
then
you're
able
to
move
into
a
leadership
role
and
share
that
same
mentorship
with
folks
that
are
falling
behind
you.
B
Docs
they're
really
important
for
that.
Docs
freeze
that
we
were
talking
about
so
there's
folks
on
the
dock
side
that
supplement
the
technical
review,
with
kind
of
a
grammatical
review
and
content
review,
and
that
sort
of
thing-
and
I
know
from
implementing
a
docs
pr
that
this
cycle,
that
they
are
extremely
helpful
as
an
engineer.
A
lot
of
these
folks
are
really
really
talented
in
how
they
write
documentation
and
they
can
help
guide,
folks
that
are
documenting
the
keps
that
they've
put
into
place
speaking
of
caps
nexus
enhancements.
B
So
these
are
folks
that
are
tracking
new
functionality,
that's
being
added
to
kubernetes
for
this
cycle
and
making
sure
those
are
implemented
in
a
timely
fashion
and
if
not
they'll
get
bumped
to
the
next
release
and
then
finishing
up
is
release
notes.
It's
been
cool
to
see,
release,
notes
working,
especially
with
release
engineering
lately
to
be
able
to
generate
some
of
the
notes
that
are
produced.
So
this
is
basically
just
telling
consumers
of
kubernetes
what
new
things
are
available
in
a
new
release,
and
that
includes
the
alpha
beta,
rc
releases,
etc.
B
So
folks
can
know
exactly
what
they're
getting
what
may
be
added.
What
may
be
be
removed
in
a
release,
all
right?
And
lastly,
just
talking
about
the
sig
release
shadowing
program
here.
I've
already
mentioned
it
quite
a
bit,
but
I
just
want
to
encourage
anyone
watching
this
talk.
Who
wants
to
get
involved
in
this
process?
I
think
you
would
be
challenged
to
find
an
open
source
community
that
better
fosters
growth
than
kubernetes,
specifically
in
the
sierra
league
shadowing
program,
and
I'm
speaking
from
my
own
personal
experience
there
joining
the
shadowing
program
two
cycles
ago.
B
There
have
been
countless
people
who
have
been
invested
in
my
growth
and
also
have
expected
me
to
contribute
a
lot
back
in
return.
So
I
am
just
really
part
proud
to
be
part
of
this
program
and
I'm
also
proud
to
be
helping
new
folks
coming
into
it.
So
please
feel
free
to
jump
in
the
slack
channel.
Get
on
the
mailing
list,
apply
for
a
shadow
position
in
a
new
release
or
feel
free
to
reach
out
to
me
or
sasha
directly.
C
Thank
you,
then
it
was
pretty
great.
So
far,
all
right,
you
should
see
my
screen
right
now.
Is
that
correct?
Just
one
question
yeah
all
right,
so,
let's
dive
into
the
kubernetes
1.19
release
cycle.
So
now
we
got
more
or
less
a
good
overview
about
the
theory,
how
it
should
look
like
or
what
the
different
parts
of
sick
release
was,
and
I
was
a
little
bit
confused
as
I
started
to
contributing
to
sick
release
from
the
beginning
of
2019.
C
I
think
so
and
but
in
general
it's
pretty
pretty
good
formalized
and
now
the
first
thing
we
have
to
do.
We
are
on
monday
april
13..
We
have
to
assemble
the
release
team
at
all
and,
for
example,
for
1.19
there
was
the
emergency
advisor
lucky
and
he
just
proposed
that
we
now
have
to
assemble
a
kubernetes
1.19
release,
team
and
roger
was
the
release
lead
for
1.18
and
he
nominated
419
taylor
and
yeah.
That's
more
or
less
the
official
process.
C
There's
also
a
release
team
selection
process
available
right
here,
which
contains
a
lot
of
information
about
timings,
membership,
also,
the
selection
criteria,
criteria
about
shadows
and
the
release
team
lead
and
stuff
like
that.
I
don't
want
to
go
through
all
of
this
right
now,
but
it's
pretty
good
well
written
down,
and
I
think
it
follows
a
good
good
example
how
the
community
works
and
how
communities
should
work
in
that
area
right
in
general,
so
yeah.
C
So
now
we
have
taylor
as
new
release
team
lead
and
then
he
can
take
over
and
just
select
or
more
or
less
in
every
previous
team
lead
or,
for
example,
jeremy
nominated
nepalin
for
enhancements
leave
419,
and
this
goes
through
more
or
less
every
section
of
the
whole
team.
And
if
we
have
the
different
leads,
then
the
leads
can
start
to
assemble
their
own
part
of
the
of
the
team
by
filling
that
up
with
the
shadow
applications.
C
So
the
shadow
application
is
not
nothing
really
special
at
all.
It's
just
more
or
less
a
google
form
and
then
the
different
lead
will
reach
out
to
you
and
we'll
talk
to
you
about
if
you're
interested
in
the
role.
What
could
you
exp?
What
do
you
expect
from
the
role
or
what
could
you
expect
from
the
overall
yeah
overall
topic,
all
right,
so
the
next
thing
we
have
to
do
and
this
ci
signal
master
blocking
tests
have
to
be
ready.
Daniel
will
talk
in
a
few
minutes
about
this
too.
C
I
just
want
to
show
you
how
I
see
a
signal.
How
it
looks
like
it's
green
is
in
general.
Pretty
good
red
is
not
so
good,
because
it's
failing
and
daniel
will
give
you
a
show
in
a
couple
of
minutes,
a
good
overview
about
how
we
can
actually
interpret
those
data.
What
we
have
to
do
if
tests
are
failing
and
yeah
what
the
work
of
ci
signal
is
in
general.
C
The
next
thing
we
have
to
do.
We
have
to
start
tracking
the
enhancements
yeah
for
tracking
enhancements.
We
have
an
enhancement
tracking
sheet
which
looks
like
this
for
119,
and
here
we
can
see
that
we
are
pretty
green,
which
is
good.
This
was
not
always
the
case
and,
for
example,
we
spoke
a
little
bit
about
zekkom
ga
and
we
should
find.
C
Second,
is
it's
not
not
part
of
this
sheet
anymore?
I
can't
find
it
right
now,
but
in
general
you
can.
You
can
see
that
the
enhancements,
every
enhancement
for
the
recycle
will
be
tracked
and
we
have
to
ensure
that
it
also
has
documentation
and
that
the
timelines
for
the
dog
status
is
right
right
on
time
and
that's
the
a
pretty
good
overview.
If
you
want
to
get
to
know
what
is
coming
up
in
the
next
release
and
what
is
new
at
all.
C
So
then
this
was
the
first
week.
There
was
a
lot
of
work
to
do
already,
and
if
we
jump
into
the
next
weeks,
then
we
will
see
that
release
engineering
will
cut
some
releases.
So
in
our
case
these
are
the
first,
the
alpha
2
and
the
alpha
3
releases.
Alpha
1
has
been
cut
directly
after
1.18.
So
that's
not
that
time
critical
for
the
release
at
all.
This
is
more
or
less
an
automated
automated
way
to
prepare
for
the
next
release.
C
C
Then
we
will
do
weekly
release
team
meetings
which
provide
status
updates
about
all
different
rules.
This
looks
like,
for
example,
we
have
a
google
docs
document
where
we
document
all
those
all
those
status
updates,
and
if
we
look
a
little
bit
down
and
we
can
see
yeah,
we
had
one
last
friday
and
every
meeting
will
be
recorded
for
sure
and
for
example,
then
we
can
see
that
the
enhancements
is
already
green,
so
we
give
a
little
status
indicator
if
he,
as
a
team,
has
green
red
or
yellow
and
for
ci
signal.
C
C
Now
we
are
already
in
week
six
week.
Six
means
that
we
have
enhancements
freeze,
so
all
enhancements
which
want
to
be
included
in
the
current
release
must
have
like
kubernetes
enhancement
proposal
in
an
implementable
stage.
This
also
includes
test
plans
and
graduation
criteria,
so
this
can
take
up
some
time
if
a
an
enhancement
proposal
should
be
really
be
implemented
in
a
release,
then
I
just
can
recommend
to
bring
it
into
an
implementable
state
in
the
release
before
otherwise,
it
would
be
really
really
stressful
for
my
perspective,
so
then
we
need
yeah.
C
C
So
if
an
enhancement
does
not
really
really
meet
those
requirements,
then
it
will
be
removed
from
the
milestone
and
we
have
to
require
an
exception.
This
is
pretty
pretty
interesting,
because
such
an
exception
would
be
something
like
this,
so
just
an
email
to
the
mailing
list
would
be
enough
and
we
just
can
raise
an
enhancement
request
except
your
request.
C
All
right,
so
then
we
cut
beta0
beta0
is
still
on
the
master
branch.
This
is
new
for
119.
Usually
we
would
create
a
release
branch,
but
since
we
just
changed
the
release
cycle
to
be
a
bit
more
longer
than
expected,
we
just
decided
to
leave
it
on
the
master
branch
and
then
go
for
the
release
candidate,
which
will
create
the
release
branch.
C
But
this
will
be
later
on
from
week.
8
on.
We
start
our
build
down
phase.
So
this
means
that
we
have
an
increased
cadence
of
the
release
team
meetings
and
we
begin
to
track
the
work
more
closely.
So
we
have
to
report
in
a
more
continuous
way
and
that
we
just
have
a
higher
focus
on
our
bug,
fixes
that
that
we
can
eliminate
test
flakes
and
generally
stabilize
the
release.
C
So
in
the
later
weeks,
we
continue
our
burn
down
phase.
We
are
now
in
week,
eight
to
ten.
You
can
already
see
that
the
release
cycle
has
a
little
bit
stretched
over
all
over
the
whole
weeks.
Usually
we
would
just
have
around
about
three
months
now
we
have
for
45
months
for
the
overall
release
and
the
release
team
lead
will
also
give
a
call
for
exceptions
for
enhancements
and
the
release
engineering
team
will
later
on
cut
better
one
and
better
too.
So.
C
Then
we
have
a
docs
deadline
and
at
least
we
need
a
placeholder
pr
for
the
documentation
for
the
enhancements,
and
otherwise
it
would
be
really
really
hard
to
bring
those
docs
into
the
into
the
release,
and
this
can
also
lead
into
yeah
that
maybe
maybe
we
have
to
drop
an
enhancement,
because
the
docs
are
not
ready.
So
this
is
a
really
really
hard
feature
gate
on
that
point.
C
C
This
means
that
we
still
so,
for
example,
if
we
cut
119
rc1,
then
we
will
create
a
release
branch,
but
this
does
not
really
mean
that
it's
not
possible
to
bring
new
and
or
bring
future
enhancements
into
the
master
branch.
We
will
sync
up
the
release
branch
and
the
and
the
master
branch
only
continuously
basis
and
yeah
until
we
reached
code
saw,
but
we
will
come
later
to
that.
C
So
then,
in
week,
14
to
week
16
the
release
branch
management
will
create
that
boss,
v2,
which
is
code
for
in
our
in
our
case.
So
then
it
it's
not
possible
to
actually
merge
something
into
the
or
submerge
something
into
the
release
branch
indirectly,
via
the
master
branch.
We
don't
sync
them
up
anymore,
but
you
are
able
to
actually
yeah
cherry
pick
your
changes
into
the
release
into
the
release
branch
yeah
that
would
be
possible
generally.
The
docs
also
have
to
be
completed
and
merged,
otherwise
yeah.
C
We
would
run
into
time
problems
and,
in
weeks
17
to
20.
We
start
our
daily
burn
down.
So
then
we
will
give
a
daily
update
about
the
release.
What
is
going
on
and
what
is
the
current
status
about
it,
so
the
cherokee
deadline
and
test
trees.
This
also
is
august,
the
6th
in
the
time.
So
this
is
in
the
future
now
and
we
do
a
small
break
for
kubecon
and
cloudnativecon
europe,
which
is
pretty
fine,
and
we
mostly
always
do
this
and
branch
management
will
release
119
on
august
27
yeah.
C
B
Awesome
thanks
sasha
I'll
go
ahead
and
share
my
screen.
I'm
just
going
to
give
you
all
a
quick
rundown
of
what
it
might
look
like
kind
of
a
day
in
the
life
of
someone
on
the
ci
signal
team.
So
I
will
go
ahead
and
share
here.
So
sasha
actually
alluded
to
this
earlier,
but
what
you're
looking
at
here
is
test
grid.
So
that's
testgrid.kates.io!
B
You
can
see
kind
of
these
different
categories
of
dashboards
here.
We're
particularly
interested
in
sig
release
and
you'll,
see
that
we
have
blocking
and
informing
jobs
for
each
branch.
So
for
most
of
the
release,
the
critical
dashboard
we're
looking
at
is
sig
release,
master
blocking
and
then
later
on,
you'll
see
we're
later
in
the
1.19
release
cycle.
So
we
now
have
1.19
blocking
jobs
specifically
for
a
while
those
will
pretty
closely
mirror
the
master
blocking
jobs,
but
blocking
jobs
basically
say:
okay,
you
can't
release
if
this
is
passed.
B
If
this
is
not
passing
and
informing
jobs
are
more
like.
This
is
giving
us
just
an
indication
of
general
health,
so
I
want
to
show
you
all
quickly,
actually
a
a
job
that
I
was
investigating
over
the
weekend,
so
we
don't
want
to
ever
have
any
red
jobs
on
the
master
blocking
dashboard
you'll
see
we
actually
have
two
right
now,
so
I
was
looking
specifically
into
this
gce
device.
B
Plugin
gpu
master
job,
so
the
first
thing
you'll
typically
do
is
go
ahead
and
open
an
issue
and
describe
what's
happening
and
kind
of
give
some
information,
and
you
can
tag
the
relevant
sig
to
say
hey.
It
looks
like
you're
responsible
for
this
job
or
test.
Can
you
please
take
a
look
at
it?
In
this
case,
a
single
test
appears
to
be
causing
this
entire
job
to
fail
so
you'll
see,
I
got
some
feedback
from
some
folks
who
are
knowledgeable
of
this
area,
specifically
from
this
feedback.
B
Here
I
was
able
to
tell
that
we're
having
an
issue
with
the
image
that
was
being
used,
so
tests
can
fail
for
a
lot
of
different
reasons.
Importantly,
let
me
move
this
out
of
the
way.
Importantly,
we
have
the
configuration
of
jobs
which
are
a
set
of
tests
to
be
run.
So
that's
what
you're,
seeing
here
all
these
different
jobs
I
mean
these
are
runs
of
a
job
and
those
would
be
in
the
test
info
repo,
so
specifically
in
config
jobs.
B
So
here
we're
looking
specifically
at
gc
device,
plug-in
gpu
master
and
a
nice
thing
that
test
grid
does.
Is
it
can
show
you
the
changes?
So
if
we
go
back
a
bit,
we
want
to
see
when
this
started
failing
and
you
can
see
that
the
infra
commit
did
not
change
so
we're
thinking,
that's
probably
not
what
caused
it,
but
the
kubernetes
kubernetes
commit
did
change.
B
So
the
first
thing
I
would
think
to
do
is
look
at
at
the
diff
between
those
two
commits
and
see
if
anything
happened,
and
often
that
will
tell
you
why
it
is
failing
so
you'll
see
here
that
I
actually
mentioned
that
here.
There's
only
one
commit
between
the
two
and
I
looked
at
what
changed
here
and
actually
all
of
these
changes
don't
appear
to
affect
those
tests.
B
So
it's
confusing
to
see
generally
usually
when
there
is
a
change
and
then
a
test
or
it's
failing
that
it
affected
that
in
this
case,
that
was
not
the
case
and
you'll
actually
see
through
some
of
the
communication
with
sig
scheduling.
Folks,
we
determined
that
in
a
separate
repo
there
is
a
daemon
set
config
that
we
actually
reference
in
one
of
our
tests.
Here
I
mean
you'll,
see
we're
looking
at
the
google
cloud
platform
container
engine
accelerators
repo,
and
we
actually
download
this
in
the
test
on
every
time.
B
So
there's
lots
of
different
ways
that
tests
can
be
implemented
in
kubernetes.
This
one
is
a
little
bit
strange,
particularly
this
is
a
remote
repo
that
then
references
the
latest
image
that
is
built
in
a
different
repo,
so
you
can
see
there's
lots
of
different
ways
that
tess
can
fail
here.
So
this
is
using
latest,
so
it's
always
pulling
the
latest.
So
if
there's
an
image
that
that
becomes
incompatible,
then
we're
gonna
have
issues
right,
and
this
this
daemon
set
particularly,
is
for
installing
nvidia
gpus
on
on
gcp
gce
instances.
B
B
B
So
you'll
see
here
that
we
found
a
version
that
worked
and
we
also
updated
some
other
components
of
it,
and
so
now,
by
default,
it's
going
to
use
this
local
one,
but
we
can
still
override
it
to
use
a
a
remote
one
as
well,
so
that
actually,
as
you
can
see
on
vr,
is
going
to
fix
the
the
test
that
was
failing.
So
if
we
look
we'll
actually
see
that
there's
a
a
per
pull
request
job
here.
That
also
runs
that
mirrors
this
one.
That's
failing
so
we're
able
to
see
that
that
passes.
B
So
that's
an
example
of
an
investigation.
So
in
that
case
it
was
an
issue
with
the
test,
not
with
an
actual
code
change,
but
sometimes
it's
a
code
change
and
then
you
may
have
a
bug
that
needs
to
be
addressed
as
well.
Sometimes
ci
signal
will
address
those
like
in
this
case.
In
other
cases,
we'll
we'll
ask
the
sig.
You
know
you
have
more
context
here.
Please
address
that.
B
So,
if
you're
interested
in
ci
signal,
that's
what
it
can
look
like,
it's
a
lot
of
just
tracking
down
different
things
that
are
failing
and
reporting
on
it.
So
it
can
be
a
little
bit
monotonous,
but
overall,
it's
a
it's
a
very
rewarding
place
to
be
because
you
get
to
see
the
overall
health
of
a
release
and
contribute
to
to
making
a
release
healthier.
B
So
with
that,
I
think
we
want
to
go
ahead
and
wrap
up
here.
It's
been
a
pleasure
to
chat
with
you
all
today
and
we're
looking
forward
to
our
kind
of
sequel
talk
at
kubecon
eu.
So
please
make
sure
to
tune
in
to
that
and
also,
please
feel
free
to
reach
out
to
me
or
sasha
directly.
If
you
have
any
questions
or
thoughts,
feedback.
A
B
Yeah,
I
can
take
that
one.
I
guess
sasha
and
I
actually
worked
on
a
a
new
feature
for
this
pr
that
I
mentioned
in
the
talk,
and
I
handled
the
docs.
For
that
I
would
say
the
earlier
the
better
honestly
there
are
a
couple
of
different
phases
of
a
docs
pr
review,
so,
first
of
all,
there's
kind
of
like
the
the
technical
review.
B
So
in
our
case
sasha
and
I
were
working
with
one
other
individual,
so
I
opened
the
docs
pr
and
they
were
the
main
technical
reviewers
and
then
on
the
documentation
side.
The
docs
team
had
some
folks
that
reviewed
it,
for
you
know
the
writing
style
and
how
a
tutorial
was
written
and
that
sort
of
thing-
and
that
can
sometimes
take
some
time
to
go
through
and
for
them
just
have
time
to
get
stuff.
B
So
I'd
definitely
say
you
know
as
early
as
you
get
the
docs
prn,
that's
great
generally,
that
can
sometimes
be
difficult
to
get
in
before
you
actually
have
the
technical
implementation,
especially
if
you're
showing
a
tutorial
of
using
it,
but
but
mostly
yeah.
The
best
policy
is
as
early
in
the
cycle
as
you
can
to
go
ahead
and
get
that
pr
open,
get
early
feedback
and
iterate
on
that,
if
necessary.
A
Yeah,
thank
you
so
as
long
as
possible
yeah.
So
the
the
next
question
question
is
still
regarding
the
ducks
you
know
in
one
in
each
release
we
will
have
different
deliverables
such
as
you
know.
You
mentioned
release
and
then
docs
generally
and
then
tutorial.
Oh,
who
is
the
decision
maker
in
which
type
of
delivery
to
be
included
in
a
certain
release.
C
It's
it's
not
easy,
so
there
are
multiple
multiple
participants
involved
right.
So,
for
example,
yeah.
If
we
speak
about,
I
mean
yeah,
I
mean
we
have
dedicated
teams.
For
example,
we
can
still
refer
some
work
to
to
to
sick
dogs
in
general,
so
they
will
really
really
help
us
in
reviewing
and
also
bringing
bringing
the
stuff
together.
But
then
we
have
a
dedicated
docs
team
in
sick
release
as
well,
who
also
yeah
have
to
ensure
that
everything
works
in
the
end.
So
all
in
all
people
have
to
communicate.
C
A
A
I
I
think
it's
really
difficult
to
you,
know,
manage
and
release,
especially
when
it
runs
different
seats
and
different
teams.
It's
it's
really
a
teamwork
to
you
know
to
get
it
done,
and
it
requires
a
lot
of
handy
tools
so
that
you
can
see
what's
the
blogger
and
something
red
in
that
page,
and
you
need
to
take
care
of
that.
C
I
mean
we
mainly
stick
to
github.
We
were
so
from
a
tooling
perspective.
We
tried
out
different
other
tools
as
well,
and
kubernetes
provides
yeah
some
sort
of
own
tool
tools
like
the
ci
signal,
dashboards
and
stuff
like
that.
But
generally
we
try
to
stick
to
good
communication
on
github
issues
and
yeah
for
immediate
or
urgent
communication.
We
usually
stick
to
slack,
which
works
pretty
well.