►
From YouTube: Are We There Yet? My Experience Leading a Kubernetes Release Team - Claire Laurence, Pivotal
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Are We There Yet? My Experience Leading a Kubernetes Release Team - Claire Laurence, Pivotal
9 months, 3 releases, 3 roles, and countless new faces - here’s a story about how I went from a brand new community member to release lead in less than a year and what I discovered along the way. Over 9 months I dove into the Kubernetes release world and dealt with the challenges of shipping a release, the tough decisions that need to be made, and how to adapt our release plan to stay on target. The Kubernetes release is at the core of the community. Managing an open source release requires flexibility, transparency, and radical collaboration. As the size of the contributors grows, how does the release process adapt? How are decisions made? How do we react when plans change? How did I quickly move from a shadow to the release lead role? In this talk I plan on addressing these questions and my experience leading the 1.15 release team
https://sched.co/UaWB
A
A
So
first
off
Who
am
I
as
Stephen
just
wonderfully
introduced
me.
I
am
a
technical
program
manager
at
pivotal
software
I'm
based
out
of
our
New
York
office.
I
am
a
former
kubernetes
enhancements,
lead
release,
team
lead
and
the
current
release
team
emeritus
advisor
for
1:17
and
I
also
just
love,
coffee
and
travel
as
a
note
feel
free
to
come
up
and
talk
with
me
at
any
point
during
the
conference
or
after
on
slack
about
any
of
these
topics.
A
So
before
we
get
started,
I
just
want
to
note
that
I'm
only
going
to
be
speaking
about
my
personal
experience
as
a
release
team
member.
So
as
such,
you
can
expect
this
talk
to
be
very
focused
on
the
enhancements
and
release
lead
aspects
of
my
experience.
Since
those
were
the
two
roles.
I
was
a
lead
for.
I
cannot
easily
speak
to
the
experience
of
CI
signal
or
Doc's
I
might
touch
upon
them
briefly,
but
will
not
be
going
into
details.
A
A
So
what
is
the
kate's
release
team?
Who,
in
this
room,
is
familiar
with
the
kubernetes
release
team?
A
couple
folks
awesome
for
those
of
you
who
aren't
aware
I'm
gonna
go
into
a
little
bit
of
detail
about
what
the
'quran
ADIZ
release
team
is.
So
the
kubernetes
release
team
is
a
sub
project
of
sig
release.
Six
stands
for
special
interest
group.
It
is
responsible
for
shipping.
The
kubernetes
release
approximately
once
a
quarter.
That's
the
cadence.
We've
been
going
on
for
the
past
few
releases.
A
This
is
the
release
team
they're
charged
with
making
sure
that
we're
able
to
ship
a
quality
release
to
all
of
our
end
users
again
on
a
regular
cadence.
The
release
team
is
mostly
staffed
by
volunteers.
Some
folks
might
get
company-sponsored
time
to
work
on
it,
but
a
lot
of
people
do
this
in
their
own
free
time
and
the
time
requirement
needed
can
be
anywhere
from
maybe
an
hour
a
week
to
a
couple
hours
a
week
and
it
usually
ramps
up
as
the
release
goes
on.
So
these
are
a
lot
of
really
kind.
A
Amazing
people
who
volunteer
their
time
for
one
quarter,
maybe
two
depending
on
how
far
you
want
to
go,
could
be
more
additionally
we're
a
fully
distributed
team.
We
have
folks
in
almost
every
single
continent
that
is
inhabited
I'm
sure
if
somebody
wanted
to
set
up
a
outpost
in
Antarctica,
we'd
accept
them
as
well
like
it
is
all
over
the
world,
and
as
such,
we
have
to
do
everything
to
accommodate
people
across
all
different
time
gos
and
do
everything
fully
remote
leveraging
tools
like
slack
github,
zoom
and
posting
all
of
our
release.
A
Meetings
on
to
the
kubernetes
YouTube
channel
and
additionally,
the
release
team
has
a
shadow
program
that
is
there
to
help
mentor
new
release
team
members.
So
those
folks
can
join
and
learn
more
about
the
release,
team
processes
and
then
one
day
take
over
as
lead
or
maybe
they
shadow
for
a
release
and
realize
oh
I'm,
actually
really
interested
in
this
other
part
of
the
of
the
kubernetes
org
I'm
gonna
move
over
there
and
that's
all
totally
cool
on
the
right-hand
side
of
this
slide.
A
I
have
a
list
of
every
single
member
of
the
current
117
release.
As
you
see,
it's
a
lot
of
individuals
and
it's
a
wonderful
team
made
up
of
brilliantly
talented
people
so
now
getting
into
a
little
bit
more
of
some
personal
stories
of
my
experience
on
the
release
team.
So
my
journey
started
in
112.
Some
co-workers
were
looking
into
the
release
team
and
we're
shadowing
on
the
release
team.
At
that
time
they
had
reached
out
because
they
wanted
to
get
more
folks
involved
and
they
talked
about
how
interesting
it
was.
A
In
my
day,
job
I
work
helping
to
Shepherd
our
clue,
our
proprietary
products
releases
and
as
a
release
manager
doing
that
I
was
really
interested
to
see.
What
are
the
differences
between
the
open-source
community
versus
what
we
do
in
our
company?
Are
there
any
cool
practices?
We
can
learn
from
the
open
source
that
we
can
bring
into
our
company
to
better
improve
our
systems
and
processes.
So
I
started
joining
some
of
those
release.
A
Team
meetings
very
covertly,
mostly
quietly
just
sitting
on
a
zoom
call
and
not
saying
a
single
word,
but
just
listening
and
then
around
September
2018.
They
asked
for
volunteers
to
shadow
different
roles
for
the
113
release
and
I
volunteered
to
be
an
enhancement.
Shadow
I
thought
enhancement
sounded
really
interesting.
A
So
they
asked
me
if
I
would
be
interested
and
I
accepted
with
a
little
bit
of
hesitation,
because
I
was
a
little
bit
nervous
to
see
what
it
would
be
about
without
shadowing
it.
But
it
was
one
of
the
most
rewarding
experiences
I've
had
in
the
open-source
community
to
date,
and
then,
after
that,
release,
I
took
a
bit
of
a
break
and
then
in
117
rejoined
as
the
release
team.
A
So
I
also
want
to
mention
that
being
on
the
release,
team
was
my
first
introduction
to
the
kubernetes
community
and
I
thought
that
it
was
a
really
great
way
to
join
this
community.
It's
a
well-defined
sub
project
with
great
mentorship
opportunities,
there's
a
very
clear
path
for
growth
and
as
a
shadow,
you're
able
to
actually
have
a
hands-on
experience
and
impact
the
release.
A
Additionally,
the
release
team
touches
so
many
different
SIG's
and
working
groups
as
enhancements
shadow.
One
of
the
first
tasks
I
had
was
going
through
every
single
issue
in
kubernetes
enhancements,
which
is
where
new
enhancements
are
proposed
to
land
in
a
kubernetes
release
and
pain
issue
owners
to
see
if
they
were
interested
in
landing.
A
That
issue
in
the
next
release,
and
that
was
one
way
to
just
meet
a
ton
of
different
people
in
the
community
and
learned
a
lot
about
all
the
different
enhancements
folks
were
trying
to
land
in
the
release
and,
additionally,
almost
every
other
role
on
the
release
team.
You
get
to
work
across
a
variety
of
different
SIG's,
and
so,
even
if
afterwards,
you
find
out
that
the
release
isn't
something
you're
interested
in
you've
had
so
much
exposure
to
the
kubernetes
project
that
it's
really
easy
to
find
other
areas
you're
interested
in
contributing
to
beyond
it.
A
A
One
thing:
that's
really
fascinating
to
me.
Is
it's
a
constantly
rotating
release.
Team
I
have
up
the
113
release
team
members
and
the
117
release
members.
There
are
only
three
names
that
are
appearing
on
both
release
teams
and
that's
over
the
course
of
about
four
releases
through
the
shadow
program,
we're
able
to
mentor
and
train
people
to
take
on
these
roles.
So
no
one
needs
to
volunteer
in
perpetuity
to
ensure
that
the
kubernetes
release
is
able
to
ship
and
you
can
volunteer
for
as
much
time
as
you
relatively
need
to
be
at
a
quarter.
A
Maybe
two
you
can
really
choose
based
on
your
own
life
circumstances,
what
works
best
for
you
and
even
rotating
these
release
team
members
we're
still
able
to
ship
a
fairly
high
quality
release
to
the
community
decision-making.
So
decision-making
in
an
open-source
community
is
very
interesting
for
me
personally,
as
release
lead.
I
had
a
lot
of
autonomy
to
make
decisions,
which
was
interesting
as
a
relatively
new
member
in
the
community
with
less
than
a
year
experience.
One
particular
example
for
115
I
had
full
autonomy
mostly
to
set
the
release
schedule
for
1:15.
A
We
had
to
deal
with
koukin
Barcelona
Memorial
Day
weekend
in
the
u.s.
coupe
con
Shanghai
and
then
wanting
to
ship
before
4th
of
July.
So
that's
four
major
events
for
the
community
within
a
really
within
a
quarter
and
having
to
balance
through
all
of
that
and
figure
out.
The
best
decision
to
make
was
definitely
difficult.
Not
everybody
was
happy
with
the
release
schedule
decision
and
we
have
a
retro
where
everyone
is
allowed
to
voice
their
very
strong
opinions
and
feelings,
but
being
given
the
autonomy
to
do
what
worked
best.
A
From
my
perspective
and
taking
into
account
all
of
the
different
aspects
going
on
in
the
community
was
very,
very
interesting
to
me
again
with
such
little
experience.
But
after
making
the
decision,
everyone
in
the
community
was
very
supportive
and
adhered
to
it
and
didn't
try
to
change
things
after
the
fact.
A
A
It's
just
very
interesting
to
see
decision
making
happen
happening
in
such
a
transparent
way
when
it's
not
always
in
other
communities
or
even
within
companies
done
that
way.
And,
lastly,
the
thing
about
decision
making
that's
super
interesting.
Is
you
very
much
see
this
push
towards
community
versus
individual
needs?
Some
examples.
During
the
release
cycle,
as
we
were
approaching
code
freeze,
once
code
freeze
heads
if
an
enhancement
isn't
fully
finished
code
wise,
it
gets
bumped
out.
We
had
quite
a
few
enhancements
that
coming
up
were
at
risk.
A
If
you
still
open
PRS
and
every
single
member
on
those
cigs,
even
members
who
are
writing,
that
code
were
able
to
make
their
own
decisions
that
they
wanted
to
bump
those
release
or
those
enhancements
out.
The
next
release
versus
wanting
to
push
that
decision
on
to
me
or
my
enhancements
lead
to
make
that
which
just
shows
that
people
prioritize
the
needs
of
the
community
and
making
sure
these
releases
have
enhancements
that
are
solid.
A
Additionally,
interesting
few
things,
bloco
kubernetes
release,
and
I
don't
know
if
this
is
because
we
have
such
an
excellent
team
of
people
who
are
really
good
at
risk
management
and
they're
constantly
on
top
of
checking
everything
from
CI
signal
tests.
Making
sure
Doc's
are
in
place,
release
notes
already.
But
additionally,
as
I
just
mentioned,
things
are
often
getting
D,
scoped
and
bumped,
because
again
folks
want
to
ensure
that
we
have
a
quality
release
that
is
able
to
ship
on
time.
A
People
are
willing
to
Dscoop
and
move
issues
out
until
the
next
release
to
make
sure
we're
able
to
ship
once
a
quarter.
Just
as
a
note,
115
was
delayed
by
two
days
and
the
thing
that
blocked
it
with
scalability
tests,
and
we
wanted
to
make
sure
that
we
understood
stood
why
those
scalability
tests
were
failing.
A
So,
even
though
it
would
have
been
great
for
me
to
have
been
able
to
ship
to
that
on
Monday,
because
I
had
a
team
off-site
that
whole
week,
I
decided
that
it
was
better
to
slightly
inconvenience
myself
in
missed
parts
of
my
team
off
site
to
make
sure
that
the
release
was
able
to
ship.
Knowing
what
the
scalability
test
failures
were.
A
Another
thing
I
find
really
interesting.
The
release
team
is
constantly
iterating
they're
open
to
new
changes
and
when
folks
come
in
and
see
something
that
could
be
improved
upon,
we
go
with
it.
So
an
example:
the
shadow
program,
as
I've
mentioned
when
I
joined
in
113
becoming
a
shadow.
All
you
had
to
do
was
comment
on
a
github
issue,
saying
you
were
interested
and
then
someone
would
slot
you
right
into
place,
you'd
be
on
the
release
team,
and
that
worked.
A
We
had
plenty
of
shadows,
but
not
every
single
role
was
fully
filled
in
terms
of
the
amount
of
shadows
they
could
take
on
and
not
every
single
shadow
fully
understood
what
they
were
signing
up
for,
necessarily
so
starting
in
114.
We
introduced
a
survey,
and
this
survey
had
lots
more
information
links
to
all
of
the
different
documentation,
'he's
about
release,
rules
that
are
in
github
and
it
allowed
the
release
team
to
better
understand
the
folks
who
are
volunteering
to
join.
A
It's
a
very
short
survey
asks
a
few
questions
about
their
interest
amount
of
time,
and
we
also
noticed
after
interviewing
the
survey
or
introducing
the
survey,
we
had
a
huge
spike
in
interest
and
that
spike
in
interest
has
continued
over
the
next
few
iterations
in
this
one.
15
1,
it's
very
small,
but
there
were
49
responses
of
interest
to
being
a
shadow
for
that
release
and
in
117
we
had
over
70
folks
sign
up
with
interest
and
we'll
have
one
come
out
again
for
118.
A
Next
up
for
iterations
as
roles
need
to
be
updated
and
revised.
We
do
that
as
a
group,
so
some
examples
from
112
to
113
the
features
lead
became
the
enhancements
lead.
The
role
stayed
mostly
identical.
It
was
a
simple
change
in
name,
and
that
was
because
the
word
enhancements
was
more
inclusive
of
the
types
of
issues
being
opened
from
114
to
115.
We
added
the
emeritus
adviser,
as
we
were
starting
to
get
more
shadows
interested.
A
We
noticed
a
need
to
have
a
role
introduced
specifically
for
mentoring
and
making
sure
that
shadows
were
getting
the
most
out
of
the
release
that
they
possibly
could.
So
we
added
an
additional
role
to
help
with
that
from
115
to
116.
The
test
in
for
ole
was
dissolved,
mostly
due
to
the
amazing
work
of
many
different
tests
and
feroro
leads
past
and
Catherine
Barry,
who
were
able
to
automate
the
roll
out
of
existence
and
anything
that
didn't
need
to
be
automated.
A
They
were
able
to
roll
over
to
the
branch
management
team,
showing
that
we're
constantly
able
to
iterate
and
improve
upon
our
processes
and
automate
things
that
can
be
automated
introducing
cups.
This
was
an
interesting
one
for
me,
so
caps
became
mandatory
for
enhancements.
Free
starting
in
1:14.
Cap
stands
for
kubernetes
enhancement
proposal
for
any
enhancement
that
wants
to
land
in
a
release.
A
They
need
to
have
a
cup
in
a
merged,
an
implementable
state
by
the
enhancement,
freeze-freeze
deadline,
and
we
introduced
this
requirement
in
1:14,
and
what
we
learned
is
that
rolling
out
new
policies
for
a
massive
org
isn't
always
smooth,
but
sometimes
you
do
really
just
need
to
rip
the
band-aid
off
and
do
it
when
we
introduce
this
policy
that
cups
were
mandatory.
A
lot
of
enhancements
were
not
did
not
have
their
cups
ready
and
in
an
implementable
state
on
enhancements
breeze,
and
we
had
to
be
a
little
bit
flexible
with
them.
A
Because
again
we
didn't
want
to
ship
a
release
that
had
no
enhancements
because
folks
didn't
have
enough
time
to
get
their
enhancements
ready,
but
starting
1:15.
We
had
fewer
and
fewer
that
needed
to
submit
exception,
requests
at
that
enhancements
freeze.
So
it
was
a
little
bit
painful
in
1:14,
but
now
looking
towards
117.
Everyone
knows
they
need
a
cup
for
enhancements,
freeze
and
it's
a
much
smoother
process
and,
as
a
result,
the
release
has
much
more
information
about
the
enhancements
landing
because
we
have
these
cups
documented
up
front.
A
It's
distributed
across
many
different
companies
and
even
folks
who
aren't
even
in
companies,
are
in
college
or
are
just
working
independently
community
over
product
or
company,
as
I
mentioned
folks
really
put
the
community
first
when
it
comes
to
the
release
over
any
of
the
needs
of
a
product
that
they're
really
passionate
about
or
the
company
that
they
work
for.
You
see
this
as
they
are
willing
to
push
enhancements
out
to
later
releases
making
sacrifices
when
needed
in
terms
of
maybe
working
late
to
help
folks
out
in
the
community.
A
You
see
this
a
lot
and
it's
really
really
exciting
to
see
automation
over
process
as
I
mentioned
automating.
The
testing
for
all
out
of
existence
is
a
big
part
of
that,
but,
additionally,
within
the
release
team,
there
are
plenty
of
places
where
folks
are
looking
to
automate
and
make
it
more
efficient.
A
A
Again,
the
team
is
constantly
evolving
constantly
looking
for
ways
to
improve,
and
if
you
want
to
learn
more
all
of
the
documentation
is
available
on
github
in
the
cig
release
repo.
We
have
a
slack
channel
highly,
encourage
everyone
to
join
if
you're
interested
in
joining
the
kate's
release
team
for
118
again
keep
an
eye
on
slack
for
a
survey
link
because
we
would
love
to
have
all
of
you
volunteer
to
be
shadows
and
yeah
I'll
give
some
time
for
questions.
So
if.
A
A
So
we
have
four
QA
work:
every
single
sig
is
responsible
for
setting
their
own
testing
up
and
we
have
Cabrini's
test
grid
and
in
addition
to
that,
their
cig
testing
on
the
release
team.
Specifically,
we
have
the
CI
signal
team,
so
they
have
a
lead
in
shadows
and
they
are
responsible
for
monitoring
the
tests
and
working
with
SIG's
to
resolve
any
issues
or
any
flakes
or
failures
on
the
tests.
B
Might
know
so
the
way
we
do
testing
is
kubernetes
uses
prowl,
which
is
a
homegrown
CI
CD
solution.
We
have
essentially
templates
that
you
can
read
it's
all
llamo
based
format,
which
is
very
close
to
what
Akoo
per
Nettie's
manifest
looks
like
anyway,
and
we
have
annotations
that
you
can
layer
on
top
of
those
job
runs.
So
things
like
making
sure
your
test
is
flagged
in
a
certain
dashboard,
making
sure
that
certain
people
are
alerted
making
sure
that
the
alerts
only
happen
after
a
certain
amount
of
test
failures.
B
D
A
A
So
I
don't
know
specifically
about
the
different
sub
projects
in
SiC
release
itself,
but
for
the
different
roles
on
the
release
team
being
a
shadow
is
usually
the
least
amount
of
time
commitment
compared
to
the
lead
rule
and
it
does
depend
based
on
the
lead
and
different
roles.
Have
different
timing
needs
at
different
points
in
the
release
so
enhancements?
Usually
you
have
the
time
commitment
up
front,
because
once
code
freeze
sets
there's
not
a
lot
for
enhancements
to
do.
A
Vs
coms
usually
is
very
heavy
towards
the
end,
because
that's
when
you're
setting
up
press
releases
and
blogs
I
would
say
across
all
of
the
different
roles,
it's
hard
to
generalize
a
time
commitment,
usually
at
a
minimum.
It's
a
one-hour
meeting
a
week
for
the
release
team,
some
roles
you
might
need
to
do
closer
to
4
to
5
hours
a
week
in
the
documentation
we
have
on
github
I.
Think
a
majority
of
them
try
to
outline
what
goes
on
week
by
week,
and
you
can
also
again
always
hop
into
slack.
A
B
C
A
Generally,
we
will
punt,
you
will
punch
the
enhancement
itself
to
the
next
release
if
there
are
any
issues
with
it
and
we
try
to
ship
the
release
on
time.
The
idea
being
that,
if
you
miss,
if
you
miss
117
118,
is
gonna
come
a
quarter
later.
You
don't
really
need
to
wait
that
long,
because
we'd
rather
have
features
that
are
fully
tested,
fully
ready
to
land
and
additionally,
as
enhancements
are
landing.
They
go
through
a
process,
so
you
don't
usually
just
submit
an
enhancement,
and
it
goes
straight
to
stable.
That
is
very
rare.