►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Release SIG - Josh Berkus, Red Hat, Chuck Ha, Heptio, Doug MacEachern, VMware, & Caleb Miles, Google
Supporting all qualified Kubernetes contributors in the Kubernetes release process, while continuing to run the release train, has been a huge effort. Increasing community stewardship of Kubernetes test and release infrastructure has been a journey which two SIG Release contributors from outside of Google, and one from The Collective, will discuss. Join us on an in depth exploration to the tooling and processes of SIG Release.You’ve heard a lot in other sessions and events about how the release cycle is managed, but not much about how the actual releases get built. And, more importantly, what this means for contributors who build Kubernetes tools, plugins, and components. Join members of SIG-Release as we discuss how the build and release actually works, and what things we’d like to change. This deep dive will describe the state of bazel builds, the release repository and the tooling around release as well as the internal-to-google workings. A member of the Kubeadm team will describe what this process means for their builds, and the Google team will explain the internal tooling they have around builds and releases. Finally, the 1.12 branch manager will talk about the Release Team’s efforts to de-Googleify many of the build and release steps.
To learn more: https://sched.co/GrdX
A
I
am
Josh
burkas
cyclical
for
release.
Team
member
former
release
lead.
We
have
here
our
panel
of
people
who
all
work
with
the
build
and
release
tools,
which
is
what
the
steep
dive
is
going
to
be
about,
including
Chuck
who's,
going
to
talk
about
all
of
the
tools.
I
Caleb
who's
responsible
for
a
bunch
of
the
tools
originally
yeah,
the
and
I
Doug
who's
been
revamping
the
branch
manager
process
and
so
I'm
going
to
talk
about
all
of
that
about
how
the
actual
bits
that
make
up
kubernetes
get
shipped
out
the
door.
A
So
here's
our
list
of
topics
when
I
talk
about
suiting
release
and
the
release
tools.
There's
we
tore
the
release
tools
is
going
to
be
sort
of
a
history
of
how
we
got
the
tools
that
we
have
now
talk
about,
how
the
the
code
you
know
binary
release,
process
works
and
then
what
the
future
is
going
to
be
into
some
Q&A.
A
B
As
you
may
have
seen
from
my
bio,
I
am
not
actually
part
of
sig
release,
but
I
will
not
be
talking
about
the
sig
release
process.
I'm
just
gonna
be
talking
about
the
release
tooling
and
before
I
get
too
far
into
that.
I
wanted
to
set
a
little
bit
of
context.
So
it's
important
to
know
what
GCB
is
and
I
think
this
image
sums
up
what
GCB
is
quite
nicely
a
ship
branded
of
kubernetes
and
packages
on
it.
Now
it's
like
a
managed
to
Jenkins
server
or
a
generic
build
job
runner.
C
B
Cool,
so
we
should
define
what
a
release
is
and
the
way
I
think
about
it.
Release
is
sort
of
five
main
components,
and
the
first
component
is
maybe
the
most
obvious
are
the
binaries,
the
kubernetes
binaries,
the
cat
distributed
building
the
binaries
from
a
specific
commit
and
putting
them
in
a
place,
a
well-known
place
that
people
can
find
them.
Secondly,
are
the
container
images
that
contain
those
same
binaries
that
are
also
pushed
to
a
registry
so
that
people
can
find
those
and
use
them
and
consume
them?
B
B
B
B
That's
not
quite
right,
because
we
don't
actually
want
to
build
anything
on
our
laptop
and
depend
on
hardware
that
one
person
has
so,
of
course,
we
want
to
actually
do
that
all
in
the
cloud-
and
this
is
where
Google
Cloud
build
comes
in,
because
that's
gonna
be
the
platform
to
build
these
things,
push
them
to
the
correct
place,
and
actually
we
have
another
tool.
The
second
tool
I
want
to
talk
about.
Gcb
manager
actually
wraps
the
entirety
of
Onaga,
so
you
don't
actually
have
to
run
an
AA
go
by
itself.
B
You
run
GCB
manager
that
will
generate
the
correct
commands
for
you
and
then
also
give
you
a
nice
interface
to
GCB
it'll.
Let
you
list
recently
run
jobs
as
you
can
see
the
status
and
you
can
tail
the
logs
of
jobs
in
case
you
need
to
debug
any
issues
that
may
arise
and
the
last
thing
linux
packaging.
Actually
the
build
process
for
that
depends
on
kubernetes
being
released.
B
So
it's
a
little
bit
of
a
chicken
and
egg
thing,
or
rather
a
dependency
graph,
but
you
have
to
first
release
kubernetes
and
then
you
can
build
the
linux
packaging
because
it
actually
goes
to
those
well-known
locations
where
the
binaries
live
to
download
and
package
them
up,
and
actually
this
is
used
quite
a
lot.
Mystic
cluster
lifecycle.
This
is
sort
of
our
desired
path
for
getting
a
best
practices.
B
Kubernetes
cluster
installed
and
the
packages
are
Kubb
ATM,
couplet,
coop
control
and
a
couple
of
dependencies,
kubernetes,
C&I
and
cry
control,
which
is
just
an
interface
to
a
container
run
time,
instead
of,
if
you
don't
wanna
use
docker,
but
those
are
the
tools
that
we
have
available
in
kubernetes
release
and
for
the
release
process.
I
think
Doug
here
is
gonna
talk
about.
C
Worry
hey
Kayla
miles
some
Google
program
manager
at
Google,
so
the
history
I
think
sig
release
and
the
release
process
is
one
of
the
clearest
examples
of
the
fact
that
we,
as
a
community,
we're
really
standing
on
the
shoulders
of
giants.
Chuck
got
into
a
little
bit,
but
at
least
for
me
Anagha
really
is
it's
a
checklist
right,
so
a
thousand
lines
of
bash.
It
is
among
the
best
bash
I've
ever
seen.
It's
like
well
factored
it's
broken
into
multiple.
C
What
you
would
consider
modules,
but
it
is
a
checklist
of
steps
that
you
would
have
to
go
to
you
know.
Checklists
have
resulted
in
many
industries,
medicine
and
aviation
being
the
primary
examples
becoming
much
safer
over
time
and
reducing
kind
of
the
ability
for
humans
to
miss
a
step
in
a
complex
sequence
is
really
helpful.
Nago
automates.
C
Most
of
that
and
also
tells
you
where
it
is
over
the
course
of
the
years
people
have
added
the
ability
for
and
not
go
to
be
reentrant,
and
that's
that's
also
great,
but
you
know
looking
back
to
where
we
started.
So
this
comes
directly
from
the
release
instructions
for
prior
to
one
four.
So
if
you
notice
here
it
says,
you
know,
releases
are
cut
directly
from
master.
You
can
imagine
some
challenges
with
doing
that,
especially
for
a
project
we
have
over
a
thousand
committers
trying
to
push
changes.
C
C
Kampf
move
changes
from
master
to
the
released
branch,
as
we
have
more
confidence
in
the
stability
of
master,
so
it
really
gives
the
release
team
a
lot
more
tools
to
be
able
to
manage
and
release
safely
yeah.
So
you
know
can
be
this
it's
fun
or
can
go
back
to
that.
You
know
to
get
history
and
sig
release
repository
community
sig
release
a
few
yeah.
C
C
We
have
done
lots
and
lots
of
work
as
a
community
as
Google
specifically
to
open
up
these
roles
to
other
people
in
the
community
to
really
make
it
a
community
driven
process.
So
this
is
really
all
forward.
You
know
historical,
your
historical
edification.
You
can
read
the
slide
on
your
own,
so
much
to
say
other
than
it
used
to
be.
Google
only
show
it
no
longer
is,
and
just
thanks
to
the
incredible
work
done
by
our
engineering
productivity
team
at
Google.
So
thank
you
to
the
entire
team.
D
D
Although
it's
you
know,
subject
to
some
change
and
once
the
release
is
actually
go
out,
sending
the
notification
to
the
communities
announced
and
the
developer
group-
and
that's
actually
I'm
sure
most
of
you
probably
seen
that
it's
essentially
the
generated
changelog
for
the
current
release,
the
other
responsibilities.
So
once
once,
we
actually
create
the
release
branch.
So
for
the
Alpha
releases,
it's
it's
all
of
a
master.
D
Once
the
release
branch
is
created,
we're
doing
daily
syncs
from
the
master
branch
to
the
release
branch
and
the
tool
is
called
branch
FF,
and
that
happens
before
code
freeze
and
then
once
once
code
freeze
actually
happens
then
or
after
go
freeze
is
lifted.
There's
no
more
fast-forwarding
master
is
kind
of
wide
open
to
all
the
stuff.
That's
been
waiting
to
come
in
for
the
next
release
and
only
cherry-picks
are
done
to
the
release
branch
and
then
there's
other
other
aspects
to
the
role.
D
D
For
my
experience,
stepping
into
the
role
you
know
before
I
spoke
with
Tim
I
had
no
experience
with
the
the
kubernetes
release
or
cig
release,
pretty
new
to
kubernetes
in
general,
and
you
know,
Tim
Tim
had
told
me
that
I
would
be
the
the
first
non
Google
branch
manager.
It
had
been
Caleb
for
a
number
of
years
and
actually
went
back
through
the
because
the
github
sig
release
repo
has
to
release
teams
for
all
the
releases
and
I
went
back
and
kind
of
verified
that
and
has
a
side
note.
I
son
released
one
six.
D
D
If
you
include
the
the
Cates
robot,
but
going
into
one
twelve,
there
was
no
there's
no
handbook
for
the
branch
manager,
and
so
what
we've
gotten
from
from
Caleb
was
this
what's
actually
still
described
in
the
repo
as
old
old
Anna
go
doc,
you're,
not
just
old,
but
old
old,
and
you
know
it's
a
bit
outdated.
You
can
still
look
at
today,
there's
a
ton
of
stuff
with
strike
throughs.
There's
internal
references
I
like
this
one
to
cube,
Ken's,
I'm,
pretty
sure,
that's
Jenkins,
something
or
other,
but
there's
some
kind
of
neat
history.
D
D
And
and
I
ended
up
and
you
know,
I
didn't
want
to
I
didn't
want
to
bug
you
Caleb,
because
you
might
well
yeah,
so
he
was
yeah.
He
wasn't
really
that
reachable,
but
then,
finally,
what
I
found
was
the
release
repo
with
all
the
tooling
and
I,
don't
get
into
that
a
little
bit,
but
you
know
kind
of
when
it
came
time
to
cut
the
release.
This
is
this
is
what
I
was
feeling
like
and
I
know
it's
kind
of
a
tired
mean,
but
there's
no
better
way
to
describe
it.
D
I
just
didn't
know
what
I
was
doing.
Tim
and
I
did
a
zoom,
and
you
know
where
were
the
shadow
roll,
so
I
said
well
Tim?
Should
we
invite
the
shadows
and
Tim
was
like,
let's
figure,
all
we're
doing
first,
but
so
so
kind
of
the
the
the
light
for
me
was.
Was
the
K
release
repo
and
you
know
the
other
guys
mentioned?
There's
a
lot
of
Bosch
code
and
they're
actually
ran
the
line
of
code
program
out.
D
There's
four
4
K
lines
in
the
repo
with
two
and
a
half
K
R
comments,
and
every
one
of
these
tools.
Branch,
FF,
GCB
manager,
all
heavy
man,
help
option
that
spits
out
manual.
So
that's
where
it
was
kind
of
like
in
the
you
know
in
the
and
the
movies,
all
kind
with
the
light,
and
that's
really
kind
of
had
everything
that
we
needed
to
get
started.
And
you
know
we
still
still
weren't
sure
exactly
what
was
going
to
happen.
How
I
was
going
to
go.
E
D
D
Really
talked
when,
on
a
now
go
in
the
release
meetings
via
GCB
manager,
so
I
wanted
to
point
out
these
tax,
because
this
is
actually
a
question
that
also
came
up
even
towards
the
end
of
the
113
release
from
people
on
a
release
team.
But
when
you
create
the
official
release,
only
1.11
official
release
was
cut.
The
v1
12
0,
alpha
dot,
zero
tag
is
created
and
it's
really
nothing
more
than
tag.
There's
no
release
associated
with
that,
but
it's
just
kind
of
marking
that
point
in
time.
D
And
then
you
know,
if
you
look
at
one
of
the
schedules,
we
usually
plan
at
least
two
I
think
there
were
maybe
three
alphas
and
in
the
past
release
and
those
are
all
kind
of
incremented
automatically
by
GCB
manager.
There's
some
metadata
that
gets
stored
and
in
google
buckets
somewhere.
That
is
kind
of
the
last
release.
But
it's
one
of
those
areas
where
there's
not
really
any
room
for
for
human
error.
D
It's
automatically
incremented,
which
is
which
is
nice
and
then,
when
it
comes
time
for
for
beta,
that's
when
the
release
branch
is
actually
created,
so
it'll
be
called
release.
Whatever
the
version
number
is
so
release
released
112
in
this
case,
and
then,
when
you
do
that
you
get
another
automatic
tag
beta
zero.
Then
we
have
a
couple
scheduled
beta
releases,
1
through
n
and
then
there's
release
candidates
and
then
finally,
there's
the
official
release
and
then,
after
the
official
release
goes
out.
D
This
is
actually
one
of
the
questions
that
came
up
was
once
113
was
tagged.
Someone
noticed
there
was
a
113
dot.
1
beta
0
tag
and
I
haven't
really
seen
any
beta
releases
of
patch
versions,
but
the
same
mechanism
is
there
if
if
it
needs
to
be
used,
but
otherwise
you
just
see
the
major
version,
dot
1
through
n
number
of
patch
releases.
C
C
So
it's
the
the
the
you
are,
as
the
branch
manager
just
hiding
more
or
less
to
bless
a
Shaw,
and
so
the
Shah
less
becomes
that
you
know
that
beta
or
the
alpha
or
the
release
candidate
and
more
changes
will
come
in
presume.
Like
generally,
you
will
either
depending
where
you
are
in
the
really
cycle.
You
either
will
just
forward
fast
forward
those
from
master
or
you
will
cherry-pick
those
on
from
master.
C
There
there
is,
there
is
a
release
x,
dot
y
branch
that
is
branched
off
of
master
and
that's
the
branch
we
are
going
to
be
stabilizing
through
the
course
of
the
release
cycle
that
gets
created.
You
can
decide.
You
know,
there's
a
little
bit
of
flexibility
when
you
want
to
create
it,
but
you
either
can
create
it.
You
can
either
do
everything
cut
everything
off
of
the
release.
D
D
Yeah
I'm
gonna
tell
our
story
coming
up
here,
so
so
yeah
so
after
after
112-
and
you
know,
Tim
and
I
had
done
these
kind
of
builds
through
through
zoom,
or
at
least
we
would
start
that
way
and
I
should
mention
the
builds.
Take
you
know
hour
and
a
half
two
hours
to
complete.
So
we
weren't
hanging
out
for
that
long
as
much
as
I,
like
you
Tim,
but
just
kind
of
pasting.
You
know
the
kind
of
status
of
the
logs
and
Tim
created
the
the
branch
manager
handbook
from
those
sessions.
D
So
it's
kind
of
really
well
documented
now
and
then,
through
the
113
release
process
with
with
Ashley
ting,
we
actually
had
a
good
process
with
the
shadows
and
and
Honus
I.
Don't
know
finances
here,
but
he's
the
next
branch
manager
and
he's
he's
all
all
ready
to
go
so
hopefully,
I'll
have
some
pictures
other
than
the
I.
Don't
know
what
I'm
doing
dog
next
year
we
talked
about
doing
a
demo.
But
again
you
know
if
we're
actually
doing
a
full
build,
takes
a
really
long
time.
D
So
I
kind
of
put
a
snapshot
in
here
of
a
couple
of
the
commands,
so
GCB
manager
stage,
you
give
it
the
if
this
was
an
alpha
2.
Actually
you
say
this
so
the
first
arguments,
the
branch
so
for
an
alpha
that
would
be
master
but
everything
else.
It's
the
release
branch
and
then
there's
a
couple
of
flags
in
this
case
official.
So
this
is
actually
the
log
from
doing
the
the
113
official
build.
D
There
is
a
mock
mode
for
doing,
builds
and
that's
the
default,
and
if
you
want
to
do
an
actual
build,
then
you
have
to
pass
an
omok
flag
and
then
build
ahead.
I'll
actually
explain
that
a
little
bit
later.
So
once
you
actually
start
this
thing,
it's
running
a
job
on
GC
B
and
then
the
output
is
going
to
be
the
next
command
that
you
see
here.
D
D
So
this
is
where
early
on
I
couldn't
have
the
right
permissions
for
some
things,
and
it
would
fail
at
that
point
and
there's
a
couple
of
checks:
GCB
related
checks,
dis
space
and
then
it's
actually
tagging
a
tagging
the
tree
locally,
and
then
it
kicks
off.
What
takes
probably
the
longest
is
actually
doing
the
cross.
D
Compiling
of
of
all
the
components
and
I
included
my
favorite
architecture,
because
I
love
mainframes,
the
SS
390,
what
its
you're
gonna
see
this
for
for
every
architecture
and
then
the
release
notes
are
generated,
they're
actually
stored
in
one
of
the
GCB
buckets
and,
and
those
are
used
for
a
couple.
Things
want
to
mention
the
actually
the
notification
and
then
eventually
those
actually
get
posted
back
into
github
and
then
staging
the
source
tree.
D
D
The
GCB
permissions
are
a
little
bit
limited,
and
this
is
where
Tim's
and
I
had
had
worked
through
an
issue
where
Tim's
and
I
forget,
who
else
you
were
working
with,
but
had
modified
the
there's
a
cloud
there's
an
image
for
actually
building
all
the
stuff.
There's
a
docker
file
in
the
release,
build
and
there's
a
new
docker
feature
to
publish,
manifests
so
so
DIMMs
have
kind
of
made
changes
locally,
but
he
didn't
have
permissions
to
actually
run
through
this
whole
tool
chain.
F
D
So
if
somebody
happened
to
want
to
do
a
patronage
patch
build
while
we're
in
the
middle
of
this,
then
that
that's
going
to
be
an
issue
another
one
that
I
ran
into
on
my
first
day
on
the
job
was
with
sending
the
release.
So
the
release
tools,
I
have
a
reference
to
you
know
the
release,
notify
uses
user
espen
sin
may
Oh,
which
of
course,
isn't
send
mail.
It's
postfix
does.
E
D
Anyway,
I
you
know
so
I
ought
to
send
the
email
I
realize
it
bounced.
I
hadn't
touched
an
MTA
in
years.
I
did
have
a
like
a
local
copy
of
the
email,
which
was
was
raw,
HTML
and
I'm
kind
of
you
know
it's
in
the
evening.
I
had
my
kids
kind
of
screaming
at
me
at
home
and
I
just
pasted
the
email
into
Gmail
and
sent
it,
and
then
it's
escapes
all
the
HTML.
It
looked
horrible,
and
but
this
is
where
you
know.
D
Right
so
yeah
so
there's
other
ways
to
do
it,
but
that
was
also
kind
of
my
I
mean
I
had
a
positive
experience
with
the
community
all
along,
but
that
was
one
where
I'm
like
man,
I,
can't
even
send
an
email
everybody's
watching,
but
everybody
was
actually
really
really
helpful
and
you
know
I
got
lots
of
emojis
I
said
I
guess
release
a
release.
Channel
gets
the
most,
so
you
know
it's
kind
of
I
feel
good.
All
around
the
other
issue,
which
I
kind
of
was
another
thing
with
working
with
dams.
C
So
that's
actually,
when
things
go
wrong,
so
a
few
releases
back
when
we
were
still
developing
GCB
manager,
I
actually
managed
to
break
it
and
have
to
do
all
of
the
last
steps
myself.
So
that's
why
the
branch
manager
retains
the
the
high
level
of
permissions
in
case
the
tooling
itself
is
unable
to
make
the
release
the
tooling
can
do
all
of
those
it
actually
has
those
permissions
as
well
and
one
day.
Hopefully,
no
one
will
need
those
anymore.
Okay,.
D
Awesome
well
in
the
meantime,
you
know
this
image.
Here
is
what
I
would
see
if
I
went
to
review
a
kubernetes
PR,
you
know
that
doesn't
have
an
LG
TM,
yet
there's
jobs
failing,
but
still
as
an
administrator,
you
may
merge
this
pull
request.
So
anytime,
I'm
reviewing
I'm
like
hit
the
mouse
by
accident
and
the
other
one
I
mentioned
earlier.
D
C
Looking
for
the
4-3
green
build
in
test
grade,
and
if
you,
if
you
know
Aaron
from
from
sick
beard,
he
will
give
you
lots
of
color
about
why
that
process
generally
does
not
succeed.
So
it
should
work
in
theory
it
just
we.
It
is
very
difficult
to
get
all
enough.
Green
runs
for
every
all
green
runs
for
all
the
tests.
That
would
not
only
be
blocking
without
having
a
human
having
to
divine
the
actual
state.
It's
there's
lots
of
challenges
there,
but
yeah.
That's
that's
what
you
see
there.
That's
the
fine
green
build.
C
C
Mean
this
is
really
just
reckless
speculation.
I
mean
since
I,
before
joining
Google
I
worked
at
core
OS
on
the
same
stuff,
since
we're
in
Google
I've
continued
on
the
mission
to
help
sig
release
in
the
release
team,
be
a
fully
community
managed
effort,
and
so
that
just
could
be
more
work
there.
So
it's
helping
to
support
the
building
of
Deb's
and
rpms
by
anyone.
There's
gonna
be
a
lot
of
work
there.
F
C
So
I
mean
I,
take
my
experience
as
a
software
developer
from
pivotal
with
me,
building
Cloud
Foundry
a
similarly
complex
project,
yeah
yep
I
was
Bosch
engineer
for
like
three
years.
It
was
a
lot
of
work,
a
lot
of
investments
in
continuous
delivery,
open
sourcing
that
kind
of
work.
We
were
doing
that
similar
kind
of
work
in
the
open,
like
we
don't
have
a
concourse
for
the
chimeric
project.
Prowl
is
a
great
tool
and
it
it.
C
But
it
is
not
that
yet,
and
so
you
know
yes,
I
do
imagine
we're
gonna
have
you
know
applying
more
pipelining,
more
continuous
delivery
to
the
release,
and
so
you
know,
I
I
have
always
believed
that
you
should
be
able
to
go.
Find
some
screen
somewhere
look
left
to
right
and,
as
you
moving
left
to
right,
do
you
see
more
and
more
confidence
as
you
as
you
sweeping
that's
the
place
I'd
like
for
us
to
get
we're
not
quite
there
yet
I
mean
it's
kind
of
it's
gonna,
be
a
big
lift.
G
So
one
thing
that
we
need
help:
one
thing
we
need
help
is:
reproducible
builds.
If
anybody
else
does
the
exact
same
bill
using
the
exact
sha
that
we
use,
you
should
get
the
exact
same
images
and
binaries.
So
we
are
in
the
process
of
doing
that.
So
you
should
be
able
to
build
this
within
your
own
organization
and
yes,
AK.
H
G
G
So
the
first
of
all,
we
need
community
accessible
repositories
for
GC,
r
and
GCS,
so
we
can
push
binaries
from
all
the
projects,
not
just
the
main
KK
and
to
the
GCS.
You
know
state
staging
areas
or
whatever
right,
binaries
release
binaries,
and
then
we
have
to
work
on
GC
r,
so
people
doing
a
cloud
provider
or
cluster
api
should
be
able
to
push
by
themselves.
You
know
without
having
to
need
Google
credentials
or
find
a
release
manager.
G
So
we
might
end
up
even
automating
that
by
saying
that
this
is
a
sha,
this
is
the
version
of
the
container
that
needs
to
be
published
and
pushed.
So
we'll
have
to
end
up
at
that
point,
and
we
need
to
make
sure
that
we
can
do
the
daka
trust
stuff
and
whatever
else
is
coming
down
the
pike
for
CRI.
We
don't
know
what
that's
going
to
look
like,
so
we'll
have
to
accommodate
that.
G
It's
the
kids,
infra
k8s,
infra
team.
You
should
be
able
to
see
it
on
slack,
so
join
that
place,
and
then
we
have
a
meeting
every
two
weeks
and
we
we
are
working
on
GC
s.
Gc
are
also
clusters
for
published,
publish
bot
and
you
know
DNS
kind
of
stuff.
So
there's
plenty
of
work.
That
means
we
do
that.