►
From YouTube: SIG Release - Deep Dive: Release Engineering Subproject - Stephen Augustus, VMware & Hannes Hörl
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
SIG Release - Deep Dive: Release Engineering Subproject - Stephen Augustus, VMware & Hannes Hörl, Pivotal
The Kubernetes SIG Release is chartered with producing project releases on a reliable schedule. A key component of this is release automation and build tooling. The process/procedures and tools used to create and maintain Kubernetes release artifacts are the responsibility of SIG Release’s recently created Release Engineering subproject. In the early days of the project this code was heavily dependent on a Google presence and is one of (if not _the_) final remnants of that historical connection to the project’s origination. As SIG Release work to shift these last pieces into a community led process and community maintained code, we need your assistance. This deep dive will present an overview of the major moving pieces in the release pipeline, detail the code/process enhancements and improvements currently underway, and share opportunities where you can join in the Release Engineering subproject to assist in making its code more robust and community sustainable.
https://sched.co/Uaff
A
Hello,
everybody
welcome
to
our
talk
about
our
deep
dive
into
the
release:
engineering
sub
project
of
the
cig
release
hi.
My
name
is
Hannes
I'm,
a
software
engineer
at
pivotal
I'm,
since
a
little
while
now
involved
in
sick
release,
the
release
team
in
parts
and
the
release
engineering
they're
a
bit
of
a
little
bit
of
stuff
in
cig
testing
and
elsewhere.
B
I'm
stephen
augustus
I
am
a
I,
am
the
lead
for
cloud
native
advocacy
and
tooling
at
VMware.
I
am
also
one
of
the
sig
release.
Chairs
have
worked
on
the
release
team
for
several
cycles.
I've
built
out
several
of
the
release
teams,
as
well
as
the
release
managers
group
that
we're
going
to
be
talking
about
in
more
detail
today.
B
A
A
A
One
of
the
things
we
need
to
focus
on
as
a
community
is
to
make
it
possible
for
non-googlers
to
do
all
the
things
our
Lee
our
release
and
to
end
this
is
currently
not
yet
the
case,
but
we
are
working
on
it.
You
might
know
historically,
kubernetes
is,
is
or
comes
out
of
Google
and
a
lot
of
stuff
a
lot
of
infrastructure.
A
lot
of
processes
are
still
basically
inside
Google's,
firewalls
and
yeah.
A
A
We
need
to
make
it
or
we
want
to
make
it
safer
and
more
maintainable
and
I'm,
not
only
talking
about
maintainable
code
or
something
like
that,
but
we
want
to
iterate
on
on
our
processes
and
generally
on
all
the
things
involved.
How
we
do
a
release
yeah,
we
we
need
to
figure
out
how
we
want
to
build
and
then
publish
artifacts
like
our
turbos,
our
images,
our
packages
in
or
on
community
owned
infrastructure
and
yeah.
A
We
also
as
part
of
the
release
process,
might
want
to
introduce
new
features,
for
example,
signing
our
artifact,
so
that
people
can
actually
check
if
the
artifact
staple
town
are
the
ones
we
we
released
and
one
of
the
other
things
we
care
quite
a
bit
about
is
make
it
easier
to
bring
on
new
people
to
the
release
team
to
any
any
part
of
the
of
the
release
process.
Quite
some
some
work
also
went
into
there
and
will
continue
go
into
there.
A
One
of
the
most
important
things
is
quite
a
lot
of
new
people
came
in
to
help
out
and
in
all
things,
release
engineering.
This
is
great.
We
have
more
people
who
can
do
more
work,
but
also
we
have
more
people
with
different
views,
expertise
and
ideas,
and
that
should
or
will
improve
our
things,
our
release
process
as
a
whole.
A
We
did
a
little
bit.
We
did
a
couple
of
tweaks
for
the
release
process
to
capture
what
is
actually
going
on.
You
might
not
really
see
it
that
well,
but
we
are
now
tracking
tracking
how
long
different
stages
of
the
release
process
took.
How
was
the
state
of
the
world
when
we
started
cutting
the
the
thing
and
and
and
things
like
that,
we
are
now
capturing
and
tracking
this
for
each
and
every
release
we
cut,
so
you
can
check
where
we
are
at
and
what
happened
and
can
compare
it
to
the
I.
B
So
so
harness
is
being
a
little
shy,
but
on
this
tool,
so
the
new
template
for
so
that's
essentially
on
the
left-hand
side
is
a
release,
cut
template.
So
we've
recently
modified
that,
so
you
can
track
even
if,
as
someone
who's,
not
a
release
manager,
each
of
the
all
of
the
logs
for
the
GCB
runs
that
we
have
the
staging
and
release
process,
as
well
as
the
locations
of
the
notifications
that
we
we
send
to
kubernetes
dev,
as
well
as
kubernetes
announced
when
a
pre-release
or
an
actual
release
of
kubernetes
comes
out.
B
On
the
right
hand,
side
you're
going
to
see
the
basically
a
nice
little
comment:
that
is
the
test.
Great
dashboard
includes.
Basically
what
we
do
is
we
scan
for.
We
scan
for
failing
tests
before
we
cut
the
release
right,
so
we
get
an
idea
of
what
tests
are
commonly
failing.
This
allows
us
to
start
building
patterns
around.
B
Are
there
specific
SIG's?
Are
there
specific
kinds
of
tests
that
consistently
have
failures
and
what
you're
seeing
is
actually
what
you're
seeing
is
a
snippet
from
test
grid,
but
it's
also
built
on
a
tool.
The
scraping
of
that
snippet
is
done
by
a
tool
that
Han
has
created
called
test
grid,
shot
that
lives
in
our
kubernetes
release.
Repo,
so
I
wanted
to
make
sure
that
he
shouted
himself
out
for
that.
A
Cool
thanks,
but
also
we
invested
a
little
bit
on,
or
quite
some
quite
a
lot
actually
undocumented.
All
the
different
roles
we
have
in
as
part
of
her
release
as
part
of
actually
cutting
the
thing,
cutting
the
release
there,
the
you
will
be
able
to
download
the
slides
later
and
and
have
the
links
at
hand
yeah
next
off
yeah.
B
So
one
of
the
big
improvements
that
we've
seen
over
the
last
few
cycles,
especially
specific
shout
out
again
to
to
Katherine
Barry,
actually,
who
received
the
CNC
F
chop
wood
and
carry
water
award
for
some
of
her
work
in
NSYNC
release.
Over
the
last
few
cycles,
she
came
onto
the
the
release
team
as
a
test
infra
shadow.
Was
it
shadow
at
the
time
yeah
and
said
this
role
needs
to
go
away,
I'm
going
to
automate
so
much
of
it
that
it
doesn't
need
to
exist
anymore
and
spent
a
cycle
doing
that?
B
Just
that,
so
one
of
the
one
of
the
tools
that
we've
been
working
on
is
the
image
builder
and
again.
This
is
a
tool
that
Kathryn,
mostly
wrote
and
what
it
allows
you
to
do
is
essentially
define
kind
of
a
yeah
mol,
manifest
right,
we're
taking
a
few
different
artifacts
we're
taking
the
the
cloud
build
yeah
Mille
that
you
would
use
to
submit
to
GC
the
GCB
for
GC
b
runs
as
well
as
a
docker
file
if
you're
building
images
and
a
variants
dot
yeah
Mille.
B
So
it's
kind
of
cool
about
the
variants
thought
yeah
Mille
is
it's
kind
of
the
way
that
we
build
a
lot
of
the
kubernetes
images
for
testing
for
intent
testing.
So
if
anyone
is
familiar
with
some
of
the
the
intent
test
runs
the
scenarios
that
we
build
out
within
tests,
infra
tests,
infra
slash
scenarios,
slash
kubernetes,
edep
right,
so
in
defining
the
variants
file.
What
you,
what
you
get
to
do
is
specify
a
bunch
of
different
variants
for
the
way
that
your
image
is
supposed
to
behave.
B
So
you
can
essentially
parameterize
a
few
a
few
things
like
say:
you're,
building,
kubernetes,
so
I
might
want
to
change
the
the
golang
image
or
I
might
want
to
or
I
might
want
to
change
the
the
Basel
version
that
we're
using
to
do
test
runs
so
or
what
have
you
right?
So
this
tool,
image
builder
that
was
written,
it's
kind
of
I,
noticed
that
it
was
going
to
be
generally
useful
for
image
bills.
So
we
have
something
called
GCB
manager
that
we
use
in
in
the
release.
B
Engineering
portion
of
the
project,
GCB
manager
is
essentially
some
bosch
sugar
around
Anagha,
which
is
more
more
scripty
doodads.
It's
a
eighteen
hundred
line
bash
script.
That's
what
we
use
to
release
kubernetes,
so
GCP
manager
is
essentially
just
meant
to
provide
the
substitutions
that
we
need
for
a
google
cloud
build
run.
So
I
saw
that
you
know.
Catherine
is
building
out
this
tool
and
I.
Think
as
like
wait.
This
is
generally
useful
to
the
release
engineering
side.
B
Why
don't
we
tweak
it
a
little
bit
right
so
now
that
tool
can
run
arbitrary,
builds
of
arbitrary
google
cloud
builds.
It
always
feels
weird
to
say:
GCP
builds,
GC,
Google,
Cloud,
build
builds,
runs,
but
anyhow
the
tasks
tasks,
let's
go
with
tasks,
so
we've
gotten
to
the
point,
where
kind
of,
in
the
background
I'm
working
on
building
out
our
build
staging
and
release
process
within
the
CNC
F
infrastructure,
as
opposed
to
the
Google
infrastructure,
and
this
tool
has
been
kind
of
key
to
being
able
to
enable
some
of
that
work.
B
We're
also
going
to
see
across
the
project.
More
people
will
start
to
leverage
this
tool.
Cluster
API
already
uses
builder
to
automatically
push
images
for
their
repos
on
post
submit.
So
what
I
would
like
to
have
is
a
periodic
job
that
runs
stages
and
build
prop
and
builds
for
kubernetes
running
in
the
background.
So
this
moves
us
a
little
closer
to
this
moves
us
a
little
closer
to
getting
nightly,
builds
right,
getting
nightly,
builds
getting
nightly
packages,
artifacts
what-have-you.
A
Yes,
speaking
of
getting
stuff
into
the
community
on
infrastructure,
we,
we
also
did
a
bit
of
work
on
with
the
workgroup
kate's
infra
to
basically
automate
all
the
infrastructure
management
things.
We
need
creation
and
management
of
our
pockets
and
GCP
projects,
and
and
all
that,
all
that
business
and
who
has
access
to
what
again.
This
is
super
important,
because
this
will
allow
us
to
way
easier
on
board
people
into
the
release,
engineering
and
release
team.
B
So
I
think
it's
something
that
we
we
often
forget
to
put
on
our
slides
and
I
think
we
might
have
this
time
too,
but
the
if
you've
ever
participated
in
on
the
release
engineering
side
as
a
release,
manager,
associate
branch
manager,
shadow
patch
release
shadow
at
some
point.
In
the
last
few
cycles,
you
would
have
noticed
that
it's
kind
of
difficult
to
get
involved
in
kind
of
gleaning
what
a
branch
manager
or
a
patch
release
team
member
does
day-to-day.
B
And
the
reason
behind
that
is
that
it's
essentially
us
giving
away
the
keys
to
the
kingdom
right
the
the
way
that
our
GCB
projects
were
configured
one
their
Google
projects.
So
we
have
limited
access
and
what
we
can
delegate
a
way
to
people
and
and
to
is
a
shadow,
a
shadow
maybe
new,
to
the
community
altogether
right.
B
So
we
wanted
to
try
to
build
a
process
that
would
allow
us
to
have
shadows,
build
trust
across
across
time
and
having
some
of
the
work
that
working
group,
Kate's
infra
has
done
in
terms
of
in
terms
of
group
management
and
bucket
management,
as
well
as
getting
access
getting
actually
getting
owner
access
to.
The
GCP
projects
that
exist
today
on
the
Google
side
have
allowed
us
to
build.
I
am
roles
viewer
only
roles
for
some
of
these
shadows
so
that
they
can
actually
glean
the
logs
and
learn
what
we're
doing
day
to
day.
B
So
it's
really
big.
Like
we've,
we've
moved
the
release
management
group
into
so
it's
now
release
managers
at
kubernetes
I/o,
and
that
is
a
list
that
we
control
via
yanil
file.
That
is
run
by
only
trusted
people
and
that
list
that
then
delegates
access
into
GCP
right.
So
there
have
been
massive
improvements
to
help
onboard
people
in
the
release.
Engineering
projects,
as
well
as
all
of
the
projects
start
that
k10
for
our
building
out
to
in
C
and
C
F
infrastructure.
A
Another
thing
we
finally
started
caring
more
about
is
how
we
build
our
packages
and
in
future
also
how
we
distribute
them.
So
what
what
happened
in
the
last
couple
of
months
was
we
kind
of
unified
the
way
we
we
do
this?
We
do
we
build
rpms
and
the
TVM
packages.
We
we
it's
not
fully
unified
yet,
but
we
we
made
the
first
steps
in
that
direction.
A
Anybody
can
run
that
thing
on
their
laptops.
It's
containerized.
It
also
runs
on
a
schedule
to
test
if
things
still
work
and
certain
packages
can
be
installed.
This
is
all
proud,
a
powered
by
prowl,
which
gives
us
all
the
nice
things
like
wewe
can
inspect
logs.
We
have
the
results
and
test
grid
and
all
that
business.
A
So
the
big
part
stephen
already
hinted
to
it.
There
are
a
couple
of
scripts
in
in
the
repo
kubernetes
slash
release.
One
of
the
maintenance
scripts
is
an
ago
which
basically
does
most
of
the
things
with
its
80
18
hundred
lines
of
weird
bash
script:
sourcing
out
of
three
library,
libraries,
which
are
depending
on
each
other
with
the
nara.
I
don't
know
fifteen
hundred
lines
of
sweet
sweet
bash.
A
We
have
another
couple
of
scripts,
which
two
different
things
and
all
in
all,
that
that
pile
of
Besh
is
not
really
maintainable
is
super
hard
to
change,
and
we
want
to
change
that
so
say
hi
to
Krell.
What
is
Krell
Carell
is
a
thing
we
recently
started.
We
wanted
to
rewrite
all
of
most
of
the
things
that
all
those
different
batch
scripts
we
have
do,
but
we
wanted
to
do
it
in
a
like,
probably
a
bit
better
suited
language,
so
we
choose
go.
A
We
use
all
the
nice
libraries
which
we
or
which
other
other
things
in
the
community
already
used.
For
example,
we
use
Cobra
and
Viper
and
try
to
be
as
close
as
close
to
humanities
or
close
to
what
Cuba
Nettie's
uses
as
we
can
yeah.
That's
gonna
be
a
major
thing.
We
want
to
asset
rewrite
all
the
all
the
scripts.
For
example,
we
have
branch
fast-forwards,
shell
script,
which
does
weird
things
and
brings
in
the
changes
which
already
have
merged
to
master
into
the
release
branch,
for
example,
into
the
release
117
in
the
in
the
window.
A
Before
we
actually
cut
the
release
after
after
code
freeze-
and
we
actually
cut
I-
think
that's
about
right,
so
I
think
that's
one
of
the
first
things
to
even
looked
into
porting
to
our
new
one-stop
shop
tool.
Krell.
Another
thing
is
like
when
the
trial
Porsche,
which
should
replace
Porsche
built,
which
just
pushes
compiled
artifacts
of
Cuban
at
ease
into
well-known,
well-known
buckets.
If
you
will
or
well-known
locations
and
yeah,
we
will
try
to
port
one
tool
after
the
other
into
this
new
world
and
trying
to
make
sure
we
don't
break
anything.
A
B
So
I
think
it's
important
to
mention
to
the
the
branch
fast
forward
process
and
Hannes
was
mentioning
that
it
does
a
bunch
of
random
things
that
maybe
it's
not
supposed
to
do
this
branch
fast
forward.
There
is
an
update,
open,
API
spec
bit
that
runs
that
changes.
The
version
make
sure
that
the
the
version
that's
landing
in
the
release
branch
is
actually
the
version
that
is
pinned
for
that
release,
as
well
as
dropping
the
change
logs.
B
So
one
of
the
reasons
I
chose
this
tool
to
start
rewriting
first,
is
that
it's
small,
it's
smaller,
it's
something
that
we
run.
We
run
fairly
frequently
as
a
manual
process
during
this
we're
actually
in
code
freeze
right
now
for
117,
so
essentially
every
you
know
day
whether
it's
myself
or
Carlos
right
now,
since
I'm
here,
do
a
branch
fast
forward
to
basically
sync
up
the
release
branch
with
current
master.
Once
we
move
into
code
thaw,
we
essentially
change
what
we're
doing.
B
As
well
as
the
branch
managers
for
that
content
to
be
allowed
into
the
the
117
cycle,
that
late
in
the
game,
so
the
push
build
scripts,
the
reason
that
I
thought
it
was
a
good
idea
to
maybe
attack
push
build
next
is
push
build,
is
very,
very
important
and
we
didn't
know
all
the
things
that
it
touched
until
I
broke
a
thing
which
was
yeah.
We
broke,
we
broke
most
of
CI
for
kubernetes
during
the
one
1210
cut
of
kubernetes,
and
the
reason
for
that
funny
enough
was
was
shell
check
updates.
B
We
were
like
hey.
Let's
make
this
code
a
little
bit
more,
a
little
bit
more
nice
to
watch
and
we're
like
started
pushing
all
these
shell
check
updates.
That's
like
perfect
112
10
is
the
perfect
time
because
we're
moving
into
an
unsupported
one.
12
is
moving
into
an
unsupported
state,
and
we
want
to
do
this
before
any
of
the
patch
releases
hit.
So
push
build.
B
Push
build
is
responsible
for
actually
pushing
artifacts
as
Hannes
mentioned,
but
it
runs
within
each
of
the
kubernetes
and
to
end
scenarios
and
ten
scenario,
tests,
which
means
everyone
who
leverages
kubernetes
ete
in
some
way
it
was
and
what
ended
up
using
push
build
and
everyone
who
is
expecting
artifacts
to
be
in
certain
locations
depended
on
that
script
right.
So
when
we
broke
push
build,
we
broke
the
rest
of
kubernetes
and
we're
like
okay
cool.
Maybe
this
is
the
tool
we
should
fix
next,
so
the
you
know,
the
effort
has
been
I.
B
Think
we've
been
the
the
branch
fast
forward
when
I
I
was
working
on
maybe
two
months
ago,
or
something
and
it's
just
about
it's
just
about
complete
I-
think
it's
ready
to
go
right.
It's
ready
to
go
ish
and
then
push
build.
Is
the
next
as
the
the
enacted,
the
next
one
to
attack
and
that's
going
well.
Daniel
and
Sasha
are
making
impressive
progress.
I
essentially
said:
go
rewrite
all
the
tools
you
guys
project,
manage
it
and
and
have
fun.
So
that's
it's!
B
It's
really
nice
to
see,
because
these
these
tools
are
representative
of
three
years.
Three
plus
years
of
technical
debt
within
the
kubernetes
project,
these
are
kind
of
tools
that
we
don't
touch
them
because
things
could
break,
and
then
we
prove
that
things
will
break
as
a
result
of
them
and
we're
like.
Ok,
you
know
exactly
what
breaks
we've
documented
it.
Now
we
can
go
about
fixing
it
so.
A
A
A
As
said
just
before,
with
Krell
I,
we
hope
to
push
more
functionality
into
Krell
itself
and
as
a
next
step,
then
Gaddis
get
corel
and
GCB
or
get
rel
into
GCP
tasks
as
much
as
we
can
for
the
generation
of
the
packages.
Steven
has
a
tool
has
started
working
on
the
tool
called
q
PKG,
which
might
end
up
being
a
relatively
flexible
and
not
so
much
bound
to
the
kubernetes
weirdness
tool,
and
maybe
other
people
can
use
that
than
to
to
bake
their
packages.
That's
one
so.
B
So
kind
of
the
idea
behind
cube
package
is
we're
so
there's
a
point
in
the
release
process
that
we
need
to
cut
dubs
and
rpms
right.
The
tricky
dance
that
we
have
is
that
the
the
optimum
repose
that
we
use
for
kubernetes
our
app
tinium
repos
that
are
again
controlled
by
Google
and
it
is
signed
with
a
package
signing
key
that
is
controlled
by
Google.
B
So
what
we
want
to
do
is
I
want
to
get
to
a
point
where
we
can
do
again.
A
nice
yeah
mole
configuration
file
right
where
we
can
define
exactly
the
package,
its
dependencies
location,
the
layout
of
what
files
it
needs
to
lay
down
and
on
the
file
system
and
have
it
have
it
be
able
to
build
arbitrary
packages,
not
just
the
ones
I
mentioned
before.
B
So
this
is
something
that
will
be
generally
useful
to
one
the
release,
engineering
team,
but
also
a
lot
of
these
sub
projects
within
kubernetes
that
maybe
don't
have
a
release
process
or
their
release
process
is
someone
cuts,
a
release
on
github
and
then
someone
else
runs
a
thing
on
their
laptop
and
you
know,
and
then
they're
like
hey.
This
is
community
safe
and
you
should
go
consume
it
right.
A
Another
thing
we
are
looking
into
or
already
started
looking
into
specifically
sasha
did
a
lot
of
work.
There
is
having
better
tooling
around
the
release,
notes,
stuff
and
also
unifying
them.
I
just
recently
found
out
that
three
friend
for
three
I
don't
know
a
couple
of
different
implementations
for
different
things
regarding
release.
Tooling,
probably
it's
on
the
long
run
nice
to
have
this
unified
in
one
tool.
So
that's
the
thing
we
probably
work
on
and
I
guess
a
couple
of
other
things.
I
don't
know
yet
we
don't
know
yet
yeah
we
will
see.
B
That's
actually
what
I
was
gonna
say.
If
you
have
ideas
about
tools
that
we
can
be
working
on,
we
are
happy
to
help
I
think
have
having
all
of
that
knowledge
from
the
community
from
people
who
are
new
to
the
community
people
who
have
been
doing
release
engineering
for
a
while
people
who
want
to
get
involved
in
doing
release
engineering
people
who
enjoy
the
idea
of
getting
rid
of
old
bash.
So
there
are
a
few
ways
to
reach
out
to
us.
The
the
first
one
is
the
cig
release
repo
the
cig
release.
B
Repo
is
the
canonical
repo
for
all
of
the
cig
release
documentation.
So
go
there
check
it
out.
You
can
read
the
handbooks
within
it's
a
sig
release,
release
engineering
and
then
handbook.
So
you
can
read
the
handbooks
for
the
branch
managers,
the
patch
release
team,
the
build
admins,
the
way
we
do
packaging.
All
of
the
the
neat
tips
that
we
give
each
other
to
like.
Hey
you've
got
to
do
this
thing
before
you.
Do
that
or
it'll
break
this,
so
you
can
read
all
of
those
cliff
notes
there.
B
The
release
engineering,
repo
is
crew,
brunetti,
slash
release,
and
that
is
that's
where
all
these
messy
tools
live.
So
there
are
a
lot
that
we
didn't
mention.
There's
fine
greed
bill,
there's,
there's
the
release,
notes
thing:
there's
compiled
release
tools.
There
is
there's
a
lot
there's
a
lot
and
it's
all
messy
and
it's
all
it
works
in
weird
ways.
B
So
that's
where
you
can
go
see
that
ball
of
wax
that
we're
trying
to
unravel
right
now
we
have
a
project
board,
so
I
manage
each
of
the
I,
try
to
manage
each
of
the
the
backlogs
for
the
sub
projects
that
we
have
across
sake
release.
So
that's
licensing,
cig,
release,
proper
release,
engineering
and
then
also
the
release
team.
So
the
project
board
for
release
engineering
and
you
can
get
an
idea
of
some
of
the
work
that
we're
taking
on
and
if
you're
interested
and
it's
not
assigned
to
anyone,
feel
free
to
come.
B
Hang
out
in
one
of
our
meetings
and
talk
about
how
you
can
you
can
jump
into
that
right.
We're
also
hanging
out
on
slack
so
on
kubernetes
sonic.
It
is
the
release
management
channel,
as
well
as
the
cig
release
channel,
and
our
groups
are
release
managers
at
kubernetes
I/o.
That
will
send
a
ping
to
everyone
who
is
part
of
the
branch
release
team,
including
myself
and
Hannes.
B
B
A
A
C
So
it's
not
a
question
more
of
a
feedback
as
a
release,
engineer
myself,
I'm,
seeing
a
lot
of
pain.
You
guys
face
that
I
face
also
like
unifying
removing
Bosch,
ISM
and
stuff,
like
that.
What
I
would
like
to
see,
ideally,
of
course,
is
to
think
about
how
this
can
be
useful
for
any
kind
of
project,
and
not
only
kubernetes
itself,
so
that
we
can
work
together
or
stuff
like
that,
like
building
signings
of.
B
Absolutely.We,
so
what
I
would
love
to
see
these
tools
be?
Is
generally
useful
for
kubernetes
the
ones
that
we
specifically
call
cube,
something
we
kind
of
only
expect
it
to
be
used
on
the
kubernetes
realm,
but
we
do
want
to
build
the
processes
around
the
tools
that
we
use.
The
way
that
we
build
things.
The
way
that
we
deploy
and
I
think
that
the
feedback
from
release
engineers
like
you
is
going
to
be
massively
helpful,
so
come
come
on
down,
come
hang
out
with
us
and
we'll
figure
it
out
together.
D
Hello,
my
name
is
Christopher
work
for
pivotal
as
a
release
engineer
for
paths.
My
question
is:
what
are
their
personas
of
the
users
that
the
release
cig
is
supporting,
so
have
we
done
like
some
research
to
like
understand,
either
like
with
personas
or
like
jobs,
to
be
done
to
understand?
Who
are
we
serving
like.
B
E
B
Find
out
so
the
LTS
survey
was
kind
of
under
the
guise
of
figuring
out
if
long-term
support
is
even
a
good
idea
for
the
project
within
that
survey,
which
we
can
we
can
share.
The
results
somewhere
right
within
that
survey
includes
a
bunch
of
the
different
personas
that
we're
targeting
right.
So
vendors
and
you
know
so
the
actual
consumers
hobbyists
of
kubernetes
and
talking
about
things
like
well.
How
long
would
you
want
to
work
or
what
are
the
reasons
that
you
do
in
don't
upgrade?
B
What's
your
risk
tolerance
for
doing
that,
and
then
you
know
the
final
one,
of
course
is
I
didn't
even
know
these
existed
right
and
55
percent
of
the
people
who
voted
on
that
said,
I
didn't
even
know
they
existed
right,
so
I
think
it's
part
of
it
is,
is
really
publicizing
that
these
are
the
things
that
we're
working
on
day
to
day
and
and
and
caring
about
those
personas.
So
we
do
we
do
care,
we
have
been
doing
work
around
that
yeah.
E
Thank
you.
This
has
to
do
that.
I
have
two
questions
actually.
First,
the
first
question
was
being
kind
of
in
the
test
and
release
side
myself.
How
do
you,
how
do
you
enforce
the
feature
developers
to
come
up
with
tests?
I
assume
that
you
know
when
you
you
guys,
are
working
towards
like
a
nightly?
No,
they
test
run.
You
probably
have
a
whole
slew
of
tests
that.
D
E
Need
to
conduct,
how
do
you
vet
vet
tests
to
ensure
that
that
you
know
that
they're
valid
you
know
obviously
you're
doing
metrics
after
after
you
run
the
tests
to
determine
you
know,
test
failures
and-
and
you
know
the
frequency
of
those.
So
how
do
you?
How
do
you
do
that
and
then,
secondly,
with
Carrell,
because
it's
kind
of
an
integral
part
of
of
the
release
process?
How
are
you
testing,
Krell.
B
So
that's
all
the
time
we
have
no
okay,
so
testing
is
hard.
There
is
so
you
saw
those
dashboards.
Basically,
what
we
do
is
before
running
any
of
the
staging
or
release
process.
We
try
to
make
sure
that
those
dashboards
are
as
clean
as
possible.
The
failures
that
pop
up
on
so
you
can
check
out
the
sig
release,
dashboards
that
is
test
grid,
kto,
slash,
sig,
release
and
you'll,
be
able
to
see
that
we
break
our
tests
into
a
set
of
master,
blocking
master,
informing
release,
117
blocking
informing.
B
Then
we
also
have
dashboard
setup
for
the
image
pushes
the
the
miscellaneous
things
that
we
do
like
making
sure
that
the
build
process
actually
works
for
like
Deb's
and
rpms,
so
we'll
be
trying
to
make
sure
that
the
things
that
are
on
release
master
blocking
as
well
as
1:17
blocking
for
1:17
are
actually
green.
What
we're
supposed
to
do
is
not
release
without
seeing
that
those
boards
are
green
right,
so
we
have
a
bit
more
flexibility
while
we're
in
a
pre-release
state.
B
But
a
lot
of
this
is
a
lot
of
this
is
not
just
technicalities
and
writing
tests.
A
lot
of
it
is,
is
cat
hurting
right,
it's
going
from
sick
to
say
we
build
an
entire
team
just
to
cat
herd
and
make
sure
that
tests
test
failures
are
knocked
out
on
the
crawl
side.
It
was
it's
increasingly
difficult
to
do
any
of
the
in
any
batch
testing
or
assume
that
someone
has
experience
doing
batch
testing,
whether
it's
using
bats
or
building
out
your
own.
B
You
know
your
own
scaffolds
for
for
this
stuff,
so
this
was
like
this
kind
of
built
the
urgency
around
rewriting
a
tool
in
a
language
that
was
fairly
easily
testable
right,
so
I
think
we're
right
now.
We're
in
that
alpha
phase
were
like.
Oh,
my
god.
Can
we
just
make
it
work
and
we're
so
I'm
I'm
good
at
I'm
good
at
the
I'm
good,
at
the
let's
make
it
work
and
I
have
a
lot
of
people
who
are
good
at
the
well
before
we
use
this.
B
This
has
got
to
be
really
well
tested,
so
I,
I
kind
of
like,
like
likely
laying
down
an
idea
so
Carell
and
the
the
Krell
fast
forward.
I
was
like,
let's
just
try
it,
let's
see
if
we
can
do
this
and
I
essentially
handed
that
project
off
to
Hannes
and
not
Hannes,
but
Sasha
and
and
Daniel
who
are
on
the
release
manager
associates
team
to
project,
manage
that
so
coming
soon.
B
A
Now
that
we
have
more
ownership
of
community
controlled
infrastructure,
we
also
have
ways
to
provision
like
staging
buckets
or
whatever
test
infrastructure
which
we'll
never
see
any
like
customer
or
use
of
our
things.
But
that's
that's
a
way
where
I
could
imagine
we
could
test
enter
and
the
release
process
by
checking
which
assets
are
there
or
not
there,
which
artifacts
came
out
at
the
end
of
the
pipeline
or
not
so.
B
Oh,
no,
it's
it's
whatever!
You
feel
whatever
you
feel
crowler
KRL.
It
stands
for
kubernetes
release,
tool
box.
So
all
right!
Thank
you!
So
much
for
attending
the
session.
I
hope
that
you
enjoy
the
rest
of
cube
con
that
you
take
a
little
time
to
provide
us
some
feedback
on
what
we
can
improve
on
as
well
as
feel
free
to
join
us
in
the
community
and
let's
work
together
to
build
the
stuff
out.