►
From YouTube: Building the Cloud Native Kernel: Kubernetes Release Engineering - Tim Pepper & Stephen Augustus
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building the Cloud Native Kernel: Kubernetes Release Engineering - Tim Pepper & Stephen Augustus, VMware
Is Kubernetes a kernel or distribution? Yes! It is necessarily both! CRD’s, out-of-tree cloud providers, and CNI/CSI/CRI abstractions evolve Kubernetes’ core toward an extensible kernel. At 2017, KubeCon NA Tim Hockin and Michael Rubin started a conversation on formalizing “Kubernetes upstream as a distro”, proposing we clean up thinking/processes, define tools/standards, incentivize distros to stay close. They argued for a Kubernetes reference distribution focused on correctness and stability. So where is it? After a slow start, we have momentum in 2019 to improve conformance, API stability, and better documented support stances. However to understand why we don’t (yet) have an upstream reference distro, we need to dive deep on build/release/test tooling. This talk will summarize Kubernetes distro issues/advances and potential contribution areas for individuals and companies.
https://sched.co/UaWx
A
Good
afternoon
everybody
and
welcome
to
the
next
talk
in
the
operations
track
before
I
introduce
the
next
talk.
Please
remember
to
rate
the
session
afterwards
and
yeah
I
think
we
will
do
the
introductions.
So
the
next
talk
is
building
the
cloud
native
kernel,
kubernetes,
release
engineering
by
Steven
and
Tim
and
Steven
is
an
active
leader
in
the
community
community
who
currently
serves
as
the
product
management
chef
for
kubernetes
and
share
for
both
the
release
and
SEO.
Six
he's
also
a
lead
cloud
native
I
advocate.
A
B
B
So
in
the
mean
time,
so
that
was
two
years
ago
in
the
meantime,
you
might
have
heard
like
kubernetes
is
the
Linux
of
the
cloud
and
that's
like
super
easy
to
say:
what
does
it
mean
like
cloud?
Is
pretty
fuzzy
on
its
own
right,
but
in
like
kubernetes?
Is
this
pretty
big
thing,
also
like
what
is
it
and
Linux?
What
does
that
mean?
So,
let's
kind
of
decompose
this
a
little
bit
kubernetes.
Is
this
big
open
source
project?
B
There's
this
huge
developer
community,
all
sorts
of
work
going
around
a
relatively
monolithic
core
and
that
core
is
a
commodity.
We
like
to
say
that
kubernetes
is
boring,
and
it's
also,
despite
that,
it's
huge
there's
millions
of
lines
of
code
and
that
code
can't
work
just
on
its
own
there's
a
whole
bunch
of
other
components
around
it
that
are
required
to
run
something,
and
then
all
of
those
things
come
together
and
what
are
called
distributions
distributions
are
opinionated.
Now
what
is
Linux
then?
Well.
Actually
this
is
a
big
open
source
community.
B
It's
a
big
monolithic
core.
We
might
call
it
a
kernel,
there's
also
sort
of
this
commodity.
We
don't
talk
about
like
do
you
really
know
what
version
of
the
Lynx
kernel
is
on
your
watch
or
on
your
phone
or
on
your
server
people?
Don't
care
it's
a
commodity.
It's
huge
millions
of
lines
of
code,
they're
a
whole
periphery
of
other
components
around
it.
They
come
together
in
opinionated
distributions.
So
it's
it's
live.
Look.
It
may
be
similar.
Now
it's
the
first
day
of
the
conference.
B
Let's
go,
maybe
maybe
it
could
be
the
Linux
or
the
cloud,
but
I've
still
been
a
little
bit
cloudy
on
one
point
and
I
started
to
allude
to
it.
What
is
Linux?
We
can
be
fuzzy
on
this,
even
all
of
those
of
us
who
have
been
around
for
a
long
time,
we
might
say
Linux
and
be
referring
to
the
kernel
or
some
specific
Linux,
Linux
distribution
or
two
Linux
distributions
in
general,
so
a
little
bit
more
on
what
is
a
distribution?
B
There's
a
lot
more
than
just
a
core
like
the
core
of
kubernetes,
the
kaykai,
what
we
call
it:
github
comm,
/,
kubernetes,
kubernetes
repository
for
the
linux
kernel.
The
core
is
out
there
on
corridors.
So,
of
course
you
got
this
core.
It's
something
that
gets
released
on
a
certain
cadence
relatively
quickly
four
times
a
year
for
kubernetes,
roughly
six
a
year
for
Linux.
There's
this
other
stuff
around
it.
These
other
components
that
I
talked
about
those
are
running
on
independent
release
cycles.
B
It
could
be
slower
than
the
core,
because
it's
some
mature,
stable
piece
of
software
or
it
could
be
running
faster,
something
where
there's
a
bunch
of
Rd
innovation
happening,
maybe
some
particular
components
a
little
more
done.
Others
are
a
little
more
raw
and
moving,
but
across
all
these
different
versions,
you
get
a
combinatorial
explosion
of
version
possibilities.
B
Then
there's
also
something
for
a
standard
library.
These
things
aren't
particularly
useful
in
isolation.
You
want
to
have
communications
you,
you
might
have
some
data
that
you
need
to
persist
somewhere
and
you
actually
need
to
run
something
the
users
stuff.
It's
not
just
running
the
system
itself,
but
usually
there's
some
other
goal
in
mind.
We
don't
get
paid,
usually
just
to
make
a
Linux
system
run
or
a
kubernetes
cluster
run.
The
point
of
those
is
to
run
other
user
value.
B
You
have
to
have
somewhere
to
run
those
things
a
platform,
and
you
have
to
have
the
the
tooling
and
frameworks
that
enable
developers
to
build
those
interesting
things
that
are
gonna
run
and
all
of
the
stuff
around
system,
administration,
DevOps,
sre.
All
of
those
sorts
of
things
that
come
together
hold.
B
The
things
there's
a
whole
bunch
of
moving
parts
in
a
Linux
distribution,
but
there's
not
a
lot
of
value.
If
you
look
at
it
just
like
this,
what
we
want
to
do
is
deliver
some
specific
value,
there's
some
content
that
we're
generating
there's
something
that
we're
serving,
and
to
do
that,
you
need
all
of
these
things
to
come
together
and
that's
where
distributions
offer
value.
There
are
purpose-built,
they're,
opinionated,
there's
some
particular
purpose,
and
with
that
you
start
to
get
optimizations,
you
can
focus
in
on
stabilizing
certain
aspects
that
are
more
important
to
your
purpose.
B
So
you
get
one
release
across
all
of
these
components
and
manage
some
of
that
comment
effect
on
dependencies.
You
get
a
coherent
self-consistent
assembly.
You
get
that
typewriter
instead
of
the
200
typewriter
parts
with
that
you
get
a
system,
that's
easier
to
install
easier
to
do
lifecycle
management,
and
that
means
you
can
test
the
thing
you
can
actually
support
it,
and
all
of
these
are
really
compelling
pieces
of
value,
and
it's
one
of
the
things,
an
open-source.
That
is
really
amazing.
You
have
this
aggregator
effect.
B
A
multiplier
effect
and
distributions
are
a
key
part
of
that
in
open
source
communities.
So
there's
a
whole
ton
of
them.
You
down
all
the
names
like
in
Linux,
you
get
the
this
uses,
the
Red
Hat's
the
canonical
is
kubernetes.
We
got
the
the
big
names
we
both
happen
to
work
for
VMware,
but
I'm
guarantee,
there's
Googlers,
Microsoft,
Amazon
folks,
others
out
there
watching
this
talk
about
this.
So
there's
a
ton
of
distributions,
all
sorts
of
unique
value,
opinionated
reasons
being
generated
cool,
but
the
benefits
of
the
distros.
B
If
we
hop
back
to
that
talk
in
2017,
the
comparison
isn't
actually
super
flattering.
We
had
a
very
coupled
core
and
non-core
component
sort
of
mish
mash
together
complicated
platform
enabling
and
the
result
was
incoherent
self,
inconsistent
assemblies
that
were
difficult
to
install
in
lifecycle
management,
rough
edges,
around
testing
and
support-
that's
not
a
fantastic
place
to
be,
but
in
the
intervening
two
years
we've
had
a
lot
of
progress.
So
today
we
actually
have
over
a
hundred
certified
conformant
kubernetes
offerings
across
vendors
distributions,
hosting
services
and
installers
as
well.
So
what
is
this
conformance
to?
B
B
You
can
get
some
synergies
there
and
interesting
things
happen
and
kubernetes
in
most
a
cloud
native
we're
using
more
permissive
licenses,
and
that
comes
with
the
risk
of
divergent
forking,
so
where
Linux
didn't
necessarily
have
conformance
testing,
we
need
something
like
that
in
kubernetes
to
make
sure
that
kubernetes
as
a
word,
means
something
that
we
can
understand
what
that
means
and
surprise.
We
have
it
over
the
last
two
years.
We
made
a
lot
of
progress
on
this
front
and
that
100-plus
certifications
are
there.
B
So
we're
doing
really
good
on
this
front
cool,
doing
good
stuff
for
distres,
but
there's
one
other
thing
that
they
talked
about
in
the
2017
talk,
they
kind
of
imagined
three
possible
community
futures.
Now
they
acknowledge
that
there's
gonna
be
other
people
who
make
distributions
okay,
but
maybe
maybe
we
should
think
about
pulling
that
back
together.
Maybe
we
want
to
just
have
one
thing
that
means
kubernetes
or
maybe
there's
some
middle
ground
there.
B
So
if
you
imagine,
other
people
are
gonna
make
distros,
you
get
this
risk
of
fragmentation
or
politicizing,
and
thankfully,
we've
evolved
conformance
and
that's
helping
a
lot
with
that
doing.
Good
they're
one
district,
a
room
all
in
the
talk
back
then
in
2017
they
said
like
don't
do.
This
is
a
fool's
errand,
it's
impossible
and
we
all
agreed.
We
did
not
go
there
so
again,
good
work
on
the
community
thumbs
up,
checkmark,
plus
one
but
find
a
middle
ground.
What
about
that
and
the
talk
they
proposed?
B
We
we
really
need
to
do
more
to
clean
up
the
core.
We
need
to
work
on
evolving
standards
and
processes.
We
we
need
to
acknowledge
that
others
are
gonna
do
their
own,
but
maybe
we
could
figure
out
a
way
to
hopefully
have
them
based
on
ours,
still
keep
a
central
community
concept
there
and
then
have
those
derived
distributions
benefit
from
staying
close
to
this
core
concept.
So
how
are
we
doing?
There?
We've
actually
made
a
lot
of
progress.
B
There's
improvements
in
cluster
install
and
management
and
common
ization
we're
breaking
up
the
monolith,
we're
splitting
out
cloud
providers,
we've
been
doing
work
to
adapt
how
our
release
cadence
works.
We
have
a
patch
release
team
now
doing
support
in
a
slightly
different
way.
That's
more
sustainable
in
the
past
and
interesting
discussions
about
further
iterations
on
that,
and
then
the
talk
also
mentioned
back.
Then
we
need
to
really
think
about
community
and
whether
we
need
different
skills
and
different
focuses
brought
into
our
community
to
be
more
inclusive.
B
So
the
talk
sort
of
planted
that
seed
and
there's
been
a
lot
of
really
constructive.
Things
happening.
Brainstorming,
work
underway
at
this
point.
So
going
back
to
these
three
points,
though
around
others
are
going
to
do
their
own
and
hopefully
based
on
ours.
There's
one
thing:
I
think
that's
maybe
slightly
missing
and
specifically
distro
skills
and
distro
focus,
there's
all
of
this
history
and
Linux
around
distros,
but
it
hasn't
quite
come
to
the
kubernetes
space
and
to
highlight
this
I'm
gonna
go
back
to
this
slide,
showing
the
the
pictures
on
the
distributions.
B
I
intentionally
left
something
out
here.
Community
distress-
and
we
all
know
them
the
the
fedoras,
the
arches,
the
Debian's,
these
other
distributions
that
are
out
there
and
it
highlights
a
distinction.
There
are
community
distributions
and
commercial
distributions
and
where
are
our
kubernetes
community
distributions
of
those
hundred
conformant
offerings?
B
So,
on
those
three
points,
then,
what
might
we
have
if
we
were
to
imagine
a
community
district
for
kubernetes
so
implied,
is
that
there
is
going
to
be
a
thing
to
stay
close
to
a
thing,
to
base
on
something
for
a
reference
implementation
and
for
lack
of
a
better
phrase
or
word
the
way
that
I'm
describing
this
is
like.
Where
is
our
debian
for
kubernetes,
like
that
that
one
opinionated.
B
Ideal
upstream,
that
is,
a
community
place
for
experimentation
and
shared
collaboration.
So
let's
go
look
at
it.
What
do
we
have
out
there?
We've
got
a
bunch
of
Galang
binaries
available
for
download
we've
got
containers,
we
got
apt
and
yum
repos,
there's
a
bunch
of
artifacts,
but
assembly
required.
These
are
at
most
parts
of
a
distribution.
What
about
our
tooling?
We've
got
this
deprecated
painful
queue.
Bob
script.
We've
got
a
bunch
of
other
interesting
tools,
cube
ATM,
cops,
cube,
spray
kind,
for
example,
but
again
these
are
assembly
required
at
most.
B
These
are
tools
for
building
a
cluster.
Not
a
distribution
implied
within
these
tools
is
that
there
is
a
distribution,
a
content,
that
these
tools
conveyed
your
cluster.
Somebody
still
needs
to
build
that
distribution,
so
I
would
posit
that
we're
kind
of
lacking
distro,
build
people
active
in
this
community,
build
tools
for
distros
as
opposed
to
components
and
then
a
derivation
model
and
tools
around
derivation
and
more
robust
dependency
management.
B
One
of
the
really
useful
benefits
you
see
and
distros
is
that
they
they
kind
of
broch
all
of
the
dependencies
and
give
you
that
coherent,
opinionated
set
of
things
that
are
going
to
work
together
so
like.
Where
is
our
kubernetes
equivalent
of
koji
or
want
to
build
or
Launchpad?
If
you
know
those
where's,
our
equivalent
of
a
Debian
PPA
or
clear,
linux's,
mixer
tooling,
so
community
distres
have
an
opportunity
to
present
some
unique
value.
B
Our
commercial
vendors
are
trying
to
differentiate,
but
if
they're
all
doing
the
same
sort
of
like,
let's
all
make
an
installer.
Is
that
really
what
you
want
your
unique
value
to
be,
or
is
there
other
more
interesting
things
that
you
could
be
doing?
Or
do
you
really
want
to
just
be
the
one
place
where
all
the
bugs
are
fixed
and
the
community.
C
B
B
It's
not
all
doom
and
gloom
like
we're
missing
a
lot,
but
we
can
sort
of
imagine
how
we
might
get
there
and
there's
a
lot
of
interesting
work
going
on
to
start
getting
us
there
and
it's
happening
a
places
like
our
cig
release.
Also
cig
testing
the
working
group
around
kate's
infrastructure,
that's
moving
our
underlying
test
automation
scheme
out
from
Google
and
into
a
CNC,
F
and
community
run
space
and
the
working
group
around
long-term
support.
B
C
So
I
guess
you
thought
I
was
hanging
out
here
just
to
watch,
Tim
talk
so
yeah,
so
we've
been
working
over
the
last
few
cycles.
It's
maybe
a
year
and
a
half
now
probably
on
building
out
a
a
release.
Managers
group
right.
So
this
is
we
some
of
these
subsets
and
characteristic
roles
within
the
release
team
around
patch
release
management
and
branch
management
really
interesting
quandary
that
we
had
for
the
patch.
C
The
patch
managed
patch
release
managers
was
that
a
patch
release
manager
would
sign
up
on
the
release
team
and
they
would
be
responsible
for
maintaining
patches
for
kubernetes
for
that
minor
set
until
it
was
out
of
support
right.
So
that's
about
a
year
someone's
signing
up
to
do
that.
Work
with
that,
without
necessarily
being
aware
of
that
all
right,
so
we
wanted
to.
We
wanted
to
find
a
way
to
make
that
more
spread
that
burden.
So
we
built
a
patch
release
management
team.
C
To
start,
we
pulled
that
out
of
the
release
team
and
made
it
part
of
the
release
engineering
sub
project.
We
had
been
talking
about
release
engineering
for
some
time.
The
general
idea
of
it
and
not
had
and
had
not
formed
something
formal,
so
that
has
happened
very
recently.
The
branch
management
role,
which
is
actually
the
role
who
cuts
the
dot
zero
of
release
so
I'm,
the
one
seventeen
branch
manager
so
after
this
stuff,
I'll,
be
pushing
a
button
in
a
few
days.
So
we
took
that
role
too.
C
That
was
also
on
the
release
team
and
we
moved
it.
We
aligned
it
with
the
patch
release
team.
We've
also
got
the
the
build
admins.
The
build
admins
as
Tim
was
alluding
to.
There
are
set
of
things
that
we
have
in
place
today.
The
app
tinium
repos
that
live
in
google
land
right.
So
this
project
has
you
know
this
project
the
project's
inception
from
Google.
C
So
there
are
a
lot
of
things
that
we
still
do
on
Google
based
infrastructure
right,
so
our
app
tinium
repositories,
the
ability
to
our
package,
signing
keys
are
located
in
google
land
as
well.
So
the
build
admins
team
is
a
set
of
Googlers
that
are
that
are
I'll.
Have
the
access
have
the
access
and
and
participate
in
the
community
to
actually
cut
our
packages
right
and
then?
Finally,
we
have
the
release
manager
associates
group
right,
so
the
release
manager
associates
group.
If
anyone
is
familiar
with
the
release
team,
we've
got
a
bunch
of
release.
C
C
There
are
their
scripts
that
you
can
check
out
that
have
copyright
dates
of
2016
and
there
are
actually
the
ones
that
are
responsible
for
for
releasing
kubernetes.
So
we
want
to
get
to
the
point
where
we
can
start
building
tearing
down
some
of
the
technical
debt
that
we've
built
up
in
the
project
over
time
and
I
was
like
what
better
way
to
get
involved,
but
by
then
by
tearing
down
some
of
this
technical
debt
right.
C
Looking
at
some
of
the
these
have
been
really
hard
problems
to
solve,
and
it's
essentially
the
infrastructure
that
we
depend
on
and
that
the
community
depends
on
that
vendors
depend
on
to
cut
kubernetes,
so
I
think.
Up
until
now,
people
have
been
a
little
a
little
afraid
of
touching
those
tools.
I'm
more
of
a
chaos
monkey
so
I'd
like
to
see
things
break
and
I
like
to
understand
the
process
behind
them,
document
that
stuff
and
and
kind
of
build
the
pathway.
C
So
the
idea
behind
building
this
release
managers
group
was
to
build
essentially
engineering
to
your
ladder
within
within
the
cig
release,
for
the
sake
of
having
people
similar
to
kate
infra.
That
would
maintain,
maintain
kubernetes
overtime
right,
maintain
the
build
infrastructure
maintain
the
tooling
that's
built
around
that
so
I'm,
really
big
on
and
any
presentation.
You'll
you'll
see
me:
do
I'm
really
big
on
these
ideas
of
people
process
tools
right.
C
We
get
things
done
by
aligning
the
right
people
putting
them
in
the
right
room,
developing
process
based
on
the
investigations
that
they've
done
over
time
and
then
writing
tools
based
on
all
of
that
stuff.
We
should
not
be
writing
tools
until
we
understand
the
processes
and
we
have
the
right
people
to
do
that
right.
So
the
next
piece
right
establish
process.
How
do
we
I?
Don't
even
do
that
I'm
very
slide
light
by
the
way
I
like
to
riff,
so
we
have
spent
time.
C
We
have
spent
time
in
the
release
team
and
a
lot
of
the
things
you'll
see
if
you
hang
out
in
cig
release,
you'll
you'll
see
lots
of
see
lots
of
contrasts
between
what
the
release
engineering
group
is
doing
and
what
the
release
team
is
doing.
A
lot
of
the
things
that
we've
done
both
within
cig
release,
as
well
as
across
the
community,
have
been
based
on
the
way
we
formed
the
the
release
team
right.
C
So
you
can
see
contributor
experience
starting
to
build
out
role
handbooks
for
some
of
the
things
that
have
like
the
issue:
triage
and
communications
and
project
management
and
stuff
like
that,
so
that
started
with
cig
release.
So
we
have
a
set
of
roles
on
the
release
team,
the
lead,
CI
signal,
enhancements,
bug,
triage
communications,
release,
notes
and
I
hope,
I'm,
not
missing
one.
Now
the
one
I'm
missing
we
got
rid
of
great
sit.
C
Am
I
missing
some
docs?
Oh
my
god,
doc,
sorry,
Jim!
So
what
we
do
with
each
of
those
roles?
Is
we
document
them
right?
We
get
an
idea
of
what
we
think.
This
role
lists
of
us
be
doing
over
the
quarter
right
over
that
cycle
of
kubernetes
and
we
iteratively
improve
on
that
documentation
across
cycles
right.
So
as
we're
building
these
teams,
where
we
have
lead
role,
we
bring
people
up
into
these
shadow
roles
and
hope
that
they
will
succeed
that
lead
role
for
the
next
cycle,
and
then
those
people
will
eventually
become
release.
C
Team
leads
if
they,
if
they
feel
like
being
a
master
cat
herder.
That's
really
that's
really
what
we
spend
our
cycle
doing
cat
herding,
which
is
it's
super
important,
so
every
time
that
we
do
this,
we
we
pull
out
something
something
really
interesting
about
the
process
and
weari
document
or
someone
asks
a
question
says
like
hey
it's
written
over
here
that
blaah
supposed
to
happen
at
this
time,
and
it
didn't
happen
this
time.
It's
like!
C
Oh,
we
don't
do
it
like
that
anymore
right
and
that's
a
it's
actually
an
avenue
for
a
contribution
because,
like
the
docks,
were
wrong
because
we
changed
the
process
but
we're
open
and
flexible
to
changing
these
processes
as
the
project
evolves
over
time.
So
we've
been
doing
that
on
the
release
team
side
and
we
started
building
out
very
similar
handbooks
for
for
the
release
managers
group
right.
So
the
patch
release
team,
the
patch
release
team,
the
the
build
admins
and
and
the
branch
managers
right.
C
So
essentially
this
kind
of
all
kicks
off
with
a
branch
management
group.
The
branch
management
group
has
a
tight
feedback
loop
between
them
and
the
release
team,
so
they're
responsible
for
cutting
all
of
the
pre
releases
of
kubernetes.
We
actually
cut
a
117
beta
2
on
Monday.
This
is
just
as
we're
kicking
off
cape
Khan
and
the
so
essentially
they
hang
out
in
the
release
and
the
release
team
meetings
get
an
understanding
of
how
the
cycle
is
pacing.
This
gets
really
important
as
we're
moving
into
or
actually
moving
into
that
part.
C
This
Friday
we're
gonna,
be
cutting
our
first
rc4
117
I'm,
so
we're
getting
close
to
the
line.
It's
really
important
that
we're
watching
CI
signal
and
different
things
like
that,
but
as
we're
building
out
these
processes,
it
gets
we
we
get
a
more
of
a
clearer
picture
about
what
these
tools
look
like
right.
So
the
tools
that
we've
been
working
on
actually
in
some
of
the
release
manager
associates
who
are
in
the
room,
have
been
working
on
them
with
me.
We
want
to
do.
C
We
want
to
find
a
way
to
rip
and
replace
the
existing
toolset
right,
so
kubernetes
is
released
by
a
tool
called
Onaga.
A
knocko
is
an
1848
line,
bash
script
that
imports
three
library
is
a
common
library,
a
git
library
and
a
release
library
that
each
are
at
least
500
lines
of
shell
code
as
well.
So
it's
time
to
not
do
that
anymore.
C
So
we've
done
that
we're
doing
push
build,
push,
build,
is
responsible
for
pushing
all
of
the
kubernetes
builds
up
to
our
GCSE
or
Google
Cloud
Storage
buckets
so
we're
starting
to
rewrite
that
one
next,
but
overall
I
had
this
idea.
I
was
like
what,
if
we
had
this
kind
of
tool,
box
right
and
I
was
trying
to
find
a
cooler
name,
but
I
came
up
with
crevel
kubernetes
release
toolbox.
So
that's
what
we've
been
doing.
Essentially,
all
of
these
tools
that
have
been
singular,
singular.
C
Shell
scripts
are
coming
into
this
toolbox
as
sub
commands
right,
so
we're
using
Cobra
and
Viper,
and
all
that
good
stuff,
and
then
the
second
one
I
you
know
myself
and
the
team
have
been
working
on-
is
a
cube
package
right.
So
the
optin
yum
repositories
that
we
were
talking
about
before
we
want
to.
We
want
there
to
be
a
dead,
simple
way
to
produce
Deb's
and
rpms
for
kubernetes.
C
We
also
want
to
try
to
build
the
stuff
in
a
way
we
actually
got
the
question
I
think
with
Sasha
who
asked
the
question,
or
maybe
it
was
you
harness
about?
Well,
why
don't?
We
put
the
Deb
in
rpm
stuff
in
inside
of
chrome,
right
and
I.
Think
it's
a
better
idea
to
have
this
separate,
because
this
is
a
more
generally
useful
tool.
C
You
know
that
we're
starting
to
move
to
the
point
where
I
think
a
lot
of
a
lot
of
companies
internally
have
built
their
own
tools
around
community
based
things,
because
we
didn't
have
anything
that
was
good
before
right.
So
we're
trying
to
kind
of
flip
that
story,
change
the
narrative
and
build
tools
actually
useful
for
not
just
the
community
but
for
for
vendors
for
hobbyists
to
consume
as
well.
C
So
it's
there's
still
quite
a
bit
of
work
to
do,
but
we're
starting
to
coalesce
a
team
processes
for
that
team
and
tools
around
around
that
effort
and
I
encourage
anyone
who's
in
the
room
who
is
interested
in
getting
involved.
It
is,
it
looks
scary.
It
looks
scary
at
first
promise.
It's
not
scary
promise.
We
are
a
friendly,
we're
super
friendly
people.
We
have
lots
of
meetings
to
go
to
on
Mondays
on
Mondays,
alternating
will
do
the
cig
release
and
then
the
release
team
meeting
on
Mondays
and
then
alternating
weeks.
C
We'll
do
the
release
engineering
meeting
and
the
release
team
meeting
back
to
back,
so
everyone
who
is
in
this
crowd
everyone
who's.
Looking
at
this
video
later,
please
come
hang
out
with
us
help
us
write
the
future
of
this.
This
kubernetes
distro
and
thank
you
to
the
people
who
have
already
started
on
that
work
and
have
spent
the
last
few
cycles.
I
love
to
see
all
of
them
in
the
room.
C
You
sometimes
right
so
like
digging
into
digging
into
the
out,
like
kind
of
like
the
outlines
of
a
Google
repository
and
and
realizing
that
we
as
sig
chairs,
didn't
have
access
to
to
push
kubernetes
or
have
access
to
like
the
release
and
testing
GCP
projects.
So
it's
been
a
it's
been
a
fun
journey
to
get
here,
but
I
think
we're.
Finally,
in
a
place
where
we
can
start
to
build
this
idea
of
like
what
kubernetes
the
distro
is,
so
you.
B
I
think
you
summarized
it
well
like
we're
we're
lucky
we've.
We
spent
a
lot
of
time
investigating
understanding
where
we
are.
We
have
a
vision
for
where
we
might
could
be
in
the
future
and
we're
starting
to
get
in
and
make
incremental
changes,
because
this
this
is
a
big,
complex
system
and
and
project,
and
we
can't
just
throw
away
our
existing
system
and
replaces
something
with
something
else
in
a
kind
of
a
monolithic
transition.
B
C
So
are
there
plans
to
move
off
of
the
Google
infrastructure?
Yes,
yes,
yes,
so
the
I
should
have
gone
a
little
deeper
into
the
roof
and
replaced
part,
so
starting
in
1:18
I
think
we
can
do
to
know
anything
so
I've
been
working
on
some
of
the
basically
mocking
the
flow
of
a
kubernetes,
build
urban
ad
stage
and
kubernetes
release
process.
Shifting
that
too.
So
it's
kind
of
still
Google
infrastructure
because
we're
running
on
GCP,
but
it
is
no
longer
Google
owned
projects
right.
C
So
the
scenes
you
have,
the
CNCs
sponsored
GCP
projects
are
the
ones
that
we'll
be
moving
to.
So
we
know
that
the
build
process
works.
It's
running
in
CI
right
now,
I'm
almost
done
with
the
staging
bits
and
then
I
think
we're
gonna
figure
out
we're
gonna
build
an
idea
of
promotion
right
so
being
able
to
land
something
and
staging
have
it
run.
Have
it
pass
a
suite
of
tests
and
then
promote
directly
into
into
prod
or
near
prod?
C
The
I
think
one
of
the
the
hugest
advantages
that
we'll
get
out
of
that
having
something
that's
closer
to
automated
is
what
we
don't
have
today
is.
If
anyone
has
ever
you
installed
kubernetes
and
done
it
via
package
manager
on
the
Debian
side,
you'll
notice
that
you
have
to
add
a
repo
called
kubernetes
annual
to
your
to
your
lists
right
and
that
repo
actually
contains
all
of
the
packages
for
for
debian.
C
It
is
not
just
for
xenial,
it
is
poorly
named
and,
and
we
it's
it's
caused
us
to
be
a
little
less
flexible
about
what
we
can
do
in
terms
of
offering
say
nightly
packages
right.
We
can't
do
that
today,
so
we're
working
on
building
new
hats
and
yum
repos
in
the
CNC
F
infrastructure
working
on
this
promotion
flow
once
that
lands
we
can
do,
we
can
do
we'll,
have
a
release
stream,
we'll
have
a
testing
stream
and
a
nightly
stream
right.
So
all
of
the
pre
releases
will
end
in
that
testing
stream.
C
B
The
the
folks
who
are
working
on
this
recording
the
the
core
of
that
so
and
maybe
just
step
back
a
piece
right
now.
We
have
a
lot
of
things
that
run
on
GCP
that
are
run
by
Googlers
who
have
access
to
special
projects
with
access
controls
and
that's
a
completely
private
thing
and
Google's
wanted
to
open
this
up
to
the
community.
Almost
a
year
and
a
half
ago,
the
open
source
summit
in
North
America
last
year
in
Vancouver,
they
pledged
I,
think
it
was
nine.
B
In
terms
of
tokens,
so
the
idea
is,
if,
if
we're
all
paying,
the
CNC
F
together,
like
any
company,
could
do
similarly-
and
maybe
we
have
other
it's
not
just
GCP,
even
underneath.
But
we
we
fund
a
neutral
place
that
chooses
how,
as
a
community,
to
do
vendor,
independent
or
more
so
vendor
independent
deployments
to
interesting
places
where
we
get
tests
feedback
and
that
we're.
B
Running
them
we
have
the
access
controls.
We
have
the
audit
ability.
We
make
sure
that
we're
doing
all
of
that
professionally
and
not
just
writing
on
the
shirt
tails
of
our
uncle
Google
who's
been
super
generous
to
us,
but
they
don't
necessarily
want
to
have
to
maintain
this
stuff.
They
want
help.
We
get
benefits
by
working
together
as
a
community.
B
The
other
interesting
thing
there
is
that,
as
we
start
looking
at
it
there's
some
common
patterns.
So
Steven
mentioned
this
idea
of
promotion.
It's
a
kind
of
normal
thing
out
there.
You
have
a
nightly
branch
where
your
automated
builds
are
going
and
the
ones
that
pass
see.
I
get
promoted
to
testing
and
people
who
dare
will
test
that
much
more
luck
than
nightly
and
at
some
point
we
bless
a
particular
one
of
those
and
say
it
is
the
release.
B
So
you
you
promote
across
these
streams,
and
we
now
have
a
generic
image
promotion
tool
for
doing
this
with
containers,
and
this
is
another
one
of
the
edge
cases.
That's
typically
a
Google
ism
right
now,
not
just
the
the
package
creation
of
the
releases,
but
any
of
the
containers
that
are
part
of
the
broader
distribution.
We
rely
on
a
Googler
who
has
the
rights
to
build
and
push
that
that
to
the
special
GCR
repo.
So
all
of
these
things
as
a
community,
we
need
to
come
together,
better
operationalize.
It.
E
B
They
have,
they
have
conformance
results
uploaded
there.
They
work
informant
I.
Think
for
one
thirteen
and
fourteen
Nettie's
there
you
could
consider
them
a
conformant
distribution
of
kubernetes.
Are
they
a
community
one
or
in
in
a
way
it
feels
like
it's
I
build
the
cap
cloud
k3s
on
github,
but
it's
also
Rancher.
Is
it
a?
Is
it
a
vendor?
Distro
I,
don't
know,
but
it's
an
interesting
example
of
the
variation
that
you
get
and
the
potential
for
innovation.
I.
Think
I.
B
C
Yeah
yeah
I
think
that
you
know
tools
like
that
make
it.
We
opened
quite
a
bit
of
possibilities,
for
you
know
stew-like,
so
I'm
a
former
field
engineer
at
some
cloud
native
companies
and
it's
interesting
to
see
when
you
have
use
cases
like
for
the
edge
right,
where
distributions
that
are
kind
of
more
fun
like
tightly
bound
start
to
fit.
Those
really
nicely
right.
So
is
that
something
that
we'll
look
at
as
a
community
potentially
right?
D
D
D
B
Starting
to
see
that
I
think,
if
you
look
at
sig
cloud
provider
which
didn't
used
to
exist,
there
was
suggesting
IBM.
There
are
sig
the
M,
where
now
there's
sig
cloud
provider
and
the
others
or
sub
projects
under
it.
If
you
look
at
the
working
group,
LTS
we've
got
folks
active
there
from
VMware
from
Red
Hat
from
Google
from
Sousa
from
we
had
interest
from
canonical.
B
We
had
interest
from
a
couple
of
large,
very
large
vendors
and
China,
who
recognized
that
there's
a
benefit
and
amortize
in
the
patch
release
process
so
that
there
everybody
is
not
independently
cherry-picking.
We
have
a
core
set
of
the
most
skilled
engineers,
doing
it
once
correctly
from
a
security
patch
perspective.
So
there's
there's
a
lot
of
things
like
that.
B
So
examples
like
the
image,
promoter
and
I
think
some
of
the
other
things
that
we're
working
on
pulling
out
of
the
giant
three
plus
thousand
monolithic
lines
of
bash,
as
we
do
that
they
become
more
usable
and
people
are
interested
because
they
have
something
similar
and
are
starting
to
wonder
if
we
could
find
a
pattern
there
and
build
something
for
a
common
tool.
Yeah.
C
So
this
is,
this
is
definitely
more
of
the
I.
Would
not
ask
you
to
run
the
thing
that
I'm
writing
today,
but
we're
getting
to
that
we're
getting
to
the
point
where
these
are.
These
are
tools
and
common
patterns
that
can
be
used
both
on
the
community
side
as
well
as
internally
for
companies
right.
So
this
is,
you
know,
we're
probably
a
year
out
at
least
yeah.
Unfortunately,.