►
From YouTube: Koping with Change: What kops Learned Adopting etcd3, cluster-api and CRDs - Justin Santa Barbara
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Koping with Change: What kops Learned Adopting etcd3, cluster-api and CRDs - Justin Santa Barbara, Google & Mike Splain, Sonos
Kubernetes continues to adds features rapidly, but this makes it hard to install and operating a reliable kubernetes cluster. kops is an official OSS installation tool that makes kubernetes installation and operation turn-key. But doing that requires that kops manage that complexity. We’ve had a busy year, migrating to etcd3, moving from an aggregated-apiserver to CRDs, starting to adopt the cluster-api. At the same time we’re taking the best pieces of kops and contributing them "upstream" into reusable modules - contributing to etcd-management and addon-management tools, as well as to the cluster-api. We’ll give an overview of these changes and some of the challenges we faced, and share our strategies for dealing with the relentless pace of kubernetes change. We’ll also talk about how the upcoming roadmap for kubernetes, kops and for the components we’re spinning out of kops.
https://sched.co/MPcD
A
A
So
this
is
just
in
Santa,
Barbara
he's
been
involved
with
kubernetes
since,
before
the
100
days
doing
a
lot
of
the
AWS
work.
So
if
you
know
what
that
means,
that
means
5
year
anniversary
this
year.
So
that's
over
5
years,
Justin
built
cops
with
a
few
others
and
which
came
out
of
the
need
for
a
better
cube
up
script
for
running
kubernetes
in
particular.
A
way
to
do
it
with
supporting
updates.
B
This
is
my
explain:
mike
has
been
using
communities
for
about
five
years
I
believe
we
decided
and
started
getting
into
cops
a
few
years
ago
to
fix
the
problems
that
he
encountered.
I
think
one
of
your
first
contributions
was
internal
load,
balancers
of
her
member
correctly,
so
I
guess
I'm,
not
sure
Mike.
We
can
really
do
pretty
fair
to
call
you
new,
given
that,
but,
from
my
point
of
view,
is
sort
of
the
older
school
original
guard.
I.
B
Think
of
you
as
part
of
the
new
wave
for
cops
and
I,
think
it's
it's
wonderful
to
see
new
people
that
bring
fresh
ideas
and
new
perspectives,
and
at
least
from
my
point
of
view,
it's
important
to
recognize
that
the
newer
people
are
the
more
attuned
they
likely
are
to
what
the
project
is
today.
So
you
likely
have
a
better
perspective
of
what
that
is,
and
so
I
think
it's
super
important
to
embrace
new
people
not
literally
and
promote
people
within
the
community,
because
they
really
do
know
the
the
space.
B
A
B
Before
we
get
into
those
I
guess
the
the
starting
point
should
be
sort
of
the
fundamental
philosophy
of
cops.
We
sort
of
take
a
very
user
focused
approach.
The
idea
is
that
a
cops
release
or
cop
zero
release
should
be
safe
to
run
in
production,
and
that
means
not
just
cops
but
also
the
version
of
kubernetes
that
were
shipping
and
the
version
of
all
the
components
that
we
bundle
in
and
in
particular
that
means
the
networking
the
C&I
providers.
B
That
obviously
means
that
cops
can't
release
immediately
when
kubernetes
releases.
There
is
a
delay
and
in
particular
there
might
not
be
a
released
version
of
cops
when
a
kubernetes
release
comes
along
and
all
the
all
the
fanfare
around
it
and
cops
users,
don't
necessarily
have
a
release
that
they
can
run
the
tasks
we
have
to
do
to
get
to
da
zero.
B
Before
we
can
release,
we
have
to
make
sure
that
cops
first
of
all
works
with
the
new
kubernetes
I
guess
that
sort
of
table
stakes,
but
we
also
to
make
sure
that
kubernetes
itself
is
stable.
Kubernetes
releases
are
primarily
time-based
and,
despite
the
fact,
despite
the
best
efforts,
often
you
know
the
two
or
three
releases,
probably
when
you
really
want
to
run
in
production,
because
there
are
typically
some
important
bug
fixes
in
those
first,
two
minor
or
whatever
the
third
one
is.
B
A
We
have
a
graph
of
that,
so
thanks
to
Peter
one
of
our
cops
contributors
for
putting
this
together.
As
you
can
see
over
time,
our
release
delay
has
gone
up.
You
know
how
many
people
here
used
cops
just
out
of
curiosity.
Okay,
we're
sorry
like
we're.
Just
we're
sorry
we're
working
really
hard
as
I
think
you
guys
probably
realize
we're
not
neither
of
us
work
on
this
full-time.
You
know,
and-
and
that
goes
for
all
the
maintainer
Xandra
viewers
and
contributors.
A
So,
as
you
can
see,
you
know,
we
kind
had
a
delay
as
as
time
went
on
even
to
the
first
alpha,
our
first
alpha
for
each
one
of
the
kubernetes
versions
took
much
longer,
so
we've
now
released
one
not
12.1
as
of
last
week,
I
believe
maybe
the
week
before.
Who
knows
very
recent
thing?
What
is
time
anymore,
one
that
thirteen
is
coming
up
soon?
We
get
Justin
cut
a
beta
on
that
yesterday
or
the
day
before
and
1.14.
A
B
I
guess
we
were
falling
further
and
further
behind
and
that
this
was
bad
I
think
for
a
number
of
reasons,
I
think
for
all
of
you
end
users.
Thank
you.
We
love
you
and
and
I.
Imagine,
right
and
I
know
as
an
end-user
myself
that
it
is.
It
is
hard
to
see
all
those
sort
of
the
press
and
the
excitement
around
the
new
kubernetes
release
and
then
be
unable
to
have
a
cops
version
that
that
supports
it.
B
So,
if
have
to
wait-
and
that's
that's
not
great-
it
also
presented
some
serious
sort
of
challenges
for
the
developers
and
the
maintainer
x'.
A
sort
of
amusing
anecdote
that
I
like
this
hell
is.
Sometimes
our
release
would
take
a
long
time
to
be
ready
and
sometimes
they'd,
even
be
ready
out
of
order.
So
cooper,
neighs
112,
for
example,
had
some
important
bug,
fixes
that
were
worshipped
and
that
they
weren't
actually
integrated
into
111
until
I.
B
So
that
sort
of
thing
made
life
hard
for
us
organizationally,
but
it
also
I
think
it
created
a
sort
of
a
sense
of
malaise
sort
of,
and
then
this
hamster
wheel
that
I
I
think
he's
common
to
software
development
in
general
and
particularly
to
people
working
on
kubernetes,
where
you
will
feel
like
we're,
always
working
just
to
keep
up
with
security
issues
or
with
the
latest
flag
changes,
and
we
were
sort
of
weren't
really
able
to
work
on
the
stuff.
That
is
more
exciting
that
we
really
want
to
work
on
the
stuff.
That's
more
gratifying.
A
So
we've
revised
their
philosophy
a
little
bit:
users
really
like
and
trust
that
our
dot
zero
releases,
our
production
ready.
You
know
we
feel
that
their
production
ready.
You
know
people
come
back
and
tell
us.
You
know:
we've
been
running
this
in
our
test
Buster's,
you
know
so
we're
gonna
try
to
get
those
out
kind
of
quicker,
but
in
a
better
way
we're
gonna
start
cutting
alphas
and
betas
earlier
in
the
process.
A
So
when
kubernetes
cuts
a
major
release,
we're
gonna
be
cutting
an
output
to
go
along
with
it
pretty
quickly
as
soon
as
we
possibly
can
so
that
you
can
start
testing,
maybe
in
your
Dever
or
test
environments.
That
way
you
know
you
can
say
actually
pretty
early
on
we're
feeling
pretty
good
about
this,
or
you
know
at
least
we
can
expose
it
to
the
users
and
get
everyone
on
the
same
page.
A
With
with
using
kubernetes
cops
versions,
we
are
working
on
fast-forwarding
into
the
current
versions
of
kubernetes,
one,
not
15,
that's
coming
up
really
quickly,
and
so.
In
order
to
get
there,
we
have
1.12
is
already
out
1.13
we're
prepping
and
1.4,
also
prepping,
to
move
to
really
fast.
So
most
new
features
are
gonna,
start
landing
in
1.15.
B
So
there
are,
there
are
some
challenges,
I'm
doing
the
challenges,
lemons,
it
leads
to
slightly
confusing
situation
on
github
I,
don't
know
if
you
visited
they
get
up
release
pages
recently,
but
we
have
a
bunch
of
alphas
and
betas
and
all
of
this
and
get
up
sorts
them
by
time,
which
is
not
necessarily
exactly
what
we
want,
and
so
that's
a
sort
of
downside
of
this
approach.
We
try
to
make
that
a
little
bit
better
than
it
could
be
by
sort
of
strategically
releasing
them
in
the
inverse
order
that
we
want
them
to
appear.
B
The
real
pain
point,
though,
is
there
are
a
lot
of
cherry
picks,
so
everything
that
goes
into
master
now
has
to
be
cherry
picked
in
probably
111-110
e
non
feature.
It
has
to
be
Turkish
Tinto,
probably
in
the
release.
111
released,
112,
release,
113
and
released
114
branches.
So
up
to
four
cherry
picks,
/
/
PR,
it's
a
bug,
fix
and
keeping
track
of.
All
of
those
and
doing
all
of
those
is
challenging,
but.
A
Justin
now
we
can
work
on
major
features
again,
but
really
we
were
held
up
a
lot
that
if
we
had
a
new
idea,
something
that
we
wanted
to
get
out
there,
we
didn't
want
to
possibly
prevent
the
next
release
from
coming
out.
So
it's
this
big
cycle
that
we
were
getting
into
so
now
we
can
land
those
right
away
and
master.
A
And
frankly
it's
like
I
said
we
don't
work
on
this
full-time,
so
it
gets
really
stressful
when
you're
trying
to
figure
these
things
out
and
how
we
can
I
want
to
run
112
113
114
in
my
clusters,
so
I
don't
but
I
don't
want
to
be
stressed
about
breaking
it
for
my
company
or
for
any
of
yours.
So
it
really
brings
down
the
stress
level
and
makes
it
much
more
productive
for
all
of
us.
A
So,
as
Justin
was
mentioning,
we
have
so
many
cherry
picks.
Justin
started
this
spreadsheet
to
handle
all
of
our
our
cherry
picks
and
our
merges.
It
is
really
a
bit
of
work
of
art.
So
thank
you
for
this,
and
our
top
contributor
Ryan
did
a
whole
lot
of
work.
Organizing
and
getting
all
of
our
pull
requests.
A
As
you
can
see,
the
left
column
that
goes
268,
there's
way
more
down
and
up
that
we
we've
been
manually
cherry-picking
and
as
a
side
note,
if
anyone
has
a
really
good
work
for
cherry-picking,
come
talk
to
us
afterwards,
because
we're
gonna
need
it
in
the
future,
but
but
yeah.
So
we've,
we've
cherry
picked
a
lot
of
this
stuff
through
113
114
are
pretty
ready
at
this
point.
Cherry
pick
wise
on
those
you
know
those
critical
fixes
and.
B
I
should
actually
give
a
real
shout
out
here
to
Ryan
and
to
Mike
who
I
started.
This
spreadsheet
half-heartedly
did
have
been
in
it.
I
forgot
what
the
colors
meant.
They
helped
me
out,
they
said
so,
can
we
get
right
access
to
it?
I
was
like
oh
I,
don't
know,
and
I
was
like:
okay,
let's
try
it
and
basically
they
then
took
it
over
from
there
and
did
like
98%
of
this
over
the
work
here.
So
they
are
the
rockstar
so
embrace
the
new
members
of
the
community,
and
thank
you
so
much.
Of
course,.
B
The
sed
team
stopped
I,
guess
wisely,
just
decided
to
stop
maintaining
at
cb2
and
moved
everyone
to
exit
III,
which
meant
the
kubernetes
of
course,
had
to
also
move.
That's
it
III
for
those
who
they
don't
know
at
cb2
and
83
are
basically
completely
different
systems,
and
the
migration
from
its
attitude
at
CB
3
is
basically
unsupported
is
certainly
not
a
smooth
one,
I
think.
B
As
a
kubera's
community,
we
probably
could
have
pushed
back
a
little
bit
more
on
that
migration,
I
think
as
a
cop's
developer,
who
had
to
do
a
lot
of
work
on
this
I
in
retrospect,
and
you
always
figured
out
in
retrospect,
I
probably
should
have
just
written
the
the
migration
support
formats
82
to
83,
because
it
would
pretty
would
have
been
less
work
than
what
we
did
in
the
alternative,
but
that
was
I.
Think
one
of
the
big
initial
sources
of
the
increasing
time
delay
that
we
had
to
a
ton
of
time.
B
Working
on
this
thing-
and
you
know
we
talked
a
little
about
frustration
or
things
that
are
frustrating
I
think
this
was
frustrating
because
it
isn't
necessarily
something
that
end-users
necessarily
care
about.
You
shouldn't
really
care
about
the
the
data
store
underlying
your
kubernetes
cluster.
You
want
to
just
work.
A
So
we
developed
sed
manager
which
is
sort
of
a
net
CD
operator
for
when
you
don't
have
a
kubernetes
cluster
to
run
on.
So
it's
it's
like
the
NCD
operator
for
EDD
for
kubernetes
right,
so
eventually
we'll
be
merging
it
with
at
CD
ADM
to
make
sure
that
ed
city
manager
features
are
available
to
everyone
and
not
just
in
cops,
but
for
now
we
we
needed
that
today.
A
So
Justin
did
a
ton
of
the
work
on
sed
manager.
So
that's
ready
in
112
a
couple
of
those
key
features:
a
net
CD
manager
are
n,
CD
backups.
So
now
you
can
have
automated
backups
of
your
cluster
and
everything
resizing
of
at
CD
clusters
is
also
supported,
and
that
makes
it
easy
to
go
from
1
to
3
to
5
at
CD
masters.
Never
do
2
or
6.
Remember
and
we've
made
it.
We've
made
it
through
the
worst
of
it.
We're
we've.
Oh
yeah,
the
other
thing.
B
And
I
think
that's
one
that
maybe
I
could
have
done
better
on
like
I.
Think
that's
one
where
we
didn't
it
like
integrate
as
well
with
the
community
as
we
as
we
could
have.
You
know
we
could
have
suggested
earlier
intervention
with
at
CD
and
or
helped
with
that
intervention,
so
hopefully
some
of
the
other
ones.
We
do
a
little
bit
better.
So
let's
try
do
better
in
featuring,
let's
cover
ones
that
maybe
we
did
better
on
so
another
one
where
so.
B
Sometimes
you
make
a
bet
on
something
and
or
I
make
a
bet
on
something,
and
it
proves
to
be
very
much
the
wrong
the
wrong
horse
to
back
you
that
I
just
got
it
wrong
and
a
good
example
of
that
is
our
aggregators
apart.
So
cops
is
a
very
early
adopter
of
aggregated
API
servers,
which
are
essentially
a
precursor
to
CR
DS,
but
a
little
bit
more
involved
in
terms
of
the
API
machinery
is
more
of
a
first
class
server
and
less
of
a
sort
of
lighter
weight
extension.
B
The
downside
of
that
first
class
support
is
that
it's
a
lot
more
deeply
integrated
and
in
the
end
we
never
got
it
to
a
really
simple
state.
It
ended
up
that
we
were
ven
during
a
lot
of
kubernetes
libraries.
They
are
was
never
a
stable
API
for
our
for,
inter
building
an
aggregated,
API
server,
it
was
always
a
ton
of
work
to
rebase
every
time.
So
this
was
another
factor
that
was
bringing
bringing
us
further
and
further
behind,
and
we
also
noticed
that
a
great
API
servers
were
not
really
being
improved.
B
The
toolkit
was
effectively
deprecated,
whereas
CR
DS
there
was
a
big
push
on
CR
DS
to
get
them
up
to
parity
and
that
there
were
some
clear
statements
that
C
RDS
were
really
the
way
of
the
future
that
that
every
API
servers
have
use
cases.
But
if
you
can
use
a
CR
D,
you
should
use
a
CR
D.
It's
api
granada
for
end-users
to
install.
B
You
have
to
end
up
setting
your
own
it
setting
up
your
own
Etsy
D,
where
C
RDS
are
fairly
easy
to
install
and
most
it
needs
pretty
lightweight
auerbach
permissions
and
you
just
drop
it
into
more
or
less
any
kubernetes
cluster
and
the
the
build
system.
The
impact
in
the
build
system
was
pretty
bad
as
well
like
our
Doc's
are
still
entwined
deeply
into
the
a
great
API
servers
and
we
are
D
twining
them
as
we
speak,
but
it
generally
was
the
wrong
horse
to
back.
A
So
we're
moving
to
cube
builder
and
CR
DS
and
it's
it's
a
lot
less
code
and
it's
simpler
and
and
frankly,
there's
a
lot
less
dependencies,
which
makes
it
easier
for
us
to
maintain
and
push
forward
over
time.
The
controller
runtime
is
a
much
more
stable
API
and
is
really
intended
to
be
user.
Facing
also
has
aa
mets
doc.
Generation.
A
Tooling
is
much
easier
than
API
machinery,
so
like
Justin
was
alluding
to
our
Doc's,
are
gonna
vastly
improve
over
time
and
and
it'll
be
easier
for
you
to
build
on
the
same
tooling
that
we're
building
on
and
frankly,
it's
much
easier
to
install
CR
D's
on
a
kubernetes
cluster,
there's
no
need
for
Etsy
D
and
it's
less
invasive
on
permission,
wise
for
using
at
CD
and
at
CR.
These
are
cool.
B
Okay,
an
amusing,
an
amazin
counterpoint
to
the
idea
of
someone
attacking
the
wrong
horse
is
sometimes
you
can
be
too
right.
You
can
happen
like
back
absolutely
the
right
horse
and
that
this
can
actually
be
worse,
like
switching
from
a
great
API
service.
Ocr
DS
is
actually
not
that
much
of
a
problem
and
it's
all
internal,
so
hopefully
most
users
won't
notice
it.
B
Whereas
a
long
time
ago,
I
proposed
an
API
object,
called
node
sets
for
creating
machines
and
says
mm
mm
15,
and
we
debated
it
for
a
couple
of
months
and
eventually
we
agreed
that
it
should
be
out
of
cork
and
so
off
we
go
and
we
create
cups
and
it's
on
its
cups
is
in
its
own
repo
and
sure
enough.
This
worked
out
pretty
well
for
cups,
it's
a
fairly
nice
model
and
some
other
tooling,
you
know
built
similar
functionality
and
so
with
a
bunch
of
different
tooling.
B
All
building
similar
things,
for
example,
gardener
in
gke
like
along,
comes
the
cluster
API
that
says
well
hold
on.
We
should
probably
try
to
compile
eyes
this
in
the
community
and
build
some
standard
objects.
That's
called
the
machines,
machines
and
machine
deployments,
and
that's
great
and
it's
wonderful,
that
everyone
is
getting
it.
B
We
instance
groups,
which
are
the
cups
equivalent
are
of
machine
deployments.
We've
shipped
those
we've
exposed
them
to
customers,
your
you
are
using
them
today,
if
you're,
a
cups
user
and
in
some
ways
these
abstractions
are
cops
so
to
replace
that
it's
gonna
be
tough.
While
the
objects
are
similar,
they
aren't
exactly
the
same
over
the
years.
B
My
cluster
pay
is
still
fairly
new
and
over
the
years
and
cups
we've
had
a
lot
of
important
use
cases
that
I'm
sure
a
lot
of
you
have
brought
up
or
contributed
PRS
for,
and
thank
you
so
much,
and
so
the
the
cups
instance
groups
are
today
richer
than
the
cluster
API
machine
deployments.
So
we
can't
just
say
bye-bye
captions,
as
groups,
we're
gonna
use
machine
deployments.
B
So
the
way
we
got
around
this
or
we
are
getting
around
this
is
we
are
going
to
shim
or
we
are
shimming
instance
groups
to
machine
deployments.
So
when
you
create
a
cop's
instance
group
CRD
a
new
component,
the
cops
controller
is
going
to
create
an
owned
machine
deployment.
So
you
will
have
the
best
of
both
worlds.
You'll
have
cops
instance,
groups
and
you'll
have
machine
deployments,
and
so
eventually
we
hope
to
get
to
a
place
where
you
can
choose
where
you
can
either
continue
to
use
cups
instance
groups.
B
If
that's
what
your
tooling
is
built
around
or
you
can
use
machine
deployments.
If
you
want
to
use
the
pure
cluster
API
and
if
you
have
tooling
that
uses
cluster
API,
it
can
address
the
Machine
deployment
and
I.
Think
cluster
era
is
still
early,
so
I
will
see
if
this
approach
becomes
commonplace,
like
whether
other
installation
tooling
adopts
a
similar
approach
or
whether
everyone
is
able
to
whether
we're
able
whether
we
are
able
to
build
up
machine
deployments
to
the
point
where
it
can
stand
in
for
everyone
else's
equipment.
A
Yeah,
so
cluster
API
is
built
around
a
server-side
model
controller
operator
model
and
people
are
porting
lots
of
API
cluster
api's
to
different
cloud
providers
right
now,
just
a
few
AWS,
a
sure
bare
metal,
digitalocean,
OpenStack,
vSphere,
there's
way
more
than
just
those
all
under
the
kubernetes
SIG's
github
repo.
So
if
you
take
a
look,
there
you'll
see
all
the
different
cluster
api's,
and
so
that
provides
a
lot
of
different
value
that
you
know
is
no
longer
in
the
core
of
cops.
We
can
abstract
that
right
over
time.
A
We
also
get
machine
set,
which
is
an
object,
we're
currently
missing
in
cops,
which
would
allow
us
to
manage
instances
similar
to
the
way
that
they
manage
deployments
right
where
we
could
in
in
AWS
world.
We,
you
know
if
you
update
your
audit
your
instance
group.
We
could
bring
up
a
new,
auto
scaling
group
and
scale
up
and
scale
down
the
way
that
a
lot
of
people
have
wanted
over
time,
similar
to
kubernetes
and
replica
sets
right
with
deployments
so
having
when
we
have
a
machine
deployment.
A
Alright,
so
be
the
change,
add-on,
bundles
and
operators,
so
bundles
are
an
area
that
we're
focusing
on
very
actively
currently
cops,
manifests
and
versions
are
all
baked
into
cops
releases.
So
that
means
things
like
our
networking
providers
core
DNS
a
lot
of
those
different
things
that
come
up
and
those
decisions
you
can
make
when
you
bring
up
a
cluster,
those
are
all
baked
into
kübra
into
cops
and
cops
releases.
So
let's
say
you're
networking
provider
has
a
critical
vulnerability.
A
We
have
to
cut
a
cops
release
to
enable
you
to
easily
deploy
that
so
now
using
bundles.
We
can
abstract
that
out
so
that
just
like,
when
we
can
push
out
a
new
cops
kubernetes
version
and
recommend
a
new
kubernetes
version
to
you
as
a
cop's
user
you'll
be
able
to
be
recommended.
Oh
you
should
upgrade
to
calico,
or
you
know,
new
version
of
calico
or
or
whatever
you
know,
dependency
downstream.
That
is,
and
we
are
also
yeah-
that's
actually,
we
all
have
to
do
a
full
release.
Sorry,
oh.
B
And
so
I
think
in
in
contrast
to
the
way
we
that
we've
built
some
of
the
stuff
in
first
in
cops
in
the
past.
This
is
sort
of
the
be.
The
change
I
think
is
is
like
the
idea
of
engaging
with
the
community
from
the
start
and
developing
add-on
operators
in
the
community
first,
and
it
has
its
trade-offs.
It
can
be
a
little
slower
sometimes,
but
it's
it's
we'll
see
if
it's
a
different
strategy
that
we're
trying
and
we're
gonna
see
how
it
works.
B
B
Operators
comprise
a
controller
and
a
CR
D,
and
the
big
advantage
here
is
that
the
CR
D
gives
us
a
really
nice
API,
surface
weather
for
end
users
or
for
tooling
to
build
on
top
and
the
controller
lets
us
continue
to
run
code
like
long
after
cops,
install
or
upgrade
has
finished.
So
you
can
check
the
health
or
status
of
your
add-ons
or
you
could
even
repair
an
add-on
that
has
gone
awry
because
we
have
code.
We
can
really
do
whatever
we
want.
B
We
can
orchestrate
complex,
rollouts
or
sequencing
things
in
the
right
order,
but
this
is
again
a
strategy.
I
guess
for
coping
with
change
is
sort
of
embracing
the
change
and
leading
the
change
and
we're
developing
in
the
open
and
that
sig
Buster
lifecycle
add-on
project.
If
you're
interested
in
this,
please
please
do
join
us.
We'd
love
to
have
you.
A
Yeah,
so
this
is
what
cops
look
like
as
components
on
the
Left
we've
built
a
lot
of
stuff
in
the
cops
community,
which
is
very
specific
to
cops.
We
just
laid
out
why,
right
you
know
over
time
over,
you
know
the
needs.
All
of
you
have
a
need.
Justin
and
I
both
have
had
needs,
so
what
we
want
to
do
is
we
want
people
to
be
able
to
use
their
own
tools
and
that's
okay.
So
we
are
splitting
cops
into
a
bunch
of
commute
community
components
so
that
everyone
can
use
those
pieces.
A
Is
this
the
right
thing
to
do?
I
don't
know,
but
I
think
it
should
allow
us
to
speed
up
development
and
allow
more
possibilities
for
all
of
our
users
to
use
cops.
However,
they
really
see
fit
so
add-ons
and
bundles
and
operators
we're
doing
community
first,
so
that
takes
a
little
bit
longer
and
it's
been
a
slower
process.
Cluster
API,
we
did
separately
as
we
just
went
over
and
you
know
so
now
the
community
developed
it,
and
this
makes
things
a
little
bit
tricky
xev
management.
A
We
did
separately
and
we
are
now
promoting
that
into
the
community
and-
and
it
seems
like
that
will
go
a
lot
more
smoothly,
but
it's
harder
to
prioritize
because
we're
okay
with
the
way
things
work.
Now,
if
you
want
to
build
your
own
tool,
you
should
be
able
to
replace
cop
CLI
with
one
or
more
of
the
component
boxes.
You
see
above
the
goal
is
really
not
to
stifle
different
approaches.
You
can
be
different
as
you
want
to
be,
but
you
shouldn't
have
to
rebuild
everything
you
might
also
notice
over
here
on
the
right.
A
We
have
a
that
little
box
called
note
up.
That's
our
node
initialization
tool
when
your
instances
come
up.
There's
a
group
in
sig
cluster
life
cycle.
That's
discussing
how
we
can
bake
images
and
what
should
really
happen
at
runtime,
so
maybe
that
last
box
might
not
be
a
component
at
the
next
cube
con
in
one
month
right.
B
A
So
kubernetes
is
changing
very
rapidly.
As
we
all
know,
we
cut
a
release
very
frequently
they're,
often
better,
but
but
sometimes
we've
we've
learned,
there's
a
lot
of
detours.
Some
changes
are
just
work.
We
we
have
a
lot
of
work
to
get
done
right
and
the
changes
duplicate
your
functionality.
Maybe
you
might
need
to
provide
a
shim
in
order
to
get
them
into
place,
so
your
users
can
start
using
them
but
participating.
Those
changes
can
really
make
things
a
lot
easier
for
our
users
and
for
everyone
and.
B
Then
I
just
add,
like
the
the
last
two
lessons
that
I
think
I've
learned
is
like
the
the
value
of
new
contributors
and
people
that
are
coming
into
the
community
and
new
are
invaluable
in
that
they
really
do
understand
better
than
the
lot
the
old
guard.
What
the
what
the
project
really
is,
and
then
this
this
tactic
that
we've
adopted
that
the
fast
forward
seems
to
be
working
very
well.
B
It
certainly
has
improved
everyone's
stress
levels
and
ability
to
land,
great
or
big
features,
so
we
effectively
declared
feature
bankruptcy
and
jumped
ahead
with
a
bunch
of
fast
forward
releases.
You
pay
for
it
and
cherry-picks,
but
it
does
seem
like
that
is
a
fair
trade-off.
It
seems
like
it's
better
for
users
who
are
getting
more
stable
releases
with
every
every
every
PR
that
goes
into
those
release.
Branches
is
very
explicitly
cherry
picked
and
looked
at
now,
and
it's
also
much
better
for
developers
and
maintained
errs.
A
So
yeah,
thank
you
all
for
all
your
contributions.
Thank
you
all,
for
you
know
working
and
using
cops
and
being
loyal
users
and
and
and
chatting
with
us
and
everything.
If
you
have
any
questions,
we
have
our
slack
Channel
just
to
make
sure
everyone's
aware.
We
also
have
a
development
specific
slack
channel.
So
if
you're
trying
to
create
a
new
PR
or
you
know,
build
on
cops
or
anything's,
we've
discussed
today
feel
free
to
talk
to
us
and
there
that
you
know
that's.
It
would
be
awesome
to
talk
to
any
of
you.
A
We
also
have
office
hours
every
other
Friday,
it's
a
little
hard
to
figure
out
exactly
what
time
and
in
place
so
just
use
that
QR
code
that'll
get
you
to
the
page
in
our
Docs.
That
explains
what
every
other
week
is
and
also
just
and
I
will
be
speaking
tomorrow
as
well
doing
a
deep
dive
on
some
more
of
this
stuff
yeah
and.
B
C
B
B
You
mean
the
binary
yeah
I
will
fix
that
you
can,
you
can
build
it
yourself.
I
think.
Is
the
I.
Don't
know
whether
we
don't
publish
as
a
release?
Yes,
I
will
fix
that
the
the
trick
that
I
would
add
is
you
can
actually
run
it
from
your
local
machine.
You
don't
have
to
run
it
on
your
your
cluster
itself.
It
talks
through
as
three
or
GCS,
so
you
can
run
on
your
local
machine.
So
it's
a
day.
D
B
So
I
think
I
think
node
pools
are
our
instance
groups
or
machine
deployments
in
that
you
can
create,
as
many
instance,
groups
or
machine
deployments
as
you
want
and
give
them
different
types
of
machines,
some
with
GPUs
and
with
not
some
spots,
I'm,
not
different
tags.
Different
labels,
different
taint
is
that
it
I
think
that
node
pools
are
the
gke
name
for
machine
deployments,
I.
Think
looking
for
other
Googlers
in
the
room
that
my
Pelton's
give
me
a
nod
or
not.
E
Thank
you,
I
think
it's
a
general
question.
How
do
you
validate
to
release?
How
do
you
test
to
release
to
gain
confidence,
but
it's
it's
stable
enough
for
production
and
and
I'm
so
impressed
by
the
cherry
picking
so
I
think
many
many
versions
of
communities
must
make
this
problem
works.
So
how
do
you
do
that?
That's
a
great
question.
A
Yes,
we
so
we
have
a
lot
of
tests
if
you've
ever
looked
at
the
PRS
you'll
see
that
there's
a
lot
of
different
tests
that
run,
we
have
an
end-to-end
kubernetes
test
that
runs
a
lot
of.
It
is
also
users.
You
know
we,
you
know
when
we
hear
you
know,
cutting
these
alphas
and
betas
should
even
help
even
more
and
and
so
that's
purposefully.
Why
we
try
to
keep
back
these
releases
sometimes
so
that
we
can
all
just
start
testing,
and
so
Justin
can
go
into
some
more
of
the
other
specifics.
A
So
that's
that's
at
least
part
of
my
philosophy
of
like
let's
get
these
alphas
and
betas
out
so
that
people
can
run
and
then
hey.
If
you
come
to
me
and
say
you
know,
we've
been
running
that
beta
for
a
week
in
our
test
environment
and
yeah.
We
had
these
few
issues
all
right,
let's
fix
it
and
then
let's,
let's
cut
the
production
release
once
we're
sure
that
that's
that's
good
for
you,
yeah.
B
I
emphasized,
yes,
we
do
have
these
automated
tests
and
there
are
PR
tests
and
there
are
periodic
tests
that
run
and
tests
are
reported
in
test
grid
that
cover
some
scenarios.
But
honestly,
we're
never
gonna
get
every
scenario,
and
if
you
have
a
staging
or
a
dev
cluster,
that
is
a
mirror
of
your
actual
configuration
running
it.
There
is
greatly
appreciated
and
also
gives
you
the
assurance
that
it
will
work
and
we
do
like
we,
we
stabilize
as
we
get
from
alpha
to
beta,
to
release.
B
C
B
It
should
absolutely
be
non
breaking,
it
should
be.
We
will
count,
it's
not
necessarily,
not
disruptive,
so,
in
other
words,
during
the
update,
you
would
expect
your
masters
to
you
have
to
bounce
all
your
masters
effectively,
but
you
but
the
but
it
is.
It
is
non,
braking
and
obviously
backup,
because
it's
always
challenging,
but
yes
it
should.
It
should
be
able
to
just
it
will
replace
your
TLS
certs.
It
will
move
you
to
the
recommended
version
of
that
CD
and
it
will
just
work
yeah
time
or
do
we
have
time
for
one
more?
B
F
So,
firstly,
great
job
on
cops.
We
use
it
to
run
lots
and
lots
of
clusters,
but
we
do
run
in
some
highly
regulated
environments
and
we
run
a
cube
bench
to
run
security
scans.
And
there
are
some
things
that
come
up
as
kind
of
red
flags
that
we
can't
change
in
cops.
So
I
was
just
wondering
if
you
guys
have
considered-
maybe
baking
that
in
or
coming
up
with
some
sort
of
reference
architecture
of
how
you
can
make
a
few
things.
Green
red
by
default.
A
Yes,
I
think
we
would
love
to
have
you
contributed
any
pr's,
yet
alright,
I
did
I
was
just
joking,
but
but
seriously
like
that'll,
be
fantastic.
Personally
I
run
Sona,
you
know,
I
do
so
knows
stuff,
so
we
don't
necessarily
need
some
of
those
high
level
things
in
place
today,
but
we
would
love
those
PRS
like.
If
you
have
any
ideas,
please
help
because
we
would
love
to
run
yes.
Yes,
yes,
yeah.
B
And
I
think
we
are
trying
to
get
to
secure
by
the
full
time.
We
are
the
back
of
the
room
now
and
it
would
be
great
and
if
you
have
particular
ones
that
are
higher
priority
for
you,
opening
like
an
issue.
Even
an
issue
is
wonderful.
I
think
we
do
actually
have
an
issue
which
this
the
cube
bench
output
was
that
you,
okay,
yeah
we're
trying
we're
trying
to
get
there.
B
It's
a
model
of
whittling
them
down
and
we
will
I
think
we
will
get
there
we're
just
sort
of
prioritizing,
but
if
there's
one
that
is
really
important
to
you,
let
us
know
all
right.
Well,
I
think
with
that
we'll
be
at
the
front.
I
can
I'll
be
at
the
front
as
answering
any
questions
and
we're
talking
about
things
and
otherwise.
Thank
you
so
much
for
coming.
Thank
you.