►
From YouTube: Deep Dive: kops (Cluster Lifecycle) SIG - Justin Santa Barbara, Google & Mike Splain, Sonos
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: kops (Cluster Lifecycle) SIG - Justin Santa Barbara, Google & Mike Splain, Sonos
We'll dig into the recent progress in kops: etcd-manager and etcdadm getting us to etcd3; support for CRDs and the server mode; starting to move to cluster API and our future plans here; starting to adopt bundles to allow for richer upgrades. Also kops is now a conformant kubernetes distribution! We'll discuss the status of our support for more clouds, including the recent OpenStack work. We'll also talk about some of our general initiatives e.g. secure by default. But most importantly let's talk about your pain points and figure out what you would like to see, and let's discuss how you can get involved.
https://sched.co/MPkU
A
A
Who
am
I
I'm,
Justin,
Santa,
Barbara,
Justin
sb
on
github
and
slack
I
am
a
cops.
Maintainer
I
originally
been
involved
in
Canaries
for
a
while,
particularly
on
the
ada
west
side
and
I
was
one
of
people
that
started.
The
cops
project
came
out
of
the
need
for
a
better
cube
up
script.
There
was
a
long
time
ago,
obviously,
and
I
work
at
Google
and
yeah,
pretty
active
generally
I'm.
B
A
A
The
goal
of
the
slides
is
basically
to
cover
the
questions
that
we
expect
that
you
know,
80%
of
people
will
have
and
sort
of
get
those
out
of
the
way
and
hopefully
leave
a
ton
of
time
for
Q&A.
So
we
can
really
dig
in
deeper
we're
gonna
cover
sort
of
the
the
new
release
thing
that
we've
been
doing
at
CD
3.
Some
of
the
work
on
CR
DS,
cluster
API,
add-ons,
the
bundles
and
operators,
the
conformance
work
and
the
ongoing
support
for
more
and
more
clouds
and
cups.
B
So
if
you
attended
our
session
yesterday,
some
of
this
might
sound
familiar
to
you.
But,
as
many
of
you
know,
we
really
try
to
make
sure
that
our
v-0
releases
are
stable.
That
is
one
of
our
core
competencies
of
cops.
We
want
to
make
sure
that
we
purposefully
release
after
kubernetes
releases,
because
we
want
things
to
be
stable
and
you
know
so.
A
The
consequence
of
that
which
I'm
sure
everyone
in
this
room
is
aware
of
is
that
we
were
lagging
further
and
further
behind
on
the
releases.
I
think
we
got
up
to
I,
don't
wanna
admit
it.
That
looks
like
more
than
six
months,
but
I
must
be
misreading
the
scale
delay
between
the
kubernetes
release
and
the
dot
zero
cops
release
thanks
to
our
contributor
Peter
for
Peter
R,
for
putting
this
together.
A
B
Since
yesterday
we've
revisited
our
philosophy
on
all
this,
so
users
really
like
and
trust
that
our
releases
are
stable
right,
but
we're
gonna
start
cutting
alphas
and
betas
quicker
and
earlier
in
the
process.
So
when
kubernetes
cuts
a
major
release,
we
should
be
cutting
an
alpha
almost
simultaneously
right,
so
that
you
know,
if
you
really
want
to
try
it,
you
can
try
it
on
some
of
your
dev
or
test
clusters.
B
Things
like
that,
and
we
want
to
make
sure
that
you
know
you
can
promote
it
up
through
your
clusters
and
and
we'll
do
the
same
with
some
of
ours
and
that
way,
then
you
know
when
the
community,
we
all
feel
like
cops,
is
ready.
We
can
cut
the
stable
release
still
in
the
normal
time
frame
that
we
aim
for,
but
hopefully
quicker
than
we
have
been
recently.
B
The
problem
with
that
is
now
we
got
to
catch
up,
as
you
saw
on
that
graph,
so
we've
been
adding,
we
won't
be
adding
a
lot
of
new
features
in
113
and
114
112
we
already
cut,
and
but
lots
and
lots
of
features
should
land
in
115.
That's
what
master
represents
now
in
the
cops
project,
so
our
new
motto
is
going
to
be
to
cut
the
branches
and
start
the
alpha
and
beta
trains
rolling
as
soon
as
possible
and.
A
We
have
done
that
and
you
may
have
noticed
a
much
higher,
much
higher
rate
of
cherry
picks.
There
is
a
gorgeous
spreadsheet
in
the
sky
that
is
tracking
all
these
cherry
picks
and
after
we
give
a
big
shout-out
to
to
Mike
and
to
Ryan
Bonham,
who
together
basically
did
most
of
the
work
on
this
spreadsheet
to
actually
track
and
go
through
all
these
cherry
picks,
so
anything
lands
on
master,
that
is
a
a
bug,
fix,
gets
cherry-pick
to
possibly
all
four
of
the
active
release
branches
right
now.
B
So,
as
most
of
you
may
know,
Etsy
D
to
support
was
deprecated
in
kubernetes
as
of
1:14,
or
something
like
that.
You
guys
can
check
that
out
yourselves,
but
we
knew
that
we
had
to
get
everyone
to
Etsy
v3
fast,
so
we've
set
up
at
CV
Manager,
which
is
a
project
that
that
the
community
has
developed
mostly
under
Justin,
and
you
know
so.
A
Yeah,
the
other
one
of
the
big
news,
even
work.
One
of
the
other
big
things
we
have
been
working
on
is
replacing
the
aggregated
API
server
machinery
that
is
in
cups
and
is
sort
of
pretty
complicated.
If
you've
ever
tried
to
do
a
a
PR,
I'm
sure
you've
like
grappled
with
that
code
generation
where
we're
switching
to
CR
DS,
they
are
basically
a
better,
simpler
approach.
A
From
our
point
of
view,
in
in
cups
as
contributors
and
maintainer,
zits,
less
and
simpler
code,
many
fewer
dependencies
and
those
dependencies
in
particular
controller
runtime,
are
actually
intended
to
be
a
stable
API.
So
one
of
the
reasons
for
that
increase
in
delay
is
that,
with
every
release
we
had
two
vendor
in
a
lot
of
decade,
the
kubernetes
libraries
and
they
were
not
necessarily
stable
api's,
and
so
we
had
to
make
a
lot
of
changes.
Hopefully,
that
will
be
easier
going
forwards.
A
A
lot
of
the
doc
generation
should
be
simplified,
so
the
the
complicated
dock
pipeline
will
hopefully
be
much
easier
now
and
from
the
end
users
end
users
perspective.
If
you
want
to
still
see
our
DS
on
your
cluster,
it
is
much
easier
to
do
than
it
is
to
install
migrate,
API
server
so
generally
around
a
good
thing.
I
think
that
we
are
making
good
progress
on.
B
We
are
also
working
on
adopting
cluster
API
and
so
we'll
be
rolling
out
machine
machine
set
machine
deployment,
we'll
talk
about
how
that's
gonna
happen
in
a
moment,
but
for
the
most
part
machine
groups
instance
groups
excuse
me,
will
correlate
to
machine
deployments.
So
since,
as
we've
discussed
a
little
bit,
we
kind
of
got
ahead
with
some
of
the
developments
in
cops.
A
There
is,
there
are
some
challenges
with
this
cops
instance:
groups
are
sort
of
much
richer.
Today,
then
machine
deployments
are
in
the
open
source
cluster
API.
You
know,
we've
had
a
number
of
years
of
building
up
the
functionality
of
cops
instance.
Groups
which
hasn't
necessarily
machine
deploy
in
seven
history
had
the
same
amount
of
time
and
PRS
against
them,
and
so
the
approach
we're
taking
is
this
one?
A
When
you
create
a
cops
instance
group,
there
is
a
controller
which
goes
and
creates
a
machine
deployment
which
backs
that
the
goal
is
that
we
get
to
a
point
where
users
can
choose.
You
can
either
create
an
instance
group
which
sort
of
will
have
the
full
functionality
or,
and
eventually
the
world
machine
prints
will
catch
up
or
you
can
create
a
machine
deployment
if
you
are
within
the
machine
deployment
area
of
functionality.
A
The
intention
is
that
the
same
tuning,
though
that
works
with
cluster
API,
because
that
machine
deployment
is
still
there
we'll
be
able
to
work
against
the
machine
deployment.
Even
though
that
machine
deployment
is
still
being
managed
by
an
instance
group,
Azure
and
I
guess
we'll
see
if
that
approach
becomes
widespread,
it's
still
early
days
for
cluster
API.
So
it
may
be
that
this
is
something
that
people
must
go.
Do
and
I
think
we
finally
get
cops
server
at
this
point
so
see
our
DS
cup
controller
cluster
API,
and
we
basically
have
this
idea.
A
That
frame
has
been
watching
Cup
server
wondering
when
it
will
ever
land
I
think
we
will
finally
be
able
to
get
it,
get
it
going
so
you'll
be
able
to.
If
you
want
to
to
have
a
communities
cluster
to
manage
other
kubernetes
clusters,
which
we've
been
talking
about
for
longer
than
I
care
to
admit,
probably
years,
but
that
should
actually
land
this
this
year.
B
So
we're
also
working
on
bundles
support
and
operators
bundles.
Currently,
we
have
a
lot
of
cops
add-ons.
The
versions
are
baked
into
cops
itself,
so
it
you
know
if
one
of
the
CNI
providers
needs
a
critical
patch
or
anything
like
that.
Yeah
users
can
kind
of
work
around
it,
but
in
reality
we
would
have
to
cut
a
new
version
of
cops,
so
we're
gonna
be
abstracting
that
out
into
bundles,
which
allows
for
controlled
releases
of
add-ons
without
doing
the
full
release
operators.
B
A
That
brings
us
all
nicely
to
this
next
sort
of
idea,
which
is
that
we
are
moving
from
a
world
where
you
know
cups
were
sort
of
its
own
project
and
when
you
adopted
cups
everything
that
you
we
had
our
own
pieces
for
everything
and
we
had
to
maintain
them
all.
And
you
know
if
you
were
using
pieces
from
community,
but
we
weren't
doing
the
same
thing
as
cops:
we're
moving
from
that
world
to
a
world
where
we
are
contributing
to
and
leveraging
community
components.
So
cluster
API,
hey!
Thank
you.
A
B
We
are
also
conformant,
and
this
is
kind
of
funny
because
we
never
really
like
we
don't
put
it
on
the
repo
or
anything
like
that,
because
technically
the
B
conformance
tests
are
part
of
the
end-to-end
suite
within
kubernetes,
but
we
now
are
actually
conformant
in
the
repo
and
we
have
the
badge
there.
We
are
test
against
110
111
112,
but
we've
always
been
conformant,
as
many
of
you
will
know,
based
on
our
tests.
So
yeah
now
now
we're
on
the
list.
We.
A
Exactly
and
we
are
adding
a
lot
of
support
for
more
clouds,
a
degress
is,
as
I'm
sure
you
know
the
most
reliable,
but
hopefully
this
pretty
soon.
We
should
be
able
to
get
GCE
support
from
beta
to
stable
it's
as
I'm
now
employed
by
Google.
That
is
an
important
thing
for
me
to
do.
We
have
some
great
contributors
doing
some
wonderful
work
on
OpenStack
and
on
alley
cloud
also,
special
thanks
to
all
of
them,
and
there
are
some
clouds
which
probably
could
use
a
little
bit
of
love
right
now,
so
digital
ocean
and
vSphere.
A
If
you
are,
if
you
know
those
clouds
and
would
like
to
get
involved
getting
involved
in
in
making
those
work
better,
that
would
be
wonderful,
there's
also
an
opportunity
on
those
on
both
of
those
to
maybe
prototype
deeper
cluster
API
adoption
they're,
not
necessarily
as
actively
used,
and
so
it
could
be.
If
you
wanted
to
get
involved
and
take
on
a
pretty
meaty
project,
that
would
be
a
great
place
to
just
start.
Yeah.
B
And
all
of
those
projects
they're
all
kind
of
feature
flag
to
the
side.
So
it's
actually
a
pretty
low
risk
thing
to
start
getting
involved
in
because
you're,
probably
just
gonna,
make
it
better.
You
know,
and
so
we
also
wanted
to
just
say
thank
you
to
all
of
our
contributors
and
and
all
of
you
for
coming
out
and
working
with
us
on
cops
without
you
guys,
like
cops,
wouldn't
be
where
it's
at
now.
We
really
really
like
real
I
on
the
commute
community,
because
you
know
none
of
us
do
this
as
our
day
job.
B
This
is
kind
of
a
tool
that
helps
me
do
my
day
job
you
know.
So,
if
you
guys,
if
any
of
you
are
working
on
cops
and
have
questions
the
cops
users
slack
channel,
we
also
have
a
development
channel.
So
if
you
have
questions
about
working
on,
PRS
and
whatnot
feel
free
to
go
there,
and
we
also
have
office
hours
on
odd
weeks
of
the
year
at
Friday's,
at
noon.
On
Eastern
Time
9:00
a.m.
B
A
B
A
So
yeah
the
yeah
okay
good.
We
made
it
through
in
about
13
in
a
bit
minute,
so
that's
pretty
good
a
little
bit
faster
than
we
thought.
So
we
have
plenty
of
time
for
questions.
Hopefully
we
covered
off
like
the
basics,
at
that,
at
the
lab
that
level
we
sort
of
level
set
and
got
everyone's
first
questions
out
the
way
or
prompted
more
questions
and
probably
more
likely,
but
I,
don't
know
if
over
over
two
people,
what
do
people
want
to
work
on
get
involved
in?
Do
you
have
questions
or
feedback
about
cops?
C
A
Think
we
talked
about
this
a
little
yesterday,
so
thank
you
for
bringing
us.
Yes,
I.
Think
it's
a
great
idea.
We
don't
have
plans
for
it.
I
think
it
would
work
very
well
to
support
that
as
an
option.
We
we
have
this
role
and
we
have
like
masters
and
nodes
and
we
have
Bastion.
So
this
could
be
a
fourth
role.
I
think
I.
A
A
B
I
think
just
to
hit
on
that
real
quick
Justin
touched
on
one
thing
that
I
think
is
critical.
A
few
people
talked
to
me
yesterday
about
this
about
like
getting
involved
in
contributing.
So
just
remember
when
we'd
say,
like
we'd
love,
to
have
your
contribution,
we're
just
saying
that
you
know
it
better
than
we
do
in
many
of
these
cases.
The
thing
you're
trying
to
solve
for
so
and
and
don't
just
go
open
a
PR
with
like
here's,
my
giant
code
change
and
it's
like
whoa
whoa.
B
You
know
so
open
an
issue
start
communicating
with
us
that
way.
We
can
help
you
make
sure
that
it's
done
the
right
way
in
the
best
way
for
everyone
right.
So
just
communication,
early
on
in
your
in
your
thought
process.
We
love,
we
love
new
ideas,
we
love
new
contributors,
so
open
issues,
slack
us,
etc.
Yeah.
A
My
rule
of
thumb
for
a
PR
is,
if
you're
unhappy,
if
you're
told
that
it
needs
to
be
completely
reworked.
Then
probably
it's
too
big,
but
if
it's
the
sort
of
thing
where
it's
just
like
I'm
gonna,
throw
it
up
there
and
like
it's
a
good
way.
It's
a
great
way
to
show
people
concretely
what
you're
talking
about
so
as
long
as
you're
prepared
to
throw
it
away,
then
by
all
means
send
to
PR.
A
A
E
Hi
I
was
wondering
how
quickly
bundles
are
coming
and
if
it's,
how
quickly
bundles,
are
coming
because
I
mean
when
now
we
can
surface
some
of
the
options
for
the
like
the
cni
plugins.
So
there
is,
we
could
do.
We
could
work
on
that.
We
could
wait
for
bundles,
more
or
less
it's
just
a
rough
estimate.
Yeah.
A
I
think
there
is
a
PR
that
is
going.
That
is
open,
that
is
target
at
115,
because
the
master
branch,
it
is
feature
Flajnik
still,
which
means
that
we
will
have
to
double
maintain
the
bundles
for
a
while.
We
have
the
version
of
the
manifests
it's
baked
into
the
cups
binary
itself
and
will
separately
have
the
versions
that
are
based
on
a
Yama
file
that
can
be
updated
separately
from
a
cops
release.
That's
not
ideal,
but
it's
a
ideal
way
to
introduce
it.
A
We
then,
in
addition,
have
to
add
the
support
for
patching,
so
a
lot
of
people
have
sent
a
lot
of
great
PRS
to
expose
additional
options
in
the
bundle
which
aren't
to
change
the
default
custom.
The
default
parameters,
for
example,
to
give
more
memory,
is
a
common
one
and
we've
sort
of
held
off
on
that
because
of
the
idea
that
bundles
and
operators
are
going
to
make
this
easier
until
we
get
bundles
and
operators
which
support
patches,
it's
not
going
to
be
there,
so
it
is,
it
is
further
off
and
so
I
think
we.
A
We
116
feels
like
we
might
get
our
first
operator
in
and
then
those
operators
will
support
patches.
We
might
be
able
to
do
something
for
patches
before
operators
as
well,
so
like
a
client
side,
cops
client,
side
patch,
but
yeah,
it's
at
least
115
116,
but
that
isn't
necessary
that
far
away
because
we
should
soon
the
intent
is
that
we
we
did
one.
We
did
113
beta
this
week,
114
alpha.
So
when
we
want
to
get,
we
want
to
try
to
catch
up,
I,
don't
think
we're
gonna
catch
up
with
115.
A
We
want
to
get
to
the
point
where
we
have
the
115
alpha
out
at
the
same
time
as
115
releases.
I,
don't
know
if
we'll
hit
that,
but
we're
not
going
to
be
too
far
off
so
it
isn't
too
far.
Although
115
feels
like
a
long
way
away.
If
you
think
about
cops
time,
it
isn't
actually
that
far
away
from
something
you
can
actually
run
it
at
the
alpha
or
beta
level.
Hopefully,
but
you
are
welcome
to
to
help
yeah
I.
D
A
Who
would
like
to
run
like
container
day,
for
example,
hands
up?
Who
would
like
to
run
for
to
the
other
ones
cryo
as
well?
Okay,
yeah,
it's
it
definitely
is
I
think
we
should
do
support
for
container
D
pretty
fast
one
of
the
things
that
that
we
do
that
we
benefit
from
is
when
GK,
for
example,
ships
something.
Then
they
put
a
lot
of
testing
into
it.
A
A
We
certainly
want
to
get
away
from
the
scenario
which
we've
had
to
date,
where
the
docker
builds
are
locked
into
the
cops,
binary
and
I
think
we
talked
about
this
yesterday,
movin
in
to
get
everywhere
so
ironically,
but
the
idea
of
putting
them
into
the
bundle
but
I
do
think
we
should
get
container
D
going
and
I
think
if
anyone
wanted
to
port
container
D.
That
would
be
a
wonderful
PR.
It's
it's
not
too
hard
to
swap
out
docker
for
container
D,
there's
actually
they're,
actually
pretty
compatible.
I.
A
Think
docker
is
technically
in
some
way
using
container
DM
to
the
car,
so
yeah
I
think
that
being
I
think
they'll
be
great,
can't
imagine.
Anyone
here
disagrees
with
the
idea
that
they
would
love
to
see
you
right
container,
decent
port,
but
now,
if
you
don't
I
will
also
do
it.
It's
it
would
be
great
yeah.
B
We
basically
said
well,
this
is
the
version
of
docker
you're
gonna
get
because
of
that's
the
version
of
kubernetes,
where
that
was
validated
against
right,
but
I
think
we
all
saw
the
exact
same
version
of
docker
ship
for
like
three
or
four
versions
of
kubernetes,
and
you
know,
there's
there's
reasons
you
might
want
to
upgrade
a
version
of
docker
and
and
say:
okay,
I'm
willing
to
take
on
the
risk
of
a
non
validated
version,
because
I
can
run
the
conformance
tests
or
I've
run
production
workloads,
that's
kind
of
good
enough.
You
know.
B
A
F
A
The
there
there
is
a
there's,
a
there
was
a
particularly
painful
occurred
with
the
HDD
migration,
and
hopefully
that
is
a
one-off
I,
think
we
we
sort
of
learned
as
a
community
that
we
should
avoid
upgrades
that
are
that
bad
in
future.
We
should
push
back
it
like
on
the
underlying
pieces.
If
that
happens,
because
they
really
like
had
a
ripple
effect
all
throughout
the
ecosystem.
I
guess
the
hope
is
that
bundles
will
allow
a
common
specification
for
what
goes
into
a
new
new
release
of
communities.
That
is
more
than
just
Cuban
eyes
itself.
A
A
The
idea
of
operators
is
to
give
some
like
right,
now,
sort
of
a
cops
operator
sort
of
fire-and-forget
like
we
update
the
we
update
the
launch,
template
or
the
the
specs,
and
then
we
and
then
we
we
also
updated
some
manifests
on
disk,
but
we
don't
really
check
that
it
worked
and
so
like.
If
cube
DNS
starts
crash,
looping,
for
example,
we
don't
have
any
way
to
recover
from
that
operators
do
allow
us
to
recover.
A
F
F
A
Sounds
like
that
yeah,
if
you
are
able
to
would
be
great.
Let's
have
an
issue
on
that
because
it
sounds
like
that
sound.
Like
a
bog.
It
sounds
like
something
I.
Hopefully
we
could
do
something
about
and
I
think
I
think
someone
hit
something
vaguely
related
where,
like
they
did
a
restore
infrared
CD
and
they
saw
weird
worried
there.
A
It
so
that
maybe
it's
related
to
that
was
a
that
was
a
more
serious
or
I
was
a
downtime
type
bug
where
they
did
a
restore
for
med
CD.
So
it
sounds
like
this
bug
might
actually
be
very
serious.
So
that's
well,
it's
yes,
but
it
sounds
like
it
also
might
have
a
more
serious
case,
so
that
would
be
it
we
great
to
get
details
from
from
both
of
you.
Thank
you.
C
G
Hi
I'm
curious
about
the
details
about
the
intervention
with
cluster
epi,
so
my
understanding
is
that
at
the
moment
cops
is
adopting
the
machining
API
from
the
classify,
and
so
you
mention
machine
deployments,
machine
sex
machine.
So
does
it
means
that
cops
is
running
the
machine
deploying
a
machine
set
controllers
from
upstream
and
the
AWS
actuator
from
a
piston
or
is
it
running
casting
implementation,
4000.
A
A
We
use
the
open-source
cluster
api
provider,
which
is
the
machine
deployment
controller
and
the
machine
set
controller.
We
use
currently
the
open-source
cluster
api
provider
GCP
controller,
except
that
we
have
patched
some
we've
added
some
extra
support
to
it.
So,
for
example,
it
didn't
support
specification
of
subnets
in
the
open
source
version,
and
so
there
are
probably
a
dozen
PRS
out
against
you
add
the
missing
support,
but
the
hope
is
that,
yes,
we
will
be
able
to
use
it.
A
The
the
unmodified
open
source
version,
the
big
change
we
had
to
make
was
to
allow
the
cop
there
was
to
allow
the
machine
to
specify
the
boot
up
script.
So
if
you
don't
specify
a
boot
up
script
by
default
because
it
goes
and
uses
cube,
ATM
and
assumes
like
a
crew,
vm
cluster
and
it's
sort
of
much
more
narrow
set
of
functionality,
then
we
need
to
support.
But
if
we're
able
to
override
the
boot
up
script,
then
we
can
essentially
call
in
to
the
cops
without
script.
A
A
Hopefully
we
can
use
that
same
approach
everywhere
and
yeah.
So
the
intention
yes
it
is.
It
is
an
open
question,
though,
in
cluster
API
like
how
are
we
going
to
have
a
cluster,
a
per
cloud
provider,
cluster
API
provider,
X
support
like
cubed
and
cube
atom
and
cops
node
up
and
other
bring
up,
tooling
I.
Think
this
approach
is
a
possible
way
to
do
it.
The
downside
is
that
you
have
this
big
ugly,
relatively
big,
ugly
script
now
in
your
machines.
So
it's
it's
harder
to
imagine
how
you
would
create
a
machine
deployment
from
scratch.
A
Today
we
probably
have
to
go
in
and
like
populate
the
script
in
there.
If
it's
a
cop's
cluster,
it's
not
ideal,
but
we
can
that's
one
of
the
things
I
think
you
know,
cluster
API,
I,
don't
know
how
maybe
we're
in
that
talk
to
us
now,
but
cluster
API
is
a
ongoing
thing.
It
just
did
v1
alpha
1
and
is
still
very
much
a
alpha
product
and
I
I'm,
actually
optimistic
that
getting
it
into
cops
will
help.
B
Imagine
the
same
things
with
your
auto
scaling
group,
so
you
make
a
change
you
we
create
a
new,
auto
scaling
group
underneath
scale
up
scale
down.
You
know
one
of
the
classic
problems
that
a
lot
of
people
run
into
is
you
know
they
want
to
roll
over
their
instance
groups
and
cops,
and
now
you
know
in
theory,
as
we
go
along
this
path,
we
can.
We
can
make
that
happen
without
having
to
do
a
lot
of
the
legwork
ourselves
and.
A
H
Hi,
thank
you
for
the
Q&A
session
here
and
there
was
recently
a
larger
outage
of
the
quiet.
I
ho
registry
and
we
ran
into
some
issues
with
starting
up
clusters
and
scaling
clusters,
and
is
there
a
way
to
somehow
mirror
the
images
that
are
pulled
or
put
them
on
one
read
registry
I
think
they
are
currently
spread
to
cry.
Oh
geezr
and
different
ones.
H
A
So
there
there
is
initial
support
for
mirroring
of
of
images.
I
don't
know
if
it
works
to
be
honest
it,
but
it
we
yeah.
We
have
a
lot
of
the
hooks
in
there
to
rewrite
the
images,
so
we
you
would
be
able
to
do
that.
It
could
use
some,
certainly
some
testing
and
some
love,
and
if
you
want
to
do
that,
that
would
be
fun
to
try
it
out.
That
would
be
great
I'm,
not
even
sure
you
actually
can
try
it
out
right
now
in
Russia,
we've
done
like
how
much
we've
really
done.
A
This
is
one
where
container
D
really
unlocks
a
lot
of
potential
here
so
container
day
lets
you
set
a
mirror
for
any
any
host,
I,
guess
or
any
registry.
So
you
can
set
a
mirror
for
k
for
k,
io
q,
yeah
yeah,
the
which
would
then
be
your
local
one
or
I,
don't
know
if
it
supports
an
s3
bucket
as
though
as
the
backend,
but
you
could
have
it
your
own
mirror.
A
Docker
does
not
allow
you
to
do
that
for
reasons
that
are
unclear,
but
container
D
does
allow
you
to
do
that
today.
The
container
deed
code
will
always
fall
back
to
the
K
key.
Do
like
canonical
source,
which
some
people
maybe
don't
want,
but
I
think
we
could,
if
that's
a
problem
for
people
who
could
probably
address
that,
but
you
could
certainly
have
for
the
availability
case
you
could
certainly
address.
You
can
certainly
have
additional
mirrors
that
that
can
cache
and
preload
this.
B
And
I
know
that
I've
heard
from
a
few
people
at
this
cube
con
alone.
Even
talking
about
that
that
you
know
they're
running
cross
region
and
they
just
would
rather
not
be
pulling
it
from
a
separate
region
and
all
that
stuff.
So
there's
definitely
a
lot
of
people
out
there
that
share
similar
problems,
and
you
know
we
can't
always
solve
all
of
them
right
away.
But
if
we
can
try
to
like
pull
everyone
together
and
see
like
who
the
the
core
people
are,
then
let's
come
up
with
the
plan
and
let's
get
it
implemented.
B
I
A
I
agree:
we
should
that
we
should.
We
should
do
that.
We
should
we
should
support
it
yeah
technically,
we
have
for
now
released
one
twelve
and
one
clear
one
we
have
113
beta,
so
we
don't
want
to
add
features
to
that.
Is
it
a
bug,
fixes
it
a
feature?
Yes
I
think
we
should
start
by
getting
it
into
the
master,
and
then
we
can
debate
how
far
back
to
cherry-pick
it.
I
think.
D
A
So
it's
it's
a
delicate
balance
and
it's
it's
sometimes
worth
holding
the
line
just
to
avoid
the
temptation,
because
everyone
wants
everyone
wants
it
right
and
it'll
be
great,
but
let's
start
by
getting
it
into
master,
and
it's
it's
actually
great.
If
I
don't
wants
to
contribute
this
another
great
one,
the
but
essentially
mirroring
what
is
done
in
the
I
guess,
still
the
cube
up
scripts,
the
clusters
directory
and
you
have
to
change
the
DNS
IP
but
other
than
that.
B
And
I
agree-
and
this
is
this-
is
one
of
those
things
that
we
will
often
hit
in
like
in
our
office
hours,
where
someone
will
say:
okay,
you
know
we
want
to
get
this
feature
in.
It's
like
okay.
Well,
let's
make
sure
the
code
works
and
it's
you
know
we
get
it
merged
into
master
and
then
we
kind
of
bring
it
to
the
room,
and
you
know
in
office
hours
and
we'll
say:
okay,
we've
heard
a
lot
of
you.
You
know
a
lot
of
people
want
this.
B
What
risk
are
we
willing
to
take
you
know,
and
so
maybe
at
that
point
once
once
the
codes
in
then
we
can
have
that
discussion
and
say
well.
Do
we
want
to
say
maybe
there's
Alfa
support
in
in
another
version,
and
we
cherry
pick
I'm
not
committing
to
that
at
all
I'm.
Just
saying
you
know
we
can
make
that
commitment
and
say.
Well,
if
you
guys,
you
know,
if
everyone
really
wants
to
test
this,
you
can.
But
it's
not.
B
You
know,
but
I'm
very
hesitant
to
say
that,
because,
as
Justin
was
saying,
we
like
to
say
that
you
know
the
beta
and
alphas,
you
know
our
feature
full
and
we
try
not
to
add
features
the
best
we
can.
But
you
know,
as
we
were
also
talking
about
earlier,
we
also
it's
a
fine
line
right
because,
like
we
want
to
make
sure
we
can
eventually
get
115
out
the
door.
So
maybe
that's
the
motivation.
Maybe
we
just
leave
it
in
115
and
we're
just
like.