►
From YouTube: Kubernetes SIG Cluster Lifecycle 20171122 - Cluster API
Description
Kuberentes
SIG Cluster Lifecycle
Cluster API Breakout Session
2017/11/22
Meeting Notes: https://docs.google.com/document/d/16ils69KImmE94RlmzjWDrkmFZysgB2J4lGnYMRN89WM/edit#
A
Okay,
I'm
recording
welcome
to
this
week's
cluster
API
breakout
session.
It
is
November
22nd
on
Wednesday,
let's
get
started
so,
according
to
our
agenda,
I
had
the
first
item.
I,
don't
know
if
people
here
who
have
attended
this
meeting
are
particularly
interested
in
this
item:
I'll,
throw
it
out
there
and
see
what
people
think
about
it.
So
in
the
in
the
machines
API
and
in
particular,
where
it
crosses
over
with
the
cluster
API.
A
We
have
kind
of
an
interesting
I,
don't
know
if
I
want
to
call
it
a
wart,
but
maybe
a
temporary
placeholder,
so
in
order
to
define
the
version
of
the
control
plane
that
you
want
to
run
and
give
it
give
users
enough
control
to
write
interesting
software.
On
top
of
this
API
we've
broken
that
out
on
a
per
machine
basis.
So
if
you're
running
in
an
H
a
mode
rather
than
have
this
global
singleton
of
here
is
the
version
of
the
control
plane,
I'm
running
and
anytime
I
change
it.
A
All
of
my
masters
update
all
at
once.
We've
actually
added
this
to
the
machines
API
as
part
of
the
spec
as
an
optional
field.
That
is
only
set
if
the
machine
is
a
master
and
as
ignored
if
the
machine
is
a
node
or
is
in
error,
States
little
unclear
and
so
each
individual
machine
that,
as
a
master,
can
have
in
its
spec
the
version
of
the
control
plane
that
should
be
running
on
that
master
and
that
lets
you
do
a
slow,
controlled
rollout
upgrade.
One
master
make
sure
that
becomes
healthy
before
the
next
one
etcetera.
A
So
so
that's
already
there
what's
interesting
is
that
this
is
one
of
the
few
fields
that
you
can't
easily
check
from
the
machine
status.
So
the
machine
status
has
an
object
ref
to
the
node
that
fulfills
the
machine
and
every
other
field
has
some
some
representation
in
the
node
spec
or
status.
The
version
of
the
cubelet
that
contain
a
runtime
all
these
other
things.
A
You
can
just
look
at
the
object
in
the
API
server,
but
if
you
want
to
check
the
version
of
the
control
plane,
that's
actually
running
as
opposed
to
what
you
declare
to
be
running.
You
actually
need
to
connect
directly
to
that
machine.
You'll
find
the
node
identity
connect
directly
to
the
API
server
running.
On
that
machine
and
ask
it
for
its
version,
which
means
you
have
to
start
a
new
client
and
and
maybe
do
off
a
little
bit
differently.
So
it's
a
little
wonky
is
achievable
today.
A
If
you
care,
if
you're
writing
an
upgrade
tool-
and
you
say
I
want
to
upgrade
that
master
and
then
wait
to
make
sure
that
it
finally
gets
reconciled,
and
it
reports
that
it's
running
the
new
version.
But
it's
also
something
we
could
just
bubble
up
to
be
part
of
the
Machine
Status
itself,
and
so
we
can
have
a
controller
responsible
for
that.
That
writes
in
the
Machine
status.
And
then
a
client
doesn't
actually
have
to
worry
about
that
logic.
A
They
just
look
at
a
field
in
a
status,
so
there
were
a
few
people
that
might
have
been
into
this
I.
Don't
know
if
anyone
has
any
anyone
on
the
call
right
now
has
any
feedback
on
that
pattern.
If
we
do
want
this
to
be
part
of
the
status
or
if
it's
okay
for
clients
to
have
to
kind
of
jump
through
hoops
to
figure
out
if
this
has
been
reconciled
yet
and
if
there's.
B
I
have
one
comment,
and
that
is
the
fact
that
we
have
distinction
between
masters
and
notes
and
in
most
cases
they
might
be
mixed
between
between
both
of
those
and
in
some
cases
like
what
we're
currently
doing.
We
don't
even
have
master
notes.
We
only
have
to
work
a
lot.
So
can
this
be
useful,
for
example,
for
them
for
the
cubelet
versions
or
and
some
other
components,
or
it's
only
need
specific
for
the
master
notes.
A
Yes,
so
we
we
could
so
I'll
say
the
way
that
you
can
get
that
right
now
is
that
is
part
of
comes
either
the
notes,
spec
or
status.
It
is
there
right
now
because
you
can
like,
when
you
say
cute
couple,
get
you
nodes.
It
no
is
the
version
of
the
cube,
that's
running
and
and
I
believe
it
also
reports
the
container
runtime.
A
So
what
you
can
do
is
get
the
machine
objects,
look
at
its
status
behind
the
node
ref
and
then
read
that
and
then
find
where
that
that
parameter
should
be
too
to
see,
but
for
to
make
it
easy,
even
easier
on
clients
for
everything
that
we
set
in
our
spec.
We
can
also
auto
populate
in
the
status
and
say
here's
the
cubelet
version,
so
you
don't
even
have
to
traverse
nodes
or
Traverse
objects
to
get
the
node
and
know
exactly
where
to
look
for
it.
A
B
A
D
In
one
question
about
this,
so
what
I
understood
was
that
you
have
the
control,
plane
party,
the
machine
spec
and
then
that
gets
applied
to
the
master
node
like
is
there
like?
There
is
the
sync
process,
always
like
unidirectional
that
whatever
you
see
in
the
machine
is
fake,
it
gets
applied,
but,
like
user
can
always
log
into
the
machine
directly
and
do
stuff
like
like
or
like
you
know,
you
may
have
unattended
upgrade
on
and
that
basically
upgrades
you
cube
later
or
not.
A
Yeah
yeah,
so
we're
using
with
machines
is,
if
you
can
think
about
the
fact
that
today,
nodes
are
just
not
declarative,
they're,
pretty
much
only
the
status,
it's
just
whatever.
Like
cublas
reported,
you
can
look
at
the
entire
node
and
think
of
that
as
the
status
field
of
a
super
object
where
the
entire
specification
comes
from
the
master,
so
the
master
or
sorry,
not
the
nice.
A
The
machine
object,
so
the
machine
everything
in
its
spec
is
the
authority
of
what
you
want
that
node
to
look
like,
and
so
whether
the
the
VM
came
up
with
incorrect
settings
or
whether,
when
logged
into
it
and
manually
messed
with
things,
the
controller,
that's
responsible
for
rectifying
machines
which
right
now,
which
we
have
a
machine
controller,
it's
constantly
analyzing,
the
actual
state
of
the
world
versus
the
spec
and
the
spec
always
wins.
So
if
it
notices
that
the
the
node
that
is
supposed
to
be
fulfilling
this
contract
no
longer
matches
what
the
spec
is.
A
D
D
A
A
So
so
let
me
just
summarize,
like
the
current
state
of
the
world
and
if
there's
a
better
way
of
doing
it,
love
to
hear
the
feedback.
So
the
API
is
you
know
as
it
is,
is
deployment
agnostic,
but
the
tool
that
you
use
to
actually
create
the
cluster
that
supports
the
API
and
sets
up
the
proper
controllers
can
do
anything
it
needs
to
under
the
covers
to
maintain
its
own
state
to
do
additional
checks,
if
necessary
in
order
to
to
fulfill
the
contract
of
the
API.
A
So
in
our
reference
implementation,
we
have
a
prototype
right
now
that
works
against
GCE.
We
do
use
cube
admin
for
for
initializing
the
control
plane
and
bringing
up
nodes.
We
don't
explicitly
track
the
version
of
cube
admin
that
we
want,
but
we
figure
out
what
version
that
we
need
based
off
of
the
rest
of
the
settings.
A
So,
looking
at
the
version
of
the
cubelet
that
you
have
and
the
version
of
cuba
that
you
want
and
the
version
of
the
control
plane
that
you
want,
we
can
determine
what
version
of
cube
admin
to
use
now,
if
it's
so
there's
other
states
that
we
want
to
keep
track
of
as
an
implementation
detail
as
part
of
this
implementation.
So
what
we
do
is
add
annotations
to
objects
which
are
completely
freeform
and
anyone
can
add
whatever
annotations.
So
we
just
have
some
special
annotations
that
represent
the
state
that
we
want
to
capture
like
it.
A
When
we
create
a
VM,
we
add
some
annotations.
So
we
remember
exactly
where
we
created
that
VM
so
later,
when
we
want
to
delete
it.
We
remember
exactly
a
project
zone,
name
where
we
created
things
like
that.
So
what
you
could
do
is,
but
this
entire
pattern
allows
someone
else
to
write
a
different
cluster
management
tool
that
doesn't
use
cube
atom
at
all,
and
so
it
doesn't
really
need
a
first-class
high-level
representation
of
cube,
admins,
Virgenes,
declarative
part
of
the
state
or
tracking
purposes.
D
The
issue,
the
last
time
I,
was
working
on
this.
So
the
issue
was
that,
like
knowing
the
version
of
to
admin
was
a
problem
because,
like
right
now,
nobody
reports
it
from
inside
right,
like
from
the
because
I
was
told
that
the
cubelet
is
not
going
to
report
that
because
it's
like
a
layering
violation,
because
cubital
I
should
not
know
about
you
or
anything
else.
So
the
provisions
say
so
then
like
and
then
and
then
the
way.
I
understand.
The
Machine
controller
also
doesn't
have
any
idea
what's
running
on
the
machine.
D
So
what
I
was
doing
at
the
point
was
basically
run
like
a
demo
set
which
will
critically
check
that
cubed
b
cubed
en
version
and
then
like
signal
annotate,
the
node
object
for
itself,
just
so
that
we
can
keep
track,
because
if
you
said
I
mean
I'm,
just
trying
to
understand,
I
understand
the
overall
architecture.
Why?
A
Two
things
to
say:
one
thing
I
strictly
disagree
with,
which
is
actually
I
I,
do-
expect
a
fairly
tight
coupling
between
the
the
mechanism
used
for
installation
and
the
mechanism
used
for
reconciliation.
So
the
Installer
that
you
use
needs
fully
aware
of
the
Machine
controller
and
all
of
its
implementation
details,
and
they
should
stay
in
sync.
So
one
of
the
things
that
we
do
right
now
is
during
installation.
A
So
that's
that's
one
thing.
The
other
thing
is
from
from
a
client's
perspective,
if
you
were
integrating
on
top
of
the
machines
API
and
on
top
of
the
cluster
API
having
to
know
that
the
cluster
even
uses
cube,
admin
I
think
is,
is
an
anti-pattern,
and
if
you
have
to
worry
about
what
version
it's
using
I
I'm
going
to
assume
that
one
of
the
problems
you
might
be
hitting
is
saying
things
like
I'm
running
the
the
cluster
is
running
1.5
like
the
control
planes
1.5.
A
All
my
qubits
1.5
I
cannot
actually
go
directly
to
1.8.
I
should
use
a
much
safer
upgrade
strategy
and
what
we're
saying
as
part
of
the
cluster
API
is.
It
is
completely
fine
to
say
my
clip.
My
entire
cluster
is
running
in
one
point:
five,
one:
five,
oh
and
now
I
in
a
declarative
way,
I
want
to
say
I
wanted
to
be
running
one
800
and
the
implementation
underneath
that
should
know
the
safe
strategy
to
get
there.
A
So
as
a
client
on
top
of
the
cluster
API
I
just
say:
I
want
to
be
this
version
and
the
Machine
controller,
and
any
other
controllers
that
you
install
in
the
cluster.
As
part
of
the
cluster
setup
should
know
how
to
get
safely
from
point
A
to
B
or
should
report
an
error,
saying
sorry
I
can't
handle
that
I,
you
don't
have
to
support
every
arbitrary
combination
of
going
from
one
version
to
another
version
upgrade
downgrade.
A
You
can
just
say
sorry:
I,
don't
support
that
southern
air,
but
if
it
does
fulfill
that
it
should
know
how
to
do
it
in
a
safe
way.
So
it
should
take
care
of
I
know
how
to
get
from
one
five
to
one
six
and
then
okay,
I'm
checkpoint
on
that
one.
Six
now
I'm
going
to
go
from
one
six,
two
one
seven
etc,
and
do
that
completely
under
the
covers,
which
gives
the
client
on
top
of
the
cluster
API
a
lot
better
UX
without
having
to
worry
about
all
those
details.
D
One
part
I'm
sort
of
still
maybe
I'm
not
really
understanding
the
implementation
or
the
design.
So
the
Machine
controller
is
just
like
any
other
controller
in
the
common
I
discussed.
Alright,
it's
not
running
on
every
machine
it
just
it
just
watches
the
machine
objects
and
like
does
what
it
needs
to
do
right,
but
it
does
have
like
direct
access
to
the
machines
like
you
like
can
Italy?
Can
it
like?
Can
you
install
cubelet
or
is
it
like
basically
brings
up
a
VM
with
like
the
proper
startup
script
so
that
the
Kuban
is
there?
D
A
Can
actually
do
both?
So
not
only
is
it
responsible
for
creating
the
VMS,
which
means
that
it
has
full
control
over
what
their
boot
up.
Script
should
be,
which
it's
gonna
have
to
feed
the
correct
data
through
the
vm
metadata
and
then
short
popular.
Does
a
template
over
ever
to
make
sure
against
the
reversion
things
for
bootstrapping
purposes.
But
one
of
the
things
that
we've
done
in
order
supports
in-place
upgrades
I
wish
fun
was
on
the
color
I.
A
Don't
think
he
is
because
he
did
a
lot
of
work,
for
this
is
actually
have
a
mechanism
to
call
out
other
VMs
and
I
think
he's
just
using
SSH
now.
But
it
could
be
some
other
agent
that
we
installed
to
allow
some
sort
of
remote
command
or
it
could
be
something
like
a
daemon
set
to
run
directly
on
each
host.
But
we
are
actually
still
running
a
centralized
machine
controller.
A
But
if
we
detect
that
we're
able
to
do
an
in-place
upgrade
like
we
can
do
an
in-place
upgrade
of
the
cubelets
or
use
cubeb
and
upgrade
to
control
to
create
the
control
plane.
It
doesn't
on
demand
ssh
to
that
machine,
and
it
knows
that
it
can,
because
it's
fully
responsible
for
creating
the
VM
in
the
first
place.
So
it
knows
that
that
VM
should
have
SSH
and
it
should
set
it
up
in
such
a
way
that
it
has
credential
status
age
there.
But
the
fact
that
it
uses
SS
ages
implementation
detail.
A
We
can
swap
that
out
for
something
else,
but
it
does
do
on-demand
pushes,
which
means
that
it
could.
The
man
asked
that
house
tell
me
every
year
if
it's
in
a
Bluetooth
host
use
F
get
to
show
me
every
package
I
have
on
the
system,
so
I
can
do
more
client-side
or
in
the
controller,
to
figure
out
what
to
do
instead
of
as
pushing
commands.
I
can
also
retrieve
data
if
it
if
it
needs
to.
A
A
Okay
say
we
had
a
little
bit
of
digression
there,
but
it's
totally
interesting
and
I
love
talking
through
the
different
patterns
that
you
can
use
with
the
API
access
I
think
it's
so
powerful
that
you
could
have
completely
different
implementations
that
still
still
make
the
API
useful,
because,
as
a
user
on
top
of
the
API,
you
just
say,
hey
I
want
to
run
this
new
version.
I
think
that's
really
cool
any
other
thoughts
on
this
particular
topic.
I'll
write
down,
notes
afterward
before
we
move
on
to
the
next
one.
A
B
B
A
Very
cool
I
know
that
Robert
Bailey
from
our
size,
I
come
from
Google
and
Chris
Nova
from
Microsoft
are
going
to
present
at
least
what
we've
done
so
far.
The
cluster
API
in
this
coming
cube
column,
which
I
think
is
in
Austin
and
so
I'm,
not
sure
I
do
does
anyone
else,
because
I
know
a
lot
of
other
people
have
now
expressed
interest
after
we've
gotten
the
prototype
moving
and
have
more
solid
idea
of
what
we're
doing
does.
A
Ideas
of
things
that
they
want
to
build
using
the
cluster
API
or
do
they
want
to
integrate
the
cluster
of
machines
API
into
existing?
You
know,
cluster
controllers
like
or
cluster
set
up
tools
or
other
projects
that
they
think
are
on
top
of
just
the
fact
that
we're
doing
the
cluster
API
itself
or
do
we
just
want
to
do
like
another
status
report,
because
it's
after
this
cute
kind
of
hey
we've
made
this
much
progress.
It's
this
much
more
useful.
C
I'm
really
interested
in
understanding
using
the
cluster
API
with
packet,
which
is
where
I
am
at
it.
We've
been
working
with
various
people.
Doing
set
up
tools
to
understand
their
plans
for
using
the
cluster,
API
and
I
think
it
would
simplify
our
engagement
strategy
if
everyone
was
using
a
similar
API
rather
than
five
or
six
different
people
staying
in
the
same
tune.
C
A
So
correct
me:
if
I'm
wrong
and
I'm
hoping
that
this
I
don't
know
how
many
people
have
context
on
this
I
believe
packet
is
purely
bare
metal,
but
gives
you
kind
of
a
cloud-like
interface
on
top
of
it
for
remote
management?
Is
that
somewhat
fair
to
say,
I,
don't
know
how
dynamically
scalable
it
is,
but
at
least
gives
you
nice,
remote
management
of
bare
metal,
yeah.
C
C
We
do
a
provisioning
step
which
is
currently
somewhere
between
five
and
ten
minutes
to
go
from
bare
metal
to
a
login
prompt.
We
think
that
the
improvements
that
Google
is
made
in
the
BIOS
world
will
eventually
trickle
up
into
faster
boot
times,
but
that's
a
side
sight
issue
and
then
once
you're
once
you've
got
a
machine
provision.
We've
got
full
control
through
an
API
for
things
like
rebooting
it
or
destroying
it
or
or
if
you
want
to
scale
up
starting
another
one.
C
C
Cube
spray
I.
Think
isn't
that
list
too
there's
a
list.
It
keeps
growing,
as
you
discover
new
ones,
to
get
involved
with
people,
as
we
you
know
understand.
What
was
it
take
to
spin
things
up.
The
other
potentially
complicating
or
more
interesting
factor
is
that,
in
addition
to
Intel
based
servers,
packet
supports
ARM
based
servers,
so
I'm
very
interested
in
the
work
the
Lukas
is
doing
in
the
arm
world
and
he's
got
a
community
stuff
running
in
our
cluster.
A
Cool
yeah,
and
so
the
question
that
comes
up
a
lot
when
we
talk
about
cluster
lifecycle
and
especially
cluster
API
machines,
API
is
how
is
this
going
to
work
well
with
on
premise
or
even
bare
metal
machines,
and
so
having?
Maybe
not
production?
Were
they,
but
at
least
a
reference
implementation
of
here's?
How
you
can
do
it
using
the
pack
of
api's
I
think
would
go
a
long
way
to
proving
out
these
ideas
or
finding
even
more
edge
cases
that
we've
missed
in
the
initial
API
design
that
we
need
to
account,
for.
C
Absolutely
and
I
think
that
the
the
API
that
we
expose
is
complete
enough
that
that
it's
and
similar
enough
to
other
some
other
providers
that
are
also
doing
either
VMs
or
bare
metal
but
different
from
the
largest
providers
in
the
world
to
be
an
interesting
like
if
you're
gonna
do
things
three
times
to
say
that
you've
done
it.
Something
like
packing
is
probably
one
of
those
three
times.
A
B
A
B
In
maybe
could
actually
shows
from
different
vendors
or
on
different
platforms
and
cetera.
So
you
can
show
much
growth
and
cooperation,
so
hopefully
more
people
will
join
this
working
group
and
start
using
this.
So
yeah
I'll
try
to
submit
some
sessions
and
I'm
not
sure
they
will
be
related
to
the
Machine
API
and
we
will
save.
A
Cool
yeah
sounds
good
and
we
don't
have
an
official
working
group
for
this
we're
still
kind
of
spin
off
of
cluster
lifecycle.
But
if
anyone
has
ideas
or
is
looking
for
co-presenters
or
anything,
a
great
place
to
coordinate
is
either
this
meeting
or
just
a
cluster
lifecycle.
Mailing
list
feel
free
to
reach
out.
If
you
have
an
idea
or
you
have
a
half-baked
idea
or
you.
A
E
I
only
have
two
sinks:
I
mean
last
time
we
discussed
about
cube,
node,
set
I
think
two
days
ago,
there's
an
article
and
the
new
stick
which
described
it
also
more
in
details.
So
if
you're
interesting
in
it
we're
currently
also
looking
into
trust
idea,
of
course,
and
probably
on
cube
con,
we
can
also
discuss
where
are
potential
points
or
what's
the
best
way,
to
integrate
or
to
join
efforts,
because
of
course
we
don't
want
to
build
a
similar
thing
again.
E
The
other
thing
is,
we
would
have
before
C
cube
cons,
there's
meetup
from
Singh
who
away
VM
there.
We
are
doing
it
and
we
do
a
talk
about
cute.
Node
sets
there.
So
if
someone
is
interesting,
it
and
there's
an
Austin
yeah
feel
free
to
join,
and
then
we
can
also
discuss
about
our
ideas
and
where
we
want
to
go.
Was
it
and
yeah,
of
course,
commonly
as
I
said?
Main
intention
is
work
closely
with
you
guys
to
to
integrate
this
and
get
something
done
together.
A
B
E
B
A
E
A
Good
yeah,
perfect
cool,
so
we're
only
at
the
halfway
point.
If
there's
nothing
else
to
discuss,
we
can
call
it
early.
I
know
in
America.
We
have
Thanksgiving
tomorrow.
So
attendance
is
a
little
spotty
this
week
and
probably
a
lot
of
people
are
happy
to
get
to
their
long
weekend,
but
anything
else
before
we
should
have.