►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cluster API Deep Dive - Naadir Jeewa, VMware & Cecile Robert-Michon, Microsoft
The Cluster Lifecycle SIG is the Special Interest Group that is responsible for building the user experience for deploying and upgrading Kubernetes clusters. Our mission is examining how we should change Kubernetes to make it easier to operate. In this deep dive, we will examine how the Cluster API simplifies the cluster management experience for cluster operators by enabling consistent machine management across environments, and bringing declarative upgrades to Kubernetes clusters.
https://sched.co/Zevc
A
A
A
Okay,
so
when
talking
about
cluster
api,
it's
important
to
understand
that
it
is
a
building
block.
So
there's
this
project
called
cubadiem
or
you
know,
cube
admin
or
qdm
atom,
depending
on
what
your
kubernetes
pronunciation
game
is.
But
it
is
another
subproject
of
sig
cluster
life
cycle
and
what
qbm
does
is
it
takes
a
computer,
a
server,
a
machine
and
it
turns
that
into
a
kubernetes
node,
so
it
really
simplified
and
streamlined
the
way
that
we
build
kubernetes
clusters
in
kubernetes.
A
A
So
that's
what
cluster
api
is,
and
it
is
modular
which
means
that
you
can
replace
the
build
your
own
provider
for
bootstrapping,
for
example,
or
use
another
existing
provider.
Besides
the
reference
cube
adm
provider
such
as
the
talus
bootstrap
provider,
and
you
can
also
plug
it
in
with
any
infrastructure
provider.
So
that
means
it's
compatible
with
any
cloud
or
any
on-prem
environments
in
cluster
api
itself.
The
life
cycle
management
part
is
really
environment,
agnostic.
A
It
is
a
growing
list,
so
I
recommend
you
check
out
the
full
list
at
the
cluster
api
book
in
the
provider
section
and
if
you
can't
find
what
you're
looking
for
you
can
also
decide
to
build
your
own
provider.
We
have
some
instructions
on
like
a
provider
contract
on
what
is
needed
in
order
for
your
provider
to
be
conformed
with
cluster
api.
A
A
The
management
cluster
can
be
any
kubernetes
cluster
as
long
as
it's
kubernetes
conformance,
so
you
can
use
kind
or
you
can
use
a
managed
kubernetes
service
such
as
aks
or
ets
and
or
you
can
use
another
cluster
api
build
cluster,
and
then
your
target
clusters
are
what
runs
your
kubernetes
workloads
what's
really
interesting
about.
This
is
that
you
can
actually
have
different
environments
being
managed
from
the
same
management
cluster.
So
in
this
example,
we
have
azure
clusters
and
openstack
clusters
and
aws
clusters
and
vsphere
clusters
all
being
managed
from
a
management
cluster.
A
So
how
it
works.
Is
cluster
api
uses
crds
to
extend
the
kubernetes
apis
to
manage
clusters
themselves
and
their
machines,
and
it
uses
controllers
to
reconcile
those
crds
and
make
sure
that
the
actual
state
of
the
world
matches
the
desired
state.
If
you're
not
familiar
with
controllers
and
crds,
I
recommend
checking
out
the
cube
builder
book.
I
think
it's
a
really
good
introduction
to
the
basics
of
how
crds
and
controllers
work.
A
A
A
So
I'm
going
to
give
you
a
bit
of
an
overview
of
you
know
the
crds
and
their
specs.
I'm
not
going
to
show
you
all
of
them,
because
that
would
take
a
long
time,
but
just
to
give
you
an
overview
of
the
important
ones.
This
is
so
this
is
the
cluster,
so
the
concept
of
cluster
is
so.
This
is
in
cluster
api.
It's
the
cluster-wide
configuration
and
it
is
everything
that
is
really
generic
and
not
infrastructure.
A
Specific
things
like
pod
and
service
ranges,
and
then
we
also
have
the
associated
infra
cluster
and
the
infrared
cluster
is
per
infrastructure
provider.
So
usually
it
will
be
named
like
something
like
name
of
the
infrastructure
and
cluster
so,
for
example,
aws
cluster
azure
cluster
and
that's
really
the
provider
specific
cluster
configuration
so,
for
example,
your
aws
region
or
your
azure
location,
or
your
vnet
name,
or
things
like
that.
A
A
So
it
has
the
replica
count,
which
is
the
desired
number
of
control,
plane
machines,
and
then
it
has
two
references:
the
infrastructure
template,
which
is
what
the
template
of
the
infrastructure
machines
and
what
each
machine
should
look
like.
So
reference
to
that
template
and
then
the
cubadium
config
and
the
cubaedm
config
is
really
what
helps
you
configure
the
bootstrapping
and
the
cluster
configuration.
A
And
then
we
have
the
machine
deployment.
The
machine
deployments
are
if
you're
familiar
with
pods
and
deployments.
Machine
deployments
are
kind
of
the
same
thing
for
cluster
api,
with
machines
and
machine
deployments,
and
so
with
a
machine
deployment.
You
can
define
a
set
of
machines
that
would
have
a
similar
configuration
and
you
will
set
a
replica
count
similar
to
the
control
planes
and
then
you'll
also
have
the
machine
spec,
which
defines
what
each
machine
and
that
set
should
look
like.
A
Okay
and
then
we
have
a
machine,
so
a
machine
is
an
individual
object
in
a
machine
deployment,
or
you
can
also
decide
to
create
individual
machine
standalone
missions
just
like
in
kubernetes.
You
can
create
standalone
pods
that
are
not
part
of
a
deployment,
and
so
this
is
really
the
configuration
for
that
specific
machine
and
the
spec
looks
very
similar
to
the
machine
deployment
spec.
We
also
have
the
cubelet
version
and
the
infrastructure
reference,
as
well
as
the
bootstrap
config
reference.
A
So,
for
example,
if
your
provider
enables
spots
virtual
machines,
that's
where
you'd
configure
that
you'll
be
able
to
set
your
instance
type
or
your
vm
size.
Depending
on
what
cloud
language
you
speak
and
different
things
like
that
that
are
really
provider
infrastructure,
specific
okay!
So
now
I'm
going
to
head
off
to
india
who's
going
to
give
you
the
show
you
the
exciting
part,
the
demo.
B
B
Is
first
going
to
use
kind
to
just
have
a
bootstrap
cluster
that
I
can
work
with,
and
then
we
use
a
tool
called
cluster
custom
which
you
can
get
from
the
link
below
this
cluster
cuttle
in
it
installs
the
components
that
are
required
to
run
cluster
api
for
each
provider,
so
we're
actually
going
to
set
up
three
different
clusters
at
the
same
time
using
azure,
aws
and
vsphere.
So
this
really
shows
the
power
of
using
cluster
api
so
to
create
a
cluster.
We
use
a
command
called
clustercuttleconfig,
and
this
just
generates.
B
Kubernetes
manifests
templates
that
have
been
released
by
the
infrastructure
providers
that
help
you
get
started,
but
there's
nothing
to
stop
you
using
your
own,
manifest
using
the
specs
that
cecile
showed
a
few
moments
ago.
So
this
is
just
an
example
of
each
of
the
azure
vsphere
and
aws
templates,
and
they
all
have
common
components
and
then
each
custom
bits
for
each
each
individual
infrastructure
provider.
B
And
then
what
we're
going
to
do
is
we're
going
to
apply
this
to
a
bootstrap
cluster,
which
in
this
case
is
the
kind
one
we
just
created
just
showing
the
logs
from
each
of
the
sort
of
controllers
that
for
each
infrastructure
browser,
so
you
can
see
what's
going
on,
and
the
first
thing
that
happens
is
that
it
creates
the
sort
of
underlying
network
infrastructure.
That's
going
to
be
needed
for
the
clusters
usually
load
balancers.
B
If
it's
public
cloud
then
anything
to
do
around
creating
private
networks,
so
that's
vm
nets
in
azure
and
vpcs
and
aws,
and
we
can
also
just
use
normal
coupe
cattle,
keep
control
to
inspect
the
objects
that
are
created
by
each
of
the
cluster
api
components.
So
in
this
case
we're
doing
couple
get
kcp,
which
is
the
cube,
adm
control,
plane
management
and
the
various
vsphere
machines.
B
In
this
experience,
and
now
we've
had
a
problem,
is
that
one
of
these
azure
machines
didn't
come
up
and
that's
because
I
don't
have
enough
quota
in
my
azure
account,
because
I
started
it
several
hours
prior
to
making
this
demo.
So
what
I'm
just
gonna
do
is
just
edit
the
machine
deployment.
So
I'm
just
gonna
make
it
two
rather
than
three.
B
So
I
want
to
basically
upgrade
my
aws
cluster,
so
I
can
do
that
by
editing
first,
the
control
plane
and
just
changing
that
version
field.
So
you
can
just
change
it
from
b1175
to
b118.2
and
once
I've
saved
that
and
exited
you
can
see
that
what's
happening
is
one
of
my
control.
Plane
replicas
become
unavailable
because
it's
about
to
be
deleted
and
I've
got
four
replicas
rather
than
three.
B
So
it's
created
one
one:
new
118
control,
plane
instance:
it's
checking
fcds
working
making
sure
that
the
xcd
cluster
is
okay
and
then
it's
just
going
to
roll
through
each
of
those
control
plane,
instances
replacing
them
one
by
one
and
the
other
thing
we
can
do
here.
It's
like
I
can
do
this
independently
is.
I
can
also
upgrade
the
working
nodes,
so
I'm
just
going
to
edit
so
the
notes
controlled
by
the
machine
deployment.
B
So
I'm
also
just
going
to
change
that
to
1
18,
2
and
exit
whim,
and
you
can
see
that
nodes
are
going
to
not
ready
scheduling,
disabled,
so
we're
doing
they're
draining
that
node
making
sure
the
workload's
moving
off
then
terminating
that
instance,
and
replacing
it
with
a
new
one,
and
you
can
control
using
the
standard
sort
of
max
burst
sort
of
settings
that
you're
familiar
with
in
pod
based
deployments.
You
can
apply
those
there
to
control
how
many
machines
get
replaced
at
once.
B
Finally,
we're
just
going
to
show
this
pivot
method,
so
what
you
might
want
to
do
is
you've
created
a
cluster
with
cluster
api.
You
want
that
to
be
the
management
cluster
for
all
of
the
other
clusters.
So
what
cluster
cuttle
move
does?
Is
it
pauses
all
of
the
resources
on
this
initial
kubernetes
cluster
and
after
we've
installed
one
cluster
cutter
in
it
on
the
aws
cluster,
which
is
where
I'm
going
to
move
everything
to?
B
You
can
then
run
classical
move
and
it
basically
deletes
the
objects
from
the
old
cluster
and
puts
them
into
the
new
one
in
this
paused
state
and
then
unpauses
it.
So
then
those
then
you
can
continue
with
ongoing
management.
Finally,
you
just
want
to
show
how
you
might
want
to
do
development
cluster
api,
so
we're
big
fans
of
using
tilt
in
development
still
allows
us
to
essentially
do
hot
work,
hot
reloading,
as
as
you
write
code
using
cluster
api.
B
So
there's
a
json
file
that
you
can
see
in
this
vs
code
that
you
will
need
to
set
up
all
the
instructions
are
on
the
cluster
api
website
and
then
what
just
do
is
just
change
a
line
of
code.
You
have
you
can
view
all
the
logs
in
the
browser
as
well,
so
you
can
just
click
on
each
of
the
controllers
that
you're
working
on-
and
in
this
case
I'm
just
going
to
change
one
log
line
and
you
can
see
that
the
controller
got
rebuilt
and
has
started
again.
B
So
this
is
most
the
sort
of
plus
api
developers
are
using
this
tilt's
great
tool
and
for
kind
of
development,
and
we
use
it
extensively
in
cluster
api.
So
with
that,
let's
talk
about
what
we've
been
working
on,
so
we've
just
done
some
new
releases
of
cluster
api.
So
in
this
0.3
series,
we've
done
quite
a
lot
of
work.
One
of
the
things
we
just
introduced
was
the
control
plane
management,
which
I
just
demoed
a
moment
ago.
So
this
is
really
about
making
sure
that
we
can
do
day.
B
Two
operations
of
clusters
managed
by
cluster
api
in
terms
of
being
able
to
do
upgrades,
remediate,
control
planes
being
able
to
scale
them
up
so
maybe
started
off
with
a
demo
one
node
cluster,
a
one
node
control,
plane
cluster,
but
it's
actually
running
production
workloads
make
it
high
availability.
That's
what
the
control
plane
controller
can
do.
B
The
machine
pools
crd
is
a
new
thing
that
was
added
in
0.3.
It's
implemented
for
azure
right
now
with
aws
is
coming
soon
and
that's
where
you
have
cloud
providers
that
provide
their
own
constructs
for
sets
of
machines
and
provide
more
enhanced
functionality
through
those
such
as
in
the
case
of
azure
vm
skill
sets
or
aws
auto
scanning
groups.
They
can
be
used
as
a
native
construct
within
cluster
api
machine
health
checks.
Allow
you
to
give
say
plug
them
on
the
model
for
a
remediating
machine,
so
you
use
it.
B
We've
done
a
lot
of
work
on
testing,
so
we've
built
a
if
you're
writing
your
own
provider
or
you're
working
on
other
bits
of
kubernetes
such
as
csi
or
cloud
provider,
integrations
elsewhere.
Then
the
cluster
api
test
framework
is
provides
some
really
nice
helpers
and
abstractions.
For
you
to
just
start
a
kubernetes
cluster
run
some
tests
on
that
class
that
you
created,
tear
it
down
again
and
just
run
things
like
conformance
checks,
etc,
failure
domains.
B
B
B
B
So
this
work
is
ongoing
within
the
cluster
autoscaling
project
right
now,
so
it
will
be
able
to
control
machine
sets
in
cluster
api
generically,
so
it
doesn't
need
to
have
any
infrastructure.
Specific
information
and
you'll
run
this
as
one
part
of
cluster
autistic
new
management
plus
at
one
point
in
workload,
cluster
and
outdoor
testing
will
be
able
to
control.
This
site
will
be
able
to
scale
up
and
down
a
clusters
managed
by
cluster
api.
B
So,
given
that
like
there's
what's
next,
so
we
would
we're
hoping
to
reach
beta
status,
that
and
by
the
beginning
next
year,
so
we're
going
to
be
working
on
stabilizing
cluster
api
and
make
sure
it
meets
the
needs
of
everyone
who
wants
the
use
cluster
api
ongoing
ux
improvement.
So
we
did
a
big
rewrite
of
cluster
ctl.
B
We
want
to
improve
that.
Not
everyone
wants
to
use
cluster
ctrl.
So
how
can
we
improve
that
experience
overall,
better
management,
cluster
add-ons,
so
things
like
cni's
or
those
sort
of
core
dns
or
coupoxy
those
core
components
that
every
cluster
needs?
We
introduced
something
called
cluster
resource
set
experimentally.
So
that's
the
ability
to
as
soon
as
you
create
cluster
apply
a
set
of
manifest
to
two
clusters
based
on
some
labels
that
you've
seen
on
them.
B
Detections
and
machine
failures
prior
to
cluster
join.
So
one
of
the
problems
we
have
now
is:
we
only
really
know
the
status
of
the
machine
once
it's
joined
a
cluster.
But
what
happened?
How
do
we
know
what's
happened
when
when
it
hasn't
joined
that
cluster?
Yet
can
we
find
out
where
that
thing
went
wrong?
So
we're
going
to
be
doing
some
experiments,
some
research
into
finding
out
how
we
can
do
that
through
the
various
infrastructure
apis
that
are
given
to
us
load
balance
providers.
B
B
Moving
machine
pool
out
of
experimental
so
complete
the
implementation
for
aws
primarily
and
make
sure
that
that
abstraction
works
across
the
various
different
types
of
public
clouds,
support
for
windows
nodes.
So
there's
that's
going
to
start
off
with
one
of
our
system.
Projects
called
image
builder,
which
provides
is
provides
mechanisms
to
build
images
for
the
various
linux
distributions,
adding
support
from
windows
to
that
and
making
sure
that
we
have
a
windows,
specific
bootstrap
provider
that
can
bootstrap
windows
nodes.
B
Essentially,
and
then
finally,
for
particularly
for
aws
or
we're
interested
in,
how
do
you
deal
with
multi-tenancy
so
with
aws
and
as
you're
in
any
large
organization?
You're,
not
just
dealing
with
one
aws
or
azure
account
it's
going
to
be
lots
of
different
accounts
and
subscriptions,
and
you
want
to
provide
a
sing
single
place
where
people
where
teams
can
manage
their
clusters.
So
we
need
to
provide
native
ways
to
be
able
to
choose
which
accounts
and
subscriptions
with
with
the
right
levels
of
security
around
them.
B
So
that's
what
we'll
be
working
on
next.
So,
if
you'd
like
to
get
involved,
then
you
can
join
us
in
our
weekly
meeting,
which
happens
every
wednesday
at
10
a.m.
Pacific
time
that's
five
1700
utc,
and
not
only
this
is
just
for
the
main
cluster
api
meeting
and
what
you'll
find
if
you
go
to
that
link
you'll
find
that
each
provider
has
their
own
meeting
specific
to
those
concerns,
so
aws,
azure,
vsphere,
openstack,
etc
so
go
and
take
a
look
and
see
what
takes
your
fancy?
C
A
All
right,
that
was
a
lot
of
information.
I'm
sure
a
lot
of
people
have
questions
so
is
here
on
the
phone
as
well,
and
we're
gonna
try
to
go
through
as
many
questions
as
we
can,
but
there
are
a
lot
so
if
we
can't
get
through
all
of
them,
we're
happy
to
continue
the
conversation
in
slack
and
you
can
find
us
on
there's
a
thread
already.
I
think
in
the
kubecon
slack,
since
you
have
slack
under
maintainers
the
maintainers
track.
C
Inaudible
api
is
doing
a
lot
to
interrogate
the
status
of
clusters.
We
do
need
to
be
able
to
connect
to
the
control
plane
endpoint.
So
if
you're
running
private
vpc,
you
are
going
to
have
to
provide
some
sort
of
connectivity
between
them,
so
whether
that's
using
vpc,
peering,
direct
connect,
vpns,
etc.
C
A
Thanks,
okay,
I
can
take
one:
is
there
tooling
for
the
bootstrap
process
of
self-managed
cluster?
Yes,
and
no,
so
there
is
some.
There
are
some
commands
in
cluster
cuddle.
Cluster
cuddle
is
the
tool
that
nader
was
using
in
the
demo,
which
is
the
cli
tool
that
was
built
to
facilitate
the
management
of
the
cluster
api
management
cluster,
and
that
does
have
an
ability
to
move
all
your
resources
to
make
in
essentially
any
cluster
self-managed.
A
So
if
you
build
a
bootstrap
cluster
and
then
you
build
a
target
cluster
like
a
cluster
api
built
cluster
with
that
target
cluster,
then
you
can
then
use
cluster
cuddle
move
to
move
all
the
cluster
api
resources
from
that
original
bootstrap
cluster
to
the
target
cluster.
So
then
that
target
cluster
becomes
self-managed.
A
I
know
there's
also
talks,
there's
an
open
issue.
I
don't
remember
the
number
of
the
top
of
my
head,
but
about
having
some
declarative
management
of
the
cluster
itself,
so
you
can
have
a
self-managed
cluster
from
zero
to
self-managed
cluster.
So
that's
still
in
the
talks,
and
I
would
expect
also
any
platform
that
builds
on
top
of
cluster
api,
to
be
able
to
automate
that
cluster
api
is
really
a
building
block.
So
you
can
do
a
lot
around
it.
A
Yeah,
that's
a
great
point,
so
how
can
you
get
involved
in
the
development
of
cluster
api?
I
love
this
question.
So
we'd
love
for
you
to
get
involved
and
I
think
the
best
way
to
start
is
to
join
the
different
communication
channels.
So
you'll
want
to
join
the
kubernetes
slack
if
you're
not
already
in
it
and
there
is
a
dedicated
cluster
api
channel
and
there
are
also
channels
for
each
provider.
A
A
C
I
think
it's
fun
to
find
a
question
about
how
the
plus
api
relate
to
other
projects
such
as
metal,
cube
or
ironic
or
tanzania.
So
these
all
plus
a
guy
is
very
much
a
building
block
and
we
fully
expect
people
to
build
other
things
on
top
of
it
and
metal
cube
is
an
example
of
using
plus
api
to
provision
to
bare
metal
tanzooz
enrolled
in
provisioning
using
clusterize
provision,
2
vsphere,
so
it's
very
much
a
fundamental
building
off.
C
A
A
It
is
still
like,
very
experimental
and
very
much
in
the
development
process,
so,
if
you'd
like
to
help
shape
it,
this
is
a
great
time
you
can
find
the
pr
in
the
kappa
repo
which
is
cluster
api
aws,
and
then
we
also
have
done
something
on
the
azure
side.
There's
an
experimental
feature
available
for
develop
for
deploying
sorry
aks
clusters,
so
that's
already
available
in
the
latest
release
under
the
experimental
folder,
so
check
that
out
all
right.
I
think
that's
our
timely
there.