►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cluster API Deep Dive - Jason DeTiberus, Equinix Metal & Marcel Mue, Giant Swarm
Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. In this deep dive, we will provide a brief overview of Cluster API and how it simplifies the cluster management experience for cluster operators by enabling consistent machine management across environments, and bringing declarative upgrades to Kubernetes clusters. You’ll also learn about the latest features and improvements to the project and what to expect in the future.
B
A
A
A
A
Additionally
class
api
aims
to
provide
building
blocks
for
higher
level
or
higher
order
functionality.
So
that
means
that
cluster
api
tries
to
set
a
level
of
groundwork
which
can
then
be
used
for
other
technologies
or
components
to
build
on
top
of
that.
So
maybe
features
that
could
be
built
on
top
would
be
more
automation
for
auto
scaling
or
better
repair
and
upgrade
functionalities.
A
A
So
there
can
be
different
bootstrap
providers
that
bootstrap
nodes
in
different
ways
and
cluster
api
aims
to
give
a
nice
interface
to
implement
these
different
providers
to
an
extent.
So
really
the
interchangeability
is
in
the
focus,
and
obviously
that
also
is
for
control,
plane,
management
and
other
possible
api
from
here.
I
want
to
quickly
go
over
how
cluster
api
looks
like
and
how
does
it
work
first
from
a
high
level,
and
then
we
will
go
deeper
into
the
individual
components
and
controller
managers.
A
So
from
a
very,
very
high
level,
we
can
see
that,
on
the
left
hand,
we
have
a
user
who
supplies
crs
so
on
custom
resources
to
the
custom
resource
definitions
that
cluster
api
kind
of
defines.
These
are
then
supplied
by
the
user
to
the
management
cluster.
We
have
a
cli,
that's
called
cluster
ctl
to
make
generating
vcrs
easier,
but
in
general
you
can
just
do
it
with
cube
ctl
inside
this
management
cluster.
There
are
then
controller
managers
which
in
turn
actually
manage
the
lifecycle
of
these
workload
clusters.
A
We
see
on
the
right,
so
we
have
components
running
in
the
management
cluster
which
the
user
interacts
with
and
those
result
in
workload
clusters
which
are
managed
by
the
management
cluster
seems
pretty
straightforward.
So
far,
if
we
take
a
deeper
look
now,
we
can
also
realize
that,
okay,
as
I
mentioned
before,
we
have
interchangeable
components
that
also
means
in
the
management
cluster.
We
can
have,
for
example,
multiple
infrastructure
providers
which
allow
us
to
manage
workload
clusters
on
different
infrastructure.
For
example.
Here
I
just
took
some
examples
of
the
available
infrastructure
providers.
A
Let's
take
a
deeper
look
inside
how
the
management
cluster
works
internally
or
what's
going
on
inside
there,
because
currently
we
just
define
it
as
kind
of
a
black
box
that
does
all
the
work
files
and
we
can
see
that
cluster
api
defines
currently
four
major
controller
managers
which
handle
basically
the
crs
and
the
reconsolation
of
these
crs
and
then
create
and
manage
the
lifecycle
of
the
workload
classes.
These
four
controller
managers
are
the
core
controller
manager,
the
bootstrap
controller
manager,
the
infrastructure,
controller
manager
and
the
control
plane
controller
manager.
A
Now
we
can
take
a
deeper
look
into
each
individual
controller
manager
to
kind
of
explain
the
structure
behind
cluster
api
and
how
they
work,
starting
off
with
a
core
controller
manager.
Each
block
you
see
here
is
one
cr
or
one
crd,
which
is
reconciled
by
the
core
controller
manager
and
therefore
each
block
is
also
managed
by
one
controller.
That's
running
inside
the
core
controller
manager.
A
The
first
block
we
see
here,
for
example,
is
the
cluster
block,
which
is
just
a
general
definition
of
water
clusters.
It
contains
some
metadata
about
the
workload
cluster
we
are
managing,
so,
for
example,
that
can
be
the
name
of
the
workload
cluster
is
stored
in
the
cid.
That's
just
called
the
cluster
crd.
A
You
know
fcr
in
this
case,
if
it's
the
instance-
and
we
can
also
see
that
we
have
some
crds
that
are
focused
around
machines,
and
here
you
can
see
an
interesting
pattern
that
is
kind
of
similar
to
how
kubernetes
does
it.
So
we
see
a
machine
deployment,
a
machine
set
and
a
machine,
and
you
should
very
easily
be
able
to
tell
that
this
is
similar
to
in
kubernetes.
You
have
a
deployment,
you
have
a
replica
set
and
you
have
a
pot.
A
We
can
now
also
continue
on
to
infrastructure,
and
here
we
see
the
first
difference
a
little
bit.
So
we
again
have
an
infrastructure
machine
which
is
actually
responsible
for
spinning
up
the
actual
infrastructure
machine.
So
if
we
go
to
a
real
world
example,
for
example,
on
aws
infrastructure
machine,
the
controller
manager
contains
the
controller
that
handles
infrastructure
machine
and
that
controller
will
be
responsible
for
every
infrastructure.
Machine
on
the
left
is
created
to
create
an
actual
machine.
A
In
your
infrastructure
provider,
for
example,
on
aws,
it
would
spin
up
an
ec2
instance
with
the
bootstrap
data
that
the
bootstrap
controller
has
provided
in
the
bootstrap
config.
So
we
can
now
see
that
these
components
are
interchangeable
and
that's
kind
of
logical,
and
we
also
see
on
the
right
that
we
again
have
an
info
machine
template.
What's
a
little
bit
different.
Now
is
that
we
also
see
that
there's
an
infrastructure
cluster.
A
What
that
or
why
that's
necessary
is
very
simple.
To
explain.
There
might
be
infrastructure
that
is
not
tied
to
a
machine
inside
a
humanities
cluster,
but
it's
just
like
relevant
to
the
cluster
or
needs
to
be
there
for
the
cluster
to
exist
so
clusters
not
only
a
set
of
machines.
For
example,
the
infrastructure
cluster
might
create
different
subnets
for
the
machines
to
then
live
in,
or
some
network
setup
or
some
load
balancers,
which
are
just
unique
to
one
cluster.
A
So
one
cluster
might
only
have
like
one
set
of
these,
but
they
need
to
be
created
as
soon
as
the
cluster
is
created
and
they're
not
directly
tied
to
a
machine.
So
you
can't
just
say
for
every
machine.
I
also
create
a
load
balance,
so
that's
that's,
usually
not
how
it
works.
So
so
that's
why
we
need
the
infrastructure
cluster
as
well
and
from
there
we
can
now
come
to
the
final
component
or
the
final
controller
manager.
I
want
to
mention
here,
which
is
the
control
plane.
A
Why
is
the
control
plane
kind
of
not
just
managed
like
a
machine
deployment?
That's
very
easy
to
tell
if
we
look
into
an
example
from
kubernetes.
You
know
that
a
deployment
you
can
just
scale
to
zero
now
for
machine
deployment
in
theory,
you
could
do
the
same.
You
could
scale
it
to
zero
or
scale
it
to
one
or
skeleton
two,
but
a
control
plane
is
usually
backed
by
lcds,
so
to
have
the
stateful
storage
of
lcd
for
your
master
for
your
control
plane.
You
are
running
these
these
cities.
A
In
the
background,
if
you
scale
down
your
ad
cities
to
zero
well,
your
data
is
kind
of
getting
lost
and,
if
you
scale,
for
example,
to
two
and
then
to
one
that
might
also
not
be
such
a
good
idea
with
ad
city.
So
there
are
some
very
practical
reasons
to
take
out
the
control
plane
component
because
it
needs
a
different
kind
of
management
than,
for
example,
just
a
machine
deployment
and
therefore
its
own
interchangeable
component.
A
And
now
we
kind
of
have
the
full
picture
of
cluster
api
on
a
theoretical
level
of
the
controller
managers
that
are
running
and
the
different
crds
that
are
provided
by
cluster
api
and
which
cigs
of
these
are
completely
replaceable,
which
would
be
the
bootstrap
infrastructure
and
control
plane.
Parts.
A
B
Thanks
again
marcel
for
this
demo,
I'm
going
to
actually
create
a
kubernetes
cluster
on
physical
hardware
that
I
have
here
located
with
me
in
order
to
do
that,
I'm
going
to
leverage
the
tinkerbell
project.
So
what
is
the
tinkerbell
project?
It
is
a
cncf
sandbox
project
that
is
looking
to
bring
cloud
native
infrastructure
management
into
the
data
center,
similar
to
what
kubernetes
does
for
applications.
Tinkerbell
is
trying
to
do
similar
for
the
actual
infrastructure
in
the
data
center,
and
how
does
it
do
this?
Well,
there
are
several
micro
microservices
that
comprise
what
we
call
tinkerbell.
B
The
first
is
the
actual
tinkerbell
service
itself.
It's
the
underlying
workflow
engine.
This
is
the
bit
that
actually
lets
you
define
what
hardware
you
want
to
manage
with
tinkerbell
and
allows
you
to
find
what
we
call
templates
and
workflows,
and
this
is
the
bit
where
you
actually
tell
the
hardware
what
you
want
it
to
do.
B
There's
also
the
boot
service
that
provides
dhcp
and
pixie
booting
services
for
the
hardware.
This
is
what
helps
get
the
actual
infrastructure
up
and
running
into
the
tinker
bell.
Environment,
there's
also
a
metadata
service
called
hegel,
and
this
allows
you
to
actually
define
metadata
on
an
instance
similar
to
what
you
would
have
in
a
public
cloud
provider.
B
You
can
define
user
data,
you
can
also
define
other
attributes,
and
then
you
can
actually
query
and
get
those
attributes
from
the
hardware
itself.
There's
also
the
worker
component,
and
what
this
does
is.
This
is
the
bit
that
actually
runs
on
the
hardware
reaches
out
to
the
tinkerbell
service
and
asks
it.
What
do
I
actually
need
to
do?
What
you
know
do
I
have
any
active
workflows?
B
And
how
does
it
look
when
we
start
putting
this
together?
Well
with
the
cluster
api
provider
for
tinkerbell,
we
provide
the
standard
resources
that
you
would
expect
out
of
a
cluster
api
infrastructure
provider.
We
have
a
tinkerbell
cluster,
a
tinkerbell
machine
and
a
tinkerbell
template,
but
we
also
have
a
shim
layer
in
there
that
provides
a
small
abstraction
layer
on
top
of
tinkerbella
itself
and
exposes
the
tinkerbell
resources
through
the
kubernetes
api.
B
With
that
out
of
the
way,
let
me
go
ahead
and
take
you
into
my
data
center
here
on.
The
right
here
is
a
small
form
factor
box.
That's
running
the
tinkerbell
services
itself
and
on
the
left
are
five
small
form
factor
machines
that
are
actually
going
to
be
the
available
hardware
for
our
kubernetes
cluster.
B
B
B
Now
to
get
started,
I
need
to
go
ahead
and
tell
create
a
hardware
resource
within
my
kubernetes
cluster
so
that
the
cluster
api
provider
will
know
what
hardware
to
use
for
that
so
that
the
initial
control
plane
instance
is
a
predetermined
machine.
I'm
actually
only
going
to
predefine
one
hardware
resource
right
now
and
I
will
define
the
rest
of
them
after
I've
had
that
hardware
defined
you
know
assigned
to
that
control
plane.
So
let's
go
ahead
and
do
that.
B
B
B
So
now
that
we've
already
defined
the
hardware,
let's
go
ahead
and
create
our
cluster
and
I'm
going
to
pre-define
a
few
pieces
of
this
here
as
well.
But
let
me
go
ahead
and
get
this
started
and
go
ahead
and
power
on
that
hardware
so
that
it
can
actually
start
bootstrapping.
While
I
talk
about
that,
so,
while
that's
booting,
I
can
go
ahead
and
see
from
the
command
line
here.
B
So
I'm
overriding
that
I'm
also
telling
it
that
I
only
want
one
control,
plane,
machine
and
I've
set
the
workers
to
zero
right
now,
because
I
haven't
predefined
that
hardware
yet
and
I'm
also
telling
it
that
I
want
kubernetes
version,
1.18.15
and
that'll
come
into
play
when
we're
trying
to
do
the
image,
lookup
and
I'll
get
into
that
in
a
bit.
So
let's
go
ahead
and
take
a
look
at
what
that
created
so
get
cluster
api
here,
and
you
may
see
that
there
is
some
content
going
on
on
the
monitor
behind
me.
B
B
So
in
here
for
these
resources,
we
can
see
that
we
do
have
a
tinkerbell
machine
here
now
and
it
is
saying
that
it's
ready
and
we
do
see
that
the
instance
id
matches
up
with
that
id
for
the
hardware
that
we
predefined
so
that
machine's
now
bootstrapping
and
getting
ready
to
go
at
the
moment.
So,
in
addition
to
those
resources,
we
did
also
create
some
tinkerbell
resources
as
well.
B
So,
in
addition
to
the
hardware
that
we
defined
earlier,
there
is
now
a
template
and
a
workflow
and,
as
I
mentioned
earlier,
the
template
actually
just
kind
of
tells
everything
what
to
do
in
the
workflow
ties
it
all
together
with
the
actual
hardware.
So
let's
actually
look
at
the
workflow
and
see
what's
going
on
there.
B
So
here
what
we
can
see
is
is
basically
in
the
spec
of
this
workflow.
We
see
that
it's
referencing
a
specific
hardware,
the
hwa
one
that
we
defined
earlier
and
it's
referencing
this
template
and
this
template
is
the
one
that
it
created
for
bootstrapping
this
instance,
and
we
can
see
that
here
in
the
status
we
see,
we
now
have
actions
defined
for
that,
and
these
actions
are
actually
going
to
feed
into
that
workflow
and
say
exactly
what
we
want
to
do
to
actually
bootstrap
this
instance
into
a
kubernetes
cluster.
B
So
the
first
action
that
we're
going
to
do
is
using
this
image
to
this
action
and
we're
telling
it
that
we
want
to
write
the
contents
of
this
image
url
into
this
destination
disk,
which
is
the
same
one
that
was
predefined.
So
if
I
had
specified
the
hardware
with
a
different
disk
device,
this
would
be
different
in
this
example
and
would
write
to
the
correct
block
device
for
a
different
type
of
hardware.
B
B
B
For
us,
the
next
thing
that
we're
going
to
see
is
that
we've
also
defined
a
default
user
here
with
some
pseudo
access,
and
that
gives
us
the
ability
so
that,
if
we
had
wanted
to
with
the
cluster
that
we
created,
we
could
find
an
ssh
key
for
this
user
or
a
password.
And
then
we
would
be
able
to
access
this
machine
if
something
had
gone
wrong,
but
I
have
trust
that
this
is
actually
going
to
do
the
right
thing.
B
B
B
B
B
But
I
don't
want
to
quite
wait
for
that.
Yet
let's
go
ahead
and
create
that
additional
hardware
that
I
had
mentioned
and
I've
already
defined
it
in
this
test
hardware
to
yaml
file.
So
here
we
created
hardware,
b,
c
d
and
e
and
that's
corresponding
to
the
other
instances
that
I
have
there
and
let's
go
ahead
and
scale
up
that
machine
deployment.
Now.
B
B
B
We
have
the
template
and
the
workflow
related
to
the
control
play
machine
that
we
previously
bootstrapped
and
we
have
similar
workflows
and
templates
defined
for
the
three
additional
instances
here,
and
you
know,
the
exact
same
process
is
going
on,
except
instead
of
doing
a
cube,
adm
init
this
time
it's
doing
a
join
just
for
a
worker
node
and
in
just
a
short
while
here
we
will
actually
be
able
to
see
those
nodes
come
up.
So
let's
go
ahead
and
run
that
get
nodes.
One
more
time.
B
All
right-
and
we
can
start
to
see
that
hardware
coming
up
now,
so
we
can
see
that
hardware
bees
already
starting
to
come
up,
and
I
expect
yep,
there's
hardware
d
and
hardware
c-
probably
isn't
very
far
about
behind
either
so
as
these
are
coming
up.
The
qubit
em
join
has
now
run
at
this
point
and
now
we're
basically
just
waiting
on
the
cni
resources
to
finish
coming
up
before
these
nodes.
B
A
A
So
the
next
upcoming
release
will
be
v1
alpha
4
or
version
0.4
in
most
likely
q2
2021.
The
focus
for
this
release
is
mostly
on
ux
and
stability,
with
inclusion
of
some
more
use
cases.
I
have
a
selection
here
as
well
that
we
will
go
over
and
previous
cube
cons
if
you've
already
seen
the
deep
dives
we've
talked
about
this
quite
a
lot
and
there's
now
a
lot
of
changes
involved.
A
Deploying
the
stack
of
controller
managers
for
cluster
api
can
be
a
little
bit
complex,
especially
if
you
want
to
update
the
version
of
cluster
api
or
you
want
to
kind
of
keep
in
touch
like
which
versions
are
compatible
and
so
on.
The
management
cluster
operator
aims
to
make
this
a
little
bit
more
simple,
so
to
make
the
deployment
of
the
components
of
cluster
api
more
simple.
A
By
having
like
an
easy
to
use,
cr
that
then,
through
the
management
cluster
operator,
deploys
the
different
controllers
that
are
necessary,
you
can
kind
of
equate
this
to
how
promethos
operator
works.
So
I
assume
most
of
you
know
from
ethos
operator
and
prometheus
operator,
really
makes
it
easy
to
deploy
pro
methods
in
a
simple
way
by
just
giving
a
high
level
abstraction
of
what
you
need
to
deploy
in
the
cr,
and
this
is
kind
of
the
similar
idea
here
to
make
really
ux
for
actually
running
cluster
api
a
little
bit
easier.
A
A
So
that
should
support
more
use
cases
in
this
area
and
then
the
final
point
I
want
to
touch
on
here
is
improved.
Multi-Tenancy
multi-tenancy
in
this
context
means
that
you,
for
example,
on
azure,
can
have
multiple
azure
subscriptions
and
the
workload
clusters
you
create
from
your
management
clusters
might
end
up
in
different
subscriptions
so
that
you
have
one
management
cluster
that,
for
example,
spawns
three
workload
clusters
in
three
different
subscriptions
for
azure,
and
this
is
currently
not
super
easy
to
do
so.
A
We
are
kind
of
overhauling
it
to
improve
the
ux
there
and
make
it
a
little
bit
easier
to
use
for
your
basic
users
of
cluster
api
and
in
the
bottom
the
slides
are
provided.
Obviously,
there
is
also
a
link
to
see
our
full
road
map
if
you
want
to
get
a
full
overview
of
what
all
the
changes
will
be
in.
We
want
alpha
4
or
the
plan
changes,
and
we
will
offer
4.
A
from
that.
I
also
want
to
quickly
talk
about
cluster
api
versioning,
so
there's
kind
of
a
discussion
going
on
right
now.
The
link
is
already
at
the
bottom.
If
you
just
want
to
click
the
link
and
check
it
out
that
kind
of
talks
about
okay,
cluster
api
version,
1.0
or
v1
beta
1
for
our
apis,
what
is
really
necessary?
What's
our
road
to
get
there
and
a
realization
that
you
know
maintenance
of
classifier
already
made
was
that
we
are
mostly
not
fully
already
operating
like
a
ga
project.
A
So
that
means
that
we
already
support
our
api
in
some
cases
for
up
to
a
year,
not
always,
but
there
is
some
support.
That's
nearly
a
year
in
most
cases,
and
we
also
support
conversions,
not
enough
conversions
like
a
ga
project,
but
we
support
conversion,
webhooks
and
the
conversion
between
our
apis
in
a
higher
level
than
what
would
be
expected
from
an
alpha
project
from
so
from
there.
The
proposal
was
kind
of
like
what
is
actually
missing
for
us
to
move
towards.
A
We
want
beta1
or
cluster
api
yeah
yeah
1.0,
and
in
this
discussion
that
was
already
brought
up
yeah.
There
needs
to
be
a
lot
of
work
on
stability,
even
though
we're
already
in
a
pretty
stable
place
and
the
foundations
are
being
kept
in
place.
We're
not
100
confident
right
now.
If
that
will
continue
to
be
that
way
or
which
additions
need
to
be
made.
A
So
for
this
it
would
be
really
valuable
for
us
if
you
would
bring
up
your
use
cases
or
your
ideas
for
cluster
api,
so
we
can
get
a
better
picture
of
what
actually
needs
to
be
improved
and
also
we
need
to
kind
of.
Oh,
there
needs
to
be
more
work
done
to
have
some
technical
depth
reduce
that
is
currently
still
in
cluster
api
and
then
the
final
things
I
wanted
to
say.
A
Yes,
obviously,
production
ready
means
different
things
for
different
people,
yeah,
so
currently
in
v1,
alpha
3
and
v1
alpha
4,
there
are
already
people
using
cluster
api
in
production,
and
this
is
a
thing
that
has
been
happening
for
quite
a
while,
but
the
version
does
not
reflect
that
properly.
So
we
want
alpha
4
we
own
alpha
3
indicates
basically
it's
not
a
production
ready
project,
but
it
is
being
used
in
production
and
with
our
work
with
conversions,
we
kind
of
acknowledge
that
it's
being
used
in
production.
A
So
for
that,
it's
very
interesting,
for
you
know
the
project
cluster
api,
to
kind
of
have
this
discussion
and
we'll
be
very
happy
for
you
to
join
in
and
transitioning
from
that
we
can
straight
go
into
well.
How
can
you
help
or
how
can
you
get
involved
in
cluster
api,
and
there
are
quite
a
few
different
ways
to
do
this?
The
first
way
would
be
you
know:
do
you
have
any
writing
skills?
We
have
a
documentation
book,
you
can
read
through
it
and
you
can
kind
of
improve
it.
A
However,
you
want
provide
a
pro
request,
that's
open
for
everyone
and
then
also,
if
you
have
product
skills,
as
I
mentioned
before,
we
are
very
happy
to
get
more
use
cases
and
and
to
kind
of
understand
what
users
of
cluster
api
might
might
want
or
might
expect
in
the
future
and
also
obviously
there's
backlog.
Grooming
is
always
a
thing
that
needs
to
happen,
so
we
have
some
milestone
maintenance
going
on
if
you're
interested
in
that
feel
free
to
contribute.
A
Obviously,
if
you
have
coding
skills,
there
are
different
ways
to
get
involved.
The
first
way
would
be
just
a
review
proquest.
We
currently
always
have
a
lot
of
open
pull
requests
and
reviews
are
really
valuable
to
us,
so
you
can
either
review
pro
requests
with
a
determination
or
the
goal
of
becoming
a
reviewer
in
the
cluster
api
project,
or
you
can
search
for
issues
that
have
the
label
help
wanted
or
good
first
issues
if
you
really
want
to
get
involved
in
coding
right
away.
A
So
these
are
two
great
ways
in
my
opinion,
to
get
started
and
then,
finally,
if
you
have
different
skills
that
I
didn't
mention
here
or
you
simply
want
to
get
involved,
we
have
weekly
community
meetings
as
well
as
a
slack
channel
and
a
mailing
list
where
you
can
easily
get
involved
and
or
obviously
on
the
cluster
api
repository.
There's
open
issues
where
you
can
just
open
an
issue
or,
like
start
a
conversation
again,
all
the
links
are
in
the
slides
as
well.