►
From YouTube: Build Your Own Cluster API Provider the Easy Way - Anusha Hegde, VMware & Richard Case, Weaveworks
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Build Your Own Cluster API Provider the Easy Way - Anusha Hegde, VMware & Richard Case, Weaveworks
Over the past year, the adoption of Cluster API (CAPI) has been growing with more end-users using it to provision their clusters. And increasingly it’s being adopted inside commercial products and other OSS projects (e.g. EKS-Anywhere). With this growth comes an increase in the variety of the Cluster API Providers you can choose from. What if none of the existing providers suit your use case? Perhaps you want to bring your own hosts or integrate with a custom infrastructure provisioning mechanism. If that's the case, this talk is for you. Come learn from the maintainers of existing CAPI providers on how to get started creating your own provider. It’s hard but at the same time easier than it sounds. Although every provider has unique considerations in its offering, there is a lot of commonality when it comes to writing a provider. This talk will highlight the common patterns, develop and debug workflows, and common pitfalls / gotchas to take into account when writing your own provider.
A
A
B
B
So
it
was
originally
designed
with
the
the
premise
that
actually
provisioning
clusters
and
it's
that
the
life
cycle
of
those
classes
is
actually
difficult.
There's
been
many
many
ways
to
provision
clusters
depending
on
your
target
environment
and
very
little
is
being
done
to
provide
consistency
from
a
from
a
user
experience
point
of
view,
and
this
is
really
where
cluster
api
comes
into
it.
So
if
you've
been
building
or
provisioning
clusters
for
a
very
very
long
time,
I'm
sure
you
you've
played
with
things
like
papanetti's
cops
and
various
other
things.
B
B
The
cluster
api
also
brings
in
higher
level
functionalities
so
as
well
as
just
the
pure
provisioning
of
infrastructure
and
then
kubernetes
on
top
of
those,
it
has
higher
order.
Functionality
like
automatically
scaling
or
automatically
doing
the
upgrades
of
kubernetes
versions,
for
example,
and
the
other
area
is
automatically
spreading
machines
across
failure
domains
because
you
don't
want
all
the
machines
in
the
same
failure
domain,
because
if
that
rack
goes
down,
then
your
whole
cluster
goes
down.
B
B
A
B
So
I'm
back
to
entertain
you
so
so,
where
were
we
so
yeah?
We
were
talking
about?
The
provider
is
basically
a
kubernetes
operator
and
as
such,
there
is
going
to
be
a
number
of
custom
resources
for
for
your
your
provider
and
those
resources
have
to
adhere
to
a
contract
and
that
contract
is
dependent
on
the
provider
type
that
you
are
building.
So
this
will
cover
provider
type
shortly
and
hopefully,
we'll
have
the
diagrams
before
then.
So
then,
they're
with,
along
with
those
custom
resource
definitions,
very
fantastic.
B
So
I
should
have
started
with
this
actually
so
a
couple
of
quick
questions
for
your
hands
in
your
hair,
who
uses
cluster
api
already
many
people
fantastic,
who
actually
contributes
to
cappy
or
a
provider
brilliant
and
actually
and
who's
thinking
of
building
a
provider
wow.
I'm
actually
quite
surprised
so
that
yeah,
that's
really
good.
B
A
couple
of
things
I
didn't
mention
here
is
so
core
to
to
the
life
cycle
of
cluster
cut,
call
to
apis
yeah,
it's
mandatory
so
spacing
operator
it
has
to
adhere
to
the
contract
and
that
contract
is
dependent
on
the
provider
type.
The
contract
is
important
because
it
allows
that
interaction
between
core
capi
and
the
providers,
so
that
things
can
talk
so
I've
mentioned
provider
types
a
couple
of
times,
so
there
are
three
provider
types
currently
in
cluster
api,
so
the
first
and
probably
the
most
widely
used
provider
type
is
the
infrastructure
provider.
B
You
can
probably
guess
from
the
name.
The
infrastructure
provider
is
mainly
concerned
with
provisioning,
the
infrastructure
that
is
required
for
your
cluster.
It
is
not
concerned
with
kubernetes
itself
and
bootstrapping
kubernetes,
it's
just
purely
building
that
environment
for
you.
So
as
an
example,
the
cluster
api
provider
for
aws
or
kappa
is,
is
an
infrastructure
provider
and
it
provisions
aws
resources
such
as
you
know,
vpcs
or
security
groups,
etc,
and
those
are
then
used
as
a
basis
to
to
create
the
kubernetes
cluster.
B
And
then
it
will
then
create
or
join
machines
to
a
existing
cluster.
There
are
generally
two
parts
to
this
bootstrap
process.
The
first
part
is
the
actual
command.
So
how
do
I
create
a
kubernetes
cluster
and
how
do
I
join
one
so
think
something
like
cube
adm
here,
so
that
is
the
nuts
and
bolts
of
how
I
create
a
kubernetes
cluster.
B
The
second
part
is
how
do
I
format
those
commands
so
that
they
can
be
run
so
generally.
This
involves
putting
them
into
a
specific
format,
so
something
like
cloud
init
or
ignition
and
making
that
making
that
available
via
a
secret
or
s3,
so
that
so
that
secret
can
then
be
used
by
the
infrastructure
provider
when
provisioning
machines
so
that
it
can
execute
those
commands
as
part
of
creating
that
machine
via
user
data
or
some
other
mechanism.
B
B
B
B
B
B
Additionally,
there's
going
to
be
some
some
kubernetes
resources
to
deploy
your
controller
into
a
cluster
instant
management
cluster.
So
this
is
plain
old,
plain
old
demo.
There
will
be
a
deployment,
a
bunch
of
our
back,
maybe
some
secrets.
You
know
whatever
just
just
normal
kubernetes
yaml,
you
have
the
option
within
those
to
tokenize
various
parts
of
it,
so
that
you
can
override
those
when
the
providers
are
installed.
B
B
So
I'm
going
to
move
on
to
actually
the
the
resource
kinds,
so
this
is
a
diagram
that
we
generally
use
the
reason
we're
going
to
cover.
This
is
because
it's
really
important
when
you
implement
a
provider,
cappy
has
a
number
of
different
custom
resources
and
are
basically
used
to
logically
represent
a
kubernetes
cluster
and
its
life
cycle.
B
So
you
can
see
this.
These
are
basically
the
grey
boxes
here
they
represent
those
the
cappy
resource
kinds,
so
the
cluster
at
the
top
there
that
represents
the
cluster
as
a
whole,
so
think
of
that
as
as
the
route
and
it
has
general
configuration
so
things
like
pod,
cider
blocks
or
service
cider
blocks
that
are
not
specific
to
to
infrastructure
or
how
you
actually
provision
or
boot
kubernetes.
B
B
So
we
have
machine
on
the
left
there,
so
that
represents
a
individual
machine
and
a
node
in
in
a
cluster.
So
there's
a
one-to-one
map
in
there.
So
then
we
have
something
in
the
middle,
which
is
called
a
machine
deployment,
and
that
represents
a
set
of
machines
that
have
the
same
template
and
you
you
specify
a
number
of
replicas
for
those.
B
B
B
B
For
example,
you
would
have
cube
adm
here,
so
capi
comes
with
a
cube,
adm
bootstrap
provider,
and
this
this
exposes
the
cube.
Adm
configuration
for
you
you'll
see
that
in
this
diagram
that
I've
highlighted
so
the
bootstrap
providers
here
are
in
pink
and
I've
drawn
and
we've
put
them
on
there,
so
they're
encompassed
by
the
cappy
resource
kinds.
Now
this
is
this
is
on
purpose.
This
is
basically
to
show
that
the
capital
capi
types
own
and
reference
the
the
bootstrap
provider's
configuration.
B
Just
to
give
an
example
here,
I've
noticed
there's
a
bit
of
a
typo
towards
the
bottom
there.
So
this
is
a
snippet
of
a
cluster,
so
so
the
top
level,
and
in
it
you
can
see
it's
referencing,
two
different
provider
types,
so
there's
an
infrastructure
provider
referenced
by
the
infrastructure,
ref
and
then
there's
a
control,
plane
provider
referenced
by
the
control
plane
ref.
B
B
B
So
it
normally
relates
to
networking
and
security
group
stuff-
it's
not
specifically
for
kubernetes,
and
it's
not
specifically
about
machines.
Generally,
you
know
there
is
it's
not
so
black
and
white
with
that.
You
know
there
are
shades
of
grey
within
that
the
infrastructure
machine
kinds
will
all
contain
configuration
that
is
specific
to
creating
whatever
compute
kubernetes
is
going
to
sit
on.
So
this
could
be
things
like
ec2
instances
or
gcp
compute
instances.
B
One
other
thing
to
note
from
this
one
diagram
is:
it
also
gives
an
example
of
some
of
the
naming
conventions
that,
as
a
provider
implementer,
most
providers
follow
so
you'll
see
at
the
top.
Here
we
have
something
called
aws
cluster
or
podman
cluster,
so
the
provider
name
is
generally
prefixed
on
your
custom
resource
types.
B
A
A
The
steps
involved
are
creating
an
infrastructure,
bootstrapping
kubernetes,
possibly
using
cube,
adm
and
then
managing
the
versions.
An
operator
codifies
all
of
these
steps
for
you
and
the
native
way
to
do
this
in
kubernetes
is
where
declarative,
apis
or
crds
or
customer
resource
definition.
As
we
all
know
it,
using
a
crd,
you
can
specify
the
infrastructure
of
your
choice,
specify
how
you
want
to
bootstrap,
kubernetes
and
specify
versions,
etc.
A
These
crds
are
then
monitored
and
reconciled
by
one
or
more
controllers.
That
brings
to
our
next
question.
What's
a
controller,
a
controller
is
nothing
but
a
control
loop
that
watches
the
desired
state
of
the
cluster
through
the
api
server
and
continuously
reconciles
to
move
from
the
current
state
to
the
desired
state.
A
One
last
bit
about
the
control
loop.
So
this
is
the
core
principle
of
kubernetes.
You
watch
for
the
changes
in
the
resources
that
you're
interested
in.
If
there
are,
if
there's
any
diff,
you
take
necessary
action
to
move
from
the
current
state
to
the
desired
state,
and
this
works
in
an
infinite
loop.
So
basically
watch
diff
act.
Repeat
is
your
mantra.
A
If
you
are
operating
on
a
cloud
or
bare
metal
service,
then
you
will
need
an
infrastructure
provider.
If
you
want
a
different
way
to
bootstrap
kubernetes
instead
of
cube
adm,
then
you
would
need
a
bootstrap
provider
and
maybe
a
complementing
control
plane
provider
as
well.
If
you
have
a
hosted,
kubernetes
control,
plane
service,
no
surprises
there.
A
If
you
want
to
provision
your
own
infrastructure
and
get
cappy
to
manage
kubernetes
on
them,
you
can
use
the
existing
bring
your
own
host
provider.
It
is
used
to
provision
kubernetes
on
top
of
your
existing
infrastructure
or
something
like
cluster
api
provider
for
aws.
Wherein
say
you
want
to
create
or
configure
your
own
vpc
and
want
to
make
that
vpc.
As
part
of
your
cluster,
if
none
of
the
existing
providers
suit
your
use
case,
then
you
may
want
to
write
one
or
all
of
the
different
kinds
of
providers
that
we've
mentioned
so
far.
A
Okay,
now
that
you
have
decided,
you
absolutely
need
to
write
a
new
provider
and
none
of
the
existing
providers
suit.
Your
need.
Let's
look
at
what
are
the
basic
steps
in
creating
a
new
provider
we
so
for
writing
a
new
provider.
Q
builder
and
controller
runtime
are
your
friends.
We
start
off
with
the
cube
builder
init
command.
This
generates
the
basic
repository
layout
for
your
provider.
A
So
all
of
the
crd
and
controller
references
will
be
with
respect
to
this
provider,
so
we
use
the
q
builder,
create
api
command
to
create
the
crds,
and
we
have
an
option
to
also
create
a
controller
for
the
crd
most
of
the
times.
You
would
need
a
controller,
because
the
controller
is
the
one
that
is
continuously
monitoring
for
the
crd
that
you
created,
but
there
are
also
times
when
you
don't
need
a
controller,
something
like
a
podman
machine
template,
so
this
provides
a
blueprint
that
can
be
create
that
can
be
used
to
create
podman
instances.
A
Out
of
so,
the
podman
machine
template
is
used
to
create
podman
machine
resources,
which
in
turn
is
reconciled
by
podman
machine
controller.
Therefore,
you
would
not
need
a
podman
machine
template
controller,
so
you
also
need
to
specify
metadata.
This
is
to
specify
the
compatibility
of
your
provider
with
the
cluster
api
contract.
A
A
So
defining
the
api,
so
cube
builder
would
have
scaffolded
some
code
for
you,
so
these
are
basically
for
an
api.
It
creates
a
apartment,
cluster
spec
and
a
podman
cluster
status,
but
there
are
some
fields
that
your
type
of
provider
has
to
be
conformed
with
cluster
api,
as
in
this
example,
we
add
a
control,
plane,
endpoint
and
this
field
represents
the
end
point
that
is
used
to
communicate
with
your
control
plane.
Similarly,
there's
a
ready
field
in
the
podman
cluster
status.
A
So
finalizers
also
play
a
very
good
role
in
writing.
Your
provider,
so
finalizers
provide
a
way
for
the
controller
to
clean
up
any
external
resources
that
it
has
created
before
deleting
the
api
resource
itself.
So,
while
writing
a
provider
you'll
mostly
end
up
creating
external
resources,
so
it's
always
a
good
idea
to
add
finalizers
to
your
api
resources
in
the
controller.
A
So
this
is
one
such
example,
so
this
is
a
reconcile
normal
function,
a
reconcile,
normal
function
is
for
a
create
or
an
update
flow.
You
would
have
an
equivalent
reconciled
delete
function
for
a
delete
api
workflow.
So,
as
you
can
see,
we
are
using
controller,
util,
dot,
add
finalizer
and
you
have
an
equivalent
remove
finalizer.
So
whenever
you
add,
this
finalizer
make
sure
are
patching
the
object
so
that
the
changes
are
persisted
in
the
api
server.
Similarly,
in
the
delete
workflow
make
sure
to
remove
those
finalizers.
A
Next,
we
go
into
implementing
the
controller
for
the
api
types
that
we've
created,
cube
builder
would
have
scaffolded
controllers.
For
you,
these
controllers
pretty
much
contain
two
functions:
one
is
reconcile
and
other
is
set
up
with
manager.
The
rest
of
the
logic
can
reconcile
should
be
filled
by
the
provider
implementer.
A
So
this
is
the
typical
set
of
steps.
If
you
are
writing
a
provider.
This
specific
example
is
for
an
infrastructure
machine
controller,
so
the
first
step
is
get
the
instance
of
the
api
type
being
reconciled
using
a
get
call
on
the
api
resource,
then
get
the
owning
capi
type
we'll
get
to
owner
references
in
a
bit
example.
If
we
are
reconciling
a
podman
machine,
get
the
machine
resource
if
we
are
reconciling
podman
cluster
get
the
cluster
resource.
A
If
we
don't
have
this
owner
reference
set,
yet
we
exit
from
the
control
loop
optionally,
you
can
also
get
cluster
and
infrastructure
cluster
objects
and
then,
from
this
top
level
reconcile
function,
we
either
break
into
a
reconcile
normal
or
a
reconciled
delete.
Now
this
depends
on
the
deletion
time
stamp
on
the
resource.
A
So
if
a
resource
with
a
finalizer
field
set
is,
is
requested
for
a
delete
call,
kubernetes
does
not
directly
delete
the
resource;
instead,
it
updates
the
resource
with
a
deletion
timestamp
field.
So
if
the
deletion
timestamp
field
is
present
on
the
resource,
then
it
means
it's
a
delete
request.
You
jump
into
the
reconcile
delete
function.
A
You
clean
up
any
externally
managed
resources
that
you
want
and
then
remove
the
finalizer
so
that
your
resources
can
be
deleted
from
the
api
server.
If
there
is
no
deletion
timestamp
on
the
resource,
then
it
means
it's
either
a
create
or
an
update
request
and
you
jump
into
the
reconcile
normal
function.
So
the
first
step
that
you
do
here
is
to
add
the
finalizer
field
and
then
proceed
towards
any
actions
for
creating
or
updating
your
resource.
A
A
A
Now
that
we
have
apis
and
controllers,
you
may
also
want
to
write
web
hooks
for
your
providers.
So
web
hooks
are
admission
controllers
that
are
used
to
add
custom
logic
or
validation
to
your
crds
queue
builder
can
scaffold
web
hooks
for
you,
you
have
to
use
the
queue
builder,
create
webhook
command
and
you
can
use
either
of
defaulting
or
programmatic
validation
or
both
of
these
flags.
The
defaulting
flag
would
create
a
defaulting
web
hook
and
programmatic
validation
would
create
a
validation,
webhook.
We'll
look
a
couple
of
examples,
so
this
is
for
the
validation
webhook.
A
So,
as
you
can
see,
queuebuilder
created
a
validate
create
function
for
us.
Similarly,
it
would
have
scaffolded
a
validate
update
and
a
validate
delete
as
well.
So
it
is
our
job
to
fill
in
the
logic
into
these
empty
scaffolded
functions.
One
thing
to
note
here
is
that
you
can
add,
as
many
number
of
rules
as
possible
to
your
validate
functions
and
aggregate
all
of
the
errors
and
return
all
of
them
together.
A
This
is
one
such
example
for
defaulting
on
the
top
we
have,
if
you
I
don't
know
if
you
can
read
it,
but
there's
a
queue
builder
annotation
for
adding
a
default
number
of
cpus
to
b2,
so
cube
builder
also
provides
a
way
for
you
to
add
defaults.
Only
if
this
is
not
enough
not
enough
for
your
use
case,
then
you
go
and
write
a
custom
default
web
hook.
B
So
I
don't
think
we're
going
to
have
time
to
cover
fully
the
rest
of
the
slicer,
we'll
probably
stop
on
this
slide
after
this
slide.
So
we'll
allow
time
for
questions
so,
once
you've
developed
your
operator,
you
need
a
you
need
a
way
to
to
test,
run
it
and
debug
it
locally,
and
this
is
where
tilt
comes
in,
so
you
can
tell
cluster
api
and
it's
tilt
file
about
your
provider.
B
B
This
is
normally
has
a
tag
of
dev
and
that
will
be
replaced
as
part
of
your
your
build
process.
The
second
important
part
is
this:
live
reload
depth,
so
tilt
has
a
iterative
development
model,
so
it
will
watch
those
files
and
those
paths
within
those
files.
If
there's
any
changes
to
those
tilt
will
automatically
recompile
your
provider,
it
will
package
it
into
your
container
and
then
it
will
instruct
the
deployment
to
use
that
new
container
all
without
reloading
the
rest
of
of
the
controllers.