►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Kubernetes (VMware) SIG - Loc Nguyen & Sidharth Surana, VMware
Deep Dive into the Cluster API Provider for vSphere The Cluster API is a Kubernetes project to bring declarative, Kubernetes-style APIs to cluster creation, configuration, and management. This is designed to support automated provisioning, scaling and updating of the underlying infrastructure supporting Kubernetes clusters. Cluster API providers exist for public cloud and on-premise platforms. SIG-VMware hosts a working group that maintains a Cluster API provider for the vSphere hypervisor. This session will take a deep dive into the design and use of the provider.
https://sched.co/MPji
A
A
Just
a
little
introduction.
My
name
is
low
Corinne
and
my
co-presenter
is
Siddhartha
Sarada
and
the
first
thing
we're
both
in
VMware
and
the
first
thing
I
want
to
do
is
just
define
some
definitions
where
we
internally
have
been
using
some
acronyms
Kathy
and
taffy.
Taffy
refers
to
just
general
cluster
API
and
Kathy
refers
to
the
Bistro
provider,
and
this
is
the
repo
for
the
link
to
the
repo
and
the
agenda
today.
Is
that
we're
gonna
go
over
a
very
brief
review
of
cluster
API,
but
not
too
in-depth
about
clustering
API.
A
A
So
now
on
to
a
brief
overview
of
cluster
API,
what
is
it
it's
a
project
and
the
goal
is
to
manage
the
lifecycle
of
kubernetes
cluster
and
we
follow
the
philosophy
of
two
bananas.
Managing
kubernetes
you'll,
see
that
in
in
future,
slides
and
the
provider
code
or
the
code
that
provides
the
glue
to
the
infrastructure,
so
cluster
API
will
use
that
glute
you
to
actually
provision
the
or
whatever
it
needs.
A
Cluster
API
right
now
is
at
V
1
alpha
1,
that's
the
API
version,
and
we
released
that
at
the
end
of
last
quarter,
q1
of
2019,
it's
built
on
top
of
C
Rd,
and
it
implements
a
set
of
container
controllers.
It
currently
supports
two
operational
model.
The
first
one
is
cluster
API
and
manages
the
cluster
that
is
deployed
on.
You
can
call
that
the
self
managing
cluster
and
the
second
model
is
cluster.
Api
manages
the
child
clusters
that
it
creates
and
you
can
call
that
centrally
managed
clusters.
A
So
when
I
said
earlier
on
this
page
that
we
were
following
the
philosophy
of
qu
Benes
management
kuba
needs,
this
is
what
a
diagram
would
that
mean
it
looks
like
so,
on
the
left
hand
side
you
have
what's
called
bootstrap
level
management
cluster
and,
on
the
right
hand,
side
you
have
to
target
workload
clusters.
On
the
left
hand
side
you
have
to
deploy
kuben
a
cluster
api
and
the
provider
on
an
existing
kubernetes
cluster.
A
In
order
to
use
it
I'll
refer
back
to
this
diagram
once
we
get
to
the
demo
so
a
little
bit
about
the
history
of
the
vSphere
provider,
so
q3
of
2018
vmware
joined
sig,
cluster
lifecycle
and
the
cluster
api
working
group
and
we
transitioned
the
the
provider
from
terraform
to
go
v,
mami
api's
and
go
v.
Mommy
api's
are
just
eight
guys
to
interact
with
the
B
sphere.
It's
a
open
source
project,
so
you
guys
can
all
access
be
sure
to
go
v
mommy.
A
Also,
q4
of
2018,
we
transitioned
the
provider
from
an
aggregate
API
server
to
Co
DS.
We
added
as
central
repo
for
official
containers
automated
at
it
automated
CIA,
III
test
and
created
and
helper
OVA,
and
we
tagged
that
as
the
Geo
dot
one
key
one
of
2019.
We
added
a
few
features,
did
some
bug
fixes
and
ensured
compatibility
with
Cathy
v1
alpha
1
and
tagged
that
as
v02,
okay
and
now
I'm
gonna
turn
it
over
to
my
co-presenter
and
he's
going
to
talk
about
the
architecture
in
workflow.
B
Thanks
luck,
so,
let's
look
at
another
covers,
what's
happening
when
you
actually
try
to
use
the
clustered
API
vSphere
provider.
So
what
you
see
on
the
screen
is
a
quick
diagram
of
an
oxide
that
you
need
a
cluster
to
run
these
controllers.
So
what
you
see
on
the
top
is
the
let's
say
the
bootstrap
pressure
or
the
management
cluster
where
your
cluster
API
pods
are
running.
So
you
see
there
are
two
parts
that
you
will
get
when
you
actually
deploy.
This
I'd
refer
that
one
is
the
kappahd
and
the
other
is
the
cap.
B
So,
first
of
all,
the
ubuntu
cloud
images
is
the
default
set
of
images
that
we
support
today,
using
which
you
can
use
to
deploy
kubernetes
cluster.
But
it's
easy
enough
to
support
other
distributions
as
well
and
will
briefly
touch
base
form
as
well
from
the
vSphere
stack
perspective
again.
Vcenter
server,
ESX,
server,
6.5
or
above
that
should
should
do
Kuban,
ADC
and
I.
Weave
is
what
we've
chosen
right
now
to
deploy
in
the
kubernetes
cluster
that
you're
going
to
get
out
of
it.
B
The
cloud
provided
the
kubernetes
cloud
provider.
Of
course,
we
configure
it
with
the
vSphere
cloud
provider
with
kubernetes.
So
there's
there
two
different
versions:
there
is
an
entry
version
and
there
is
an
out
of
free
version
right
now
we're
using
the
entry
version
of
the
cloud
provider
that
you
get.
But
again
we
we
have
our
work
cut
out
to
move
to
the
out
of
the
tree
version
of
the
cloud
provider
as
well
cloud
in
it.
You
will
be
familiar
with
some
of
these
tools.
B
Cloud
knit
is
the
primary
tool
that
is
being
used
to
literally
like
bootstrap
the
VM
when
it
powers
on
QB
ADM
is
another
cube,
atom
work.
Your
ATM
is
the
another
tool
that
we're
using
to
bootstrap
the
kubernetes
cluster
itself.
Inside
that
we
am,
if
you
want
to
do
a
quick
start,
we
also
have
a
helper
OVA
that
is
part
of
the
repo,
and
there
are
instructions
how
to
build
that
real,
quick
so
that
you
can,
you
know,
get
jumpstart
on.
You
know
trying
it
flying
things
out.
B
Let's
quickly,
look
at
some
of
the
resource,
yeah
Mel's,
since
its
kubernetes
as
all
Yama's.
A
few
very
important
means
I
want
to
just
touch
base
upon
first
is
provided
compliments
at
Yambol
now
this
is
a
file
that,
when
you
go
through
the
instruction,
this
will
be
a
generated
file,
but
the
key
thing
to
notice
this
particular
file
is
responsible
for
installing
and
setting
up
your
cluster
API
controllers
on
to
your
target
cluster
like
bootstrap
cluster
or
management
cluster.
B
Similarly,
the
other
two
main
ones
that
I've
highlighted
here
and
again.
This
is
not
entirely
a
comprehensive
list,
but
this
is
the
main
ones
cluster
out,
yeah
Mel
and
the
Machine
bottom
out.
These
are
primarily
what
gives
you.
What
is
your
cluster
level
things
that
you
want
to
find
and
similarly
on,
the
machine
yeah
defines
what
is
that
one
machine
is
going
to
look
like
and
we'll
look
into
in
the
next
slide,
some
more
details
of
what
these
are.
B
So,
let's
look
at
this
is
the
sample.
Clustered
Gamal
would
look
like,
on
the
right
hand,
side
that
you
see
two
main
pieces
of
information.
First
is
the
cluster
networking
so
some
bit
of
exciter
selection
and
things
like
that,
and
the
second
part,
is
the
provider
spec,
which
is
more
provided
specific
information.
For
our
case
we
have
like,
where
is
every
center
server?
What
are
the
credentials
that
you
want
to
use
to
access
those?
These
are
the
two
pieces
of
information
that
is
that
we
will
find
in
the
cluster
data
Yambol
moving
on
machine
level.
B
This
is
how
it
looks
like
as
an
example.
This
is
really
what
is
giving
you
the
how
what
what
is
the
shape
in
the
size
of
the
VM
that
you
ultimately
are
going
to
deploy
when
you
use
this
machine
Yama,
so
it
has
things
like
data
center
data
store
resource
pools
like
which
defines
where
that
VM
will
be
placed,
for
example,
then
some
bit
of
sizing
like
CPU
memory
and
so
on.
For
then
it
also
has
some
specific
things
like
which
VM
template
that
you
want
clone.
B
This
is
going
to
point
to,
for
example,
whatever
the
template
that
you
have
prepared
in
vSphere
things
like
NTP
servers
or
certain
certain
injection.
So
this
is,
if
you
have
let's
say,
custom
shirts,
that
you
want
to
make
sure
that
you,
you
know
inject
when
the
VM
comes
up,
so
this
is
where
you
will
define
them
that
will
be
used,
similarly,
again,
networking
information
and
so
on
so
forth.
We
both
support
like
DHCP
as
well
as
static
IP,
from
the
networking
point
of
view
for
these
VMs.
B
So
so
this
is
just
in
a
general
sense
like
how
the
bootstrap
workflow
words,
so
you
define
the
machines
George
Jamell,
which
we
just
saw
a
moment
ago,
which
is
your
intent
to
what
my
VM
should
look
like,
and
then
we
have
some
scripts
which
are
right
now
existing
in
a
file
called
templates
or
go,
which
is
in
the
source
code.
But
these
are
essentially
what
defines
your
cloud
net
based
configuration
and
all.
Ultimately,
these
are
the
two
source
of
input
that
gets
in
the
machine
controller
running
running
as
part
of
the
captive
part
processes.
B
These
resources,
clone
is
responsible
for
cloning,
the
VM
generating
the
metadata
that
needs
to
be
injected
in
that
VM
and
then
ultimately
powers
that
off
once
that
VM
is
powered.
On
first
thing
cloud
it
kicks
in
cloud
net,
takes
care
of
a
networking
and
anything
like
certain
ejection,
whatever
the
things
that
you've
specified
cloud
new
takes
care
of
that,
then
it
also
emits
out
a
shelf
square.
Boot
got
a
search
in
temp
location,
which
is
basically
has
all
these
steps
that
is
needed
to
be
executed
to
bring
this
node
online.
B
Ultimately,
at
the
end
of
the
day
that
boot
data
set
contains
calls
to
Cuvier
DM,
you
know
you
can
imagine
the
last
call
as
the
cube
a
DM
join
or
qvd
a
minute
essentially
start
here
to
turn
this
VM
into
a
kubernetes
node.
For
you,
a
little
bit
of
the
where
they're
headed
towards
so
from
a
Cappy
perspective,
there
is
V
1
alpha
2
is
the
current
release
that
is
being
worked
on
at
the
moment
and
there's
a
lot
of
redesign
that
is
happening
is
part
of
even
alpha
2.
B
Some
of
the
work
streams
have
listed
out
on
this
slide
here
things
like
the
data
model,
which
is
your
cre
definitions
itself.
Some
of
them
might
undergo
some
changes.
How
the
how
the
extensions
are
being
used
like
how
is
the
provider
talking
to
the
main
controller?
Those
things
are
also
under
consideration
for
changes.
B
Similarly,
the
control,
plane
lifecycle
or
the
node
life
cycle
individually
and
the
last
not
really
is
the
image
building
like
there
is
a
again
versioning
for
image,
stamping.
How
do
you
reliably
reproduce
or
generate
an
image
that
you
can?
You
know
reliably
use
in
in
the
capi
world
from
the
cap
V
perspective,
more
specific
to
the
vSphere
provider?
First
and
foremost,
caffeine
needs
to
align
with
whatever
the
changes
that
come
out
of
the
Vivaan
alpha
to
redesign
of
the
cap
e,
then
other
than
that
we
want
to
align
the
cap.
B
We
feature
said,
along
with
the
other
providers,
for
example,
things
like
add
high
availability,
multi
master
support.
We
don't
have
that
yet,
but
that's
something
that
we
probably
going
to
get
soon
enough
again,
as
I
mentioned,
we
want
to
switch
to
using
the
out
of
three
version
of
the
vSphere
cloud
provider
instead
of
using
the
entry
version,
which
is
now
getting
ping
getting
deprecated
at
the
moment,
enabled
zone
support
again
vSphere
cloud
provider
already
has
these
own
support
capability,
but
it's
just
that
in
the
cap
v-world.
We
have
not
yet
enabled
that
configuration
option.
B
B
Last
point
here
is
more
touches
upon.
If
you
go
back
to
the
last
diagram
that
I
saw
showed
like
the
scripts
in
template,
dot
go
here.
It's
not
very
useful
because
right
now,
they're
kind
of
hard
built
in
the
go
templates,
I
figure
out
to
modify
in
order
to
adapt
something
you
can
I
have
to
modify
the
source
code,
which
is
less
than
ideal.
A
A
C
A
Okay,
so
in
in
here
I'm
at
the
root
folder
of
the
the
repo
and
in
this
folder
we
have
a
make
file
and
I'm
gonna
step
through
how
you
would
build
it,
how
you
would
deploy
it
and
then
we'll,
and
how
do
would
you
interact
with
the
the
cluster
that
we
created
it?
So
in
this
we
have
a
make
file
and
inside
this
make
file.
A
A
So
this
is
what
I
have
for
my
container
on
docker
hub
and
once
you
once
you
put
that
information
into
this
make
file.
I'm
not
gonna.
Do
it
here,
I'm
just
gonna
talk
through
it,
you
can
say
he
make
Deb
PMO
and
what
that
would
do.
Is
it's
going
to
generate
an
outfitter
with
sample
yellow
files
and
it
will
generate
some
certificates
for
you
that
cluster
API
will
use
to
set
up
the
SSH
for
the
the
nodes
of
the
target
cluster?
A
Once
you
do
that,
then
you
can
say:
dev
build
yamo
and
that
will
build
the
container
and
tag
it
with
the
name
that
you
specifying
in
last
of
all
make
push
ammo
and
that
will
push
it
up
to
your
to
your
to
your
registry
and
and
then
you
can
use
it.
So
I've
already
generated
a
the
sample
folder
and
inside
of
out.
You
see
that
there
are
a
bunch
of
a
bunch
of
yellow
flowers
right
here.
A
You
know
when
Siddhartha
was
talking
and
he
mentioned
the
provider,
components,
machines
and
cluster
gamal.
There
are
a
couple
other
yeah
mo
files,
but
we're
in
our
than
discussing
them
today,
because
I
really
just
want
to
focus
on
showing
a
basic
bringing
up
a
cluster.
I
won't
go
back
to
this
diagram
that
we
had.
We
showed
earlier
where
we
had
kubernetes
managing
kubernetes
right.
A
So
for
today,
I'm
going
to
use
kind
as
my
bootstrap
cluster
or
for
those
of
you
guys
who
don't
know
what
kind
is
kind
as
a
project
in
the
community
that
basically
instantiates
a
kubernetes
cluster
on
top
of
docker
containers?
Okay,
so
it's
really
handy
and
in
order
to
save
time,
I
went
ahead
and
created
a
time
cluster.
Previously
I
didn't
want
you
guys
to
watch
it.
A
A
A
A
So
one
thing
that
was
mentioned
earlier
was
that
we
currently
support
vCenter,
6.5
and
above
we
also
support
ESX
for
development
purposes.
So
for
those
of
you
guys
so
right
here,
I
have
ESX
running
in
a
fusion,
so
I
have
nessip
ESX
and
right
here,
I'm,
showing
that
we
don't
have
any
VMs
running
so
now
we'll
go
ahead
and
create
a
machine
for
that
cluster.
A
A
A
A
A
A
A
Okay,
okay,
so
apparently
it's
not
there,
this
time,
we'll
just
go
ahead
and
use
coop,
CTL
passing
the
target,
Tech
Talk
file,
and-
and
we
see
here
that
the
machine
is
up
and
running
so
now
you
can
interact
with
that
that
target
cluster
using
that
target
that
conf
file,
okay
and
so
now
the
last
thing
I
want
to
do
is
I'm
going
to
delete
the
delete
the
machine
and
in
order
to
delete
the
machine,
you
actually
have
to
do
it
from
the
original
bootstrap
cluster.
So
remember,
when
I
do
it's
the
GL.
A
A
We
back
here,
you
see
that
machine
is
there
and
the
machine
is
gone.
So
what
I
was
demonstrating
me
said
if
you,
if
you
ever
get
cloned
the
main
tapi
repo
you'll
you'll,
see
that
there
is
that
CLI
tool
plus
your
CTL.
It
does
a
lot
of
this
stuff
for
you,
but
I
wanted
to
show
you
guys
how
everything
underneath
what
is
doing
from
scratch
using
coups
CTL.
A
A
A
C
A
B
So
so
the
kubernetes
api
sell
them
in
itself
is
not
doing
it.
So
remember
in
the
one
of
the
diagram
we
had
like
the
two
parts
here,
for
example,
so
the
parts
that
are
running
it
that
they're
running
some
controllers
here
right.
So
when
you
submit
that
machine
dot,
yeah
Mel
with
Q
CTL
apply
that
that
creates
a
new
custom
resource
of
type
machine
and
that
type
resource
is
being
processed
by
the
controller
that
is
running
of
the
s
part
of
the
cap
real
part.
So.
B
B
A
B
B
So
in
here,
okay,
so
we
saw
this
machine's
dot
yeah
mill
file.
If
we
look
at
that
quickly,
you
know
so
this
is
very
similar
to
what
we
show
what
you
saw
in
the
slide
now.
The
interesting
part
is
now
when
you
want
to
add-
let's
say
more
worker
nodes
for
your
cluster.
You
cannot
define
almost
identical
to
this
file,
how
your
worker
nodes,
what
sizing
and
things
like
that,
what
you
want,
the
only
difference
between
the
master
node
and
the
vector
node,
for
example,
will
be
in
this
last
line.
B
Here,
we've
seen
in
the
versions
section
for
a
master
node.
It
also
includes
something
called
a
control,
plane
version
which
kind
of
identifies
that
this
is
a
master
and
signifies
that
so
for
a
vertical
mode.
You
basically
just
remove
this
particular
line,
for
example,
and
basically
resubmit
this
back
and,
of
course,
if
you
have
static,
IP
change,
the
IPS
and
all,
but
the
moment
you
do
that.
The
controller
that
is
running
as
part
of
the
cap
V
part
that
is
responsible
for
generating
an
on-the-fly
cube
ADM
program
from
the
master.
B
It
ensures
that
that
cube
ATM
token,
that
it's
going
to
utilize
is
valid
at
a
minimum
for
fifteen
minutes
and
I.
Don't
think
it
caps
up
at
like
a
twenty
minute,
so
within
that
time
range.
If
the
VM
has
that's
the
time
frame,
when
you're
working
nodes
can
basically
clone
powered
on
and
the
cube
ATM
in
it
automatically
will
happen
to
join
the
main
cluster.
C
B
So
essentially,
even
if
even
when
we
do
support
the
multi
master,
remember
like
the
Li
from
the
a
cube,
and
it
is
API
perspective.
You
still
will
be
a
talking
to
just
one
community's
APN
time,
because
even
with
the
multi
master,
then
you
have
to
put
in
into
the
picture.
Is
the
load
balancer
for
the
API
server?
So,
yes,.
D
B
So,
in
the
current
implementation
that
exists
as
of
today,
cluster
API,
like
say,
the
cap
V
has
already
made
certain
choices
that,
unfortunately
right
now
you
cannot
like
configure
so,
for
example,
in
your
cluster
or
jam,
and
you
cannot
specify
oh
I.
Don't
want
vivre
I,
won't
let
say
calico
as
a
CNI
that
flexibility
today
doesn't
exist,
but
we
I
mean
I,
think
Cappy
in
general.
B
It
doesn't
even
exist,
I
think
in
general,
as
a
Cappy
level
itself
like
across
the
various
cloud
providers,
it's
not
a
general
option
that
you
will
find
anywhere
at
the
moment.
However,
that
is
going
to
change
and
as
we
even
alpha-2
comes
along
the
way,
I'm
sure,
if
not
and
even
alpha-2,
but
immediately
the
nips
at
subsequent
releases.
B
There
will
be
available
these
kind
of
choices
in
a
such
that
when
you
are
defining
your
cluster,
for
example,
the
cluster
or
DML,
you
should
potentially
be
able
to
specify
I
want
calico
or
I
want
be,
but
I
want
planner
or
whatever
the
choice
of
your
CNI,
with
whatever
configuration
you
might
want
to
pass
being
made
available.
As
you
know,
intent
based
definitions,
but
they
don't
exist
as
of
not
today.
B
Today
those
choices
are
kind
of
fix
you,
but
you
can,
of
course,
when
you
are,
if
you're
developer,
you
can
of
course
clone
the
repo
make.
Those
final
modifications,
the
remember
I
point
out
like
this-
is
a
template.
Go
file,
that's
really
where
the
main
definitions
are.
So,
if
you
want
to
let's
say
in
your
environment,
deploy
calico
in
that
file,
you
will
find
you
can
easily
find
the
snippet
Yama's
snippet
that
basically
instantiates
we've
today,
you
can
easily
replace
that
with
the
scene.
B
B
B
But
hopefully
again
you
know
I,
don't
know
really
what
the
future
would
be,
but
hopefully
you
know
I
think
it
might
become
reality
that
you
know,
along
with
the
other
CNI
choices,
NCP
will
or
potentially
may
become
a
possible
CNI
deployment
option.
If
you
have
any
sixty
already
in
your
environment,
the
mechanics
of
that
might
again
I,
don't
know
really
at
the
moment
what
that
would
be,
but
definitely
should
be
possible
in
somewhat
near
future.
B
A
C
B
So
right
now
it
uses
username
password
based
credentials,
which
of
course
always
have
the
caveat
that
okay,
whatever,
if
somebody
rotated
the
password
or
something
like
that,
and
now
how
do
you
update
it?
So
so
those
are
some
of
these
operational
challenges,
but
unfortunately,
at
the
moment
they
can
again
only
talking
about
what
the
current
code
is.
It
doesn't
really
handle
like,
for
example,
the
password
rotation
you
deploy.
You
can,
of
course,
manually
go
in
update
that,
but
it's
not
really
automated
very
well
now.
B
As
far
as
other
type
of
credentials
mechanism
that
may
be
supported,
things
like
you
know,
exchange
I
think
start
base.
I
serve
based,
authentication
things
like
that
I
think
cap
V
is,
is
really
more
of
a
tooling
right
now
to
kind
of
in
able
the
cluster.
At
the
end
of
the
day,
those
credentials
are
being
used
by,
let's
say
the
vSphere
cloud
provider
now,
so
we
switch
the
moment
like
I'm,
not
entirely
sure
if
the
vSphere
cloud
provider,
especially
in
the
out
of
the
tree
version
of
that
currently
supports,
let's
say
a
solution.
D
D
B
So
essentially
so
in
theory,
then,
essentially
even
with
the
with
what
you
saw
here
then,
because
because
what
the
credentials
and
how
they're
used
are
pretty
much
identical
to
how
any
vsphere
api
is
how,
when
you're
consuming
we're
going
to
be
used,
so
I
guess
even
today,
when
you
have
so
the
cap,
we
supports
the
secret
based
definition,
which
is
very
identity,
which
is
actually
identical
to
how
you
know
I
think
the
visa
cloud
provider
supports.
So
you
cannot
generate
it
in
the
same
way.
A
Well,
okay,
I
think
we're
at
time,
or
we
actually
a
little
bit
over.
So
if
there
are,
if
there
are
any
other
questions,
well,
we'll
stick
behind
and
answer
them,
but
the
last
just
a
last
thing:
I
went
we
want!
Thank
you
guys.
If
you
guys
want
to
reach
us
there
are.
There
are
two
slack
channels.
Under
the
kubernetes
organization,
cluster
API
and
SiC
VMware,
we
have
bi-weekly
meetings
on
Monday
at
1:00
p.m.
Pacific,
Standard
Time.
The
next
one
is
on
June
4th
and
last
of
all,
this
is
the
git
repo.