►
From YouTube: VMware SIG: Cloud Provider Moving Out of Tree+CSI - What it Mean... Steven Wong & Fabio Rapposelli
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
VMware SIG: Cloud Provider Moving Out of Tree+CSI - What it Means to Users - Steven Wong, VMware & Fabio Rapposelli, VMware
Kubernetes cloud providers and volume plugins used to be "in-tree" meaning that their source code is included in the main Kubernetes repo. They were compiled in, and shipped only in a Kubernetes release. The drawbacks of this monolithic approach were that Kubernetes was larger than needed, and feature + patch activity was locked to Kubernetes release schedules. Going forward, new features are exclusive to the new replacements: an out-of-tree vSphere cloud provider + a CSI storage plugin. Legacy implementations remain for the short term but destined are for deprecation. Agenda: - Deep Dive : Install and configure of out-of-tree cloud provider + CSI storage - Migration options for current users This session will be useful to: - Users running Kubernetes on VMware infrastructure - Authors of installers and Kubernetes distributions which target the vSphere platform
https://sched.co/Nrr2
A
B
My
name
is
Fabio
Abu,
celli
I
work
for
Yammer
as
well
I'm,
an
engineer
working
in
kubernetes
upstream,
so
my
day.
Job
is
basically
to
make
sure
that
kubernetes
is
a
health
bar,
healthy
project
and
I
am
you
know,
we're
able
to
contribute
new
features
and
so
on.
Sam
I'm,
mostly
active
in,
say,
cloud
provider.
Obviously
sick,
VMware,
I'm,
a
co-chair
with
with
Steve
and
a
sick,
lush
lifecycle
and.
B
A
Today,
we're
going
to
talk
about
why
the
kubernetes
cloud
provider,
not
just
the
VMware
one,
but
all
of
them,
are
moving
out
of
tree
and
what
that
means,
then
we're
going
going
to
go
into
a
deep
dive
of
installing
and
configuring.
The
new
out
of
tree
cloud
provider,
along
with
CSI
storage
and
Fabio,
is
going
to
give
a
demo
of
that
and
address
the
migration
options
available
for
people
who
are
already
using
the
entry
version.
So
the
original
architecture
of
kubernetes
started
with
what
are
called
the
cloud
providers
built
right
into
kubernetes.
A
A
If
you
don't
know
this
already,
is
that
it's
the
abstraction
layer
that
makes
kubernetes
portable
so
that
you
can
take
kubernetes
and
have
the
identical
experience,
whether
you're
running
on
an
Amazon
public
cloud
at
Google,
public
cloud
Azure
or
on
your
own
on
Prem
hardware,
with
or
without
a
hypervisor,
and
this
cloud
provider
sits
there
and
makes
it
contains
the
code
that
makes
everything
uniform
so
that
when
you
write
or
deploy
apps,
the
apps
aren't
even
able
to
tell
where
they're
running.
Unless
you
do
something
stupid
and
intentionally
try
to
pierce
these
layers.
A
But
the
premise
is
that,
in
order
to
be
cloud
native,
the
app
should
run
identically,
regardless
of
where
they're
hosted
and
the
cloud
provider
layer
is.
What
does
this
mapping
to
whatever
the
underling
infrastructure
actually
is?
So
these
eight
cloud
providers
are
the
ones
that
are
built
into
kubernetes
automatically,
whether
you
like
it
or
not,
as
a
version
114.
Now
it
turns
out
that
having
these
cloud
providers
built
into
kubernetes,
it's
easy
if
you're
on
one
of
those
eight.
A
But
there
are
some
problems,
these
these
cloud
providers,
if
code
in
the
cloud
provider
has
a
bug
or
needs
a
security
patch,
they
can't
be.
They
can't
actually
be
published
independent
of
a
full
kubernetes
release
so
and
for
any,
given
user
you're
only
going
to
be
using
one
of
these
cloud
providers
at
a
time,
yet
the
binary
distribution
of
kubernetes
includes
them
all.
So
it's
bloated
unnecessarily.
A
A
And
finally,
the
steering
committee
and
the
architects
have
come
to
the
conclusion
that
really
kubernetes
should
just
be
the
standardized
orchestration
kernel
or
code,
and
these
drivers
are
cloud
providers
should
be
completely
independently
maintainable
by
people
familiar
with
that
platform,
even
free
to
declare
independent
release
cycles
so
that
if
we
on
the
vSphere
cloud
provider
wanted
to
get
a
release
out
two
weeks
after
kubernetes
itself,
we'd
have
the
freedom
to
do
that
if
the
code
isn't
tightly
coupled.
So
this
move
of
the
cloud
riders
out
of
the
tree
isn't
really
urgent,
meaning
it's
the
standard.
A
Kubernetes
deprecation
policy
applies,
meaning
users
get
it
a
minimum
of
a
one-year
notice
that
this
is
happening
and
aren't
forced
to
march
overnight
to
the
new
one.
So
there
will
be
a
period
when
both
are
available
and
that
we're
in
that
period
right
now
where
the
entry
cloud
provider
for
vSphere
is
still
alive
and
well
and
present
and
supported,
but
the
auto
tree
version
is
also
there.
There
hasn't
been
a
firm
decision
on
when
the
cutoff
date
is
going
to
be,
but
you
will
buy
kubernetes
policy
get
at
least
a
year's
notice.
B
A
The
way
this
move
is
structured,
it's
based
on
a
number
of
these
caps,
these
kubernetes
enhancement
proposals,
I'm
not
going
to
go
into
the
details,
but
the
link
I'll
give
you
up
a
shortened
link
to
this
whole
deck
at
the
end,
and
you
can
go
look
at
those
caps
that
have
detailed
designs
as
to
what's
going
on
with
this
move
out
of
tree.
One
thing
that
is
happening
in
conjunction
with
the
cloud
provider
is
for
exactly
the
same
reasons.
A
Is
architecture
is
changing
to
use
something
called
CSI
the
container
storage
interface
that
you
can
get
detailed
information
on
CSI
from
presentations
done
by
the
kubernetes
storage
saying,
but
the
gist
of
it
is
that
CSI
is
an
Orchestrator,
independent
industry
standard.
Where
you
write
a
driver
and
it
is
capable
of
supporting
not
just
kubernetes
but
Apache
mezzos,
docker
and
Cloud
Foundry,
and
it
enabled
storage
vendors
to
write
one
driver
that
supports
more
than
just
kubernetes,
but
kubernetes
put
in
place
a
storage
interface
to
use
these
out
of
tree
storage
implementations.
A
And
when
you
move
to
the
out
of
tree
cloud
provider,
you
have
to
move
to
the
out
of
tree
storage
plug-in
as
well.
There
they're
a
couple
that
move
is
coupled
together,
so
I
guess
with
with
that
I'll
move
on.
So
the
schedule
here
is
that
the
CSI,
the
storage
driver
for
vSphere
is
this
is
in
the
first
release.
Now
we've
declared
it
alpha
version
zero
to
zero,
anticipate,
AGA
and
August
a
couple
months
from
now.
It
does
have
some
enhanced
features.
A
So
one
of
the
things
that's
happened
with
both
the
auditory
cloud
provider
and
the
out
of
trees.
Storage
plugins
is
that
the
old
versions
are
no
longer
having
feature
enhancements
and
new
features
are
going
strictly
into
the
out
of
tree
and
those
new
feature
ads
have
been
happening
since
about
the
beginning
of
the
year.
A
So
already
you
are
in
a
situation
where
you
can
do
things
with
the
out
of
tree
that
are
possible
with
the
in
tree
and,
what's
going
on
with
the
entry
is
bug
fix
only
so
no
new
feature
ads
are
being
admitted
into
those.
So
if
you
want
these
new
features,
you
might
have
a
motivation
to
move
to
the
outer
tree
versions.
Right
now,
even
though
you're
not
forced
to
what
we're
going
to
have
going
forward
is
VM
independent
volume
management.
A
What
the
so
called
first-class
disk,
that
is
something
that
came
in
the
later
editions
of
vSphere,
where
it
was
possible
to
define
storage
volumes
that
aren't
strictly
tied
to
one
VM.
They
they
have
their
own
existence
independently,
so
that
these
volumes
can
be
created
and
handed
off
from
one
VM
to
another.
A
We
also
have
support
for
straddling
kubernetes
clusters
across
multiple
V
centers.
Multiple
data,
centers
and
Fabio
is
going
to
give
a
demo
of
that
later
in
this
presentation,
and
we
can
also
handle
both
conventional
and
what
are
called
raw
volume
outs.
The
difference
is
that
a
conventional
mount
attaches
in
the
file
system.
A
raw
one
appears
as
strictly
a
device,
and
there
are
certain
types
of
apps.
B
B
So
what
we
have
is
we
can
define
arbitrarily
in
vSphere
the
type
of
zones
that
we
want
to
have,
so
we
can
specify
with
using
tags
a
data
center
and
the
zone
where
the
data
center
lives
and
using
those
tags
we
can
schedule
pods
where
the
specific
stories
the
lives
in
that
specific
region
or
a
specific
zone
without
further
ado
I'm,
just
gonna
go
on
with
the
demo.
I
had
to
record
a
demo,
because
I
wasn't
sure
if
I
were
able
to
you
were
able
to
connect
to
our
system.
B
So
it's
recorded
demo
has
a
lot.
Lots
of
pauses
in
the
middle
is
gonna
step
ahead
and
just
I
like
the
the
major
things
in
the
video.
So
what
we
have
is
we
have
our
our
environment
here
is
made
up
with
two
V
centers,
so
we
have
to
be
centers.
Think
of
them
as
two
independent
control
planes,
the
first
one
we
have
here
as
a
single
data
center.
It
has
a
cluster,
has
a
bunch
of
hosts
and
it
has
part
of
our
kubernetes
cluster.
B
Let
me
just
go
get
closer,
so
this
is
Region
II
you
so
the
European
one,
and
we
have
a
zone
that
says
Kate's
region
EU
all
because
we
don't
have
a
specific
region
for
this.
We
just
have
one
the
other
vSphere,
the
other
V
Center
is
connected
to
two
data.
Centers
one
data
center
is
in
the
region
us
and
in
the
zone
us
West.
So
this
this
data
center
is
on
the
west
coast
of
the
US,
while
the
other
one
is
on
us
East.
B
So
it's
still
in
the
region
us,
but
it's
in
a
different
zone,
and
here
we
have
the
rest
of
the
nodes
for
our
kubernetes
cluster
that
are
spread
across
the
2v
centers.
So
the
first
thing
we
want
to
do
here
is
we
want
to
deploy
our
CCM,
which
is
our
our
clock,
controlling
manager.
So
the
the
the
auditory
call
provider
and
apologies
for
this,
but
you
know
this
is
basically
a
bunch
of
commands
we're
going
to
just
deploy
the
manifests
that
we
have
here
and
this
is
going
to
deploy.
B
This
is
the
early
manifest
that
are
coming
from
directly
from
the
repo
where
the
CCM
lives.
If
you
look
at
the
pod,
so
we're
gonna
see
that
we
have
a
clock,
control,
a
manager
being
deployed,
there's
only
one
controlling
manager
deployed
because
there's
really
one
that
needs
to
be
running
or
it
has
to
run
on
every
master
node.
If
you
have
a
multi
master
cluster.
B
The
second
step
is
to
deploy
the
CSI
driver.
So
what
we're
gonna
do
is
we're
gonna,
get
the
manifest
for
the
CSI
driver
and
deploy
on
top
of
our
cluster
that
we
have.
Maybe
it's
not.
You
know
visible
here,
but
we
were
coming.
We
have
in
the
same
configuration
file
for
both
the
CCM
and
CSI
is
called
vSphere.
Kampf
is
the
same
file
that
is
deployed
both
as
a
configuration
for
a
CCM
and
for
CSI,
because
we're
sharing
that
configuration
file
and
what
we're
doing
here
we're
basically
just
creating
and
applying
those
those
yellow
files.
B
Now
that
we
applied
them,
we
make
sure
that
they're
running
correctly,
we're
gonna,
see
a
demon
set
deployed
for
CSI,
because
CSI
requires
to
have
pod
running
on
every
single
node,
because
this
pod
here
is
responsible
for
attaching
and
detaching
the
disks.
So
we
see
there,
our
controller
managers
are
still
running.
We
have
a
CSI
Thacher.
We
have
all
the
nodes,
we
have
the
provision
and
running
so
at
this
point,
we're
all
good.
We
have
a
working
environment
with
CSI
we're
able
to
provision
storage.
B
And
this
is
highlighting
the
fact
that
we
have
storage
cluster
data
store
clusters,
so
this
is
another
object
in
in
vSphere,
so
we
basically
have
I
scuzzy
one
cluster,
which
is
a
datastore
cluster
that
contains
data
data
stores
that
are
shared
in
the
specific
data
center.
So
there's
a
bunch
of
data
stores
and
we
want
to
target
that
specific
cluster.
So
we
want
to
target
this
I
scuzzy
one
cluster
on
region,
us
us
east.
B
So
we're
going
to
open
up
and
apologies
if
it's
not
there's
a
little
light
here,
but
this
is
a
storage
class.
It's
a
type
of
vSphere
fcd,
which
is
first
class
disk,
is
going
to
have
a
parent
type
of
data
store
cluster
and
the
name
is
ice
CAHSEE,
one
cluster,
which
is
the
one
that
we
just
saw
on
V
Center
we're
going
to
mark
that
with
two
labels,
we're
going
to
say
that
the
figure
domain
the
zone
for
the
failure
domain
is
Kate's
zone.
Us
east
in
the
region
is
Kate's
region
us.
B
And
now
we
are
ready
to
deploy
our
simple,
simple
application.
The
simple
application
is
just
really
just
a
busy
box
container
that
sleeps
the
application
itself
is
not
doing
anything,
but
the
important
part
is
that
we
are
mounting
slash
data
here,
which
is
coming
from
my
FC
volume
and
my
F
CD
volume
is
a
persistent
vault.
Persistent
volume
claim
they're
going
to
see
after
this,
that
is
a
five
gig
virtual
disk.
That's
going
to
get
attached
to
this
application.
B
B
This
is
the
persistent
volume
claim
that
this
is
the
Briton
volume
claim.
We
have
VCR
CSI
PVC,
it's
a
type
rate
right
once
and
it's
a
five
gig
disk,
so
we
create
our.
We
apply
our
test
body
ammo.
So
this
is
our
application.
We
just
go
get
pod
we're
going
to
see
that
we
have
a
my
CSI
application
up
here.
B
B
B
If
you
want
to
know
more
about
this,
if
you
want
to
try
it
yourself
here,
there's
a
bunch
of
references.
Obviously
you
know
you
don't
have
to
write
them
down.
You
can
look
at
the
slides
after
we're
done
here.
There
is
there's
decent
documentation
for
all
this.
It
will
be.
It
will
get
better
we're
actually
working
on
a
new
documentation
website
with
all
this,
and
you
know
much
better
walkthrough
and
some
workflow,
and
you
know
some
other
type
of
configuration
right
now,
there's
just
a
bunch
of
deployment
models
that
we
support.
Well,
that
we
document.
B
So
if
you
don't
find
what's
in
there
or
if
you
don't
find
this
useful
for
your
setup,
feel
free
to
open
an
issue
and
get
up
or
just
reach
reach
out
to
us.
On
slack
on
the
kubernetes
slack,
there's
a
channel
called
cig
vmware
there's
a
channel
called
there
will
be
a
channel
called
provider.
Vsphere,
there's
gonna
be
opener
up
soon,
there's
also
a
bunch
of
other
presentation
that
we've
done
in
the
past.
B
A
Thing
I'd
like
to
point
out:
I
guess
we
should
have
put
it
in
that
slide,
but
it
was
so
crowded.
There
wasn't
room,
but
we
did
a
presentation
specifically
on
the
CSI
out
of
tree
storage
just
a
month
ago
in
Barcelona
and
that
got
videoed
in
the
deck
is
published
at
the
Q
con
Barcelona
website.
So
you
can
find
it
there
and
that
was
a
whole
hour
strictly
on
the
storage
part.
So
it's
a
it's
a
deeper
dive
yeah.
So
you
might
be
interested
in
that
any
questions.
D
D
D
A
No
in
general,
its
that's
a
bad
idea.
It
is
possible
with
the
kubernetes
architecture
just
because
they
did
it
a
different
way
at
the
beginning.
To
do
things
like
create
a
volume
first
and
then
specifically
indicate
that
you
want
that
exact
volume
in
the
pots
back
or
the
container
spec,
but
that's
generally
deemed
to
be
a
bad
idea.
So
there's
something
called
storage
classes
where
you
define
classes
that
you
should
label
with
meaningful
names.
A
If
it
was
based
on
those
labels,
you
could
run
it
on
vSphere
and
an
admin.
Maps
storage
classes
to
physical
data
stores
or
classes
and
the
users
consume.
Based
on
the
outcome
specifications
you
could
take
that
to
a
public
cloud
like
Amazon
and
have
a
mapping
defined
to
use
those
same
class
names
and
just
works.
It
gives
you,
the
you
know,
maps
the
expensive,
fast
storage
for
your
database
indexes
and
the
cheap
storage
for
your
log
files,
and
it
is
portable
if.
B
I
take
the
same
example
that
are
shown
in
the
demo,
with
what
you
were
asking
you
would
see.
Basically,
we
had
three
workers
in
that
specific
region.
So
if
you
were
to
drain
a
node,
let's
say
drain
Kate's
worker
nine,
which
is
the
one
that
the
pub
was
scheduled
on.
If
you
drain
that
node
and
you
remove
that
node,
the
pod
will
be
rescheduled,
the
disk
will
be
moved
and
reattached
to
another
worker
inside
that
region.
A
Under
the
covers,
kubernetes
as
an
Orchestrator,
along
with
the
CSI,
is
handling
these
detach
and
reattach.
Now
they
they
can't
defeat
things
like.
If
you
have
data
plane,
communication
limitations,
where
certain
volumes
aren't
simply
aren't
detachable
in
some
zones
or
regions,
they
aren't
going
to
create
some
miracle
where
it's
going
to
hook
those
up,
but
in
general,
if
you
intelligently
define
these
things
will
just
work,
and
you
don't
have
to
manually
deal
with
attach
the.
B
Attach
as
long
as
you
have
the
underlying
infrastructure,
the
underlying
vSphere
design
done
correctly,
where
you
have
multiple
workers
that
are
able
to
access
the
same
VMDK.
So
they
have
a
path
to
access
that
VMDK.
Then
it's
gonna
be
automatically
dealt
with
by
kubernetes
and
CSI.
Obviously,
that's
CSI
is
doing
the
work
really
so.
A
They
have
a
plan
to
eventually
replace
the
existing
entry
with
stubs.
That
would
simply
call
the
CSI
implementation,
but
that
isn't
implemented
yet
so
you
can't
get
that
today.
But
that
is
that's
the
closest
thing
to
a
long-range
plan
that
has
been
stated,
and
this
isn't
unique
to
VM
to
the
vSphere
implementation.
The
plan
is
to
pull
this
off
or
all
forms
of
storage.
E
E
A
And
when
it
comes
to
admin,
there's
a
concept
of
both
kubernetes
admins
as
well
as
admins
of
the
underlying
platform,
and
it
depends
on
your
organization
how
fragmented
they
are.
You
know
I've
seen
some
where
you
have
admins
for
Network
admins,
for
storage,
admins
or
compute,
and
others
that
that's
one
role
where
they
have
hundred
percent
Villas
visibility
to
define
these
storage
classes
in
kubernetes.
You
have
to
be
a
kubernetes
admin.
A
B
A
The
kubernetes
project
is
reforming
these
SIG's
that
are
related
to
cloud
providers.
So
by
the
end
of
the
year.
The
plan
is
that
there
will
be
a
user
group
and
then
the
group
doing
of
developers
doing
the
VMware
cloud
provider
are
going
to
be
folded
into
a
group
underneath
the
cloud
providers
sake
so
expect
this
to
change.
But
for
now
those
links
are
good,
join
the
group
and
join
our
meetings.
If
the
timezone
is
bad
because
you're
in
China,
the
meetings
are
recorded,
both
notes
and
YouTube
videos
and.