►
From YouTube: Building a Controller Manager for Your Cloud Platform - Fabio Rapposelli, VMware & Chris Hoge
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building a Controller Manager for Your Cloud Platform - Fabio Rapposelli, VMware & Chris Hoge, Openstack Foundation
The Cloud Controller Manager (CCM) concept was created to allow cloud specific vendor code and the Kubernetes core to evolve independent of one another, with Kubernetes v1.11, CCM has graduated to Beta and in upcoming releases, it will be the preferred way to integrate Kubernetes with any cloud. This talk will expand on the CCM documentation available online and explore in detail how a Cloud Controller Manager is built, what are the testing strategies and how it can be deployed alongside Kubernetes.
https://sched.co/MPdH
A
All
right,
good
afternoon,
everybody
hope
you
know
you
got
over
to
food
coma
after
lunch.
You,
you
know
you're
gonna,
get
better
after
this
session,
so
we're
here
today
to
talk
about
building
your
clock,
enthralling
manager
for
for
kubernetes.
So
there's
been,
you
know
a
lot
of
discussion
around
this.
If
you've
been
attending,
you
know
the
various
sakes
like
sig
note
and
say
cloud
provider
and
the
various
you
know:
claw
provider
SIG's
like
Sig
VMware,
seeing
OpenStack
CWS
and
so
on.
We're
we're
gonna,
go
there.
A
The
documentation
on
this
specific
part
of
the
kubernetes
codebase,
where
the
company's
ecosystem
is
a
little
bit.
You
know
foggy
so
we're
we
wanted
to
give
some
documentation
some
overview.
Some.
You
know
more
information
on
what
is
what
is
happening
in
this
space
before
we
get
started.
I
just
want
to
briefly
introduce
ourselves
again:
I
guess
my
name
is
Fabio
poiseuillie
I
work
at
VMware
software
engineer,
I'm,
working
in
upstream
kubernetes
and
also
on
product
on
the
side.
I'm
also
co-leader,
co-chair
of
cig
VMware,
and
we
have
Chris
Chris.
B
A
So
today
we
have
like
agenda
with
it's
divided
in
three
major
blocks,
so
we're
going
to
talk
about
what's
the
print
status,
then
we're
going
to
go
over
on
how
to
build
your
color
controller
manager
and
there's
a
third
one.
We're
gonna
give
you
some
examples.
You
know
taking
you
the
color
provider
OpenStack
in
a
club
provider
via
vSphere
as
examples,
so
let's
get
started
with
the
current
state
of
the
cloud
providers.
So
today
you
know
when
you,
when
you
you
know.
Sometimes
we
hear
the
term
entry.
A
But
right
now
we
have
this
8
club
providers
built
inside
of
the
kubernetes
core,
so
whenever
you
download
it
even
with
114,
so
the
very
rest
of
these
even
115,
now
you're
gonna
have
all
the
code
for
this
club
providers
bundling
to
your
kubernetes
refineries
and
whenever
you
around
them,
you
have
all
this
unnecessary
code,
because
you
know
you're,
probably
running
kubernetes
on
one
of
these
clouds
yourself
or
maybe
not
even
using
this
code
potentially.
So
it's
a
lot
of
code.
That
is,
you
know.
Sometimes
it's
maintained.
A
Sometimes
it's
you
know
less
maintain
the
quality
of
the
testing
for
all
these
provider
is
very
I
would
say
it's
a
mixed
bag.
Ok,
and
when
it
comes
to
maintaining
this,
it's
really
not
something
that
people
that
are
not
working
with
a
platform
itself
can
do
so
it's
all
delegated
to
other.
You
know
people
that
have
the
domain
knowledge
for
that.
So
why
is
there's
a
problem?
Obviously
the
problem
is
that
you
know
kubernetes
should
not
be
in
the
business.
Copernicus.
A
Corps
should
not
be
in
the
business
of
maintaining
this
drivers,
so
drivers
should
be
maintained
independently
by
the
companies
or
the
people
that
know
that
platform
very
well.
They
have
the
domain
knowledge
for
that
platform.
And
again,
if
you,
you
have
a
deployment
of
kubernetes,
possibly
you're,
not
running
that
on
all
these
different
clouds
at
the
same
time,
you're
running
on
one
single
cloth,
so
you
want
to
have
a
more
you
know
specific
configuration
for
just
that
cloud,
so
you
don't
want
to
have
all
this
craft.
A
A
You
know,
given
that
a
code
is
part
of
kubernetes
poor,
it
kind
of
implicitly
implies
that
it's
supported
or
endorsed
by
the
kubernetes
poor.
You
know
team,
and
this
is
not
necessarily
true-
I
mean
the
code.
Is
there
because
you
know
it
was
useful
at
the
time
and
it's
still
useful,
but
does
not
imply
that
kubernetes
is
supporting
that
in
any
specific
way.
A
One
very
big
problem
that
we
have
in
the
past,
specifically
in
the
past
I,
guess
through
three
minor
version
of
kubernetes
being
that
there's
been
some
critical
fixes
that
had
to
be
delivered
in
those
cloud
providers,
so
think
of
it
like
if
there's
a
there's,
a
CVE
that
has
to
be
fixed
and
the
code
of
the
CVE
is
specifically
for
that
cloud
provider.
It
has
to
go
through
the
normal
release
cycle
of
kubernetes,
which
means
you
know
it
has
to
go
into
a
patch,
so
he
has
to
go
or
into
a
new
minor
release.
A
So
it
takes
time
to
get
there
and
has
to
be
reviewed.
It
has
to
go
through
the
you
know,
traditional
review
process
of
kubernetes,
which
is
you
know
fast.
You
know,
if
you
look
at
that,
if
you
compare
that
to
a
traditional
software
development
cycle,
but
it's
still
a
long
time
if
you
need
to
go
and
fix
one
specific
problem
that
he'd
have
so
in
some
of
the
work
I
think
I
mentioned
this
before
some
of
the
work
on
some
of
the
providers
has
actually
stopped.
A
You
know,
maybe
months
or
years
ago,
and
some
of
those
providers
aren't
they're,
just
you
know
being
there
and
lying
there
as
element
in
code
and
people
are
you
don't
really
want
to
hold
a
band-aid
because
sometimes
when
you
pull,
you
know
rip
off
some
of
the
code
that
is
there.
You
end
up
with
unintended
consequences
that
you
really
want
to
get
into,
especially
if
you're
working
on
a
release
or
kubernetes.
So
if
we
want
to
fix
this,
we
need
to
decouple
the
code
that
is
cloud
provider
dependent
from
the
core.
So
to
fix
this.
A
A
Obviously,
this
is
a
is
proved
to
be
a
very
long
and
painful
process
because
again
ripping
the
band-aid
removing
the
code.
You
know
decoupling
that
code
that
is
tied
into
so
many
part
of
the
kubernetes
core
is
not
an
easy.
You
know
an
easy
thing
to
do,
especially
since
you
know
some
of
the
providers
took
some
liberties
in,
in
terms
of
you
know,
adding
stuff
inside
other
part
of
the
kubernetes
1/4
enemies
code
base,
so
ripping
those
out
as
being
proved
to
be
very
challenging.
A
I
think
this
process
has
been
going
on
for
I
think
at
least
two
years
and
now
we're
getting
to
a
point
where
we've
started
to
move
this
code
out.
So
all
the
major
providers
have
an
out
of
tree
implementation,
so
they
have
a
CCM,
create
it.
They
can
run
the
grown
as
a
sidecar
on
the
notes,
instead
of
being
bundled
inside
the
kubernetes
core
and
we're
actually
moving
we're
starting
moved
the
cloud
provider
code
that
is
part
of
entry
into
staging
repositories.
A
So
we're
trying
to
you
know
rip
the
band-aid
slowly,
instead
of
just
you
know,
taking
one
go.
The
idea
here
is
that
you
know,
through
the
with
the
help
of
cyclop
Revital,
which
is
kind
of
you
know,
pulling
the
strings
of
all
this
work.
We
want
to
have
by
the
end
of
2019,
so
with
the
release
that
will
be
released
at
the
end
of
2019
to
remove
all
the
entry
collateral.
B
Thanks
so
nice,
so
we
kind
of
laid
out
the
problems
of
why
you
know
the
problem
that
we
were
facing
is
having
entry
providers
and
how
to
make
a
more
level
playing
field
and
a
more
sustainable
ecosystem
for
the
multitude
of
cloud
providers
that
are
coming
in,
and
you
know
one
thing,
that's
really
great
about
the
community.
They've
actually
set
up
a
huge
amount
of
help
for
you
to
be
able
to
build
external
providers
that
can
allow
kubernetes
to
to
be
aware
of
the
resources
that
are
that
are
within
your
cloud.
B
So
we
have
this
Cloud
Controller
manager,
which
replaces
the
former
keep
controller
manager
and
as
a
daemon
that
enables
the
following
cloud:
specific
alerts:
there's
a
node
controller
in
it,
there's
a
route
controller
and
a
service
controller.
Now,
if
you're
familiar
with
the
cube
controller
manager,
it
also
had
a
volume
controller
in
it,
and
you
may
be
asking
yourself:
where
is
the
volume
controller?
And
this
work
has
been
largely
taken
over
by
the
zorka's
largely
been
taken
over
by
the
open
storage
I
mean
by
the
CSI
interface,
the
container
storage
interface?
B
And
so
you
know
by
loading
the
CSI
and
external
CSI
into
your
into
your
cluster.
You
should
be
able
to
get
a
volume
access
and
again
the
nice
thing
about
this.
Is
you
you
can
use
any
sort
of
volume,
implementation
that
you
want,
and
it
doesn't
have
to
necessarily
be
tied
to
your
specific
cloud
implementation,
and
so
it
gives
you
a
lot
of
freedom
and
a
lot
of
choice.
B
Okay,
so
you
say
you
want
to
build
your
own
controller
manager,
they're
only
three
steps
involved
in
this.
The
first
is
you
create
a
go
package
that
satisfies
the
cloud
provider
interface,
it's
very
basic
interface,
it's
pretty
clear
on
how
to
set
it
up
and
you
provide
an
implementation
for
that
and
you
have
a
go
package
and
then
there
is
a
you:
create
a
copy
of
the
cloud.
Controller
manager
main
go
function,
which
is
part
of
the
kubernetes
distribution
and
import.
Your
package,
making
sure
that
there's
an
init
block
available
and
what
this
does
is.
B
This
creates
an
application
that,
when
you
run
it
will
connect
to
your
cube,
API
server
and
provide
the
services
that
you
need.
Then
you're
done
right,
there's
no
step
two,
except
for
the
building
and
testing
and
packaging
and
everything
else
you
have
to
do
to
have
a
successful
cloud
provider,
but
but
at
its
added
edits
at
its
first
level.
These
are
the
things
you
have
to
do
and
the
tools
are
there
to
help.
You
succeed
in
it,
and
so
those
are
some
of
the
things
that
were
going
to
cover
over
the
next
few
slides.
B
So
if
you,
if
zones,
don't
make
sense
for
your
cloud
provider,
then
you
don't
need
to
implement
zones,
and
when
we
look
at
the
interface
right
here
like
you
can
fit
the
entire
interface
on
a
single
screen,
and
so
at
every
point,
these
functions
have
a
bool
associated
with
it.
That
tells
you
whether
or
not
that
interface
is
implemented,
and
so,
even
though
you
have
all
of
these
things,
there,
you
only
implement
what
you
need
or
you
can
deliver.
B
Okay,
so
the
next
thing
that
you
need
was
to
build
a
cloud
controller
binary.
The
external
clock
controller
runs
as
a
separate
binary
that
interacts
the
kubernetes
api
service
and
it's
been
speak,
it's
configured
with
the
standard
set
of
options,
and
so
when
you
build
one
of
these
binaries,
if
you
actually
run
it
and
ask
for
help,
you're
gonna
get
a
long
list
of
options
that
look.
B
There
are
some
additional
things
that
you
need
to
do
on
this,
like
the
application
by
itself.
Isn't
the
best
thing
for
deploy
into
into
a
cluster
it's
nice
to
be
able
to
load
this
into
a
docker
container
and
then
and
then
run
that
in
your
cluster
and
we'll
talk
about
that
a
little
bit
more
later
in
the
talk,
but
the
one
thing
that
all
of
this
doesn't
do
for
you
is
providing
testing
yeah
right
and
there
are
two
major
types
of
testing
that
you're
gonna
need
to
do
you
know
and
and
likely
more.
B
B
You
should
test
them.
You
know,
and
this
is
pretty
straightforward-
you
know
use
whatever
mocks.
You
need
to
do
also
write
whatever
functional
testing
you
need,
but
you're
also
gonna
have
to
set
up
end-to-end
testing
you'll
want
to
make
sure
that
when
you
run
your
Cloud
Controller
manager
on
your
cloud
that
you're
providing
all
of
the
services
in
the
way
that
Corona
expects
them
to
be
provided
and
the
way
you
that
you
do
this
is
with
the
end-to-end
testing.
You
have
a
cloud
available
to
you
or
you
stand
on
the
test
cloud.
B
B
And
the
nice
thing
about
that,
too,
is
that
you
can
report
that
the
test
grid
and
and
and
and
with
enough
tracks,
you
know
if
you,
if
you
have
a
large
enough
footprint
on
your
cloud,
you
can
actually
be
common
gating
job
for
kubernetes
release
make
sure
that
communities
will
be
released.
Yeah,
there's
a
break
in
kubernetes
that
breaks
on
your
cloud,
then
that
will
be
fixed
okay.
So
to
run
this,
the
main
arguments
are:
when
you
start
your
cube,
API
server,
you
need
to
provide
cloud
upfront
provider
external
formally.
B
You
would
provide
cloud
provider
in
the
direct
name
of
the
provider.
In
this
case
you
say
it's
external
and
then,
when
you
start
your
CCM
binary,
you
take
that
same
cloud
provider
argument
and
give
it
your
cloud
provider
name
the
one
that
you
define
in
that
application.
We
showed
you
earlier
and
then
point
to
your
cloud
config
and
then
what
other
options
that
are
necessary.
Typically,
these
you
know
these
are
things
that
are
specific
to.
B
You
know
behaviors
that
your
cloud
expects
that
you
can
set
in
there
and
again,
the
similarity
of
these
options
comes
from
the
scaffolding
code
and
there
may
be
a
lot
of
unneeded
options
and
at
an
earlier
session
from
this
week,
people
actually
raised
some
concerns
that,
like.
Why
is
it?
When
I
run
the
Cloud
Controller
binary,
then
I
get
all
of
these
other
things
associated
with
it,
and
so
it's
something
that
sig
cloud
provider
is
aware
of
it
and
we
might
be
able
to
turn
our
attention
to
in
upcoming
releases.
B
B
B
The
second
is
cloud.
Information
about
nodes
in
the
cluster
is
no
longer
going
to
be
provided
using
local
metadata,
but
instead
of
API
calls
to
retrieve
node
information
will
go
through
the
cloud
controller
manager,
and
so
the
API
server
is
going
to
start
pinging
your
cloud
and
a
might
start
in
your
cloud
a
lot,
especially
as
a
number
of
nodes
and
your
workload
increases,
and
so
you
need
to
be
aware
that
if
you're
running
particularly
large
workloads,
you
may
need
to
adjust
the
the
rate
limiting
on
the
API
server.
B
Just
you
know
that
tells
you
how
to
set
it
up
and
how
to
run
it
so
that
you
can
load
your.
You
can
build
a
docker
container
with
your
cloud
controller
image
in
it
and
how
to
run
it,
and
these
slides
will
be
available,
and
these
are
also
all
available
on
the
on
the
kubernetes
documentation
sites
and
will
be
able
to
fight
the
links
in
the.
A
Slides
to
those
can
I
just
make
one
point
here,
given
that
when
the
cluster
comes
up,
it
has
a
cane
that
says
no
schedule
effectively.
We
want
to
schedule
a
pot
on
it
right
now,
because
we
want
to
schedule
the
CCM
in
order
to
remove
that
taint.
So
it's
a
sort
of
a
chicken-and-egg
direct
intuition.
So
in
order
to
make
this
possible
there's
the
oration
part
that
says:
oh,
this
yamo
five,
this
is
this
deployment.
B
Okay,
thinking
about
some
implementation
details,
you're,
probably
going
to
need
a
library
that
interacts
with
your
cloud
right,
and
so
that's
one
of
the
first
things.
You're
gonna
have
to
find
as
a
library
a
go
library
that
interacts
with
your
cloud
and
that's
the
first
step,
because
there
are
likely
many
other
things
within
the
kubernetes
ecosystem
that
you
want
to
be
able
to
interact
with
your
cloud,
and
so
it's
best
to
have
some
sort
of
library
to
interact
with
that,
rather
than
to
implement
all
those
details
directly
into
your
cloud
provider.
B
The
second
is,
you
want
to
think
about
how
you
handle
authentication
and
authorization.
It's
likely
that
your
Cloud
Controller
isn't
the
only
thing
that's
interacting
with
your
cloud,
especially
with
cluster
api
coming
along.
This
is
going
to
be
an
incredibly
important
API
that
you're
gonna
want
to
support
within
the
kubernetes
equal
system,
because
their
goal
is
to
become
the
de
facto
way
that
you
do
not
only
cluster
installation
but
also
upgrading
and
auto-scaling,
complete
lifecycle
management
of
your
cluster,
and
so
this
is
something
you
want
to
pay
attention
to.
Along
with
your
cloud
provider,.
B
You
know
so
you
want
to
consolidate
you
out
your
efforts,
otherwise
you're
gonna
wind
up
with
fragmented
and
inconsistent,
often
configuration
that
leads
across
your
projects.
Speaking
from
experience
on
that
one
and
speaking
about
experience.
Moving
on
to
part
three
of
the
talk
were
just
some
examples
that
are
drawn
from
our
own
experiences
and
developing
these
providers,
and
so
the
first
is
with
the
OpenStack
cloud
provider.
B
We
use
gopher
cloud,
it's
the
OpenStack
SDK
that
we
use.
We
have
a
number
of
different
sub
controllers
that
are
independent
of
one
another
and
so
we're.
Actually
it
makes
it
easier
if,
for
us
to
kind
of
load
and
unload
things
that
are
running
or
nod
and
have
a
little
bit
better
separation
of
concern
across
the
codes,
we
also
don't
implement
zones.
B
It's
zones
has
an
open
stack,
abstraction
would
be
a
kind
of
multi
cloud
type
of
situation
and
since
there's
really
no
commonly
adopted
way
to
federates
across
the
OpenStack
zones,
aren't
something
that
are
you
know
something
that
we
think
about.
So
it's
just
not
implemented.
You
know
and
things
work
fine.
B
B
Finally,
we
we,
we
reuse
the
distant
tools
for
standing
up
the
various
test
environments.
So
you
know,
testing
is
really
important
for
us
and
we
put
a
lot
of
thought
and
effort
into
this
and
we
actually
allow
for
a
matrix
of
compatibility
testing
by
building
upon
the
tools
that
are
already
out
there.
So
the
first
thing
that
we
do
is
we
bring
up
a
test,
OpenStack
cloud
using
something
called
dev
stack,
which
is
just
the
standard
tool
that
we
use
to
drink
up
clouds.
B
Then
we
run
local
up
cluster,
which
is
one
of
the
standard
kubernetes
tools
for
standing
up
test
clusters
of
kubernetes
inside
of
that-
and
this
is
actually
a
really
cool
script
to
look
at.
You
should
pay
attention
to
this
because,
when
you're,
when
you're,
when
you're,
when
you're
developing
your
Cloud
Controller
manager,
there
is
code
in
here
that
kind
of
shows
you
how
to
set
those
things
up
and
have
them
set
up
your
tests,
and
there
are
environment
variables
that
you've
said
to
bring
it
up,
and
so
it's
actually
really
nice
tool
to
do.
B
Testing
on.
We
used
it
launched
at
kubernetes
cluster
on
the
OpenStack
cluster,
using
the
custom
cloud
controller
manager
and
then
finally,
we
use
e
to
e
go
to
run
and
and
tests
on
it
join
the
conformance
testing
on
the
cluster,
with
the
results
reported
back
to
test
grid,
and
all
of
this
is
orchestrated
with
ansible
playbooks
and
launched
on
the
sole,
and
so
you
know
the
level
of
effort.
You
know,
the
actual.
You
know
unique
code
that
we
had
to
get.
This
running
was
pretty
small.
A
lot
of
this
RDS.
A
So
vSphere
of
the
club,
a
lot
of
useful
information
is
relatively
new
because
we
we
had
the
vSphere
support
inside
the
kinetics
course
of
entry
for
a
very
long
time,
and
it
was
mainly
used
because
it
also
supported
storage
inside
the
entry
provider.
So
part
of
that-
and
the
very
first
bullet
point
on
this
slide
is
actually
one
of
the
points
I
was
making
you
know
in
earlier
and
during
the
presentation.
A
So
one
of
the
big
problems
we
have
with
the
code
inside
the
kubernetes
core
is
that
some
of
these
SDKs
that
we're
importing
in
are
huge.
One
of
the
main
offenders
is
actually
our
own,
so
gooey
mommy
is
our
Go
Go
SDK
for
the
vSphere
API,
and
the
types
package
is
huge.
It's
gigantic.
It's
23
megabytes,
so
actually
I
heard
that
the
golang
team
is
using
it
as
a
benchmark
when
they
run
benchmark
for
go
because
it's
so
big,
it's
Auto
generated
by
the
way.
A
It's
not
we're
not
doing
it
by
hand,
but
anyway,
regardless
the
the
problem
is
that
we
have
all
this
all
these
SDKs
that
are
built
into
the
core.
So
R
is
one
of
the
largest
GCE
is
also
very
big.
So
if
you
add
you
know,
we
have
eight
of
them.
So
if
you
add
up
all
these,
it
becomes
gigantic,
so
keeping
them
out
of
the
core.
It's
always
a
good
idea
in
terms
of
the
actual
club
provider
for
vSphere.
A
A
We
don't
have
that
so
we're
not
supporting
that
directly,
but
there's
more
to
come
in
that
we
support
vSphere
from
version
6.5,
which
is
you
know,
it's
been
around
for
quite
a
while
now,
so
we
feel
like
it
was
a
good
starting
point
for
for
support
and
we
do
share
the
secret.
So
when
you
configure
the
claw
provider
vSphere,
we
do
have.
We
manage
the
credentials
to
connect
to
our
cloud
so
to
get
the
vSphere
through
a
violence
inside
kubernetes.
A
We
share
the
same
secret
file
with
CSI
vSphere,
which
is
for
storage
support
in
terms
of
supporting.
We
also
have
support
for
zone
which
was
recently
introduced.
I
think
it
was
a
3-4
months
ago.
This
was
actually
a
big
deal
for
us
because
we
didn't
support
it
initially,
because
the
problem
with
availability
zones
is
that
vSphere
is
an
on-prem
software
that
you
install
and
you
architect
yourself.
So
you
don't
have
one
one
type
of
a
little
bit
zones.
You're,
not
you
know,
buying
service
from
a
cloud
that
defines
the
availability
zones
for
you.
A
You
have
to
define
them
for
you
and
it's
a
pretty
interesting
helping
in
pretty
large
topic.
I
mean
there
are
entire
books
about
it.
So
there's
a
lot
of
there's
a
lot
of
work
and
a
lot
of
different
opinions
on
how
to
architect
fault
domains
for
vSphere,
so
we
had
to
find
a
way
to
actually
map
those
concepts
through
the
you
know,
fixed
concept
that
we
have
in
kubernetes,
so
zones
and
regions.
So
we
use
vSphere
attacks.
A
So
you
can
go
into
vSphere,
you
can
tag
your
data
centers
with
zones
and
in
region
tags
and
we
will
take
them
and
use
them
inside
the
kubernetes
club
provider
vSphere
to
map
those
availability
zones
between
the
two
so
again,
I
put
the
new
acronym
there
so
bring
your
own
topology.
So
you
make
your
own
topology
in
vSphere
and
you
figure
out.
How
do
you
want
to?
A
You
know,
use
that
into
your
inside
your
Q
bananas
clusters
that
you're
running
on
it
it
even
zone
support
is
actually
important
in
the
club
provider
because
it
enables
the
same
support
for
CSI.
So
when
you,
when
you
want
to
run
when
you
want
to
do
something,
particularly
with
the
storage
topology,
you
need
to
have
that's
on
this
information
that
zones
information
comes
from
the
CCM.
So
it's
it.
It's
symbiotic
alliance
between
the
two
and
then
what
about
starts
so
well
in
controller
group
is
not
implemented.
A
So
this
was
actually
again
another
big
pain
point
for
us,
because
again
we
had
a
you
know:
long-standing
entry
implementation
that
was
mainly
used
for
storage
and
we
had
to
find
a
way
to
transition.
You
know
you
know
end
users
that
are
using
call
provider
vSphere
entry
today
to
the
auditory
model.
So
building
a
CCM
was
not
enough.
We
had
to
build
a
CSI
plugin.
A
So
today,
if
you're
running
vSphere
entry
with
of
entry
claw
provider-
and
you
want
to
move
to
the
auto
tree-
you
have
to
move
to
both
CCM
and
CSI
at
the
same
time,
and
this
for
us
is
also
a
problem
in
terms
of
support
ability,
because
this
adds
another
dimension
to
the
number
of
versions
that
we
have
support.
So
if
you
look
at
the
support
matrix,
we
have
you
know
kubernetes
version
and
now
out
of
tree
CCM,
which
is
another
version.
A
Then
we
have
CSI,
which
is
another
version
and
plus
you
know,
since
it's
a
it's
a
it's
a
non-print
product,
we
also
have
vSphere
version,
so
we
have
so
many
dimensions.
The
support
makes
it's
kind
of
a
nightmare
right
now,
we're
trying
to
standardize
that
and
making
sure
that
we
can
support
anymore,
I,
would
say
linear
way
and
future
things.
A
So
this
is
actually
an
interesting
thought
that
we
got
some
feedback
from
users
of
vSphere
on
AWS
vSphere
edibility
asked
for
people
that
don't
know
it's
basically
a
vSphere
cluster
running
on
a
device
which
means
that
you're
you
have.
This
risk
release
your
cluster,
that
has
you
know,
access
to
some
advanced
services
coming
for
an
FPS
directly,
so
I'll
say
that
before
you
know,
vSphere
on-prem,
the
plain
vSphere
doesn't
have
a
little
dinosaur
facility,
for
example.
A
So
you
have
to
you
know:
either
build
your
own
or
bring
your
own
from
you
know,
and
insects,
sort
of
some
other
commercial
product,
vSphere
and
ABARES
has
access
to
elastic
load
balancer
from
a
table.
Yes,
the
problem
is
that
it's
two
different
cloud
providers
right.
You
have
vSphere
and
you
have
AWS.
How
do
we?
How
do
we
make
sure
that
we
can
consume
both
resources
in
the
same
kubernetes
closet?
So
this
is
still
an
open
question
which
is
researching.
You
know
discovering
how
to
do
that,
but
it's
an
interesting.
A
You
know
hybrid
situation
that
we're
getting
into
that
was
not
explored
before
so
and
again.
If,
actually,
if
people
have
any
questions
or
feedback
on
this
be
happy
to
hear
that
either
directly
or
during
one
of
the
sig
calls
SiC
la
provider,
speaking
of
seek
la
provider,
you
know-
and
this
is
the
last
lie-
were
you
any
longer
on
this
say?
A
Claw
provider
is
actually
the
focal
point
for
anything
that
is
happening
around
these
discussions,
so
everything
that
is
and
is
happening
in
terms
of
going
out
of
Tresor,
removing
all
this
code
building
new
cloud
providers,
even
if
you
have
a
if
you
are
a
cloud
provider,
if
you
are
a
new
cloud
provider
and
you're
building
your
own
clock,
controlling
manager,
I
make
sure
you
join
SiC
claw
provider,
because
everything
is
and
everything
is
happening
in
there
in
the
convenience
community.
We
do
have
three
major
caps,
so
obviously
they
will
be
available.
A
This
light
will
be
available
in
in
the
conference
page,
but
there's
three
major
caps.
You
want
to
look
at
if
you're
interested
in
this.
We
have
you
know
the
main
cap
for
plug
controller
manager,
which
is
the
very
first
one.
We
have
the
tested,
conformance
requirements,
which
are
you
know,
as
was
aligning
before
you
know.
A
So
again,
if
you're
interested
in
this
topic
make
sure
you
go
checked
out
these
caps
and
if
you
have
feedback,
you
know,
sick
la
provider
meets
every
other
week
on
Wednesday
make
sure
you
join
the
call
and
get
your
feedback
and
wait
at
that
said,
we
are
wrapping
up
the
session.
Any
questions
minutes.
C
D
B
A
Terms
of
recommend
it's
a
recommendation
versus
you
know
a
requirement,
so
it's
not
a
requirement
to
run
it
as
a
daemon
set.
You
can
run
it
as
a
static
pot.
I
tend
to
recommend
the
daemon
set,
because
you
can
run
the
demon
set,
all
the
master
notes
that
you
have
and
that
will
make
it.
You
know
more
available,
I
guess
using
a
static,
but
again
there's
no
strict
requirement
on
how
to
run
it.
You
can
even
run
it
as
a
binary.
You
know,
alongside
the
cubelet,
for
example,.
E
B
So
I
think
so
it's
my
understanding
that
the
long
term
plan
for
for
cluster
auto-scaling
is
actually
going
to
be
very
much
baked
into
the
cluster
API
project,
because
once
you
have
do
you
have
machine
sets
and
everything
kind
of
like
comes
down.
Who
you
want
to
like,
say:
oh
I
need
to
auto-scale.
You
ask
for
more
machine
sets
that
implementation
happens
at
the
cluster
API
level,
and
then
everybody
can
use
the
same
techniques
for
auto
scaling.
B
A
One
thing
that
I
didn't
mention
their
presentation,
but
it's
actually
something
that
people
might
be
interested
in
doing
it.
You
know
this
obviously
is
open.
Source
code
is
available
and
can
be
run.
You
know
without
forking
the
kubernetes
core.
So
if
you
have,
if
you're
running
on
a
club
provider,
you
there's
there's
a
feature
that
you
really
want
to
have
or
like
an
integration
that
I
really
want
to
have.
You
can
fork
the
code
for
the
auditory
Club
provider.
Add
your
own
code
to
it
and
then
run
it
as
a
pod.