►
From YouTube: Cloud Provider Extraction: What We’ve Done, Where We Are and What's Left- W Fender, S Wong, N Turner
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cloud Provider Extraction: What We’ve Done, Where We Are and What's Left! - Walter Fender, Google; Steven Wong, VMware; Nick Turner, Amazon
Cloud Provider code allows Kubernetes to run on top of different platforms. Originally, support for all variations was built directly into a K8s release. This brought bloat - a running cluster used only a subset of the code within the release. Also, any cloud specific bug fix or enhancement demanded a new full K8s release as a carrier. This will be a status report on removing the cloud provider code from the main Kubernetes repository. Significant early milestones were recently achieved: the API Server no longer needs the cloud provider library; SSHTunnels have been deleted. For each in-tree cloud provider, we will report on efforts, accomplishments, and roadmap for getting "out-of-tree". We’ll also discuss the plans to handle the speed bumps that are left - including interesting topics like multi-repo e2e testing and removing the cluster directory.
A
We
also
have
people
representing
many
other
popular
cloud
providers
who
couldn't
be
with
us
here
physically,
but
who
graciously
prepared,
lightning
talk,
recordings
or
slides
which
we'll
be
sharing
in
the
middle
of
this
talk,
if
you're
wondering
about
the
cloud
suit,
of
course
it's
sig
cloud
provider,
but
it's
just
a
demonstration
that,
in
spite
of
walter's
rule
with
an
iron
fist,
as
a
co-chair,
we
do
have
some
fun
and
levity
in
the
group.
So
come
to
meetings
and
enjoy
yourself.
A
The
way
we'll
start
out
is
I'll
I'll
quickly
explain
what
a
cloud
provider
is
about,
just
in
case
anybody
here
wandered
in
and
isn't
familiar
with
the
kubernetes
architecture
and
then
we'll
move
on
to
a
background
on
our
strategic
mission
to
move
the
cloud
provider
specific
code
out
of
the
main
kubernetes
source
tree,
then
nick's
going
to
give
a
general
status,
followed
by
cloud,
the
cloud
provider,
specific,
lightning
talks,
and
then
finally,
walter
will
close
covering
roadmap,
interesting
topics
and
how
you
can
get
involved
with
the
group,
so
the
cloud
provider.
What
is
it
well?
A
This
is
the
part
of
the
architecture
that
allows
kubernetes
apps
to
be
portable
across
various
public
and
on-prem
clouds.
The
goal
is
to
allow
a
well-written
app
to
run
anywhere
and,
for
the
most
part,
not
even
able
to
tell
where
it's
running
sig
cloud
provider
works
closely
with
a
few
other
sigs
to
make
this
happen.
For
example,
most
of
the
related
abstraction
is
actually
done
by
sig
storage,
but
there
are
cross-sig
integration
efforts
and
design
reviews
that
go
into
this.
A
B
So,
as
many
of
you
probably
know,
the
cube
controller
manager
is
a
basically
a
collection
of
control
loops
and
the
control
loops
might
be
acting
on
kubernetes
api
resources
and,
in
some
cases,
they're
also
going
to
be
creating
or
updating
or
deleting
cloud
resources
as
well.
So
the
whole
effort
to
migrate
the
cloud-related
code
out
of
tree
affects
the
cube
controller
manager
and
the
way
that
we're
going
to
do
that.
Migration
is
by
moving
those
cloud
loops,
those
any
of
the
loops
that
actually
touch
cloud
resources
to
a
vendor-specific
binary
called
the
cloud
controller
manager.
B
So
this
effort
is
pretty
far
along.
Many
vendors
already
have
cloud
controller
managers
at
various.
You
know
some
rga.
Some
are
alpha
alpha
summer
beta,
but
you
know
the
vendors
that
you're
interested
in
you
can
find
those
repositories
and
look
at
them
and
and
see
how
far
along
they
are
so
specifically
for
the
cube
controller
manager.
You
would
set
the
cloud
provider
flag
to
be
external
and
when
you
do
that
those
control
loops
are
disabled,
and
so
you
would
do
that
when
you
would
start
the
cloud
controller
manager
along
with
it.
B
The
first
is
the
ssh
channel
functionality,
and
this,
as
of
122,
has
already
been
dropped
and
is
replaced
by
the
network
proxy
project,
also
known
as
the
connectivity
agent.
If
you
have
any
questions
about
that,
walter
is
your
guy
and
the
second
area
is
the
persistent
volume
labeling
admission
plugin.
B
So
this,
unfortunately,
is
taking
a
little
bit
longer.
It's
probably
going
to
be
the
long
pole
with
extracting
cloud
provider
related
code,
but
luckily
we
did
get
a
kepp
merged
in
123
which
will
make
this
migration
a
little
bit
easier.
So
it's
going
to
build
some
some
framework
that
allows
cloud
providers
to
build
web
hooks,
as
replacement
for
this
admission.
Plugin
and
that's
related
to
labeling
manually
created
persistent
volumes
with
these
topology
labels
and
then
another
component
that
is
important,
is
cubelet.
B
So
a
cubelet
can
initial
initialize
a
node
without
setting
the
addresses
and
the
cloud.
Node
controller
can
asynchronously
populate
those
addresses
and
then
another
aspect
of
cubelet-related
code
would
be
or
cloudflare-related
code
in
cubelet
would
be
the
external
image
credential
provider
providers-
and
this
is,
I
think,
beta
going
beta
in.
C
B
And
we're
adding
a
feature
gate
to
sort
of
push
people
along
which
will
start
out
being
disabled,
but
it
will
eventually
become
enabled,
which
will
then
go
and
disable
the
credential
plugins.
So
when
that
reaches
beta,
you'll
have
to
actually
go
and
flip
that
feature
gate
to
to
false.
If
you
want
to
continue
using
it
yeah.
So
take
a
look
at
that
cap
and
the
entry
volume
plugins
are
the
final
piece
of
cubelet,
so
these
are
being
replaced
by
csi
in
terms
of
queue,
controller
manager,
migration.
B
D
B
So
take
a
look
at
that
release
on
github
if
you're
interested
and
the
ebs
csi
driver
recently
released
version
131
with
multi-arch
os
image
manifest
and
dcr.
So
take
a
look
at
that
as
well,
if
you're
interested
and
what's
coming
soon
so
the
cloud
controller
manager
we're
working
on
a
improved
upstream
test
framework,
we'll
be
taking
advantage
of
the
last
known
good
testing
proposal
and
you
can
look
forward
to
the
cloud
controller
manager
being
enabled
on
eks
in
a
version
soon.
B
The
aws
load,
balancer
controller,
has
a
release
coming
out
pretty
soon,
also
a
bunch
of
features
loaded
into
that
release,
so
so
give
it
a
look
and
then
csi
will
be
enabling
csi
migration
on
eks
in
one
of
the
next
couple
releases
and
cubelet
image,
credential
provider
we're
working
on
the
first
release
of
that
with
binaries
and
documentation.
If
you
want
to
build
it
yourself,
it's
there
in
the
github
repository.
E
All
right,
my
turn
to
talk
assuming
this
is
on
yeah.
Okay,
my
turn
to
talk
we'll
get
to
find
out
quickly,
whether
or
not
as
a
googler,
I'm
you
know,
and
the
chair
I'm
capable
of
being
somewhat
neutral
on
these
slides.
So
the
azure
folks
actually
provided
these
slides,
andy
and
ping.
Fay
actually
work
out
of
china.
E
So
it's
a
little
hard
for
them
to
show
up,
but
they've
actually
been
doing
a
lot
of
work,
trying
to
make
sure
that
the
abstraction
layer
under
azure
works
and
that
the
microsoft
code
is
being
pulled
out.
So
we
see
a
lot
of
stuff
from
them
for
the
for
the
csi
drivers,
a
lot
of
that
getting
the
cloud
provider
abstraction
the
reference
implementation
for
microsoft,
so
that
any
of
you
can
go
and
look
how
to
bring
up
a
kubernetes
cluster
on
microsoft
and
get
it
all.
E
Working
they've
had
multiple
plot
patch
releases
and
they
actually
have
just
recently
gone
ga
with
a
121
release.
So
and
then
you
know
the
one
of
the
things
that
nick
mentioned
is
this
idea
about
the
credential
provider,
and
so
their
own
external
credential
provider
needed
to
go
to
acr
should
be
coming
out
very
soon.
E
E
E
E
As
far
as
bringing
the
cluster
up
right
now,
we
use
the
cube
up
script.
That
is
the
standard
for
that
has
been
the
standard
for
kubernetes
for
quite
some
time.
There
are
newer
technologies
out
there
and
we
are
looking
at
using
one
of
the
alternates,
but
that
that's
an
investigation
that
is
still
ongoing
right
now,
I'm
bringing
up
a
control,
plane
and
three
nodes,
and
so
we
will
see
that
come
up
soon.
E
Once
it
comes
up
we'll
go
ahead
and
check
to
see
that
the
that
all
the
nodes
that
I'm
expecting
are
there
and
then
we'll
take
a
look
at
the
pods
and
make
sure
that
the
key
pods
that
are
needed
for
extraction
are
there
so
great
we're
up
and
we're
running
now
we'll
go
ahead
and
take
a
look
at
the
nodes.
Okay,
all
four
of
my
nodes
are
there
and
look
to
be
ready?
Take
a
look,
we
see
the
ccm
is
there.
We
see
the
connectivity
agent
and
connectivity
servers.
E
E
E
Gcp
is
going
to
have
a
prototype
and
then
we're
hoping
to
roll
this
out
to
all
the
other
cloud
providers.
We'd
also
like
to
more
immediately
start
dealing
with
the
issue
of
the
cloud
provider
and
credential
provider
feature
gates
which
are
currently
blocking
the
extraction
effort,
and
after
that
we
should
be
good.
E
E
They
have
a
ccm
implementation
and,
interestingly,
their
implementation
has
a
couple
of
control
loops
that
actually
come
from
their
managed
offering
that
have
since
been
open
sourced,
so
that
anyone
who
is
interested
in
knowing
how
ibm
does
their
managed,
node,
control
or
load
balancing
can
now
see
how
they
do
that
their
csi
stuff
is
also
coming
along.
Quite
well,
and
you
know
there
they've
got
some
interesting
things
coming
up
with
the
api
provider.
So
definitely
you
know
if
you
are
at
all
interested
in
that
or
in
open
shift.
F
Hello,
my
name
is
nico.
I
worked
on
cd
jr
working
team
at
vmware,
so
today
I
will
share
a
few
updates
on
cloud
provider
this
year
since
we
1.18
there
are
multiple
new
features
implemented
in
this
report
and
the
first
part
is
we
implement
zone
and
region
support
so
that
we
can
initialize
a
node
with
cloud-specific
domain
region
labels.
F
F
These
same
tags
are
mapped
to
labels
in
quantities
allowing
placement
of
nodes
and
thus
pause
and
persistent
volumes
in
the
appropriate
zone.
Origin
and
we
also
add
new
instance,
type
labels
which
can
be
used
for
nodes.
This
is
useful
if
you
want
to
target
certain
workloads
to
certain
instance.
Types
like
small
and
medium
center
instance
type
offers
different
compute
memory,
storage
and
network
capabilities.
F
We
also
add
initial
support
for
nsxt
rapid
pass
so
that
past
ips
are
relatable
and
accessible
externally
from
the
cluster.
We
introduce
initial
support
for
vsphere
power.
Virtual
cloud
provider,
so
currently
we
have
two
modes
in
cloud
provider,
vsphere
projects
which
are
called
vsphere
and
vsphere
para
virtual.
It
has
different
implementations
for
cloud
provider
interfaces
like
instances
and
routes.
F
F
F
I'd
like
to
share
the
road
map
of
vsphere
cloud
provider
as
well,
so
first,
there
is
a
plan
for
the
cpi
migration,
the
cloud
provider
vsphere
runs
mo
a
node
route
and
load
balancer
control
rules
in
two
modes,
which
are
called
bispheria
and
paramount,
and
we
want
to
merge
those
two
nodes
into
a
single
one
so
that
we
remove
one
mode
then
freshly
install
the
other
one.
To
do
this,
we
might
also
need
a
new
manifest
for
different
resources
in
the
future.
F
E
All
right
so
we've
had
the
lightning
talks
for
the
cloud
providers
we've
sort
of
seen
what
each
of
the
individual
cloud
providers
is
interested
in
doing
in
the
effort
to
get
extracted.
But
I
think
it's
also
interesting
to
look
at
what
kubernetes
would
like
to
get
done
for
cloud
provider
in
upcoming
releases,
so
we'd
like
to
get
the
code
out,
why?
I
think
quite
a
bit
of
that
was
gone
over
by
steve,
but
I
mean
there
are
other
concrete
things
to
think
about.
E
E
So
what
did?
What
are
the
immediate
things
that
we
as
the
cloud
provider
and
we
as
the
community,
need
to
do
well?
We've
added
two
new
feature
gates
and
the
purpose
of
these
feature
gates,
the
disabled
cloud
provider
and
the
disabled
cubelet
cloud
credential
provider
is
to
see
what
happens.
If
you
delete
that
code.
We
do
it
by
removing
by
turning
off
off
everything
within
the
binaries,
and
if
you
don't
turn
it
off,
then
we
kill
the
binary.
E
E
E
E
So
we
either
need
to
rewrite
those
tests
to
run
in
a
non-google
specific
way,
or
we
need
to
just
realize
that
some
of
our
testing
has
to
happen
on
individual
cloud
providers
and
if
that's
true
that
what
that
means
is,
I
got
to
build
kubernetes
as
a
kernel,
and
then
I've
got
to
find
a
way
to
take
that
kernel,
put
it
into
a
cloud
provider,
bring
that
cloud
provider
up
and
then
run
my
test
there.
We
have
a
proposal
for
this.
I
mentioned
it
earlier,
I'm
mentioning
it
again.
It
is
the
last
known
good
proposal.
E
E
You
know
we've
got
assorted
cleanup.
We
really
do
do
a
lot
of
things
that
cover
a
lot
of
the
sigs,
as
was
mentioned
earlier.
One
of
these
is
the
cluster
anyone
who's
dealt
with
the
cluster
directory.
Anyone
who's
had
to
support
it,
I'm
one
of
them,
but
there
are
about
four
sigs
that
deal
with
that
clock.
That
directory.
We
all
want
to
delete
it.
We
want
it
gone,
but
we
all
have
dependencies
there,
and
so
all
those
dependencies
need
to
be
fixed.
E
We
have
all
of
these
reference
implementations,
so
I
mean
we've
gone
through
a
bunch
of
the
reference
implementations
that
gave
us
slides,
but
I
mean
huawei
and
rancher,
and
a
lot
of
other
cloud
providers
have
their
own
reference
implementations,
and
so
we
want
to
work
out
how
to
help
them
integrate
into
prowl.
We
want
to
standardize
on
how
branching
works.
E
We
want
to
work
out
how
to
get
good
docs,
so
any
user
who
wants
to
bring
up
kubernetes
on
a
cloud
can
go
to
kk
find
that
page
that
says:
here's
your
cloud
provider,
here's
how
you
build
it,
here's
how
you
bring
it
up.
We
also
know
that
you
know
whether
it's
ipv6
new
ways
to
think
about
load
balancers,
whatever
it
had
new
ways
to
do,
you
know
firewalls,
etc.
E
There
are
going
to
be
things
that
we
want
to
have
as
a
standard
part
of
kubernetes
that
require
deep
integration
with
the
cloud
provider
and
working
out
how
to
do
that
in
a
neutral
way
is
what
our
sig
is
for.
So
even
when
this
was
extraction,
effort
that
has
gone
for
a
few
years
now
finally
gets
done.
We're
going
to
have
this
need
to
keep
making
sure
that
kubernetes
remains
neutral
and
functional
across
all
the
cloud
providers.
E
Now,
if
any
of
that
sounds
interesting
to
you,
I
would
highly
recommend
that
you
join
the
sig,
so
you
know
the
standard.
How
do
you
join
the
sig
slide?
The
first
thing,
I'm
going
to
say,
is:
join
the
google
group
right,
the
the
kubernetes
sig
cloud
provider.
As
soon
as
you
join
that
group
you'll
be
on
the
emailing
list.
You'll
get
calendar
invites
most
of
the
information
you
need
to
become
a
contributing
member
is
is
just
going
to
show
up.
The
other
thing
I'll
mention
is
join
our
slack
channel.
E
Most
of
ours
are
america
times,
but
you
know
go
to
the
slack
channel
and
say
hey,
I'm
in
asia,
and
I
really
or
I'm
in
I'm
in
eastern
europe,
and
I
really
would
like
to
join
we'll
set
up
meetings
for
you.
We
have
been
doing
it
so
you
know
and
take
a
look
at
the
recordings
we
have
various
recordings
of
previous
systems
and
even
dives
into
various
bits
of
the
technology
like
the
ccm
and
the
api
server
network,
proxy
or
connectivity
system.
So
with
that,
does
anyone
have
any
questions.
E
E
E
E
Wait
for
the
signal
because,
as
we
know,
most
of
the
signal
is
going
to
come
from
the
managed
folks
and
the
managed
folks
tend
to
be
about
a
release
or
two
behind
to
know
that
we've
actually
gotten
it
all
right
and
that
it
is
then
safe
to
delete
so
I'll.
Let
you
do
the
math.
The
answer
is
later
than
I
like
cool
any
other
questions.
E
E
Sure
connectivity
system,
so
ssh
tunnels,
the
bane
of
my
existence
for
quite
a
while,
was
a
very
specific
google
solution
to
a
google
problem
having
to
do
with
where
the
control
plane
lived
and
where
the
customer's
cluster
lived,
and
it
was
a
lot
of
difficult
code
that
no
one
liked
having
to
support
not
even
the
googlers
and
everyone
wanted
it
gone,
took
a
couple
of
revs.
We
put
out
a
cap,
we
built
the
connectivity
system.
E
What
we
have
are
these
agents
that
depends
on
the
cloud
provider
how
they
get
installed,
but
they
run
them
next
to
the
cubelet.
They
connect
to
the
connectivity
server,
which
is
running
next
to
the
api
server,
and
then
the
api
server
connects
to
it,
and
that
is
how
the
traffic
is
tunneled.
It
can
be
either
done
via
grpc
or
http
connect.
There
are
a
couple
options
on
how
the
the
authentication
works
for
that
and
then
one
of
the
interesting
things
here
is
so
this
was
originally
designed
as
an
ssh
tunnel
replacement.
E
There
are
several
things
that
have
been
done,
such
as
the
ability
to
have
it
intelligently
route,
your
traffic,
which
ssh
tunnel
could
never
do
and
even
understand.
If
you
have
failure
zones
that
cannot
communicate
with
each
other.
It
understands
that
and
will
do
the
routing
correctly,
with
that
set
of
provisions
there's
even
one
of
the
companies
in
the
deck,
but
I
won't
say
which
one
who
is
looking
at
what?