►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes Manages More Than Just Containers - Ashley Schuett, Containership
Kubernetes is known for managing, scaling and deploying containerized applications, but it can also be leveraged for managing your infrastructure. When running a cluster there is the option of running a cloud controller manager(CCM) which can make creating resources and creating hardware synonymous. The CCM can provision loadbalancers, set up routes, and tear down vms. In this talk we will do a deep dive into the different controllers that make up a CCM, and how they are keeping the state of your infrastructure consistent with the clusters resources. The main controllers behind the CCM are cloud agnostic and live in core kubernetes, but each provider has to specify its own implementation. We will examine how provider specific logic hooks into the core and how to leverage the framework to write a custom CCM for any provider or bare metal solution.
To learn more: https://sched.co/GrYM
A
Hey
everyone,
I'm
gonna,
go
ahead
and
get
started
thanks
all
for
showing
up
we're
gonna
be
talking
about
CCM
today,
how
you
would
run
one
the
features
of
a
CCM
and
then
how
you'd
create
your
own
as
well
as
why
you
would
want
to.
But
first
of
all
about
me.
I
am
a
engineer
at
containership
and
what
I
do
is
I
write
software
for
provisioning,
urbanized
clusters
on
multiple
providers,
as
well
as
the
software
that
runs
on
those
clusters.
A
One
so
running
because
of
this
I
have
had
to
play
with
cloud
controller
managers
for
four
of
the
providers
that
we
support
as
well
as
I
ended
up
writing
my
own
for
packet
sense.
When
we
were
going
to
integrate
with
them,
they
didn't
have
one
from
doing
this.
I've
really
had
to
dig
into
the
code
and
they're
not
about
them.
A
Ccms
were
first
introduced
in
kubernetes
in
version
1.6,
and
it
runs
a
daemon
that
embeds
cloud
specific
control
loops.
This
was
pulled
into
its
own
binary
to
allow
cloud
providers
develop
and
release
at
a
different
pace
than
the
kubernetes
project.
Instead
of
having
to
do
the
quarterly
releases
with
kubernetes,
they
could
release
faster.
A
This
also
allows
you
to
have
some
of
your
hardware.
Properties
be
declarative,
manifests,
and
this
can
be
an
advantage
for
cost-saving
reproducibility
and
those
advantages.
One
such
thing
is:
if
you
have
your
cloud
running
and
a
dev
environment
and
production
environment,
you
know
that
you're
going
to
have
the
same
environment
across
those
two
clusters
for
node
labels,
load,
balancers
and
those
other
feature
sets
that
we'll
get
into
I.
A
Think,
first
of
all,
it's
important
to
know
how
and
what
happens
when
you
do
run
one
so
there's
two
different
types
of
CCMS:
there's
entry
and
out
of
tree,
and
if
it's
entry,
that
means
that
the
code
is
actually
in
kubernetes
kubernetes
and
it
is
built
as
part
of
the
binary
when
you
deploy
your
kubernetes
cluster.
If
it's
out
of
tree,
it
is
not
in
the
kubernetes
repo
and
it's
in
a
separate
repo.
An
example
of
this
would
be
digital
ocean,
as
well
as
Oracle
I
mean
there's
some
others
as
well.
A
Now,
if
you
run
out
of
tree,
you're
still
gonna
have
to
specify
that
cloud
provider
flag
on
your
cubelet
and
specify
it
to
external.
This
is
because,
by
doing
that
on
your
cubelet,
it
sets
all
of
your
nodes
to
have
this
tape,
which
says
that
that
note
is
not
schedulable
until
the
CCM
initializes,
your
node,
as
well
as
it
just
lets.
Your
CCM
know
that
it
needs
to
do
something
with
this
node,
so
I'm,
sorry,
I'll,
take
this
off
here.
Scratching.
A
A
This
is
what
registers
that
that
data
is
there
and
then
how
to
go
from
there
that
it
needs
to
do
something.
You
can't
just
ignore
this
node
and,
first
of
all,
what
it's
going
to
do.
Is
it's
going
to
set
of
provider
ID
on
it,
which
Maps
a
kubernetes
node
to
the
node,
that
is
in
your
provider
and
making
those
two
things
one?
A
It
also
adds
a
zone
and
region
label
for
identifying
that
node
for
management
purposes.
Writing
the
nodes
network
address
and
host
name
to
the
status
object
so
that
you
have
all
the
information
about
your
node
through
cube
CTL
by
just
looking
at
your
nodes
as
well
as
it
gives
you
node,
rectus
a
reconciliation.
So
when
you
do
cube
CTL
get
knows
or
if
you're
looking
through
the
API,
you
know
those
nodes
that
are
being
returned
to
you
actually
exist.
A
So
this
is
what
the
node
controller
kind
of
looks
like
in
a
generic
way.
So
the
node
controller,
if
you're
not
familiar
with
the
NGO
frameworks,
has
a
controller
which
has
a
sync
Handler
and
all
that
sync
handler
is
doing.
Is
it's
taking
the
state
that
your
cluster
is
in
and
taking
it
and
comparing
to
a
desired
state?
A
It
then
goes
and
looks
for
the
instance
type
and
adds
label
to
your
node,
so
you
can
identify
your
nodes
and
know
that
this
is
a
plus
a
node
with
two
CPU
or
however,
these
nodes
are
labeled
in
your
provider.
Is
going
to
give
that
same
information
as
if
you
were
looking
at
it
through
the
digital
ocean,
dashboard,
the
AWS,
dashboard,
etc,
as
well
as
it's
gonna
give
you
the
labels
with
the
zone
failure
domain
as
well
as
own
region.
A
A
After
it
goes
through
all
those
checks,
it
will
remove
the
taint
and
then
I
have
update
note
on
there,
because
it
does
the
same
thing
on
any
note
update
it's
gonna
go
through
that
loop,
however
it'll
get
to
that
check
cloud
tight
and
since
it's
been
removed
as
a
process
and
initialize,
it's
just
not
gonna
do
anything,
and
so
when
I
say
node
reconciliation.
What
it
does
is
it
sets
off
the
separate
go
routine.
That
is
another
node
controller.
A
Another
really
powerful
feature
that
the
cesium
is
gonna,
give
you
is
load,
balancer,
management
and
I.
Think
it's
probably
the
most
powerful
feature
of
the
CCM.
Your.
What
it
does
is
when
you
do
a
cube,
CTL
create
service
lb,
is
gonna,
go
and
create
a
load
balancer
on
your
provider.
If
the
CCM
supports
that
so,
what's
really
cool
about
that
is
if
your
provel
pers,
whoever
has
access
to
your
provider,
goes
on
there
and
deletes
your
load
balancer,
because
they're
like
hey,
this,
isn't
being
used.
A
The
CC
I'm,
just
gonna,
see
that
it's
just
gonna
bring
it
right
back
up.
You're,
not
gonna
experience
like
a
day
of
downtime
of.
Why
isn't
this
server
working?
That's
just
it's
gonna
come
back
up
in
the
same
state
running
on
the
same
port
in
another
really
cool
thing
about
that
is
depending
on
your
provider.
Unfortunately,
this
isn't
standardized,
but
you
can
add
annotations
to
your
service
and
you
get
all
these
special
features
you
get
for
AWS.
You
can
add
which
type
of
traffic
you're
routed
to
that
load.
A
Balancer
you
can
get
HTTP
HTTPS
TCP
and
then,
if
you're
running
that
HTTP,
you
can
map
that
to
a
cert
in
your
provider
and
it
automatically
is
going
to
be
added
to
your
load
balancer.
It's
not
an
extra
step
so
you're
when
you're
committing
this
into
your
github,
your
get
lab
whatever.
How
have
you
managed
your
code?
You
go
to
deploy.
It's
gonna,
be
the
same
thing
every
time
you
have
that
repeatability
and
even
across
cloud
providers.
A
A
Another
feature
that
the
CCM
gives
you
is
route
management.
This
is
required
for
GCE
and
the
only
provider
I
know
that's
using
it
as
GCE
and
that
allows
pod
networking
between
your
nodes.
I
was
talking
to
some
people
from
cloud
provider
earlier,
and
it
seems
like
this
was
pretty
much
like
GCE
was
the
first
provider
using
the
CCM,
so
it's
just
one
of
those
features
that
they
needed
and
they
put
in
there
and
no
one
else
has
really
used
from
my
understanding,
but
some
pain
points
with
this
documentation
is
sporadic
because
it
is
across
providers.
A
It's
hard
to
know
exactly
where
to
look
for
each
individual
piece
that
is
configurable
they're,
definitely
doing
a
lot
of
work
on
this.
There's
a
PR
out
for
getting
set
up
with
a
CCM
right
now
to
revamp
the
documentation
for
that.
So
it
is
more
easily
understood.
However,
it's
not
there
right
now
and
when
I
was
trying
to
figure
out,
it
was
pretty
much
trial
and
error
launched
a
cluster
with
this
and
did
it
work.
Okay,
nope
tear
that
one
down
try
again
see
if
I
can
get
this
on
the
first
shot.
A
A
This
is
really
going
to
improve,
though
so
you're
gonna
have
a
much
better
user
experience.
Some
other
pain
points
I've
had
is
this.
Was
this
project
was
originally
something
that
you
would
set
on
each
piece
of
your
kubernetes
cluster,
so
they
came
with
cloud
or
with
volume
management
which
isn't
really
the
case
anymore.
A
However,
if
you
run
it
through
the
cubelet
for
a
provider
such
as
AWS,
you
get
that
volume
management,
but
then,
when
you
run
it
as
a
deployment
or
a
daemon
set,
you
don't
get
the
volume
management
and
until
recently,
AWS
didn't
have
a
CSI.
So
you
didn't
really
have
the
option
of
how
you
wanted
to
run
this
binary
on
your
cluster
and
that
could
be
kind
of
frustrating
to
figure
out
as
to
why
what
you
were
expecting
to
happen
wasn't
happening
as
well
as
I
took
a
screenshot
of
this.
A
This
with
the
documentation
I
looked
at,
and
it's
like
these
links
will
vary
for
every
cloud
provider.
That
was
there.
You
didn't
go
in
the
plac
cloud
provider
like
hey.
This
is
how
you
should
set
them,
or
this
is
what
they'll
do
is
just
kind
of
like
have
fun.
So
that
was
a
good
time.
So,
as
I
was
saying,
the
CCM
used
to
be
part
of
API
cube
controller
and
the
cubelet,
and
it
was
a
cloud
provider
flag.
You
had
on
all
three
parts
of
your
infrastructure
for
it
to
work
properly.
A
A
The
plan
right
now
is
for
them
to
each
have
their
own
repo
under
kubernetes,
slash
cloud
provider,
the
provider
name,
some
providers
have
it
in
their
own
repo,
so
like
digitalocean,
has
it
under
their
own
digitalocean
github
and
that's
kind
of
based
on
tooling
them
how
they
are
interacting
with
the
community
and
just
other
options.
But
you
can
read
more
about
where
they're
planning
on
going
and
their
cups,
but
yeah
like
I,
said
the
documentation
is
definitely
improving,
there's
PRS
up
and
yeah,
so
that's
pretty
cool
to
see
so
creating
your
own.
A
Creating
your
own
is
actually
pretty
simple,
mainly
because
there's
already
examples
out
there
that
are
really
well
documented,
as
well
as
these
interfaces
are
very
well
split
apart
as
to
what
each
piece
is
doing.
That
makes
it
really
easy
to
understand
and
then
you
can
implement
which
features
you
want,
as
I
talked
about,
there's
different
features
within
that
CCM
and
you
don't
have
to
implement
them
all
for
it
to
be
at
CCM.
A
It
gives
you
so
much
power
to
have
repeatable
infrastructure
that
you
have
things
that
you
can
depend
on
in
it.
So,
even
if
you're
running
on
Prem,
you
can
implement
the
label
management
label
management
to
point
to
a
certain
rack,
a
certain
server.
So
you
just
know
by
looking
at
this
node
where
to
go
and
which
server
you're
actually
talking
to.
As
long
as
you
have
an
API
to
give
you
information,
you
can
write
a
CCM
and
have
better
more
repeatable
infrastructure.
A
So
to
do
that,
your
linear
main
file,
you're
gonna,
include
the
course
ECM
code
to
run
and
you're
gonna
import
your
API
with
all
the
functions
that
you
have
implemented
for
the
interface
and
then
to
run
the
CCM.
You
do
that
app
run,
which
is
the
CCM
code
that
you've
imported
and
you
can
add
other
options
as
well.
If
you
want
to
look
at
examples
of
this,
you
can
look
at
the
packet
CCM
with
the
digitalocean.
Ccm
are
the
ones
I'm
most
familiar
with,
but
there's
others
out
there
as
well.
A
So
this
is
an
interface
that
you
would
implement
in
each
of
these
pieces.
So,
like
the
load
balancer,
you
would
return
another
interface
that
it
implements
for
it
to
be
a
load
balancer
and
then
a
boolean
value.
So
if
you
don't
want
to
implement
that
interface,
you
just
return
false.
If
you
want
to
implement
it,
you
return
true
with
it
and
the
interface
that
you
are
implementing.
A
So
to
initialize
your
provider,
you
are
going
to
want
an
init
block
in
that
main
source
code
that
you're
importing
with
the
underscore
and
that
registers
your
cloud
provider
with
your
provider
name.
So
it
knows
about
it
and
you're.
Gonna
return
that
cloud
provider
interface
that
you
have
implemented
with
the
information
about
the
other
interfaces
that
you
have
decided
to
implement
and
then
the
how
you
are
going
to
manage
and
keep
track
of,
the
information
that
you
need
to
and
if
you
can
look
at
the
sides,
I've
posted
them
and
link
to
them.
A
So
an
interface
within
that
that
you
can
implement
is
the
zones
interface
to
have
that
label
data
on
your
nodes.
You
can
use
it
for
syncing
metadata,
it's
synced
by
the
node
controller
and
you're
just
gonna.
Add
some,
since
you
feel
like
an
example
is
like
the
failure
domain,
the
region
that
would
then
be
on
every
node,
that's
running
the
cesium,
and
you
can
always
count
on
that
label
to
be
there
for
your
node
management
to.
A
Zone
interface,
it's
just
three
functions
and
it's
gonna,
be
the
get
zone
getting
zone
by
provider
ID
and
gets
owned
by
no
name
and
again
I've
linked
out
to
like
the
specific
file
for
the
packet
cesium,
where
I
implemented
this
and
yeah.
So
you're
gonna
just
use
your
providers
API
to
get
that
information
in
this
case,
I
actually
just
ended
up
using
a
little
metadata
server
to
get
metadata
off
the
machine
and
then
write
it
and
return
it
as
the
zone.
A
So
I
didn't
even
really
interact
with
the
packet
API
for
some
of
us,
but
you
then
implement
like
getting
that
device
ID
from
the
provider
ID.
So
you
have
to
know
how
to
map
those
two
together.
So
whether
you
have
that
as
the
node
name
that
people
provide
for
your
provider
or
the
ID
just
being
able
to
map
those
two
together,
I
know
in
some
cases
people
let
you
provide
the
private
IP
and
how
you
would
talk
to
the
cluster
for
those
mapping
purposes.
A
It
doesn't
have
to
be
the
node
name
was
just
kind
of,
but
you
specify,
and
then
you
have
to
document
what's
allowed.
Otherwise,
the
CCM
won't
know
what
device
on
the
provider
it's
talking
to
and
then
you
would
get
the
zone
by
the
node
name
as
another
option
instead
of
the
provider
ID
and
returning
that
information
as
well
and
again,
this
just
returned
that
struct,
that
has
the
region
or
the
failure,
device
and
maps
that
and
adds
it
to
the
label
that
will
be
put
on
the
node
yeah.
A
So
that's
CC
I
mean
you
like.
What
are
you
going
to
do
after
this?
You
can
go
and
build
your
own
open
source
yeah
for
people
to
use,
if
that's
a
metal
solution,
something
for
behind
the
cloud.
If
you
can't
talk
to
an
Internet
just
being
able
to
have
that
power
to
have
those
repeatable
steps
for
your
clusters,
but
this
sig
is
definitely
looking
for
help
with
Docs
and
I
have
linked
to
the
sig
cloud
provider
that
manages
all
of
this
as
well
as
entry
cloud
provider.
A
One
interesting
thing
that
I've
was
kind
of
thinking
about
is
something
you
could
do
is
foreign-trained
load
balancers.
You
can
really
take
that
block
and
do
whatever
you
want.
If
I'd
be
kind
of
interesting
to
have
it
spin
up
a
service
or
some
nginx
load
balancer
and
you
don't
even
need
a
physical
load
balancer
then,
and
just
you
know,
you
can
play
those
like
different
ideas.
You
don't
necessarily
have
to
be
doing
something
on
a
cloud
provider.
You
can
just
make
it
work
for
you.
So
questions.
B
A
What's
the
deployment
you
can
specify
just
running
one
replica,
the
reason
I
put
that
difference
is
I
know
in
digital
oceans.
Examples
they
have
it
as
a
deployment.
However,
the
documentation
has
as
a
daemon
set
that
runs
on
all
the
masters,
doing
like
a
leader
election
format.
So
you
get
those
a
provided
benefits
from
kubernetes.
So
if
you're
running
a
che,
you
can
have
your
CC
Maj
as
well
yeah.
C
There
a
way
to
add
like
arbitrary
data
to
the
couplets
information
when
it's
registering
as
a
node.
So
if
I
wanted
to
know
the
exact
hardware
type
and
things
like
that
to
carry
that
into
the
controller
manager
for
how
we
make
decisions
on
what
region
it
is
or
what
type
of
daemon
sets
would
automatically
get
created
for
that
type
of
node?
C
A
The
couplet
you
can
specify
whatever
labels
you
want.
That
is
a
cubelet
feature
within
the
CCM.
If
you
wanted
to
implement
certain
things
for
your
nodes,
you'd
have
to
do
that
under,
like
the
framework
of
instance,
type
and
return
that
information
there,
but
it
wouldn't
be
done
in
the
cubelet
necessarily
except
for
there's
a
label
spike
and
thank
you.
But
if
you
just
wanna
put
everything
there
as
well.
A
A
But
if
you
wrote
your
own
TCM,
you
could
probably
implement
that
you
would
put
it
in
the
node
controller
and
have
it
interact
that
way,
but
you
would
have
to
know
how
to
talk
to
the
API
and
how
to
create
and
edit
node.
So
it
wouldn't
be
like
a
great
place
for
it.
But
if
you
wanted
to
do
it,
you
could
I'd
look
at
more.
So
at
some
kind
of
autoscaler.
E
So
likewise,
I
didn't
see
any
provision
in
the
interface
for
configuring
network
objects
or
anything
like
that.
Besides
the
load
balancers
themselves,
so
I'm
thinking
in
Amazon,
you
know
it
would
be
awesome
if
we
could
have
kubernetes
itself
manage
things
like
the
subnets
in
which
it
would
be
instantiating
those
instances
or
even
the
VPC
itself,
and
if
you're
talking
about
something
within
a
separate
control
plane.
Is
there
any
provision
for
that,
or
is
that
sort
of
just
down
that
you?
A
That
wouldn't
be
present.
Generic
crating
of
the
load
balancer
but
AWS
by
far,
has
the
most
annotations
that
you
can
add
to
your
services
to
get
custom
functionality
so
depending
what
you're
looking
for,
you
might
be
able
to
accomplish
it
with
that.
But
yeah.
You
might
want
to
write
your
own
thing
or
there
might
be
some
other
project
for
that,
but
I
haven't
done
anything
with
it.
A
Well,
cluster
api
is
purely
for
provisioning
on
any
provider
where
this
is
more
like
that
second
step
of
how
you're
gonna
interact
with
that
cluster,
once
it's
up
having
that
those
label
informations
and
the
load
balancers.
As
far
as
I
know,
cluster
API
is
only
touching
creating
the
instances
it
wouldn't
be.
The
load
balancers
and
then
there's
separate
project
CLI,
which
also
falls
under
cloud
providers
for
creating
and
managing
volume.
Data
and
I.
Think
that
is
all
separate
from
the
cluster
API.
A
A
So
the
cue
controller
manager
sees
that
you're
not
running
a
cloud
provider
and
it
doesn't
do
anything
with
the
volume
management
and
then,
when
you
run
on
the
cloud
controller,
it
doesn't
have
the
same
controllers
to
run.
So
that's
when
this
yes.I
comes
into
play,
so
OpenStack
would
need
a
CSI
for
it
to
work
and
that's
part
of
the
thrashing
that's
kind
of
going
on
between
taking
those
entry
providers
out
of
tree
thanks.