►
From YouTube: Roll Out the Red Carpet for Production Kubernetes Clusters with a Kube-vip - Dan Finneran, Equinix
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Roll Out the Red Carpet for Production Kubernetes Clusters with a Kube-vip - Dan Finneran, Equinix
This talk begins with the presenter trying to improve upon a problem, namely deploying Kubernetes clusters on bare-metal and taking them into production. Often tasked by customers of making their on-premises clusters as "public cloud like" as possible, a lot of work went into writing solutions for these functionality gaps. Time and time again ensuring highly available access to clusters proved problematic to implement and implement into lifecycle patterns. Kube-vip slowly evolved from trying to fix that one use case into a widely used project that provides highly available Kubernetes clusters for various infrastructures. This talk will cover that journey, and show how from on-prem to the edge you can implement both highly available networking and extend that same functionality to "inside" the cluster with load balancer functionality for Kubernetes services with Kube-Vip.
A
A
Today,
we're
going
to
be
talking
about
rolling
out
the
red
carpet
for
production
kubernetes
clusters
with
a
cube
vip.
My
name
is
daniel
finneran,
also
known
as
the
bsd
box
on
github
and
and
twitter
and
other
places,
and
I'm
currently
leading
the
engineering
efforts
in
developer
relations
at
equinix
metal.
A
So
to
kick
things
off
a
bit
of
background,
I
guess
into
the
the
cube
vip
project
many
many
roles
ago,
I
was
predominantly
focused
on
helping
customers
and
end
users,
roll
out
bare
metal,
kubernetes
servers
and
clusters,
and
from
that
I,
I
kind
of
spawned
a
bare
metal
provisioning
project
in
order
to
kind
of
alleviate
a
lot
of
the
issues
that
I
was
finding
getting
these
these
clusters
rolled
out.
So
it's
quite
a
simple
project
to
automate
the
deployment
of
bare
metal
servers.
A
A
A
So,
typically,
in
order
for
a
highly
available
kubernetes
cluster,
we
obviously
need
more
than
one
control
plane
node,
but
we
also
need
a
number
of
other
kind
of
components
in
order
to
provide
highly
available
access
and
redundancy
in.
In
the
case
of
in
the
case
of
failure
previously
or
in
a
lot
of
examples,
you
will
find
people
having
additional
nodes
that
sit
in
front
of
their
kubernetes
clusters
that
provide
things
like
load,
balancing
or
highly
available
ip
addresses.
A
So
you
know
if
we
kind
of
look
at
all
this,
and
these
are
kind
of
the
issues
that
I
was
facing.
This
incurs
kind
of
a
lot
of
operational
overhead
from
an
automation
perspective.
There
are
a
lot
of
additional
pieces
of
software
that
are
required
to
provide
that
functionality.
We
need
to
automate
the
load
balancing
part
of
things
we
need
to
load
while
we
need
to
automate
the
clustering
technology
and
and
the
the
tooling
that
we
use
to
provide
virtual
ip
addresses,
and
things
like
that.
A
A
All
of
that
makes
it
kind
of
quite
hard
to
automate,
so
that's
kind
of
where
the
project
kind
of
spawned
from
and
then
you
know
kind
of.
I
realized
that
we
could
kind
of
take
it
a
little
bit
further.
So
we
have
our
kubernetes
cluster
up
and
running,
but
you
know:
pods
inside
of
kubernetes
cluster
typically
can't
be
accessed
from
outside
the
cluster
and
prior.
We
would
need
additional
technologies
in
order
to
expose
these
to
outside
worlds
out
the
outside
world.
A
So
I
kind
of
realized
that
you
know
kind
of
already
implemented
a
lot
of
the
technology
in
order
to
do
that,
and
that
is
where
cube
vip
went
from
not
only
providing
just
highly
available
kubernetes
clusters,
so
production-ready
kubernetes
clusters,
but
also
being
able
to
provide
load
balance
of
functionality
to
allow
external
access
to
pods
inside
your
kubernetes
cluster.
So
this
is
kind
of
what
I'm
going
to
be
talking
about
today.
A
So
cuba
background
I've
already
done
that,
but
we're
going
to
be
looking
at
kind
of
the
architecture
of
cube
vip
and
the
protocols
that
it
uses
in
order
to
expose
things
to
the
outside
world.
We'll
look
at
how
the
highly
available
kubernetes
clusters
actually
look
like
with
cube
vip,
we'll
discuss
kind
of
the
load,
balancer
services.
What
that
looks
like
and
how
that
works
a
little
bit
of
the
road
map
in
terms
of
things
that
we're
working
on
next
and
then
hopefully
time
for
some
questions.
A
So
in
the
initial
design,
cubevip
was
designed
to
sit
outside
of
the
the
kubernetes
cluster
and
in
order
for
it
to
work,
we
needed
a
way
of
having
a
leadership
election.
So
we
originally
opted
for
using
raft,
which
already
runs
inside
kubernetes
through
xcd
raft,
requires
an
odd
number
of
members
in
order
for
the
election
process
to
actually
work
and
raft
effectively
works
by
having
elections
on
a
regular
basis
and
if
one
of
the
nodes
becomes
ill
or
doesn't
respond
in
time,
etc.
A
A
A
So
using
you
know
kind
of
things
like
the
the
kubernetes
go
sdk
we
can
effectively
have
some
code
that
will
connect
to
the
api
and
say
that
it
wants
to
it
wants
to
participate
in
this
leader
election,
so
typically
cube
vip.
Has
this
code
within
it,
and
a
number
of
kind
of
cube
instances
will
connect
to
the
kubernetes
api
and
say
I
want
to
hold
this
lease
I
want
to
be.
I
want
to
be
the
leader.
A
A
If
it
becomes
unresponsive,
then
a
timeout
will
occur
and
the
process
will
restart
where
all
of
the
other
nodes
that
participate
in
that
leader
election
will
also
ask
for
the
lease,
and
one
of
the
other
one
nodes
will
then
get
that
lease.
So
this
allows
us
to
have
that
technology
of
a
having
one
of
the
participants
become
the
leader
and
then
in
the
event
of
failure
or
upgrades
or
or
anything
lifecycle
related.
A
So
when
a
new
node
becomes
a
leader
for
the
first
time,
it
will
need
to
inform
the
network,
the
wider
network
and
the
traffic
should
come
to
its.
There
are
two
distinguished.
There
are
two
technologies
that
we
focus
that
we
rely
on
within
cube
vip.
The
first
one
is
arp,
which
is
a
layer
2
protocol
and
effectively
arp
allows
a
node
on
the
network
to
effectively
broadcast
and
update
that
network,
so
that
that
network
knows
to
send
traffic
to
it
when
it
is
wanting
to
send
traffic
to
a
particular
ip
address.
A
A
It
can
broadcast
to
the
network
that
to
get
to
this
ip
address,
send
your
layer
to
traffic
to
this
mac
address,
so
it
effectively
updates
the
network
to
say
ip
address
to
this
mac
address
and
that
effectively
is
how
layer,
2
traffic
actually
works.
When
we
want
to
send
traffic
to
an
ip
address.
We
look
up
on
an
internal
table
where
which
piece
of
hardware
do
we
want
to
actually
send
the
traffic
to
and
in
the
event
that
we
have
a
leader
election?
A
A
A
It
also
has
additional
ip
addresses,
so
it
has
the
ip
address
10.0.2.5
and
it
is
peering
and
advertising
to
the
router
that
all
traffic
that
needs
to
get
to
that
10
address
should
be
sent
to
the
192.21
address.
So
what
that
allows
the
router
to
do
is
know
which
node
to
send
traffic
to
in
order
to
get
to
that
additional
to
that
next
step
ip
address.
A
A
One
additional
benefit
of
bgp
is
that
it
typically
offers
load
balancing
out
of
the
box
via
the
router.
So
we
can
see
we
have
multiple
nodes,
all
advertising
that
10.0.2.5
address
to
the
router
and
then
a
client
that
goes
to
that
address
will
be
sent
to
one
of
those
nodes
that
is
participating
in
that
peering.
A
A
That
supports
that
is
actually
required,
and
some
hardware
vendors
may
require
additional
licenses
in
order
for
it
to
work.
Arp
poisoning
can
disrupt
a
network,
so
arp
poisoning
is
where
a
malicious
actor
on
the
network
starts.
Advertising
false
ip
addresses
to
false
mac,
addresses
and
effectively
what
that
would
allow
a
user
to
do
is
kind
of
black
hole
traffic.
A
However,
bgp
can
mandate,
authentication
and
you
can
have
acls
and
rules
on
the
routers,
which
would
stop
malicious
routes
being
advertised
arp.
Some
switches
in
virtual
software
and
physical
switches
actually
can
restrict
our
broad
updates.
So,
for
instance,
if
things
are
changing
too
rapidly,
some
switches
will
start
denying
those
broadcasts
from
it
going
going
further
on
the
network
bgp,
because
it's
layer,
3
requires
firewall
access,
possibly
on
a
on
a
udp
port
to
to
the
router.
A
A
So
two
common
methods,
either
a
static
pod
or
a
demon
set
and
both
of
those
methods
both
come
with
their
own,
unique
quirks
to
a
certain
degree,
so
static,
pods.
Originally
with
cube
vip.
I
was
using
cube
adm,
but
still
I'm
using
cube
adm,
but
that
effectively
would
come
with
a
chicken
and
egg
scenario
in
the
cube
adm
in
it
has
a
check
which
tries
to
connect
to
the
virtual
ip
this
floating
ip
address
as
part
of
the
installation
procedure.
A
A
So
what
we
can
do
is
we
can
kind
of
simulate
the
the
behavior
of
cube
adm
in
it
in
the
cube
adm
in
it
will
effectively
populate
a
number
of
manifests
in
the
the
kubernetes
manifests
folder
and
then
kubelet
is
effectively
told
to
start
up
by
cube,
adm
init
and
then
cube
adm
in
it
will
start
all
of
those
static.
Pods
that
are
those
manifests
in
the
manifests
folder.
A
What
we
can
do
is,
we
can
add
our
own
static,
manifest
for
cube
vip
inside
the
manifests
folder
before
we
do
the
cube
adm
in
it,
and
this
basically
allows
cube
adm
in
it
to
sit
alongside
all
of
the
control,
plane,
components
and
cube
that
will
just
start
up
alongside
of
all
of
those,
so
the
api
server
will
start,
the
schedule
will
start
at
cd
will
start
and
cube.
Vip
will
also
start
at
the
same
time,
which
means
it
can
connect
to
the
api
server
and
it
can
expose
that
control
plane.
A
So
we
can
stand
that
cluster
up
and
then
once
the
the
cluster
is
actually
up
and
running,
we
can
do
a
cubesat
cubectl
apply
of
the
cubevip,
manifest
as
a
daemon
set,
bring
up
those
additional
control,
plane
nodes
and
it
will
just
scale
across
it
will
come
up
and
it
will
start
advertising
that
10.0.2.5
address
as
a
as
a
highly
available
virtual
ip
address.
So
it
really
depends
on
you
know,
kind
of
a
the
kubernetes
distro
that
you're
deploying
or
the
installation
mechanism
that
you're
actually
looking
at
going
down.
A
So
you
know,
as
I
mentioned,
with
some
of
the
the
technologies
mentioned,
we
now
have
all
of
the
components
required
to
provide
high
availability.
We
have
that
clustering
algorithm
that
will
ensure
that
in
the
in
node
failure
and
upgrade
another
node
will
take
leadership
and
continue
to
receive
traffic,
and
the
networking
topologies
will
be
updated.
Should
there
be
any
changes.
A
So
what
does
it
all
actually
look
like?
In
this
example,
we
have
our
three
control
plane
nodes.
We
no
longer
require
additional
nodes
that
sit
atop
to
provide
high
availability,
and
things
like
that.
So
we've
already
reduced
the
amount
of
nodes
that
are
actually
required
to
provide
that
high
availability.
A
So
we
can
see
we
have
our
three
nodes
as
part
of
the
control
plane.
They're
all
running
the
control
plane,
components
they
all
have
static.
Pod
manifests,
which
means
that
cube
vip
is
actually
running
on
them,
and
we
can
also
see
the
first
node
has
been
elected
leader,
which
means
that
it
carries
the
virtual
ip
address.
A
They
will
ultimately
be
given
the
lease
once
they
have
been
given
that
lease
they
will
start
advertising
that
to
to
get
to
the
control
plane.
I
have
the
vip
and
send
traffic
to
me
so
from
an
end
user
perspective,
you
may
see
one
or
two
ping
loss
during
the
the
failover,
but
it
is
very
quick.
In
some
cases
almost
instantaneous
the
leader
election
happens
as
soon
as
the
other
node
is
is
removed
and
arp
broadcasts
will
ultimately
update
the
network
immediately
from
a
bgp
perspective.
A
Either
a
demon
set
or
we
have
static
pods
here,
all
three
of
them
carry
the
vip,
but
they
carry
it
on
a
internal
adapter.
So
what
that
means
is
that
that
10.0.2.5
address
isn't
directly
accessible
by
the
outside
world.
Otherwise,
we
would
end
up
in
a
position
of
conflicting
ip
addresses,
that's
something
to
to
be
aware
of,
but
it's
not
something
that
that
would
trouble
you
due
to
the
architecture
of
this
and
effectively.
A
What
that
means
is
that
and
a
client
that
wants
to
connect
to
the
kubernetes
control
plane
that
wants
to
get
to
10.0.2.5
will
connect
through
the
router
and
that
router
will
ultimately
send
traffic
to
one
of
any
of
the
nodes
that
is
peering.
So
we
get
load
balancing
from
that
router
automatically
in
the
event
that
one
of
the
nodes
dies.
So
we
can
see
here
number
node,
node
1
has
become
unaccessible.
A
That
node
will
stop
advertising.
It
will
stop
being
part
of
the
peering
for
those
those
routes
to
the
ip
address,
at
which
point
the
router
will
no
longer
be
able
or
will
no
longer
have
that
route
and
will
no
longer
send
traffic
to
that
node.
So
as
soon
as
that,
node
disappears
from
being
able
to
appear,
traffic
will
no
longer
be
sent
to
that
node.
A
So
this
provides
instantaneous
failover
in
the
event
of
upgrades
or
failure,
and
things
like
that,
so
we've
kind
of
covered,
you
know
kind
of
the
highly
available
part
of
cube
vip,
how
it
how
it
typically
works.
Typically,
it
sits
alongside
the
control
plane,
components
participates
in
leader
elections
and
advertises
a
control
plane
ip
address
to
the
outside
world
and
updates
in
the
event
of
failure,
and
things
like
that.
A
One
is
a
ccm
which
is
a
cloud
controller
manager
and
then
the
second
is
something
to
provide
that
networking
magic,
so
in
this
example,
cube
vip.
So
what
is
a
cloud
controller?
Well,
a
cloud
controller
is
effectively
the
secret
source
when
you
want
to
run
a
kubernetes
cluster
on
your
own
or
other
people's
infrastructure.
A
A
It
means
that
when
I
want
to
do
something
in
a
kubernetes
cluster,
the
ccm
can
kind
of
translate
that
into
the
infrastructure
of
where
it's
running
so,
for
instance,
on
places
like
aws
or
google
cloud
and
other
cloud
providers.
The
cloud
controller
will
allow
us
to
speak
directly
to
the
infrastructure
of
aws.
So
if
I
require
an
external
ip
address,
an
eip
and
things
like
that
that
cloud
controller
can
speak
to
the
aws
or
cloud
providers
apis
and
get
that
bit
of
information
for
us.
That's
specific
to
that
infrastructure.
A
A
It
needs
to
be
configurable
for
different
networks
and
network
ranges,
and,
ideally
you
know
kind
of
capable
of
plugging
into
things
like
existing
ipam
or
other
infrastructure
management
tooling.
So
there
is
work,
that's
taking
place
at
the
moment
on
the
cluster
api
project
in
order
of
having
an
ipam
controller,
which
typically
would
allow
you
to
you
know,
give
it
the
knowledge
of
your
network
ranges
and
it
will
provide
ipam
functionality.
A
A
So
if
we
look
at
what
has
been
created
here,
we
can
do
a
get
service
and
kind
of
describe
it.
We
can
see
here
that
we
have
a
service,
but
it
has
not
been
given
a
load
balance
ip
address.
So,
as
I
was
saying,
with
an
eip
from
aws,
for
instance,
the
ccm's
role
is
effectively
to
be
able
to
link
the
two
together,
so
you
know
kind
of
what
what
happens
here.
A
A
So
we
have
done
that
expose.
We
have
a
ccm
running,
so
cube.
Vip
has
its
own
ccm,
however.
Qvi
is
designed
in
a
way
that
doesn't
tie
it
to
any
particular
ccm
and
I'll
kind
of
show
why,
in
a
second,
but
this
allows
end
users
to
create
their
own
ccms,
and
a
number
of
people
have
done
that
so
harvester
has
its
own
ccm.
That
cubevip
will
work
with
equinix
metal
has
a
ccm
that
qvic
will
work
with
and
cubic
has
its
own
ccm
and,
as
mentioned,
the
ccm's
role.
A
A
So,
additionally,
cubevip
was
then
kind
of
updated
to
work
in
a
hybrid
mode,
and
this
is
mainly
for
kind
of
edge
deployments.
This
allows
cubevip
to
do
both
h
a
and
services
through
the
control
plane,
so
they
will
all
participate
in
that
leader
election
for
for
the
highly
available
vip.
So
traffic
will
go
to
the
cubic
pods
when
trying
to
hit
the
control
plane,
but
also
also
when
traffic
is
coming
into
a
kubernetes
service.
A
A
One
additional
feature
that
we
added
was
dhcp
load
balancers
again.
This
is
kind
of
useful
for
edge
deployments
so
effectively.
What
this
allows
us
to
do
and
we
can
see
in
the
example
there
we've
specified
a
load
balancer
ip
address
of
0.0.0.0,
which
is
a
valid
invalid
ip
address.
But
what
that
effectively
means
is
that
the
kubernetes
api
will
accept
it
as
an
ip
address
for
a
service,
and
what
actually
happens
here
is
when
we
specify
a
a
service
without
ip
address.
A
A
It
has
given
cube
vip.
An
ip
address
of
1.123
cubevip
now
will
update
the
service
with
the
ip
address
and
then
use
that
to
expose
the
service
to
the
outside
world.
So
we
effectively
get
free,
topology
information
from
the
dhcp
server,
and
this
works
great
in
small
networks,
where
edge
environments,
where
we
can
leave
all
of
the
ipam
knowledge
to
whatever
it
is.
That's
providing
dhcp
functionality
inside
that
network.
A
So
that's
some
of
the
additional
functionality
around
cubic,
so
we've
covered
a
bit
of
the
back
story.
We've
covered
how
high
the
high
availability
works,
we've
covered,
how
ccms
work
and
how
qbip
can
use
that
information,
then
to
expose
access
into
the
services
network
for
services
of
type
load
balancer,
so
that's
kind
of
where
qvip
is
at
the
moment.
From
a
roadmap
perspective.
A
few
months
ago
we
submitted
cubevip
to
the
cncf
sandbox.
A
A
A
A
Everything
for
code
is
all
on
the
cubevip
repositories
with
that.
Thank
you
very
much
and
if
you
are
using
cube
vip,
I
hope
you
enjoy
it
and
it
is
doing
what
it
is
meant
to
be
doing
for
you.
If
not
please
raise
issues,
and
let
me
know
thank
you
very
much
and
enjoy
load
balancing
with
cubevip.
Thank
you.