►
Description
IPVS for Scaled Private Cloud Load Balancing [I] - Kimberly Messimer, Comcast VIPER
In this talk, we discuss how Comcast VIPER addresses network-scaling bottlenecks using IPVS as a load balancer. We discuss the benefits including enabling us to achieve over 500Gbit/s egress from a modest Kubernetes cluster as well as dynamic VIP allocation.
Using a tool we’ve written in-house called Kube2IPVS, which utilize Kubernetes config maps, we can assign an ingress IP address and port to a Kubernetes service, and our load balancer will automatically reconfigure, with no downtime, to load balance traffic into a service.
We go into technical detail in this presentation, starting with a brief overview of IPVS and why it’s useful. We will then descend into the requirements surrounding L2 adjacency and why ARP is your frenemy. We’ll discuss how we addressed the challenges of running IPVS internal to Kubernetes, including IPVS master and backend co-location.
Finally, we'll highlight how port forwarding is not possible using IPVS in direct-reply mode, and then discuss how Kube2IPVS manages iptables rules to route packets directly into Kubernetes service chains, effectively bypassing this restriction.
We expect to have Kube2IPVS open-sourced in 2017.
About Kimberly Messimer
Platform Engineer, Comcast VIPER
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
Vipre
stands
for
video,
IP
engineering
and
research,
and
we
manage
IP
video
delivery.
We're
a
small
office
base
at
a
Denver
Colorado.
This
group
started
in
2010
with
a
just
a
few
people
and
we've
grown
to
over
200
people.
Since
then,
what
this
group
does
is
take
the
core
products
that
Comcast
offers
and
implements
them
in
software
and
serve
them
as
IP
video.
This
is
different
from
traditional
cable
services,
in
that
it's
a
unicast
transmission.
A
So
I
briefly
want
to
touch
on
how
and
why
we
began
to
use
kubernetes
in
mid-2015
VIPRE
was
developing
an
application
called
C
DD
R,
which
is
DVR
on
the
cloud.
The
application
requirements
were
pretty
aggressive,
with
a
need
for
around
300
gigabits
per
second
of
network
egress
and
at
the
time
the
architecture
had
over
15
microservices.
A
A
So
what
do
we
need
out
of
this
load?
Balancer
we
needed
something
that
would
support
at
least
CVC
DVRs
300
gigabits,
a
second
of
egress.
It
needed
to
handle
TCP
and
UDP
connections
they
needed
to
handle
many
connections,
possibly
over
a
million
concurrent
and
multi-tenancy
it
needed
to
be
configured
rapidly
and
online
with
no
downtime.
And,
lastly,
we
needed
a
solution
that
was
going
to
provide
some
government
governance.
Most
of
our
tenants
are
not
data
center
operators,
so
restricting
the
pool
of
IP
addresses
available
for
load
balancing
was
really
important.
We
started
looking
for
options.
A
We
looked
at
a
lot
of
things
and
came
to
the
clusion
that
there
weren't
really
many
solutions
for
us.
Any
doctor
or
kubernetes
specific
solutions
either
didn't
exist
at
the
time
or
were
pretty
alpha.
We
couldn't
run
with
a
layer,
7
load
balancer
because
it
wasn't
going
to
need
our
requirements
and
our
throughput
requirements
specifically
and
most
options
didn't
support
online
configurations.
A
So
our
quest
for
solution
landed
us
on
IP
vs
for
anyone
not
familiar.
It's
a
network
load
balancer,
that's
implemented
in
kernel
space.
It
provides
online.
Flexible
configuration
supports
TCP
and
UDP
for
both
ipv4
and
ipv6
and
because
the
request
routing
is
done
in
kernel
space,
it's
really
success.
We
also
chose
IPS
because
it
was
a
known
known
in
VIPRE.
We
already
use
it
in
production
for
most
of
our
video
applications
and
we
knew
that
it
would
support
the
use
case.
A
So
a
little
bit
more
about
ipbs.
It
offers
a
few
different
options
for
load,
balancing
and
I'm
not
going
to
go
into
here,
but
the
fastest
and
most
familiar
to
Viper
was
direct
reply,
direct
reply,
let's
of
scale
egress
out
to
the
some
of
the
interface
bandwidth
for
all
of
our
back-end
nodes,
meaning
that
if
I
have
10
nodes
with
10
gigs
10
gig
interfaces,
I
have
100
gigs
of
egress.
A
It
works
by
enabling
back-end
notes
to
apply
it
directly
to
the
client
instead
of
proxying
through
the
master
load
balancer,
where
you
might
have
a
limit
of
10
gigs
in
this
mode.
The
IQ
vs
master
owns
visit
virtual
IP
and
advertises
that
zip
to
the
router.
All
the
backends
are
configured
with
that
this
one
at
sleazebag
interface,
when
a
request
comes
into
the
master.
The
request
destination
MAC
address
is
replaced
with
the
MAC
address
of
the
backend
and
sent
to
that
chosen.
Back-End.
A
The
backend
node
sends
its
reply
directly
back
to
the
client,
not
through
the
load
balancer.
So
in
this
load
you
can
actually
handle
a
more
traffic
into
the
cluster
than
the
IP
vs
masters
interface
actually
supports,
but
all
this
kind
of
amazing
performance
has
a
cost
and
there
are
two
main
limitations.
A
First
is
a
requirement
for
layer,
two
adjacency
between
the
master
and
the
nodes,
so
it
limits
you
to
the
number
of
nodes
in
your
cluster
2
as
a
physical
signal
and
also
there's
no
disassembly
of
the
packet
header
in
direct
reply.
So
you
can't
rewrite
the
destination
port
and
all
go
into
the
details
of
the
implications
of
that
a
little
bit
later,
but
we
had
chosen
a
solution
and
once
we
had
we
needed
to
get
it
running
alongside
of
Cooper
at
ease.
So
this
is
the
docker
run
command
for
our
initial
implementation.
A
The
container
runs
in
privileged
mode
so
that
this
can
be
configured
on
the
primary
interface
of
the
master
and
the
loopback
interface
of
the
backends
in
the
master
mode.
Ipv
sadm
is
used
to
configure
the
front-end
and
port
to
listen
on
and
also
configures
the
backends
to
route
traffic
to
in
back-end
mode.
The
bit
just
gets
added
to
the
loopback
interface
and
any
inbound
traffic
coming
in
on
the
defined
port
gets
routed
into
kubernetes
you'll
notice
that
all
of
the
information
here
is
explicitly
passed
into.
The
container
has
command-line
arguments.
A
So
we
had
load
balancing
great
it
worked
and
it
scaled,
but
we
quickly
realized
that
the
solution
could
only
work
for
a
single
cluster
single
application
deployment.
Why
was
this?
Well?
Let's
start
with
the
fact
that
we
didn't
have
the
option
to
port
forward
with
ipbs,
which
meant
we
needed
to
use
node
ports.
A
Ip
vs
could
not
listen
on
a
standard,
HTTP
port
and
forward
to
cooper,
nedeth
noght
support,
which
meant
that
whatever
no
port
stuff,
whatever
a
node
port,
was
assigned
to
the
service
that
we
were
routing
to
IP
vias
had
to
listen
and
forward
on
that
same
port.
So
this
didn't
really
give
us
the
flexibility
that
we
needed
at
the
time.
A
We
had
some
additional
problems
with
this
implementation,
so
node
ports
are
applied
to
the
cluster
across
name
space.
No
two
tenants
could
use
the
same
node
ports
and,
as
it
turns
out,
humans
are
pretty
bad
at
being
random
and
a
lot
of
our
developers
were
very
prone
to
choosing
the
same
node
ports
for
their
services,
and
this
caused
a
lot
of
collision
inside
the
cluster.
A
The
easy
solution
to
that
use
some
randomization
that
kubernetes
supports,
but
node
ports
can't
be
randomized
if
we
wanted
to
ensure
that
a
crew
brandy
service
didn't
get
a
new
port
every
time
the
service
got
recreated
when
we
needed
to
provide
a
reliable
way
to
retrieve
information
from
a
known
address
and
port
so
developers
because
of
those
had
to
statically
assign
node
ports
to
their
services.
In
addition,
we
were
seeing
inconsistency
and
node
port
assignments
for
single
applications
across
site,
so
we
had
to
have
specific
configurations
for
each
deployment.
A
A
A
So
we
needed
to
improve
this.
We
needed
to
implement
something
that
addressed
the
problem
of
application,
coupling
and
static
nodes
to
port
note,
our
port
to
no
port
load
balancing.
So
we
had
a
VIP.
We
had
a
port
on
that
bit
and
when
traffic
came
into
the
VIP,
where
it
went
beyond
that,
we
wanted
to
be
defined
by
name
and
for
each
service
that
runs
interpret
Nettie's.
A
So
we
decided
to
make
this
better
and
we
came
up
with.
We
call
cube
tight
idea,
cube
to
IBBs
uses
kubernetes
and
the
kubernetes
api
to
store
load,
balancer
configurations
and
retrieve
them
when
they
change
it
enabled
us
to
port
forward.
It
enables
us
to
add
and
remove
backends
from
Louis
to
backends.
It
enables
us
to
load
balanced
traffic
into
all
of
our
kubernetes
tenants
from
a
single
distributed
application.
A
So
to
understand
how
we
do
this
freaking
a
little
bit
of
primer,
about
how
cube
proxy
works
with
IP
tables,
IP
tables
works
by
chaining
different
rule
sets
together
and
matches
those
rules
against
an
incoming
packet.
An
incoming
packet
has
metadata
associated
with
it,
such
as
a
destination.
Address.
Ip
tables
will
compare
the
packet
with
the
pre-configured
rule
set
and
if
the
packet
matches
will
send
the
packet
to
a
target
chain,
cue
proxy
uses
IP
tables
to
route
traffic
down
for
a
kubernetes
service
into
an
endpoint
for
that
service.
A
A
A
If
a
packet
matches
on
a
destination
port
address
and
port,
it
will
be
sent
into
the
cube
the
cube
service
chain
specific
to
that
endpoint.
The
I'm.
Sorry
to
that
service,
specific
queue
service
chain
contains
a
list
of
endpoints
which
correspond
to
pods
that
are
available
to
host
the
service.
The
chain
will
pick
one
of
those
rules
at
random
and
send
the
packet
to
the
chosen
cube
service.
Endpoint
here
queue
proxy
sets
a
masquerade
bit
on
the
packet
and
destination.
Ask
the
packet
directly
into
the
pods,
that's
associated
with
that
end
point.
A
If
the
incoming
packet
doesn't
match
on
a
service
report
in
the
cube
services
chain
and
pre
routing,
it
will
attempt
to
match
in
the
last
chain
and
pre
routing
up
called
cube,
shared
ports.
If
the
incoming
packet
matches
the
node
port
in
this
chain,
it
will
forward
to
the
specific
cube
service
and
that's
associated
with
it
and
complete
the
rest
of
this
for
the
flow
through
IP
tables,
ultimately
getting
sent
to
the
pod.
A
So
what
cube
to
IBBs
does?
Is
it
injects
its
own
service
chain
into
pre
routing
that
bridges
traffic
found
for
this
address
directly
into
the
specific
cube
service
chain
for
a
configured
service?
This
means
that
packets
go
in
to
keep
pre
routing,
then
into
cube
to
ipbs,
where,
if
they
match
as
if
and
port,
they
get
routed
into
the
service.
This
approach
is
transparent,
the
kubernetes
built-in
service
load
balancing.
So
if
the
package
doesn't
match
nineteen
cubed
IP
vs,
it
gets
processed
through
the
queue
proxy
chain,
normally.
A
A
A
So
in
IP
vs
master
service,
the
IP
vs
master
manages
the
front-end
in
bits
and
ports
that
ipbs
listens
on.
It
queries
the
queue
API
or
the
config
map
populates
the
currently
configured
discs
and
adds
them
to
the
list
of
furnance
for
each
VIP
that
the
master
finds
in
the
config
map.
It
adds
that
bit
to
the
primary
interface
of
the
nodes,
the
master
also
creates
the
API
server
for
a
list
of
available
nodes
to
add
to
the
I
PDS
configuration
as
back-end
servers
for
each
of
these
furnace.
A
So,
in
order
to
maintain
the
list
of
backends,
the
ITV
Assessor
will
create
the
API
for
list
of
nodes
in
the
cluster
will
filter
the
nodes
and
they
tame
a
list
of
backends
of
my
PDF
configuration
based
on
those
health.
As
you
can
see,
the
last
node
and
the
Kagan
nodes
I
have
visited
not
ready
state
and
in
the
screen
shown
on
the
right.
A
You
can
see
that
the
IPPS
master
has
ensured
it's
not
in
the
list
of
healthy
back
in
the
IP
vs
master
service
will
also
remove
any
nodes
that
are
cordoned
or
unschedulable
and,
as
you
can
see
in
the
screenshot
on
the
right,
the
list
of
backends
is
missing
the
dot
158
node.
This
is
because
the
icbs
master
service
is
scheduled
and
running
on
that
node
and
when
the
IPPS
master
comes
up,
it
will
check
its
own
IP,
address
and
filter
out
of
the
list
of
available
nodes.
A
A
So
we
separated
it
pulled
by
VLAN
so
that
when
I
new
abode
Balan
services
added,
keep
to
IBBs
will
only
add
that
VIP
to
nodes
that
have
the
correct,
VLAN
node
label,
and
this
is
something
that
we
do
to
address
the
layer
to
adjacency
requirement
of
I
PBS,
the
IP
vs
backends
and
masters
watch
the
config
map
in
the
API
and
dynamically
update
their
bits
and
IP
tables
rules
as
a
change.
So
we
essentially
have
real-time
configuration
a
roller
balancer.
A
In
order
to
operationalize
the
updates
to
the
config
map,
we
created
a
UI
developers
or
operating
teams
can
use
to
manage
their
own
bits
on
a
per
site
basis.
This
gives
us
the
ability
to
run
the
config
map
in
a
secure
namespace,
so
that
only
the
platform
and
data
center
teams
can
manage
the
pool
of
IPs
available.
The
config
map
can
be
dynamically
dynamically,
updated
by
a
specific
teams
for
their
load
balancing
needs
without
directly
interacting
with
the
underlying
infrastructure.
So
this
enables
different
teams
to
be
responsible
for
their
own
load.
A
We
were
running
happily
for
a
while
when
QA
began,
recording
performance
degradation
during
load
testing
and
the
troubleshooting,
we
found
that
all
traffic
was
coming
into
and
going
out
of
the
same
server
and
more
interesting
and
interesting
enough.
It
was
not
my
PDF
master
and
we
immediately
knew
this
smelled
like
ARP.
So
what
was
happening
was
a
race
condition
where,
if
the
loopback
address
was
brought
up
by
a
cube
to
IPPs
on
any
back-end,
it
would
announce
itself
to
the
router.
A
If
this
timing
happened
before
the
IPPS
master
announced
itself,
its
ownership
of
the
VIP,
the
router
would
send
all
traffic
destined
to
that
IP
address
to
the
loopback
interface
on
the
backend
node
and
with
IP
tables
rules
being
set
traffic,
never
left
that
single
node.
So
a
quick
read
through
IP
VSS
documentation
led
us
to
the
familiar
arc
rules
which
made
the
loopback
interface
on
the
back
end.
Ignore
any
ARP
requests
on
the
network.
It
would
also
disable
our
pronounced
when
it
came
online,
so
this
list
left
the
IPPS
masters.
All
of
this.
A
The
next
issue
we
ran
into
was
when
we
attempted
to
co-locate
the
master
and
back-end
server
for
ipbs.
We
were
initially
explicitly
defining
a
set
of
nodes
for
the
masters
and
a
set
of
nodes
for
the
back
end,
but
we
wanted
to
reduce
the
operational
cost,
so
we
tried
running
them
on
the
same
set
of
nodes,
but
when
these
services
got
scheduled
on
the
same
node,
they
conflicted.
The
zip
address
was
assigned
to
the
primary
interface
as
well
as
the
loopback
address
and
similarly
to
ARP.
Iptables
rules
were
set
and
traffic
never
left
the
node.
A
So
we
devised
the
way
for
the
master
to
advertise
its
presence
and
for
the
presence
of
a
master
to
disrupt
the
back-end
service.
So
when
a
master
came
up,
it
opens
a
port
at
the
backend.
Servers
would
check
if
the
backend
server
receives
a
response
on
that
port.
It
halts
itself
on
the
node,
removes
its
VIP
IP
tables
rules
and
our
pools
if
it
does
not
receive
a
response
that
continues
running,
and
we
continuously
on
this
port
check.
A
In
our
initial
implementation
of
coop
to
IBBs,
we
are
relying
on
IP
tables
out
of
the
box.
Functionality
to
make
rule
set
changes
in
this
mode.
Ip
tables
uses
X
table
blocks
to
perform
an
individual
check
and
set
for
each
rule
that
we
wanted
to
create
and,
at
the
time
to
proxy,
was
doing
the
exact
same
thing.
A
So
in
cases
where
we
had
hundreds
or
even
thousands
of
services
queue
proxy
and
keep
to
IBBs
would
fight
each
other
over
who
owns
the
IP
tables,
locking,
and
this
contention
resulted
in
neither
to
proxy
or
cubed
IBBs
being
able
to
make
changes,
and
ultimately
this
resulted
in
cube
services
becoming
stale.
So
to
resolve
this,
we
switch
to
IP,
table,
save
and
restore,
and
using
this
method
instead
of
using
thousands
of
requests
for
each
change,
a
single
request
would
be
made
every
time
a
change
was
necessary.
A
And
lastly,
once
our
developers
started
using
cube
to
ipbs,
they
began
to
facilitate
communication
between
services
and
different
namespaces.
Using
this,
and
occasionally
we
would
see
requests
from
a
pod
to
a
fifth
timeout
and
we
realized
this
was
occurring
when
the
pod
that
was
making
a
request
was
scheduled
on
the
same
note
as
the
ITV
master
and
request
from
the
private
IP
of
the
pod
were
unable
to
reach
the
VIP
on
the
notes.
Primary
interface
and
the
coup
Brandis
cubelet
actually
provides
an
option
to
run
a
hairpin
mode
to
handle
these
edge
cases.
A
Okay,
so
we're
looking
at
the
Qi
PBS
UI
here
this
is
a
load
balancer,
that's
running
in
a
VLAN
that
we
call
green
in
the
mythical
namespace.
The
service
that
we
have
configure
here
is
mermaid's,
the
port
name
is
HTTP
and
the
dip
on
the
right
is
listening
on
port
80.
So
let's
take
a
look
at
this.
This
looks
like
in
the
back
end
here,
I'm
doing
a
watch
and
the
IPPS
ATM
command
inside
of
the
master
container.
A
A
At
the
bottom
here
these
are
the
IP
tables
rules
that
are
on
the
back
end
note
itself.
So
for
each
service
that
we
bring
up,
we
have
two
IP
tables
rules
and
this
destination
import
is
the
zip
address
that
we
have
configured
in
the
UI,
and
this
is
the
specific
Tube
service
entry
in
IP
table
so
where
the
VIP
gets
routed
to.
A
Up
on
the
top
right
here
is
our
config
map.
You
can
see
we
have
a
list
of
IP
addresses
that
are
in
the
VIP
pool
and
the
configuration
that
describes
the
service
that
we
configured
in
in
the
UI.
So
this
is
named
state
service,
name
and
port
name.
You
can
see
that
we're
not
using
any
node
ports
here
and
all
in
our
definition,
so
I'm
going
to
add
another
another
service.
I
have
a
service
called
unicorns
of
running
in
this
cluster
I.
A
A
A
So
now
we
have
two
of
its
pointed
up
to
the
unicorn
service
on
two
different
ports.
We
come
back
here.
The
IPPS
master
has
been
updated
with
the
Sun
proxy
I've
been
port.
Our
back-end
has
the
new
IP
tables
rules.
Configured
and
I
just
want
to
point
out
here
that
the
service
on
the
right
is
the
exact
same
service
that
is
getting
ready
to
in
the
backend.
So,
even
though
it's
listening
on
two
different
ports,
it
will
route
to
the
same
Cooper
nettie
service.