►
Description
Scale Kubernetes to Support 50,000 Services [I] - Haibin Xie & Quinton Hoole, Huawei Technologies
Kubernetes currently has two load balancing modes: userspace and IPTables. They both have limitation on scalability and performance. We introduced IPVS as third kube-proxy mode which scales kubernetes load balancer to support 50,000 services. Beyond that, control plane needs to be optimized in order to deploy 50,000 services. We will introduce alternative solutions and our prototypes with detailed performance data.
About Quinton Hoole
Technical Vice President, Huawei
About Haibin Michael Xie
Senior staff engineer/architect, Huawei Technologies
I am a senior staff engineer and architect from Huawei PaaS team based in Seattle, WA, USA. Main main focus is scalability and performance for PaaS platform and APM technologies. Prior to joining Huawei, I worked on Windows Storage Server, Windows Azure monitoring/analytics and Bing Ads user behavior targeting at Microsoft.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Most
of
the
hard
work
here,
I
must
point
out
was
not
done
by
myself
at
all,
but
by
my
colleague
Michael,
who
is
one
of
our
senior
research
network
research
engineers
in
Seattle.
He
unfortunately
could
not
make
it
today
he's
sitting
in
Seattle
without
a
visa,
so
the
task
fell
for
me
to
present
his
good
work.
A
It's
not
a
particularly
long
talk,
which
is
probably
a
good
thing,
given
that
I'm
the
only
thing
standing
between
you
and
kobir
at
this
point,
so
I
will
take
that
responsibility
very
seriously,
and
there
should
be
significant
time
at
the
end
for
questions.
I
think.
Maybe
it's
easier
if
we
lead
them
towards
the
end,
but
certainly
prepare
your
questions
along
the
way.
We
should
have
a
good
ten
minutes
or
so
at
the
end,
all
right
with
no
further
ado.
A
So,
first
of
all,
not
all
of
you
are
necessarily
that
familiar
with
the
way
kubernetes
internal
cluster
service
load,
balancing
works
so
we'll
just
go
over
that
so
you've,
probably
familiar
with
having
kubernetes
master
inside
the
master.
We
have
an
API
server,
tralala
manager,
scheduler,
HCD,
persistent
store,
etc,
and
then
we
have
the
the
nodes,
each
of
which
have
a
bunch
of
agents
running
on
them,
one
of
which
is
Q
proxy
to
proxy,
essentially,
programs
of
local
load
balancer
on
each
node
and
that
then
load
balances
to
the
pods.
A
So
a
couple
of
problems,
first
of
all,
all
of
the
control
plane
related
stuff-
has
potential
bottlenecks
involved
in
it
and
then
once
we
have
deployed
the
load
balancers
that
the
traffic
obviously
needs
to
flow
through
these
things-
and
there
are
performance
issues
there
too,
so
deploying
services
and
pods
is
potential.
Scalability
bottleneck,
propagating
endpoints,
so
endpoints
of
the
you
know,
final
IP
addresses
that
the
traffic
ends
up
at
after
load
balancing
these
obviously
change
over
time
when
your
pods
are
deployed
or
torn
down.
A
All
the
changes
to
these
endpoints
need
to
be
propagated
to
all
nodes
in
a
cluster
and
then
the
actual
data
plane
itself.
We
need
to
add
and
remove
services
to
the
load
balancers
on
each
node
in
some
configurations,
as
IP
tables,
we'll
talk
a
little
bit
about
exactly
what
the
challenges
are
there
and
then
accessing
the
services
themselves.
Each
packet
in
your
network
stream
needs
to
Traverse
some
sort
of
rules
which
determine
where
it
gets
load
balance
to,
and
these
can
also
incur
performance
problems
diving
into
the
control
plane
a
little
bit
further.
A
When
a
service
gets
deployed,
it
gets
stored
in
sed
by
the
API
server
that
then
triggers
the
endpoints
controller,
which
then
creates
endpoints
are
also
persisted
in
sed
by
the
API
server.
These
are
then
picked
up
by
all
of
the
nodes
which
listens:
the
cubelet
on
each
node,
X
or
sorry,
Q,
proxy
and
each
node
listens
against
the
endpoint
lists
and
updates
its
local
load
balancer.
Accordingly,
then,
whenever
we
schedule
pubs,
we
obviously
create
a
part
in
the
API.
A
This
again
will
trigger
a
change
to
the
input
where
the
influence
controller
will
then
pick
that
up,
create
or
remove
whatever
endpoints
might
be
necessary,
which
again
same
old
story,
gets
propagated
down
to
all
of
the
nodes.
So
it
doesn't
take
too
much
complicated
math
to
work
out
that
if
you
have
n
nodes
in
every
cluster-
and
you
are
trying
to
schedule
some
number
of
pods
per
second,
the
QPS
against
the
API
server
is
n
times
n.
That's
how
many
endpoints
per
sorry.
A
Yes,
that's
how
many
endpoints
per
second
are
created
against
the
API
server,
so
just
to
make
that
a
little
bit
more
concrete.
You
create
services.
So
many
of
you
have
done
this
before
the
structure
on
the
Left
gives
you
an
idea
of
what
those
things
look
like.
They
specify
things
like
ports,
protocols,
etc,
selectors
for
which
pods
form
part
of
this
service.
You
get
a
cluster
IP
allocated
to
your
service
and
then,
on
the
right
hand,
side.
A
Essentially
so,
you
can
see
that
that
can
pretty
quickly
get
out
of
hand
in
clusters
with
large
numbers
of
services.
So
we
kind
of
brainstormed
a
few
different
approaches
to
solving
the
control
plane,
alternatives
solutions.
We
can
partition
the
endpoints
into
multiple
objects.
This
you
know
fairly,
obviously
has
some
pros
and
cons.
You
reduce
the
number
of
the
size
of
the
endpoints,
but
you
increase
the
number
of
objects,
so
the
net
savings
is
not
very
significant.
A
A
Clearly,
this
reduces
the
number
of
connections
and
requests
against
the
API
server,
but
it
introduces
an
extra
hop
in
your
traffic
routing.
You
need
these
things
to
be
highly
available.
If
you
only
had
a
single
one
of
these
load
balancers,
obviously,
if
it
went
down
all
of
the
services
in
your
cluster
would
go
down.
That
turns
out
to
be.
A
You
know,
non
severely
difficult
and
perhaps
most
importantly,
the
scalability
of
that
is
kind
of
difficult,
because
you
know,
as
you
have
more
services
or
more
service
traffic,
you
need
to
deploy
more
of
these
load
balancers
and
manage
them,
etc.
Now,
kubernetes
clearly
can
make
that
a
lot
easier
for
you.
However,
if
you
just
have
a
load
balancer
on
every
single
node,
that
kind
of
scales,
naturally
in
accordance
with
the
amount
of
activity
going
on
in
your
cluster,
so
that
seems
like
a
nicer
solution.
A
third
option
that
we
considered
was
batching
the
updates.
A
A
A
We
reduced
the
QPS
in
our
tests
by
98%,
so
that
was
pretty
cool
and
if
you
look
at
the
graph
at
the
bottom
and
that's
for
a
relatively
small
number
of
services,
you
can
see
that
our
optimized
system
at
the
bar
bottom,
the
red
line,
is
pretty
much
flat,
whereas
with
the
pre
batching,
we
ended
up
with
a
you
know,
essentially
approaching
exponential
growth.
You
can
extrapolate
that
to
the
right
to
50,000
services,
and
you
can
see
that
the
number
looks
pretty
ugly.
A
In
turned
latency,
so
this
is
the
latency
of
of
updating
our
load
balances.
Again,
these
are
fairly
small
numbers
of
services,
I'm,
just
giving
you
a
flavor
of
some
of
the
stuff
that
we
tried
and
they'll
get
into
some
bigger
numbers
later.
In
the
talk
to
illustrate
how
this
translates,
when
you
have
tens
of
thousands
of
services,
we
I'm
not
going
to
go
through
all
the
details.
Here
we
had
a
simulation.
We
had
10
pods
in
each
service.
A
We
had
you
know
hundreds
of
services
across
couple
of
points
of
data
there
and
you
can
see
the
before
and
after
numbers
of
we
managed
to
reduce
the
latency
by
you
know:
60
percent,
which
was
good,
so
the
more
interesting
stuff
I
think
is
the
data
plane.
So
at
the
moment
the
commonly
used
configuration
of
Q
proxy
is
to
program
IP
tables
on
each
node
to
perform
the
load
balancing
within
the
cluster.
A
For
those
of
you
may
not
be
intimately
familiar
with
what
exactly
I
P
tables
is
it's
a
user
space
application
that
allows
you
to
configure
internal
firewall
rules
which
are
implemented
on
top
of
net
filter,
which
is
a
part
of
the
Linux
kernel,
and
these
rules
are
contained
in
things
called
chains
which
are
essentially
tables.
Hence
the
name.
Ip
tables
and
netfilter
in
turn
is
a
framework
provided
by
the
Linux
kernel
that
allows
you
to
customize
things
like
packet,
filtering
and
network,
address,
translation,
etc.
A
So
there
are
two
pretty
major
challenges
related
to
IP
tables
and
they're,
both
kind
of
fundamentally
a
result
of
the
fact
that
these
things
are
tables
and
lists,
and
you
probably
remember
from
your
computer
science,
one
courses
that
inserting
and
removing
from
lists
and
tables
is
not
the
most
efficient
of
operations.
So
when,
when
those
tables
become
extremely
large,
which
they
do
in
the
case
of
kubernetes,
you
can
have
some
pretty
serious
problems
there.
A
The
second
related
issue
is
that
every
packet
has
to
go,
and
you
know
match
against
some
rule
in
that
table
before
it
gets
load
balanced
and
if
it
does
a
linear
traversal
through
the
table
to
find
the
rule
that
matches
if
that
list
is
50,000
long,
each
packet
has
to
incur
that
latency,
which
translates
into
lower
throughput
etc.
So
those
are
the
problems,
here's
an
example.
So
this
is
the
actual
IP
tables
rules
that
are
configured
by
few
proxy,
and
this
particular
example.
We
have
two
services.
A
A
So
what
does
that
mean
long
lists
of
rules
in
a
chain
produce
some
pretty
horrifying
things
in
particular,
if
you?
So
not
only
do
you
get
poor
performance,
but
you
also
get
variable
performance
because,
depending
on
where
your
service
landed
in
that
chain,
which
is
probably
you
know
related
to
when
it
was
created
relative
to
the
other
services,
the
performance
can
vary
quite
dramatically.
So
you
can
see
in
the
graph
there.
We've
got
a
bunch
of
services
of
a
number
of
services
scales
to
the
right.
A
So,
where
exactly
is
this
latency
generated,
and
particularly
around
manipulating
these
IP
tables
rules?
So,
first
of
all
the
implementation,
and
for
those
of
you
might
have
gone
to
the
excellent
talk
that
came
before
this
one
about
IP
vs.
They
also
touched
on
this
problem
where,
because
of
the
fact
that
every
time
the
tables
need
to
be
manipulated
in
any
way,
they
need
to
be
locked.
If
you
have
multiple
concurrent
processes
trying
to
manipulate
your
tables,
you
look
at
lock
contention
and
failures,
etc.
A
A
We
make
all
the
changes
and
we
save
all
the
rules
back
again,
which,
as
you
can
imagine,
has
raised
conditions
and
other
problems
associated
with
it
and
and
I
bedeviled
is
entirely
locked
during
this
whole
process,
so
that
you
know
again,
when
you
have
fifty
thousand
services
and
each
one
of
them
has
pods
joining
and
leaving
the
service.
The
rate
of
change
is
pretty
high
and
each
one
of
those
operations
requires
a
lock
as
well
as
traversals
of
these
very
long
chains.
A
So
you
can
see
the
numbers
they're,
adding
one
rule
when
we
have
five
thousand
services
which
results
in
forty
thousand
iptables
rules,
details
of
which
we
aren't
going
to
now,
but
you
could
see
for
the
small
number
of
services.
We
had
an
example.
We
had
more
than
that
number
of
rules.
It
takes
eleven
minutes
to
add
one
rule
to
the
iptables
configuration
so
clearly,
this
is
not
tenable.
If
you
have
twenty
thousand
services,
you
have
sixty
thousand
rules,
that's
five
hours
per
update,
obviously
not
usable.
A
So
we
considered
a
couple
of
different
options.
We
one
option
which
is
actually
being
discussed
I
think
before
is
the
possibility
to
restructure
iptables
into
a
some
kind
of
a
search
tree.
That
way
you
can
address
both
of
the
issues
that
we
mentioned,
one
being
that
the
that
any
individual
chain
in
iptables
will
become
much
shorter
and,
secondly,
the
traversal
of
those
chains,
so
the
insertions
are
done
per
chain.
So,
if
you're
inserting
into
a
short
chain,
the
insertions
are
a
lot
quicker
and
similarly,
the
packet
reversal
through
those
chains
are
a
lot
faster.
A
The
other
approach
that
we
considered
was
replacing
IP
tables
with
ipbs
at
the
virtual
server.
This
gives
a
pretty
dramatic
performance
improvement,
probably
and
we
never
actually
prototyped
the
restructuring.
The
IP
tables
option,
but
but
ipbs
does
at
least
as
well
as
that.
In
addition
to
that,
it
has
much
smarter
load
balancing
options
available,
you
can
do
weighted
load
balancing
you
can
do
retries,
etc.
So
it
has
much
more
sophisticated
load.
Balancing
IP
table
rafter
always
was
not
really
designed
for
load
balancing
design
firewalls,
whereas
ipbs
was
designed
for
load
balancing
and
con
firewall.
A
A
My
animations
are
not
great
here,
but
you
can
see
that
the
number
of
chains,
the
number
of
rules
you
end
up
traversing-
is
a
lot
smaller
and
the
size
of
each
one
of
those
chains
for
update
purposes.
In
particular
the
the
search
tree
based
service
routing
time
complexity
is
order
of
square
root
of
n
or
n
root
of
n
for
a
given
tree
depth.
A
So
that's
clearly
a
significant
improvement
over
the
numbers
that
reported
earlier
a
little
bit
of
a
background
to
ipbs,
and
this
is
not
a
deep
dive
into
ipbs
by
any
means,
but
it's
a
transport
layer
load
balancer
which
directs
requests.
It
supports
TCP
UDP
and
it's
very
close
to
real
service
as
part
of
the
Linux
virtual
server
project.
It's
been
around
for
many
many
many
years
in
the
kernel,
very
heavily
used
considered
very
stable,
and
so
we
thought
this
was
kind
of
worth.
Trying
out
see
what
see
how
it
works.
A
So,
just
to
have
a
quick
look
at
the
comparison,
so
IP
tables
can
manipulate
tables
provided
by
the
learning
side
wall.
It's
much
more
flexible
as
I
mentioned
earlier,
because
you
can
manipulate
packets
at
different
stages
of
the
writing
process,
pre,
routing,
post
routing,
etc,
and
it
has
way
more
operations
that
can
be
performed
on
packets.
You
can
do
source
snap
destination
mass.
You
can
reject
things.
A
You
can
do
poor
translations,
all
sorts
of
things,
none
of
which
we
actually
necessarily
need
for
pure
load,
balancing
I
think
the
big
plus
for
ipbs
is
that
it's
hash,
based
and
rather
than
list
based
or
chain
based
and
obviously,
as
I
mentioned.
It
also
has
a
lot
more
load
balancing
algorithms
by
virtue
of
the
fact
that
it's
designed
as
a
load
balancer
another
firewall.
A
So
you
can
do
least
load
East
connections,
locality
based
Wake
based
many
different
load.
Balancing
algorithms
supports
health
checks,
connection
retries,
sticky
sessions,
all
of
these
things
of
which
are
very
useful
in
load
balancing.
If
you
have
multiple
backends
and
single
front-end,
you
would
like,
ideally
to
have
that
client
directly
to
the
same
server
for
caching
reasons
and
numerous
reasons.
Why
that
why
sticky
sessions
are
useful
and
we
can't
do
those
with
IPPs,
at
least
with
IP
tables,.
A
A
Semantics
supports
all
of
the
normal
things
you
would
expect
of
commedy
services.
What
we
did
was
we
made
changes
to
the
queue
proxy
agents
that
runs
on
the
nodes,
so
that
agent
at
the
moment
already
supports
two
modes,
a
probably
familiar
with
go
to
user
space
mode,
which
was
historically
the
first
one,
I
think
happy
the
kubernetes
version,
one
only
supported
that
there
were
performance
problems
with
that
and
various
other
things
later.
A
An
IP
tables
mode
was
added,
which
is
what
we've
been
talking
about
now
much
faster
all
in
kernel,
but
with
all
the
challenges
we
mentioned,
what
we
did
was
we
added
the
third
mode,
IP
vs,
the
original
feed
proxy
program
was
about.
12,000
lines
occurred
roughly,
so
we
added
another
680
lines
of
code
to
implement
the
IPPS
logic,
and
it
has
an
external
dependency
on
a
seesaw
library.
A
So,
let's
have
a
look
at
some
of
the
comparative
Layton
sees
this
I
guess
should
be
in
a
graph,
but
we
didn't
get
there,
but
you
can
see
that
for
even
reasonably
modest
numbers
of
services.
If
you
have
5,000
node
cluster,
it's
likely
that
you're
running
you
know
tens
of
pods
per
node.
You
can
imagine
that
5,000
to
10,000
to
50,000
services
is
not
unreasonable
on
the
cluster
that
size
and
we
do
run
clusters.
This
big.
A
On
the
other
hand,
irrespective
of
the
number
of
services
that
you
have
has
a
essentially
a
linear
to
millisecond
update
rate,
which
is
by
virtue
of
the
fact
that
it's
hash
table
based
and
not
insertions
and
deletions
from
lists,
what
else
did
I
have
to
say
there?
Yes,
I,
guess:
that's
that's
enough!
For
today.
We
then
ran
some
benchmark
tests
using
Q,
/
f.
You
can
see
that
the
we
had
various
QPS
Ray's
Layton
sees
throughput,
etc.
They
all
show
very
similar
patterns
and
you
can
see
the
top
two
lines.
A
There
are
the
IPPS
based
implementation,
so
we
managed
to
scale
it
up
to
50,000
services.
The
throughput
is,
you
know
approximately
half
of
what
it
would
be.
If
you
had
one
service
which
is
still
pretty
bad,
and
this
we
could
do
some
more
work.
There
I
think
we
can
probably
get
that
that
red
and
purple
line
relatively
flat,
horizontal
the
other
two
you
can
see
even
at
5,000
services,
which
is
not
a
lot,
the
throughput
is
pretty
bad
and
basically
there
is
no
throughput
beyond
about
10,000
services,
so
we
were
pretty
happy
with
that.
A
I
guess.
The
other
important
thing
to
note
there
is
that
the
first
in
the
last
service
in
the
IBBs
case
are
essentially
equally
performant,
whereas
there's
a
fairly
substantial
difference
between
the
services
throughputs
of
different
services,
any
iptables
case
that's
about
as
much
as
there
is
interesting
to
say
on
that
slide.
I
think
we've
also
done
a
bunch
of
other
scalability
work.
We
worked
on
scaling
the
number
of
nodes
and
pods
in
a
single
cluster.
A
lot
of
this
work
has
been
sort
of
paralleled
in
the
open-source
community
for
a
variety
of
historical
reasons.
A
This
was
not
open-source,
but
we
probably
should
have
and
will
certainly
do
better
in
the
future.
With
these
kinds
of
things
we've
reduced
various
enter
and
Layton
sees
we.
We
did
quite
a
lot
of
work
on
scheduling
performance
improvement
because
we
have
these
very
large
clusters
and
pretty
high
rates
of
change.
We've
got
an
optimistic
concurrent
scheduler,
which
actually
is
a
very
small
change.
The
existing
urban
Eddy
scheduler.
Again
we
happy
to
open-source
that
and
probably
will
be
doing,
that
in
the
coming
months.
A
We
also
have
a
pretty
interesting
flow
graph
based
high
performance
scheduler
that
we're
working
on
together
with
the
University
of
Cambridge
and
that
can
bring
you
know
very,
very
dramatic
improvements.
The
optimistic
on
parent
stuff
essentially
enables
you
to
schedule
multiple
pods
at
the
same
time,
and
we
employ
some
basic
heuristics
to
reduce
the
probabilities
of
collisions,
multiple
concurrent
schedulers
scheduling
the
same
or
different
pods
onto
the
same
resources
underlying
resources,
and
that
turns
out
not
to
be
terribly
difficult.
A
Actually,
if
you
introduce
a
little
bit
of
randomization
into
things,
the
probability
of
collision
is
very
low
and
then
you
just
retry
and
the
performance
of
that
was
you
know
an
order
of
magnitude
or
more
better
than
the
existing
scheduler
flow
graph
based
high
performance
scheduling
is
many
many
orders
of
magnitude
faster
than
what
we
have
at
the
moment
and
has
a
variety
of
additional
benefits
like
facilitating
rescheduling
much
better
than
we
do
at
the
moment
and
we're
happy
to
open-source.
All
of
this
work,
in
fact
we're
the
University
of
Cambridge
work.
A
We're
doing
is
definitely
already
open
sourced
and
we
will
continue
to
work
with
the
community
on
and
with
that
I
leave
you
with
at
least
nine
minutes
more
of
beer
drinking
time,
less.
Whatever
questions
we
have
anybody,
questions
comments,
yeah
we'd
like
to
grab
that
microphone
there
or
somebody
could
positive.
A
How
many
notes
you
had
how
many
nodes
did?
We
have
I
believe
and
I'm,
not
a
hundred
percent
in
these
particular
tests.
That
I
was.
These
were
very
small
numbers
of
nodes.
This
was
just
a
test
to
verify
so
that
the
performance
we
expect
to
I
mean
that
should
be
comparable,
irrespective
the
number
of
nodes.
A
The
point
of
mentioning
the
number
of
nodes
was
just
that
when
you
have
large
numbers
of
nodes
you
likely
to
have
large
numbers
of
services
and
parts
as
well,
but
we
just
simulated
that
by
putting
large
numbers
of
very
small
pods
on
small
numbers
of
nodes,
because
we
didn't
have
10,000
nodes
to
test
on,
but
we
do
have
deployments
we
have
tested
in
other
scenarios
with
thousands
of
nodes.
I
don't
have
the
exact
results.
I
could
get
them
for
you
if
you
like,
but
I,
think
they're
in
the
region
of
5,000.