►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scaling Kubernetes Networking Beyond 100k Endpoints - Rob Scott & Minhan Xia, Google
As Kubernetes continues to grow, scalability is increasingly important. One of the key bottlenecks that emerged as clusters got larger was related to network endpoints. They became quite slow at scale, and had limitations preventing them from growing beyond approximately 5,000 endpoints per Service. In this talk, Rob will show how the scalability of network endpoints has been dramatically increased in Kubernetes. He’ll cover the design and implementation of EndpointSlices, along with the impact they’ve had. He’ll walk through the process of profiling key components in the Kubernetes codebase to identify areas that could be optimized. Finally, he’ll show how all this work has improved Kubernetes network scalability, including real world performance metrics from large clusters with over 100,000 endpoints.
https://sched.co/Zeta
A
A
Google,
today's
talk
is
going
to
be
divided
into
six
different
sections.
We've
got
a
problem
statement
which
will
kind
of
give
some
background
into
why
we
did
all
this
work.
Then
an
endpoint
slice,
api
introduction,
just
a
very
high
level
idea
of
why
we
designed
the
endpoint
slice
api
and
what
it
can
accomplish
and
then
I'll
talk
about
how
it
works.
A
Some
of
the
key
functionality
involved
and
then
I'll
dig
a
bit
deeper
into
how
we
could
profile
kubernetes
to
dramatically
improve
the
performance
of
these
implementations
and,
finally
putting
it
all
together,
talk
about
how
that
translates
into
performance
at
and
even
beyond
a
hundred
thousand
endpoints
and
in
conclusion,
I'll
talk
about
what's
next,
what's
coming
up,
that
is
going
to
be
based
on
this
functionality
and
there's
some
really
exciting
stuff.
But
with
that
I'll
hand
it
over
to
minhon
to
give
some
background.
B
All
right,
thank
you,
rob
for
the
intro
so
before
we
go
to
our
solution.
So
let's
understand
what
the
exact
pain
points
our
users
have.
So
there
are
mainly
two
pain
points
when
using
kubernetes
service
at
scale.
The
first
one
is
that
there's
currently
a
limit
for
number
of
endpoints
within
a
single
service,
so
this
is
around
5000
endpoints
in
a
single
service.
So
the
second
one
is
that
the
user
may
observe
a
significant
performance
degradation
in
a
very
large
cluster.
B
So
both
of
these
questions,
these
problems
are
originating
in
the
same
thing,
which
is
the
endpoints
api.
The
api
was
designed
in
the
very
early
days
of
kubernetes
and
the
scalability
wasn't
an
important
area
of
consideration
at
those
days.
So,
in
short,
the
problem
lies
in
the
fact
that
the
endpoints
object
contains
a
full
list
of
all
endpoints
behind
a
given
service.
B
B
B
So
one
and
a
half
megabyte
of
endpoint
object
can
roughly
contain
5000
endpoints
inside
a
service
and
that
there's
a
various
aspects
like
that
can
influence
the
size
of
the
endpoint
object
like
the
side,
the
length
of
the
name
and
the
name
spaces
of
those
back
and
pod.
Is
it
the
ipv6
endpoint
or
the
ipv4
endpoint.
B
B
So,
let's
take
a
look
at
the
second
aspect
of
the
problem:
that
is
the
performance
degradation.
So
in
order
to
understand
that
we
need
to
do
a
quick
recap
on
how
does
the
service
control
actually
works
within
the
kubernetes
behind
the
scene?
So
when
the
service
gets
created,
the
endpoints
object
gets
created,
so
the
pod
gets
created.
Then
pod
gets
scheduled.
B
The
cuber
watches
its
own,
the
parts
running
on
itself
and
then
cubeless
signals
the
container
runtime
to
basically
in
the
implement
this
container,
and
this
part
and
the
container
runtime
says
it's
run
and
it's
running
and
healthy.
Then
the
cube
updates
the
pot
status
to
be
ready
and
the
endpoints
controller
watches
the
pod,
and
then
it
updates
the
endpoints
correspondingly
and
q
proxy
pick
it
up
and
program
iptables
ipvs
and
to
implement
kubernetes
services
and
then
eventually
the
endpoint
is
programmed.
B
So
what
happened
is
that
each
q
proxy
running
on
each
node
needs
to
watch
all
endpoints
of
the
whole
kubernetes
cluster
right.
So
so.
This
is
because
it
has
this.
This
in
these
information
is
necessary
for
it
to
basically
implement
kubernetes
service
on
each
node,
and
that
translates
to
there
are
n
number
of
nodes,
means
n
number
of
watchers
right
and
because
there's
no
magic
in
sending
update
through
the
network.
B
So
any
number
of
copies
object,
copies
will
need
to
be
sent
from
the
api
server
to
each
watcher
and
combined
with
the
number
back
in
pod
and
the
size
of
the
endpoint
object,
which
is
proportional
then
to
the
number
back
in
pod
and
times
the
number
watcher.
So
the
total
bytes
transmitted
per
update
is
o
n
times
p.
B
So,
let's
give
some
real
numbers
here.
So
the
number
of
nodes-
let's
say
it's
a
large
cluster
with
five
thousand
five
thousand
nodes
and
the
size
of
the
endpoint
object,
is
one
megabyte
which
contains,
let's
say,
5000
endpoints,
then
in
each
update
we
need
to
send
five
gigabytes
of
data.
That
is
basically
the
size
of
the
full
dvd
right.
So
for
one
endpoint
object,
one
endpoint
update
for
a
large
service
and
let's
do
a
further
estimation
right.
Let's
say
total
by
transmitted
per
update,
is
five
gigabyte.
And
what
about?
We?
B
Do
a
rolling
update:
let's
do,
let's
do
a
rollout
to
this
large
service
and
this
service,
let's
say,
have
5
000
endpoints
and
that
roughly
translate
to
more
than
3
000,
maybe
more
or
maybe
less
depending
on.
What's
your
rolling
update
strategy,
then,
let's
assume
it's
5
000
updates
right,
then
that
translates
to
25
terabyte
of
data.
B
That
is
a
lot
of
data,
in
the
contrary,
the
user.
What
user
expects
is
that
it
wants
to
run
large
services
in
large
clusters
and
it
wants
to
be
agile
as
well
right.
It
wants
to
enable
auto
scaling,
it
wants
to
roll
out
frequently,
and
that
translates
to
high
churn
within
the
service,
and
all
of
this
should
just
work
and
kubernetes
should
provide
this
kind
of
capability.
B
B
Let
me
quickly
walk
through
what
is
endpoint
slice
api,
so
the
goal
of
the
endpoint
api.
The
primary
goal
is
to
address
the
scalability
constraints
that
we
listed
out
in
the
previous
section,
that
is,
to
support
tens
of
thousands
of
backend
endpoint
in
a
cluster
with
thousands
of
nodes
since
we're
basically
reinventing
and
redesigning
the
whole
endpoint,
the
endpoint
api
we
throw
in
a
bunch
of
other
extension
points
which
we
wanted
to
address.
Dual
stack
and
topological
aware
services,
and
things
like
that.
B
B
A
So
there's
three
core
components
that
I
want
to
talk
about:
there's
an
endpoint
slice,
controller
that
watches
services
and
pods
and
creates
or
updates
endpoint
slices.
This
is
very
similar
to
the
endpoints
controller.
It's
just
managing
endpoint
slices.
Instead,
then,
we
have
a
newer
controller,
called
the
endpoint
slice,
mirroring
controller
that
watches
custom,
endpoints
and
mirrors
them
over
to
endpoint
slices.
A
A
So
we
wanted
to
reduce
endpoint
slice
churn
this
lines.
Up
with
what
menhan
was
saying
earlier,
every
node
is
going
to
be
watching.
Endpoint
slices,
so
updates
are
very
expensive.
We
want
to
minimize
the
number
of
endpoint
slices,
we're
updating,
because
every
single
time
we
update
it
it
has
to
get
sent
out
to
every
node.
A
Then
we
want
to
limit
the
request
per
second
to
api
server.
Endpoint
slices
that
are
too
small
will
result
in
too
many
resources
being
updated
and,
on
the
other
hand,
if
we
have
endpoint
slices
that
are
way
too
big.
It
will
result
in
a
dvd's
worth
of
data
getting
sent
across
the
cluster
even
for
the
tiniest
of
changes
and
we'll
be
right
back
to
the
problem.
We
had
that
to
begin
with,
so
with
that
background,
I
wanted
to
explain
one
relatively
unintuitive
part
of
the
how
this
controller
works.
A
So,
let's
imagine
an
example
where
you
have
two
endpoint
slices
now
keep
in
mind
that
endpoint
slices
by
default
can
hold
up
to
100
endpoints,
so
we
have
one
endpoint
slice
with
70
endpoints
and
one
endpoint
slice
with
80
endpoints,
and
we
have
50
endpoints
to
add
now
you
would
think
well.
This
is
a
really
straightforward
update.
You
can
just
update
those
two
endpoint
slices
and
everything
will
work
fine,
but
instead
what
the
controller
is
going
to
do
is
create
an
entirely
new
endpoint
slice,
because
we
prefer
a
single
create
over
multiple
updates.
A
There's
a
lot
of
extra
detail
and
logic
here.
That
means
that
this
rarely
actually
happens,
often
in
an
individual
sync
we're
also
updating
endpoint
slices,
and
if
we
are,
we
fill
them
up
as
much
as
we
can.
But
if
we're
not
updating
anything,
we
prefer
to
create
something
new,
as
opposed
to
run
multiple
updates.
A
and
it
was
it
worked.
That
was
the
good
news,
but
when
we
actually
started
to
look
at
performance,
things
were
just
slower.
Coup
proxy
was
very
noticeably
slower
when
using
endpoint
slices
compared
to
endpoints
and
part
of
this
made
sense.
You
know
the
implementation
was
more
complex.
There
was
more
to
do,
and
that
was
almost
fine,
but
endpoint
slice.
Implementations
really
should
have
been
faster
because
we
really
wanted
to
scale
this
out.
A
So
I
did
some
profiling
with
a
really
cool
tool,
called
pprof
against
cube
proxy
just
to
see
if
there
were
any
bottlenecks
that
we
could
get
rid
of,
so
if
you've
run
prof
before
you're,
probably
familiar
with
something
that
looks
a
bit
like
this.
This
is
the
result
of
running
a
p
prof
report
on
coop
proxy,
specifically
for
cpu
time.
It
can
profile
any
number
of
components,
but
in
this
case
I
was
interested
in
cpu
time,
and
with
that
background
I
found
three
really
important
ways.
We
could
improve
performance
here.
A
First,
we
had
a
endpoint
dot
ip
function,
which
was
taking
43
of
total
cpu
time.
That's
an
absolutely
massive
amount
for
doing
something
that
shouldn't
have
taken
any
time.
I
had
thought
that
this
function
was
just
getting
a
string,
that's
it,
but
what
it
was
actually
doing
was
it
was
calling
net
parse
ip
that
parse
an
ip
out
of
a
string,
port,
iport
combination,
there's
no
reason
we
couldn't
have
just
used
the
iport
string
here
so
making
that
really
simple
change
ended
up,
really
improving
performance.
Here.
A
A
A
But
if
you're
at
this
talk,
you're-
probably
not
just
interested
in
ten
thousand
points
because
well,
we
know
that
this
works
at
ten
thousand
endpoints
and
it's
pretty
reliable
at
this
point,
but
we
really
want
to
understand.
Well
just
how
far
can
we
push
this
thing?
So,
let's
talk
about
a
hundred
thousand
endpoints,
but
before
we
get
too
far
into
this,
I
have
to
provide
some
caveats.
A
These
results
are
absolutely
not
scientific.
This
is
just
what
happens
when
we
wanted
to
know
what
would
happen
at
the
scale.
This
is
at
by
no
means
an
endorsement
of
running
a
service
with
100
000
endpoints,
at
least
not
in
production,
and
I
did
absolutely
nothing
to
tune
this
cluster
for
better
performance
at
scale.
It
was
running
with
all
the
standard
e2e
test
settings,
so,
if
anything,
it
probably
was
tuned
against
scale.
There's
some
things
like
really
verbose
log
levels,
which
have
to
have
some
kind
of
negative
impact
on
performance.
A
So,
let's
talk
just
briefly
about
how
I
set
this
up,
so
I
used
coop
test
with
most
defaults
used
for
e2e
test
clusters,
and
I
ran
this
on
a
1.19
alpha
free
release,
so
this
is
after
code
freeze,
but
before
1.19.0,
so
somewhere
in
that
mix,
I
have
a
specific
commit
there.
If
you're
interested
is
what
this
was
run
against
and
you
can
see
the
command
I
used
to
spin
up
this
cluster.
A
A
So
I
learned
pretty
quickly
on
that.
Although
a
hundred
thousand
pod
deployment
doesn't
work,
I
it
at
least
in
my
case
was
impossible
to
actually
get
any
information
about
it,
because
if,
as
an
example,
you
try
to
run
coop
cuddle
get
pods
and
at
least
in
my
experience,
it
would
consistently
time
out,
because
every
pod
inside
a
deployment
shares
the
same
set
of
labels.
A
It
was
very
difficult
to
just
get
an
individual
pot
or
a
small
subset
of
pots,
so
instead,
what
I
ended
up
doing
is
10
deployments
with
10
000
pods,
each
and
surprisingly
enough.
There
weren't
any
issues
listing
10
000
pods
at
a
time,
and
I
targeted
all
those
pods
all
100
000
pods,
with
a
single
node
port
service.
A
I
have
to
mention
this
is
again
by
design,
because
the
controller
is
minimizing
updates
and
it's
always
going
to
create
a
new
endpoint
slice.
Instead
of
updating
multiple
endpoint
slices
that
those
are
the
two
options,
so
not
every
endpoint
slice
was
completely
full
but
reasonably
good
distribution.
Here
now
you
have
to
you
have
to
wonder
at
this
kind
of
scale,
how
crazy
does
ip
tables
look
and
well?
The
answer
is
pretty
crazy.
Four
four
hundred
thousand
total
lines
and
over
a
hundred
thousand
of
those
lines
were
probabilities.
A
Now,
if
you're
familiar
with
how
cube
proxies
high
p
tables
implementation
works,
it
is
really
just
decreasing
probabilities.
So
the
very
first
item
in
this
list
would
have
a
100
000
chance
of
going
to
that
end
point
and
if
that
isn't
picked,
it
goes
one
further
down
the
list
and
there's
a
one
in
ninety
nine
thousand,
nine
hundred
and
ninety
nine
chance
that
it
will
go
to
that
end.
Point
all
the
way
down
the
list.
So
just
a
really
really
long
chain
of
probabilities.
A
A
A
Well,
I
sent
50
000
requests
across
the
cluster
to
this
service
and
they
averaged
a
883
second
millisecond
response
time,
so
not
exactly
blazing
fast,
but
it
did
work
and
I
specifically
used
minhan's
fork
of
hay
because
it
includes
this
really
cool
functionality
that
allows
you
to
see
a
response,
distribution
and
used
appropriately.
You
can
see
exactly
how
many
different
endpoints
unique
endpoints
respond
to
your
request,
and
so
in
this
case
my
fifty
thousand
requests
were
distributed
across
twelve
thousand
seven
hundred
pods.
A
A
A
We
had
a
883
millisecond
response
time
on
average
and
the
requests
were
distributed
to
12
700
pods
and
the
way
I
was
able
to
determine
that
was
with
minhan's
fork
of
hay,
which
made
it
very
straightforward
to
see
just
how
many
pods
we
were
able
to
reach
with
these
requests,
so
12
700
unique
pots.
So
this
is
not
exactly
blazing
fast,
but
it
did
work.
A
And
then
for
ipvs,
I
had
to
try
out
the
ipvs
proxy
remote
and
for
that
we
sent
another.
Fifty
thousand
requests
again
with
the
same
library
and
the
average
response.
Time
was
a
bit
slower,
but
the
requests
were
more
distributed
going
to
seventeen
thousand
pots,
so
again
continue
to
work,
but
a
bit
slower
with
ipps
in
this
specific
instance,
but
of
course,
you're
probably
curious.
I
know
I
said
more
than
a
hundred
thousand
beyond
a
hundred
thousand
endpoints,
so
this
all
continues
to
work
well
with
a
hundred
and
twenty
thousand
pots.
A
A
There
were
errors,
and
I
would
be
remiss
not
to
mention
so
first,
it's
worth
mentioning
that
the
api
server
would
timeout.
This
is
one
example
of
it
timing
out
on
an
endpoint
slice
watch,
but
there
are
other
examples
too,
where
it
would
timeout
and
eventually
a
connection
would
be
re-established
and
things
would
work.
But
these
errors
became
more
common.
The
further
I
pushed
the
limits
so
the
closer
I
got
to
120
000,
the
more
frequently
at
observe
errors
like
this
and
then
there's
kind
of
a
trickier
error.
A
Around
endpoint
slice
updates
the
way
kubernetes
controllers
work.
Is
they
use
this
shared
cache
of
resource?
This
dramatically
decreases
load
on
api
server,
but
it
does
mean
that
sometimes
a
controller
may
be
operating
on
resources
that
are
not
completely
up
to
date
and
if,
for
example,
say
the
endpoint
slice
controller
chooses
to
try
to
update
a
resource
that
it
has
a
stale
copy
of
you'll
get
this
kind
of
error
message.
The
object
has
been
modified.
Please
apply
your
changes
and
try
again.
A
A
This
could
happen
and,
as
you
would
imagine,
the
more
and
more
endpoint
slices
you
get
for
service
the
higher
the
likelihood
of
these
events
happening
is
and
then,
finally,
as
I've
already
mentioned,
the
endpoints
controller
just
kept
on
running
into
that
lcd
object.
Size
limit,
we're
hoping
to
truncate
endpoints
in
the
future,
so
that
when
you
get
to
this
large
scale,
the
controller
stops
trying
at
a
certain
point,
but
right
now,
it'll
just
keep
on
trying
and
failing,
and
that's
just
unfortunately,
what
the
story
is
at
this
scale
right
now.
A
A
A
So
with
that
said,
I
want
to
highlight
a
couple
things:
ipvs
update
times
were
dramatically
faster
5
seconds
instead
of
25
or
29
seconds
with
ip
tables,
and
you
can
also
see
the
ip
tables
performance
hit
as
you
gradually
increase.
The
number
of
end
points
becomes
more
and
more
significant
at
scale,
whereas
ipvs
performance
is
relatively
flat,
regardless
of
scale.
A
So
that's
great:
it
works.
It
works
well
reasonably
well
at
high
scale,
but
what's
next,
let's
talk
about
the
really
cool
features
that
endpoint
slices
are
going
to
enable
we
have
some
really
exciting
goals
for
kubernetes
1.20.
I
realize
there's
a
chance.
Not
all
of
these
are
going
to
make
it
in
kubernetes
1.20,
but
at
this
point
we're
hopeful
so
we're
trying
to
work
towards
a
plan
for
automatic
topology,
aware
routing
and
endpoint
slice
subsetting.
A
Now,
if
you
take
this
idea
and
add
this
next
idea,
you
have
significant
ability
to
save
costs
and
decrease
api
server
load
with
subsetting.
The
idea
is
what,
if
not
just
topology
or
where
routing,
but
what,
if
coup
proxy,
could
just
select
the
endpoint
slices
that
were
relevant
to
it,
so
the
ones
that
represent
the
closest
endpoints
to
it.
A
A
Hey
thanks
for
being
part
of
our
talk.
We
have
time
to
answer
a
couple
questions
that
came
in
thanks
for
all
the
questions.
Again,
I
think
you'll
see
in
chat
we're
also
going
to
be
on
the
kubernetes
networking
channel
and
cncf
slack.
If
you
have
additional
follow-up
questions
that
we
can't
get
to
here.
A
What
we
can
say
is
the
endpoints
api
is
no
longer
the
bottleneck
with
endpoint
slices.
We've
solved
one
of
the
bottlenecks
here,
but
as
this
talk
kind
of
covered,
there
are
other
bottlenecks
that
we'll
run
into
as
an
example.
Current
implementations
of
coup
proxy
were
really
not
intended
to
reach
the
scale
of
a
million
endpoints
and
also
there's
some
some
other
load
say
api
server
that
may
need
some
addressing.
So
those
are
some
high
level
things
that
I
know
we
need
to
address.
A
B
B
B
It
really
depends
on
how
your
deployment
model
like
we,
we
take
these
a
lot
of
questions
similar
to
this
before
so,
basically,
why
do
you
need
to
run
like
one
million
endpoints
behind
a
service
in
the
first
place
right,
I?
Why
can't
you
chop
it
up
so
like,
for
instance,
one
one
aspect
is
to
one
aspect
to
consider
is
like
how
to
deploy.
B
So
so
the
answer
is
probably:
it
depends
based
on
your
requirements
of
your
reliability
and
how
agile
you
want
and
then
and
then
there
are
some
also
some
other
bottlenecks
on
the
deployment
controller
itself,
that
it
can't
support,
let's
say
a
million
pods
behind
the
deployment.
If
you
scale
a
deployment
to
a
million
pods,
you
will
basically
keep
on
creating
parts
in
the
loop
and
take
for
take
a
while
to
actually
realize
those
ten
like
a
million
pods,
so
yep.
So
sorry,
we
are
at
time.