►
From YouTube: Kubernetes Scalability: A Multi-Dimensional Analysis - Shyam Jeedigunta & Maciek Różacki, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes Scalability: A Multi-Dimensional Analysis - Shyam Jeedigunta & Maciek Różacki, Google
‘How much do our clusters scale?’ has been of interest to the developer and user communities of kubernetes for a long time. This is super important if you want to better understand how to architect your workloads by understand where are the bottlenecks of the system. We will begin this talk from formulating a ‘scalability envelope’ across attributes and features of scalability dimensions. We will then illustrate limits that are currently known to us through scale-testing experiments and user experiences.
To learn more: https://sched.co/GrXy
A
B
A
As
some
folks
are
grabbing
some
seats,
a
quick,
maybe
demographics,
check
who,
in
the
room
is
a
user
of
kubernetes
and
you're
running
your
own
clusters.
Okay,
so
looks
like
half
at
least
and
who,
among
you,
is
running
clusters
of
1,000
nodes
or
more
okay
and
2,000.
Anyone
are
just
two
hands
so
3,000
for
now.
Okay,
anyone
among
those
of
you
who
did
not
raise
your
hands
is
anyone
considering
it
or
planning
that
you
run
such
large
workloads
in
the
future.
A
That's
actually
more
than
I
expected.
Oh,
if
you
want
it
to
be
great,
if
you
could
also
network
a
bit
then
after
the
session,
but
now
it's
maybe
then
moved
to
two
what
we
prepared
to
do
today
for
you
of
sham,
so
we
will
be
talking
about
scalability
of
kubernetes
and
we
will
cover
how
you
should
think
about
it
and
what
are
the
current
limits
and
then
what
does
it
actually
mean
for
you?
So,
first,
maybe
with
a
little
bit
of
background,
why
why
we
prepared
this?
Why
we
prepared
this
material?
A
So
we
are
getting
quite
a
lot
of
questions
from
the
community
from
developers
from
users
of
kubernetes
on
what
does
it
actually
mean
that
kubernetes
scales
was
the
definition
of
scalability
in
the
context
of
this
product?
What
are
the
limits?
Why,
even
though
I'm
running
our
a
cluster
that
is
smaller
than
5,000
nodes,
I'm
still
hitting
scalability
limits
and
why
workloads
that
are
very
similar
to
mine
are
not
actually
part
of
testing
of
of
from
scallop
scalability
perspective
and
to
respond
to
these
questions,
and
also
many
other
that
we
had.
A
So
we
documented
the
outcomes
of
this
study.
We
consolidated
it.
We've
also
some
experience
and
and
the
work
that
we
did
before
and
with
kubernetes
users
and
with
development
of
the
product.
And
through
this
presentation
we
would
like
to
explain
to
you
how
how
you
should
model
scalable,
scalable,
kubernetes
configurations
from
scalability
perspective,
how
to
think
about
it.
How
to
plan
your
architecture?
A
So
maybe
first,
let's
talk:
what
does
it
actually
mean
that
kubernetes
scales
and
how
to
define
it
so
first
important
misconception.
Is
that
it's
a
single
number?
So
it's
not
we
very
frequently
in
our
materials
and
in
the
conversations
we're
using
the
number
of
nodes
as
a
proxy
or
a
simplification
of
the
scale
and
complexity
of
your
cluster
and
in
many
cases,
actually
does
good
enough.
A
So
a
better
way
to
think
about
scalability
of
kubernetes
is
that
within
all
possible
configurations
of
your
cluster,
there
is
a
cube
which
wraps
an
envelope
of
those
configurations
which
will
offer
you
good
stability
and
good
performance,
and
as
long
as
you
are
within
that
enveloped,
within
the
limits
of
scalability
on
on
multiple
dimensions,
you
are
safe.
You
can
say
that
you
are
within
the
scalability
limits
and
you
will
not
run
into
any
trouble
neither
from
performance
perspective,
not
from
stability
of
the
cluster.
A
One
may
be
note
the
example,
the
dimensions
that
we're
showing
here
on
the
slide.
These
are
examples.
It's
not
an
exhaustive
list
in
the
next
slides,
we'll
talk
a
bit
more
about
how
to
think
about
them
and
which
ones
are
most
relevant
that
you
should
consider
when
architecting
or
developing
features.
A
Yeah
so
first,
let's
spend
a
couple
of
minutes
talking
what
are
the
properties
of
this
envelope
and
of
this
cube
that
defines
scalability
limits
of
kubernetes?
So,
first,
it's
not
actually
a
cube.
By
that
we
mean
that
the
dimensions
that
describe
your
configuration-
they
are
not
independent
and
you
may
run
into
a
situation
that
if,
on
one
dimension,
you
have
a
limit
a
on
another
dimension,
you
have
a
limit
B
and
you
max
out
on
both
of
those
dimensions
with
your
configuration.
A
Then
it's
probable
that
your
cluster
will
be
already
outside
of
the
scalability
envelope
and
you
will,
you
might
run
them
into
trouble
even
more.
The
shape
of
the
envelope
is
not
convex,
so
taking
a
more
conservative
approach
and
going
for
averages
of
a
and
B
so
that
you
stick
somewhat
in
the
middle
on.
Both
of
these
dimensions
still
might
take
you
outside
of
outside
of
the
envelope,
and
you
will
breach
the
scalability
limits
of
of
kubernetes
next
property.
B
B
So
we
go
ahead
and
look
at
some
of
the
small
envelopes,
especially
like
for
Cuba
notice,
but
I
don't
make
a
few
things
clear
before
we
actually
look
into
those
limits.
The
first
thing
is
that
we
are
going
to
do.
The
limits
are
we're
going
to
talk
about
here
are
purely
specific
to
Cuban
interest
control
plane.
So
these
do
not
have
anything
to
do
with
the
cloud
provider.
These
come
basically
out
of
the
kubernetes
design.
B
There
may
be
some
limits
that
come
from
the
underlying
infrastructure
of
your
cluster,
but
here
we
are
going
to
only
talk
about
the
kubernetes
plane
limits
and
these
limits
don't
form
an
exhaustive
list.
These
are
just
the
ones
we've
discovered
so
far,
so
this
will
likely
change
over
time
and
also
they
they.
They
basically
form
a
rough
sketch
of
what
is
our
idea
of
safe
configurations,
and
this
is
based
on
primarily
scalability
tests
that
six
scalability
has
been
doing
in
some
of
the
user
cases.
B
In
practice,
you
may
be
able
to
actually
push
beyond
some
of
these
limits.
We
try
to
be
conservative
at
a
few
places
and
we're
making
recommendations
based
on
based
on
things
that
you've
tested
on
upstream
officially,
but
you
may
be
actually
be
able
to
push
beyond
these
limits,
and
even
it's
possible
that,
within
these
limits,
by
doing
some
really
peculiar
things
or
some
different
kinds
of
configurations,
you
can
screw
up
so
in
general,
like
use
discretion,
and
if
you
have
any
questions
like
please
feel
free
to
reach
out
to
us.
B
Six
scalability
is
very
happy
to
answer
such
questions,
so
the
next
limit
or
the
first
limit,
is
actually
the
number
of
nodes
versus
the
number
of
parts
that
you
can
put
on
a
node
which
a
lot
of
you
may
also
call
a
spot
density,
and
you
see
on
the
screen
graph
and
there
is
a
green
shaded
region
which
basically
represents
our
scalability
envelope
and
it's
bounded
by
three
limits.
There
is
a
limit
on
the
x-axis.
B
There
is
a
limit
on
the
y-axis
and
there
is
a
limit
which
is
a
function
of
the
X
and
the
y-axis
which
in
this
case,
is
the
curve
that
you
can
see.
Now.
Let's
talk
about
each
of
these
limits
individually,
so
the
first
one
is
the
limit
on
the
x-axis
of
five
thousand.
Third,
you
can
see
for
the
number
of
nodes,
and
this
is
the
officially
tested
limit,
and
this
is
what
we
test
for
in
upstream
kubernetes
for
every
release
as
part
of
six
scalability
validation
process
that
we
do
so.
B
B
Now
the
limit
on
the
y-axis,
which
is
the
number
of
pods
per
node,
is
110.
This
again
is
an
official
limit
and
something
with
scale
test
for
also
by
default
in
Cuban,
it
is
cubelets,
are
provided
with
a
max
pods
argument
of
110,
which
doesn't
let
you
start
more
than
these.
If
you
try
it
to
go
further
beyond
this,
you
can
run
more
number
of
pods
on
your
node.
Then
the
load
on
your
cubelet
and
docker
is
going
to
increase
and
their
responsiveness
can
fall
down.
B
B
So
here
we're
going
to
talk
about
cluster
IP
services
and
I'm
not
going
to
step
into
the
domain
of
cloud
provider
based
load
balancers,
because
that
is
like
I,
said
earlier:
something
cloud
provider
dependent
and
can
change
from
provider
to
provider
some
feeling
holistic
to
the
internal
service
load
balancing
mechanism
in
kubernetes.
Now
you
can
see
that
there's
a
limit
of
10000
services
on
the
x-axis.
This
is
again
something
we've
tested
for
in
in
fix
scalability
as
part
of
our
upstream
Kuban.
It
is
scale
testing.
This
limit
mainly
arises
because
of
IP
tables.
B
Whenever
you
add
a
new
service,
when
you
create
a
new
service,
you
add
a
rule
to
the
cube
SVC
iptables
chain,
and
this
means
that
when
you
have
too
many
services
the
size
of
this
chain
increases
which
affects
your
iptables
performance
in
two
ways.
The
first
is
basically
the
performance
of
iptables
operations
themselves,
which
is
IPTA,
will
save
and
restores
it
takes
more
time
to
do
those
operations
when
the
strain
increases
and
the
second
is
packet.
B
So
whenever
you
add
a
new
part
to
a
service
you're
going
to
recalculate
the
whole
endpoint
object
for
the
service,
add
a
new
or
new
IP
entry
to
this
object
and
then
post
it
over
to
all
your
queue
proxies.
So
this
really
creates
a
very
quadratic
kind
of
traffic.
So
it's
we
recommend
to
not
have
too
big
services,
because
then
your
control
plane
can
get
all
ordered
because
of
endpoints
traffic,
but.
B
Is
not
a
hard
limit?
You
can
try
to
push
further,
but
just
make
sure
that
if
you
are
trying
to
create
too
many
services,
which
are
big,
try
to
have
fewer
number
of
I
mean
when
you're
trying
to
create
services
that
are
bigger
than
this
try
to
have
fewer
number
of
those
services.
So
you
can
see
a
configuration
on
on
a
node
that
I
mentioned
on
the
right
side,
which
is
a
configuration
we
have
tested
for.
B
So
what
we
do
is
we
test
with
a
really
large
number
of
backends,
but
we
have
majority
of
those
backends
as
part
of
small
services,
a
few
of
them
which
are
part
of
medium
services
and
a
very
few
of
them,
which
are
part
of
large
services.
So
this
is
something
which
kind
of
works
well
in
practice,
so
yeah,
that's
basically,
okay,
sure.
B
B
And
when
you
go
beyond
a
threshold
which
is
5,000,
the
part
starts
to
crash,
because
the
kernel
cannot
load
the
process
into
memory
because
there's
just
not
enough
space
in
the
buffer
for
the
process
which
holds
environment
variables.
So
you
want
to
make
sure
that
this
is
satisfied
and
if
I
am
NOT
wrong
like
in
Cuban,
it
is
released
1.13.
We
have
a
change
which
makes
it
possible
to
have
service
linked
environment
variables
optional.
B
Fortune,
this
is
something
which
has
been
asked
or
again
and
again
by
so
many
people,
and
so
many
users
are
interested
in
this
metric,
so
soap
or
churn
is
defined
as
the
total
or
the
number
of
pod
creates,
update
or
deletes
that
you
can
do
a
second
and
in
Cuban.
It
is
we
by
default
limit
this
to
20
per
second
and
the
reason
why
that
limit
arises
is
basically
a
limit
on
the
QPS
for
the
controller
manager.
B
So
the
controller
manager
by
default
cannot
make
more
than
20
calls
to
the
API
server
per
second,
which
basically
limits
your
controllers,
that
that
take
care
of
these
create
update
and
delete
operations.
But
there
are
a
few
caveats
based
on
which
you
can
kind
of
go
above
or
below
this
threshold.
I
just
mention
a
few
here.
So,
for
instance,
if
you're
going
to
create
parts
manually
which
I
don't
think
is
a
very
common
use
case.
B
But
if
you're
going
to
do
that,
then
you
basically
bypass
the
controller
manager,
and
you
can
have
a
very
high
rate
of
pod
creations
and
for
deletions.
If
you're
going
to
depend
on
the
Kuban,
it
is
garbage
collector
to
delete
your
orphan
parts,
then
you
actually
don't
have
even
a
lower
limit
of
ten
per
second,
because
the
way
the
garbage
collector
works
right
now
in
cuban
itis
is
it
makes
two
API
calls
for
deleting
a
single
object.
B
So
that
basically
means
you
can
only
achieve
a
throughput
of
ten
per
second
for
deletes
and
in
general,
when,
like
I
said
earlier,
if
you
have
pods
that
are
part
of
really
big
services,
then
pod
shown
can
create
a
lot
of
endpoints
traffic.
Because
of
this
massive
endpoints
objects,
and
that
can
overload
your
control
plane
and,
as
a
result,
you
may
not
actually
be
able
to
achieve
the
churn
of
20
per
second
safely.
B
Yeah,
so
this
limit
is
about
the
number
of
nodes
versus
the
number
of
conflicts
you
can
have
per
node.
So
the
number
of
conflicts
that
per
node
is
basically
the
total
number
of
secrets
and
config
maps
that
a
node
needs
in
order
to
be
able
to
start
its
bottles.
So
the
way
it
works
until
release
1.11
is
cubelet,
periodically
goes
and
fetches
these
conflicts
and
secret
Maps
yeah.
So
so
here
the
limit
on
the
x-axis
of
5,000
nodes
or
something
already
talked
about
before
the
limit
on
the
y-axis
is
interesting.
B
So
we
say
that
the
number
of
conflicts
per
node
can
be
more
than
200,
and
this
limit
is
basically
because
of
cubelets
api
QPS
limitation
and
cubelet
by
default
is
configured
to
not
have
more
than
this
QPS.
So
it
makes
it
hard
for
it
to
refresh
periodically
the
secrets
and
config
maps
because
it
has
to
mean
get
calls
to
the
API
server.
B
The
the
curve
here
represents
the
product
of
both,
which
basically
is
the
aggregate
number
of
conflicts
that
are
needed
across
all
your
nodes,
and
this
we
limit
to
150,000
the
reason
being
that
just
now
the
thing
that
I
told
you
that
cubelet
needs
to
make
periodic
calls
and
the
bulk
of
these
get
secrets
and
concept
map
calls
in
your
cluster
becomes
so
big
that
your
or
that
your
master
can
get
overloaded.
Because
of
this.
So
luckily
the
good
news
is:
this
limitation
does
not
exist
anymore,
starting
from
release
1.12.
B
B
B
Is
to
turn
off
the
default
service
account
token
mounting
for
your
pod.
This
is
basically
the
default
in
cuban.
It
is
that
the
pod
is
mounted
with
the
secret
corresponding
to
the
namespaces
identity,
so
that
it's
able
to
talk
to
the
api
server,
which
basically
adds
one
extra
secret
for
a
part,
and
it
basically
affects
the
cement.
So,
if
possible,
you
don't
if
your
parts
don't
need
namespace,
spaced
identity
or
they
don't
need
to
talk
to
the
API
server,
then
consider
turning
this
off
totally.
B
So
the
question
is:
what
is
the
limit
for
the
cubelet
QPS
in
1.11
or
before
I?
Believe
the
limit
is
200
or
actually,
you
know,
I
think
it's
5,
which
is
per
second,
and
this
basically
shows
per
midnight.
Refresh
rate
I.
Think
it's
five
but
like
I
need
to
check
that
I'm
100%
sure
looks
like
so.
This
limit
is
across
the
number
of
namespaces
versus
the
number
of
parts
you
can
have
power
namespace.
The.
B
It
is
not
a
hard
limit,
it
is
something
we've
empirically
observed
in
our
scalability
tests
that,
as
you
have
more
and
more
number
of
pods
in
your
namespace,
the
responsiveness
of
your
controllers
can
go
down,
because
a
lot
of
reconciliation,
loops
in
controllers
depend
go
through
all
the
parts
of
a
particular
namespace
which
can
kind
of
make
things
a
bit
slower.
So
this
is
a
recommendation.
B
You
probably
can
go
higher
or
you
like,
he
can
go
higher
than
this,
but
if
possible,
try
to
break
your
pods
into
multiple
namespaces
yeah
and
the
curve
which
is
the
product
of
both,
which
is
the
total
number
of
points,
is
limited
250,000.
This
is
what
we
talked
about
earlier
already,
let's
go
to
the
next
limit.
Oh
that's
awesome!
So
with
so
one
thing,
I
want
to
say
is
like
with
with
the
knowledge
of
with
a
better
perspective,
of
what
scalability
really
is
and
with
the
knowledge
of
these
limits.
B
A
Just
to
add
to
what
Shyam
said
a
couple
of
remarks
in
terms
of
workouts,
for
you
one
is
that
if
you
are
kubernetes,
developer
and
you're
building
feature
features
and
improvements,
please
make
sure
that
you
test
them
from
scalability
perspective
on
six
site
on
github,
you
can
find
instructions
in
the
whole
process,
how
to
go
about
it
and
how
to
test
and
which
tests
to
use.
Well.
A
Scalability
is
a
horizontal
concept,
so
it's
not
in
the
hands
of
a
single
special
interest
group
to
make
sure
that
kubernetes
scales
it
is
in
the
it.
It
is
a
topic
for
all
of
the
teams
working
on
improvements,
both
to
make
sure
that
you
maintain
the
existing
scalability
and
that
we
push
the
limits
further.
A
Secondly,
if
you
are
covering
this
user,
please
make
sure
that
you
account
for
its
scalability
tests
in
your
projects
when
you're
architecting
your
infrastructure
and
your
systems.
This
is
because
you
may
have
certain
specific
aspects
of
your
configuration
that
will
change
the
shape
of
this
envelope.
There
might
be
topics
associated
with
your
specific
infrastructure
and
networking
security
and
other
that
may
be
affecting
your
the
scalability
limits.
That
would
apply
to
your
specific
system,
so
please
make
sure
that
you
test
it
and
further.
A
Is
that
please
stay
in
touch
with
us
with
the
whole
team
or
human
scalability.
We
need
both
engaged
developers
who
are
building
features.
Who
would
collaborate
with
six
scalability
to
make
sure
that
limits
are
maintained
and
expanded
and
maybe
even
more
importantly,
we
need
users
who
want
to
run
workloads
which
are
beyond
the
current
scale
ability
envelope
who
would
be
interested
in
collaborating
with
us
in
pushing
those
limits.
So
on
this
slide,
we
put
key
contact
channels
to
get
in
touch
of
us.
A
A
B
Someone
had
a
question
about
IP
vs,
oh
yeah,
so
yes,
we
know
ipbs
performs
better
than
IP
tables
because
it
has
a
better
implementation
of
these
rules.
I
think
it's
based
on
hash
tables,
but
we
aren't
yet
tested
for
IP
V
s
for
scale
testing,
but
we
do
expect
it
to
show
a
lot
of
improvement
or
the
performance.
B
That's
a
very
good
question,
so
I
don't
have
too
much
expertise
in
storage,
but
I
can
definitely
connect
with
you.
With
someone
from
the
storage
team,
there
are
some
recommendations
that
I
know
of
storage,
which
are
like
kind
of
way
out
for
specific
limits
because,
like
some
not
on
the
committee's
level,
I'm
aware
of
none
but
like
I'll,
maybe
reach
out
to
you
with
someone
from
six
storage.
B
Okay,
so
the
question
is:
if
I
can
explain
the
formula
on
the
number
summation
of
the
number
of
conflicts
per
node,
and
the
answer
is
yes,
I
can
explain
and
it
is
it
is.
It
is
basically
the
sum
of
the
number
of
conflicts
that
each
node
needs
across
all
the
nodes,
so
node
a
needs.
Some
number
of
conflicts
right,
some
number
of
secrets
and
config
maps
for
for
the
pods
that
run
on
that
node.
Its
summation
of
this
across
all
your
nodes
in
the
cluster.
C
B
B
D
D
D
B
In
general,
they
get
six
scalability
were
like
really
happy
to
take
your
contributions
and
your
input.
If
you
have
had
some
experiences
with
some
fear,
scalability
tests
on
your
own
in
your
own
production
environment.
We
are
always
happy
to
hear
about
it,
and
indeed,
we
kind
of
seek
manpower
in
this
kink,
because
it's
kind
of
very
niche
right
now
and.
D
B
So
the
question
is:
are
there
some
tests
in
the
conformance
suite
that
someone
can
run
as
a
vendor
and
I'm,
assuming
it's
in
the
context
of
scalability
tests?
So
the
answer
is
right.
Now
we
don't-
and
this
is
a
discussion
we've
been
having
in
the
past
with
a
bunch
of
people
like
if
we
should
start
involving
scalability
test
as
part
of
the
conformance
suite.
This
is
an
ongoing
discussion.
D
B
Yes,
oh
yeah,
thanks
Bob,
so
q
mark,
like
like
I,
was
saying
earlier.
It's
kind
of
simulated
kubernetes
cluster
that
you
can.
You
can
basically
done
a
large
simulated
cluster
on
just
a
few
nodes
with
the
nodes
node
part
of
the
cluster
hollowed
out,
but
your
control
plan
being
real.
It's
something
that
you
can
test
even
locally.
B
B
For
instance,
the
pod
shown
that
I
was
talking
about
it's
a
it's
a
QPS
limit
on
controller,
so
you
can
try
to
change
the
default
of
the
controller,
but
you
have
to
be
careful
about
the
resources
your
masters
has
because
when
you
give
it
higher
QPS,
there
is
more
stuff,
that's
happening
within
the
master,
so
you
also
have
to
scale
it
vertically
and
there
are
other
things
like
the
number
of
nodes.
If
you
want
to
increase
it,
you
probably
it
makes
sense
to
vertically
scale
your
control
plane
but
yeah
in
general.
B
A
B
Okay,
so
just
to
repeat
the
question,
the
question
is:
if
we
have
some
kind
of
knowledge
or
have
we
tested
anything
with
scalability
of
cron
jobs?
No,
we
haven't
really
done
that
like
yeah,
but
I
am
not
surprised
by
the
thing
you
say
about
having
too
many
cron
jobs
can
affect
your
controllers
because
of
too
many
calls.
E
B
So
the
question
is:
if
there
are
a
lot
of
conflict
maps
that
have
become
stale
over
time
and
they
haven't
been
cleaned
up,
does
that
affect
performance?
The
answer
is:
if
they
are
stale
and
not
really
being
used
by
anyone,
they
just
lie
around
wasting
get
CD
space,
but
otherwise
they
they
shouldn't,
really
affect
performance.
B
So
there
is
a
resource
on
the
community
repository
under
six
scalability,
which
talks
about
the
recommended
configurations
for
our
scale
tests,
but
I
believe
these
particular
limits
have
been
tested
with
64
cores
and
of
something
more
than
100
gigs
of
memory
and
per
master
yeah.
So
I'm
talking
the
master
by
the
way.
Sorry,
so
the
this
testing
was
done
both
with
the
single
master
and
under
H,
a
which
is
three
postures.
Yes,.
B
D
B
So
these
are
for
worker
nodes
in
your
q-
cluster,
like
to
throw
some
light
on
this.
We
basically
take
a
really
small
node
and
run
pause
spots
on
them,
which,
like
Bob,
said
to
nothing
and
they
are
like
given
very
minimal
resource
requests,
so
likes
few
milli
cores
and
like
very
little
memory,
and
you
just
start
them
on
the
node.
This
is
to
test
the
control
plane
scalability
for
the
node,
which
is
Kuebler,
docker
and
other
stuff.
We
don't
actually
go
into
application
space,
which
is
like
what
exactly
are
those
containers
doing
and.
A
E
B
B
No
so
so
I
think,
strictly
speaking,
that
falls
under
the
purview
of
sig
auto-scaling,
which
basically
takes
care
of
like
dynamically
changing
the
resource
requests
of
your
your
user
space
pods
or
even
the
control
space
pods
based
on
usage.
We
basically
try
to
stress
everything
and
like
give
some
really
maximal
values
to
each
of
these
and
like
test,
how
I
mean
if
we
give
enough
resources?
How
much
can
we
push
on
each
of
these
fronts,
but.
B
So
to
know
about
what
are
the
things
that
you
need
to
push
to
be
able
to
support
higher
limits?
You
can
come
to
scalability
and
to
know
about
how
to
increase
dynamically
and
when
to
change
the
resource
requests
of
your
pods
or
what
matrix
to
auto
scale
on
I.
Think
auto
scaling
is
the
better
thing
to
reach
out
for.
B
So
yeah,
so
the
question
is:
why
does
endpoints
traffic
increase
quadratically
with
the
number
of
backends?
It's
basically
because
of
the
way
the
API
is
designed
right
now
we,
the
endpoints
object
for
a
service,
is
basically
I
mean
at
a
high
level,
a
list
of
all
the
IPS
of
pods
that
are
part
of
the
service
and
whenever
a
new
pod
comes
up,
you
basically
add
one
entry
and
update
this
object.
So
instead
of
maintaining
individual
endpoints
or
separate
objects,
you
update
this
whole
thing
and
patch.