►
From YouTube: Latest Kubernetes Scalability Improvements - Shyam Jeedigunta, Amazon & Yassine Tijani, VMware
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Latest Kubernetes Scalability Improvements - Shyam Jeedigunta, Amazon & Yassine Tijani, VMware
As the kubernetes project evolved, it started to increasingly gain adoption by enterprise and large scale users. Kubernetes, with a series of performance and scalability improvements, had come to support high scale an various fronts. However, a more recent onset of large-cluster use-cases highlighted some newer scalability limitations when adopting the “one big cluster” pattern. The project has made impressive progress in the last few releases on those. As scalability is a cross-cutting effort across different verticals, we would like to use this opportunity to walk you through the recent scalability/performance improvements made to kubernetes in different areas and introduce the new APIs and patterns that led to these. We will then discuss key elements that Kubernetes needs to address in order to improve adoption pace of its large-scale users.
https://sched.co/MPce
A
Hi
everyone
thanks
for
joining
us
on
this
talk
about
latest
scalability
improvements
that
have
been
happening
with
kubernetes
I'm
Sean
I
am
one
of
the
developers
on
the
kubernetes
project.
I've
been
working
on
critters
for
a
little
more
than
two
and
a
half
years
and
I'm,
one
of
the
chairs
for
six
scalability.
A
A
Until
that
point,
we
were
mostly
looking
at
scalability,
as
mainly
about
the
size
of
the
cluster
or
the
number
of
nodes
in
the
cluster,
but
scalability
is
really
much
broader
than
that,
because
there
are
so
many
dimensions.
As
you
can
see
on
on
that
picture,
there
are
so
many
dimensions
that
Cuba
netis
has
and
you
can
stress
along
each
of
those.
So
you
you
have
number
of
nodes.
You
have
number
of
pods
of
pods
on
Pol,
Pot
density,
number
of
services
and
and
a
lot
of
these
things
so
Cuba.
A
So
scalability
is
really
a
multi-dimensional
problem
and
we
realize
that,
with
the
bottlenecks
that
we
came
to
know
from
a
lot
of
users,
we
realize
that
there
are
some
improvements
that
we
need
to
make
in
different
areas
and
we
ended
up
improving
a
lot
of
these.
So
I'd
go
ahead
and
we'll
go
and
look
at
some
of
these
new
and
exciting
improvements
that
have
happened
in
the
last
few
months.
A
So
the
first
one
is
about
having
too
many
node
revisions
in
Cuba,
knitters,
cubelets,
post
periodic
heartbeats
to
the
control
plane
to
tell
the
control
plane
that
the
node
is
still
alive.
So
this
is
done
every
10
seconds.
So
this
means
that
the
cubelet
posts,
a
new
version
of
the
node,
object
once
every
10
seconds
and
each
of
these
objects.
A
Each
of
this
revision
that
is
posted
is
stored
as
one
revision
on
HCD
and
if
your
node
object
is
really
big,
probably
because
there
are
too
many
images
that
have
been
pulled
onto
the
node
or
there
are.
There
are
a
lot
of
volumes
that
have
been
attached
to
the
node,
which
can
make
the
node
object
really
big,
like
10,
KB
or
even
more.
A
Your
cluster
is
pretty
much
frozen
with
respect
to
rights,
because
it's
silly
can't
take
anymore
right
requests.
It
enters
a
read-only
model
because
of
this
problem.
There
are
a
few
dimensions
that
software,
it's
mainly
the
number
of
nodes
and
also
the
number
of
images
or
volumes
that
you
can
have
per
node.
So
so,
how
did
we
solve
this
problem?
We
started
out
with
a
simple
idea
of
splitting
away
heartbeats
from
the
node
objects,
because
conceptually
heart
beats
is
something
different
from
the
specification
of
the
node
like
the
spec
and
the
status
of
the
node.
A
A
So
even
if
you
have
many
revisions
of
this,
it's
going
to
be
okay
and
the
node
object
now
can
only
cubelet
can
only
make
updates
to
the
node
object
when
there's
actually
a
meaningful
change
in
the
status
no
in
the
node
status.
So
we
go
ahead
and
look
at
this
new
API,
but
just
a
little
small
node
on
the
feature
availability.
This
is
in
alpha
in
113
and
in
114.
This
is
in
beta,
then
the
node
heartbeat,
and
it
is
also
turned
on
by
default.
A
So
if
you're
running
your
114
cluster
expect
to
find
these
heartbeat
objects
in
your
clusters,
so
let
me
quickly
go
through
this.
New
API
I
won't
go
into
too
much
details,
but
the
the
object
on
the
left
is
the
node
heartbeat.
The
blue
fields
represents
basically
whatever
the
things
that
you
should
be
focusing
on.
It's
a
pretty
small
object.
A
A
So
there
are
some
short-term
stopgap
s--
to
kind
of
mitigate
this
issue.
To
some
extent,
we
introduced
caching
on
the
client
side
for
these
secrets
and
config
maps,
and
we
reduce
the
pole
frequency
for
larger
clusters
to
like
reduce
the
effect
of
these
large
number
of
calls,
but
they're
not
really
solving
the
problem
from
the
root
and
they're
more
like
just
band-aids
to
the
problem.
So
so,
because
of
this,
the
dimensions
that
suffer
are
mainly
the
number
of
nodes
and
the
number
of
secrets
and
config
maps
that
are
needed
by
your
nodes.
A
A
A
So
the
bug
itself
is
that
golang,
110
and
111
made
a
breaking
change
with
the
HTTP
client
library
that
when,
when
all
the
streams
of
your
TCP
connection
are
exhausted,
the
client
does
not
automatically
start
a
new
TCP
connection
and
in
Kuban
it
is.
We
multiplex,
watches
on
to
a
single
TCP
connection,
and
each
watch
uses
one
stream.
So
as
soon
as
like
there
are
250
secrets
or
config
Maps,
you
have
250
watches
and
that
that
eats
away
your
TCP
connection
in
the
node
can
then
go
dead.
This
was
fixed.
A
A
It
is
to
program
the
networking
layer
on
the
nodes
for
service
routing
and
load
bar
and
sing
within
the
cluster
and
IP
Devils
has
been
around
for
a
long
time
with
the
Linux
kernel,
which
of
course
comes
with
its
own
advantages
that
it's
a
really
mature
tool
and
it
doesn't
have
many
bugs-
and
it
is
quite
feature-rich,
but
as
come
with
the
advantages.
There
are
also
some
disadvantages
of
legacy
systems
and
that
it
was
not
very
well
written
with
respect
to
performance.
A
A
So
when
their
large
number
of
services,
this
chain
becomes
really
big,
and
this
affects
both
your
IP
tables
data,
plane
operations
and
control,
plane
operations,
so
data
plane
operations
is
basically
routing
of
the
packets
which
is
done
by
evaluating
those
rules
and
because
it's
a
linear
list,
you
have
to
go
through
the
whole
list,
so
it
can
be
really
slow
for
to
to
route
the
packets
and
the
control
plane.
Operations
are
changing.
These
rules
themselves,
which
is
updating,
IP
tables
and
IP
tables,
doesn't
allow
you
to
incrementally
add
new
rules.
A
So
what
you
need
to
do
is
you
need
to
like
get
the
current
chain
and
add
a
new
rule
and
just
post
this
whole
thing,
which
can
take
up
to
a
really
long
time
and
it
can
take
up
to
like
30
seconds
if
you
have
many
services
like
of
the
order
of
thousands
of
services,
so
because
of
this,
some
of
the
dimensions
that
suffer
of
the
number
of
services
and
the
size
of
services
like
how
many
pods
you
can
have
in
a
service.
So
how
did
we
solve
this?
A
Just
yet
because,
partly
because
the
community
is
still
on
its
way
to
decide
and
make
up
its
mind
on
if
this
is
actually
the
route
we
want
to
take
down
the
lane.
If
this
is
the
long
term
path,
we
want
to
choose,
and
also
because
IP
vs,
even
though
it
is
just
like
GA
in
111,
it's
it's
it's
something
that
is
still
kind
of
like
considered
in
the
experimental
mode,
because
not
many
providers
have
like
yet
started
supporting
it
in
production
and
also
it's
a
relatively
newer
thing
is
compared
to
iptables.
A
So
it's
still
like
going
through
the
bug,
stabilization
phase,
alright.
So
so
what
is
IP
vs
IP
V
s
is
an
alternative
to
IP
tables,
especially
for
the
service
load
balancing
part.
It's
mainly
a
it's
mainly
written
keeping
load
balancing
part
in
mind,
so
it
also
holds
the
holds
rules,
rules
for
packet
routing,
but
it
stores
them
in
a
different
way
as
compared
to
IP
tables.
It
uses
better
data
structures
which
is
it
uses.
A
Hash
tables
and
hash
tables
are
much
more
efficient
as
compared
to
linear
list,
so
so,
basically
that
that
allows
for
much
more
efficient
operations
with
I
with
data
plane
and
control
plane
operations
that
I
was
talking
about
earlier,
and
it
gives
really
high
scale
with
respect
to
services.
It's
still
missing
some
functionality
and
still
a
way
for
it
to
mature.
But
this
is
one
really
good
option
for
us
right
now.
All.
B
Right,
so
one
of
the
issues
that
we
encountered
is
also
scheduled
in
performance.
Some
workload
really
needs
some
high
scaling
performance
and
on
large
clusters,
scheduling
throughput
is
usually
around
90
pot
per
second
on
a
2,000
clone
cluster
and
30
pots
per
second
on
a
$5,000
sir,
when
we
use
the
affinity
feature,
it
drops
down
to
5
parts
per
minute
on
a
5,000
Kuster,
so
here
the
dimensions
that
are
suffering
our
pattern
and
the
number
of
nodes.
B
So
how
did
we
solve
this?
First
of
all,
one
of
the
optimization
that
we've
done
is
we
score
only
a
subset
of
note,
meaning
that
we
have
the
notes
of
our
cluster.
We
start
filtering
the
notes
using
predicates,
and
once
we
found
a
subset
of
notes,
we
start
only
scour
a
scoring
priority
function
only
against
this
subset,
so
it
makes
it
cheaper.
Also,
we
improved
the
affinity
computing
by
split
it
in
splitting
it
in
two
phases.
B
First,
one
is
that
we
find
all
the
parts
that
are
matching
the
affinity
term
and
then
on
all
the
subsets
of
pods,
the
art
that
are
matching
the
affinity
terms.
We
start
filtering
by
topology
key,
so
this
actually
also
make
it
cheaper
and
for
the
pod
scheduling
latency
by
profiling,
the
scheduler
we
found
on
one
scheduling
cycle
the
scheduler
spends
most
of
its
its
time
snapshot
in
the
scatterers
cache,
so
it
can
get
a
consistent
view
of
the
cluster
state.
B
So
one
of
the
optimizations
that
we've
done
is
we
change
the
data
structure
of
the
schedulers
cache
to
make
the
snapshot
an
even
faster
as
for
feature
availability,
the
scoring
and
the
affinity
are
available
since
112
and
the
cache
snapshotting
is
available
since
114
okay.
So
another
issue
also
is
the
way
we
handle
events
in
kubernetes
we've
encountered
multiple
scalability
issues
with
events
either
they
fill
the
HDD
database,
our
also
overloading
the
API
server
by
making
requests
my
clients
making
requests
to
create
events.
B
Also,
we
had
a
very
poor
user
experience
to
query
events,
use
tools
such
as
monitoring,
tracing
and
automation,
to
use
these
events-
and
there
are
some
dimension
that
we
wanted
to
improve,
which
are
the
number
of
the
API
calls.
The
number
of
note
the
arm,
for
example,
emitting
events
and
pot
chart
the
way
we
solve
this
is
basically,
we
introduced
a
new
events
API
and
with
this
new
events
API,
we
introduced
a
new
deduplication
logic.
This
logic
introduces
a
concept
which
is
called
zoomorphic
events.
B
Two
events
are:
is
amorphic
if
they
have
the
same
subset
of
fields,
for
example
the
reporting
controller,
the
reporting
instance,
the
type
of
the
event,
the
action
that
was
taken
and
so
on.
We
also
try
to
avoid
aggregation
by
you.
Creating
a
new
event
object
on
top
of
all
the
events
that
create
that
we've
created
to
say,
hey,
I,
just
created
all
the
events.
B
B
When
we
will
try
to
back
off
and
try
to
recall
the
image,
we're
gonna
check
if
the
event
is
amorphic
and
if
it's
the
same
event
we're
gonna
start
what
we
call
an
event
theory,
which
is
basically,
we
increment
account
on
the
non
the
object
of
the
event.
We
write
it
to
the
client-side
cache
and
then
we,
we
patch
the
API
server
to
increment
account
and
whenever
we
try
to
back
off,
we
check
if
this
is
an
isomorphic
event.
B
If
it
is,
we
write
it
to
the
client
cache,
but
we
do
not
make
extra
calls
to
the
API
server
and
when
we,
when
we
try
when
we
when
we
detect
that
the
series
has
ended-
and
we
do
not
see
any
event,
theory
anymore,
we
write
what
we
have
on
the
client
cache
to
the
API
server
to
reflect.
The
latest
count.
B
The
issue
here
is
that
whenever
we
restart
the
watch
connection
which
overloads
the
API
server,
for
example-
let's
say
we-
we
want
to
watch
for
a
number
of
of
pods
that
has
a
specific
label
and,
for
example,
10
pots
are
matching
of
100,
and
these
Jen
pots
are
not
changing,
but
the
other
pots
are
a
lot
changing.
So
this
list
gonna
have
what
we
call
a
resource
version
to
indicate
the
version
of
the
risk
pot
less
resource
and
since
the
10
pots
are
not
changing
and
the
others
are
changing
the
pot
list.
B
B
This
is
a
feature
that
needs
to
be
explicitly
obtained
by
the
clients,
so
we
don't
break
the
existing
clients.
So
when
you
establish
a
watch
connection
on
one
of
the
options
that
you
can
specify
is
a
low
watch
bookmarks,
the
initial
back
benchmark
shows
40
times
in
provement
and
decrease
in
wasteful
event
processing.
As
for
the
availability
of
this
feature,
it's
going
to
be
available
in
115
as
an
alpha.
B
So
let's
take
an
example.
We
we're
watching
for
a
subset
of
resource,
so
we
make
the
list
call
to
get
the
list
and
the
resource
version.
We
start
watching
against
the
API
server
with
the
resource
version
X
and
specifying
a
low
watch
bookmark
to
true
right
before
the
timeout
timeout
two
seconds.
The
API
server
gonna
send
a
watch
event,
a
new
watch
event,
type
called
bookmark
that
reflects
the
latest
resource
version.
A
A
So
currently,
the
endpoints
API
is
not
very
performant
for
the
way
it
has
been
designed
because
for
a
service
you
have
one
single
endpoint
object,
which
stores
a
list
of
all
the
eyepiece
of
the
pods
that
belong
to
that
service.
So
that
means
that,
even
if,
like
one
port
goes
up
or
comes
down,
you
have
to
update
this
object
and
repost.
This
whole
thing,
even
if
all
the
other
ip's
didn't
change.
A
So
this
this
becomes
kind
of
costly
as
you
go
to
bigger
clusters
and
as
you
have
bigger
services,
because
it's
not
just
about
updating
these
objects,
it's
also
about
then
publishing
these.
The
newer
version
of
these
endpoints
to
all
the
nodes,
because
queue
proxies
are
watching
for
endpoints
yeah.
So
the
next
one
is
to
allow
for
a
higher
pot
density.
So
port
density
is
the
number
of
pods
that
are
allowed
to
be
run
on.
A
One
would
currently
it's
limited
to
110
by
default
and
kubernetes,
and
officially
we
support
for
like
a
maximum
110
pods
per
node,
but
because
of
some
of
the
recent
improvements
with
docker
continuity
and
and
even
improvements
in
cubelet
itself
of
we
are
now
able
to
go
higher
and
there
are
a
bunch
of
folks
who
are
already
trying
like
200
to
300
parts
per
node.
So
we
might
soon
increase
this
default
limit,
or
at
least
like
go
go
higher
for
bigger
instance
types.
A
The
next
one
is
about
rethinking
or
current
affinity
and
anti
affinity
logic
in
scheduler,
because
this
is
started
to
hunters
and
large
clusters,
because
this
this
has
become
a
bottleneck
for
our
scheduler
throughput
in
large
clusters,
because
the
way
we
are
computing
affinity
and
anti
affinity-
currently
we
are,
we
are
considering
the
state
of
the
whole
cluster
for
every
single
for
evaluating
the
validity
of
every
single
node.
For
the
part-
and
this
this
becomes
a
very
computationally,
expensive,
so
yeah.
So
we're
still
thinking
about
ideas
of
what
we
can
do
here
to
to
improve.
A
This
may
be,
there
might
be
some
significant
changes
and,
in
the
feature
itself
something
to
think
through
and
the
last
one
we
have
your
is
watching
for
arbitrary
fields.
This
is
again
something
far
term
where
we
still
again
thinking
about
this-
we're
not
yet
hundred
percent
convinced
if
we
want
to
do
this,
but
this
is
about
watching
for
particular
fields
of
an
object
instead
of
watching
for
the
whole
object.
A
Consequently,
your
controllers,
because
you
have
bigger
objects
to
process
and
more
memory
allocations
to
make
yeah
and
I,
put
those
three
dots
there,
because
maybe
there
is
something
that
someone
among
you
have
ideas
about,
and
you
might
want
to.
Let
us
know,
because
this
is
a
good
opportunity
and
thank
you.
A
C
D
We
are
basically
not
focusing
that
much
on
memory.
It's
not
still
that
not
not
one
of
the
things.
That
is
it's.
What
most
users
are
affecting
out,
though,
like
we've
go,
112
that
we
started
using
in
114
go
112
is
way
way
better
with
handling
memory,
like
the
drop
that
we've
seen
in
our
like
continuous
testing
is
something
like
between
2
&,
4
X.
So
once
you
get
go
to
like
get
to
this
version,
you
will
see
significant
improvement.
E
So
I
had
a
question
about
the
numbers
you
showed
four
before
the
fixes
for
pod
scheduling
on
five
thousand
out
clusters,
especially
with
the
anti
affinity
feature,
so
you
said
something
like
five
pods
per
minute.
That
sounds
like
very,
very
slow
I'm
wondering
if
you
could
expand
a
bit
on
what
was
happening
to
cause
such
a
slow
processing.
Speed.
For
you
know,
5,000
elements
in
the
list
doesn't
sound
like
that
much
so
what
sort
of
thing
was
happening?
That
was,
if
you
can
I,
don't
know,
maybe
it's
too
large
to
explain
that
so.
A
So
that
throughput
of
five
boards,
less
than
five
parts
per
minute,
is
the
throughput
when
these
pods
are
using
anti
affinity
feature.
So
also.
This
is
like
before
some
of
the
improvements
were
made,
but
the
reason
why
it
was
so
slow
is
because
the
the
scheduler
predicated
and
priority,
which
actually
evaluate
like
which
took
evaluate
this
code
for
the
node,
the
the
way
they
are
written,
is
not
very
efficient
because
how
it
works
is
like.
Let's
say
you
have
you'd
if
you
define
a
deployment
of
pods
with
the
anti
affinity
ad
at
the
node
level.
D
Basically,
the
main
problem
here
is
that
an
TF,
importante
affinity
is
like
symmetric
here.
So
when
you
are
scheduling
a
pot
you,
we
are
not
only
looking
into
how
how
it
affects
other
pot
like
we
are
all.
We
are
also
looking
into
like
whoever
scheduling
our
pot
will
not
break
any
other
assumptions
of
any
other
pot
anti
affinities
that
are
already
scheduled
in
the
system.
D
So
if
you
are
scheduling
a
pot
that
in
a
cluster
that
has
let's
say
10,000
pots
already
running,
then
we
need
to
check
whether
my
pot
and
any
of
those
other
hundred
10,000
pots
aren't
breaking
the
assumption
for
any
of
the
existing
like
anti
affinity
of
anything
else.
So
that's
kind
of
like
more
than
quadratic,
or
something
like
that
that
is
like
has
significantly
changed
like
when
I
was
last
looking
into
the
numbers,
I
think
in
113
we
are
somewhere
around.
D
Between
10
and
20
pots
per
second
already
with
ante
affinity
and
there's
what
there
is.
There
are
a
couple
more
things
coming
around
15,
so
we
are
getting
to
at
least
the
same
order
of
magnitude
with
ante
affinity,
and
there
are
like
couple
plans
to
not
support
ante
affinity
on
any
arbitrary
topology,
also
which
I
was
suggesting
for
quite
a
long
time.
It
seems
that
it's
currently
getting
some
traction
so
so
yeah.