►
From YouTube: Deep Dive: Scalability SIG - Shyam Jeedigunta, Amazon Web Services & Wojciech Tyczynski, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Scalability SIG - Shyam Jeedigunta, Amazon Web Services & Wojciech Tyczynski, Google
This session will focus on the vision and strategy for the scalability effort. We will touch upon work the Scalability Special Interest Group has been doing over the past year. This involves work done along various fronts like improvements to scale-testing tooling, introduction of newer and more user-focused SLOs and performance/scalability improvements to Kubernetes. Following that, we will discuss what’s next in our roadmap. We will leave time for Q&A and receive input from the audience that can help guide our priorities.
https://sched.co/MPkd
A
A
A
Okay,
so
first
some
small
background
any
work
that
we
do
at
six
scalability
kind
of
falls
into
at
a
high
level
one
of
these
categories,
so
we
are
responsible
for
defining
and
driving
the
scalability
definitions
and
goals
for
the
Cuban
Indus
project.
So
this
involves
building
an
understanding
of
what
scalability
is
and
building
an
understanding
of
what
metrics
we
want
to
look
look
at
and
what
are
the
long-term
and
short-term
goals
with
respect
to
scaling
better
the
next
cutting
the
next
one
is
monitoring
and
measuring
the
performance
aspects
of
the
system.
A
So
we
we
actually
continually
monitor
what
different
kinds
of
metrics
in
our
system
and
see
how
they
are
changing
over
time,
look
at
trends
and
patterns,
and
often
based
on
that
we
end
up
identifying
some
of
the
things
that
we
want
to
improve
with
respect
to
performance
and
scalability,
which
is
where
we
come
into
this
coordinating
and
contributing
part.
We
do
some
improvements
ourselves
and
some
some
of
the
improvements
we
basically
get
done
with
help
from
others.
Other
things
the
fourth
category.
A
This
is
the
other
side
of
the
scalability
story,
where
we,
we
are
continually
persevering
to
try
to
protect
the
project
from
scalability
regressions
entering,
and
this
happens
all
the
time
we
we
have
scalability
regressions
coming
in
and
we
are
like
actively
battling
those.
The
fine
was
the
final
one,
which
is
something
you're
always
doing
in
the
background
is
consulting
the
community
with
respective
scalability
and
performance
topics
and
providing
reviews,
and
just
educational
talks
and
blog
posts
and
stuff
like
that.
A
So
one
particular
thing
I
would
want
to
specifically
call
out
is:
please
don't
confuse
six
scalability
with
sync
auto-scaling.
Both
of
them
have
scale
in
common,
but
that's
pretty
much
everything
that's
in
common
and
scheme
six
scalability
is
about.
How
much
is
this?
How
much
is
the
system
able
to
scale
when
we
are
throwing
enough
resources
at
it?
So
we
are
really
talking
about
like
the
absolute
heart
bounds
for
us
and
auto-scaling
is
more
about
how
dynamically
the
system
is
adjusting
to
load
and
yeah.
B
Okay,
so
scalability
definition
and
girls.
This
is
a
topic
I'm
not
going
to
talk
that
much
today.
I
have
a
separate
presentation
tomorrow,
fully
about
this
topic,
so
I
will
focus
a
bit
more
on
here
on
just
like
some
kind
of
a
teaser
for
that.
So
basically,
defining
scalability
is
very,
very
complex
topic.
We
are
how
we
are
doing.
That
is
that
we
are
basically
formulating
a
set
of
SLI
Sandnes
allows
service
level
indicators
and
several
service
level
objectives.
B
You
can
think
about
them
as
like
metric
SLI
being
a
metric,
some
metric
and
SLO
being
some
metric
with
a
threshold
basically,
and
what
we
are.
What
we
are
doing
is
like
we
are
defining
the
scalability
by
defining
a
set
of
those
sls,
and
we
are
saying
that
your
cluster
scales
are
a
given
set
of
scales
when
those
SLS
or
subs
are
satisfied.
B
Those.
This
definition
has
a
lot
of
drawbacks,
I'm,
going
to
talk
a
lot
about
those
tomorrow,
so
I'm
not
going
to
get
into
deep
to
details
today,
but
we
went
quite
a
long
route,
even
though
like
it's
way
more
in
front
of
us,
but
currently
we
significantly
or
visibly
improve
those
as
those
as
allows.
B
B
We
ensured
that,
like
all
the
people
understand
them
the
same
way,
not
that
us
in
the
at
the
beginning,
like
I,
bet
that
if
I
took
any
of
you
and
myself,
you
will
understand
them
differently.
There
are
also
five
more
isolationist
allows,
defined
and
approved,
mostly
in
the
networking
area
and
one
more
one
in
storage
area.
They
are
still
like
work-in-progress
ones,
because
although
we
agreed
on
the
exact
formulations-
and
we
discussed
them
with
users-
that
those
are
the
things
that
they
care,
they
really
care
about
care
about.
B
We
still
don't
measure
them
properly,
so
we
want
to
first
measure
them
would
like
see
where
exactly
we
are
tune,
the
exact
fresh
cult
and
and
only
then
make
them
really
official
ones
yeah.
So,
as
I
said,
like
I
will
talk
for
pretty
much
half
an
hour
about
this
like
tomorrow.
So
if
you
are
interested,
I
I
recommend
this
duck.
B
There's
also
like
like
based
on
those
sls.
We
are
able
we
introduced
this
concept
of
like
scalability
envelope,
so
scalability
is
basically
not
a
single
dimension,
it's
not
only
size
of
the
cluster.
It's
also
like
number
of
pots
in
this
cluster
number
of
namespaces
number
of
Secrets
and
many
others.
There
are
dozens
of
those
dimensions
and
basically,
what
we
are
saying
is
that.
B
If
you
like,
we
are
building
this
concept
of
scalability
envelope
in
a
way
that,
if
you
are
within
in
all
the
dimensions,
if
you
are
within
this
envelope,
then
we
guarantee
that
your
that
the
SLS
that
we
have
will
be
satisfied,
and
there
was
also
a
talk
by
Shyam
and
and
magic
on
last
cubic
on
that
was
presenting
like
the
what
what
is
the
characteristic
of
this
envelope
this?
It's
not
that
the
dimensions
here
and
in
are
independent.
It's
very
complicated
so
also
like.
Where
are
we
with
respect
to
each
dimensions
and
stuff
like
that?
A
Alright,
so
so,
once
we've
defined
and
identified
our
goals
and
definitions,
the
next
part
is
to
actually
go
ahead
and
monitor
and
measure
various
metrics
of
our
system,
and
this
is
where
we
have
a
bunch
of
infrastructure
that
we
built,
especially
for
scalability
tests
and
some
really
good
tools
to
help
us
do
this.
So
I
go
through
some
of
the
the
first
one,
of
course,
is
like
automating
all
of
our
scalability
testing,
so
we've
made
we've
included
scalability
testing
as
part
of
the
project
CI.
A
So
now
all
the
changes
that
get
into
the
system,
all
the
things
that
get
into
the
Cuban
artists
repository,
are
being
scaled
tested
at
both
Esprit
submits
and
post
submits,
and
this
is
a
really
big
dashboard
which
has
grown
over
time,
especially
in
the
last
few
months.
We've
added
many
different
jobs,
and
these
jobs
are
like
from
different
different
scales.
They
start
from
hundred
nodes
and
go
all
the
way
till
five
thousand
nodes,
and
we
have
benchmarks
running.
A
We
have
a
simulated
cluster
test
running,
which
is
what
to
be
called
as
cube
mark
and
some
experimental
tests,
and
these
so
yeah.
These
tests
are
basically
running
as
for
summits
and
periodically,
and
we
are
publishing
the
metrics
and
yeah
we're
basically
publishing
the
metrics
from
those
tests
publicly.
A
It's
you
can
go
to
that
URL,
it's
really
in
small
font,
but
you
can
go
to
that
URL
and
you
will
find
a
bunch
of
very
interesting
metrics,
so
these
are
from
our
scalability
tests
and
things
like
API
call
it
in
C,
spot
startup,
latency
is
scheduled
or
through
port
resource
footprint.
Cpu
memory,
usage
of
various
components
in
the
control
plane
are
all
shown
here.
The
next
one
is
gathering
of
profiles.
We
also
do
this
I
think
right
now
to
do
this
only
for
API
server
unless
it
changed
recently.
It's.
A
That's
awesome
so
yeah,
so
we
we
here
we
get
CPU
profiles
and
memory
profiles
and
we
from
time
to
time.
This
gives
us
very
useful
signal
and
knowing
where
are
we
spending
most
time
using
up
CPU
cycles
in
these
components?
And
where
are
we
doing
some
of
the
biggest
memory
allocations?
And
sometimes
this
helps
us
catch
bugs,
and
sometimes
it
just
suggests
general
improvements
that
we
can
do
based
on
these
yeah.
This
one
is
a
big
one,
so
trusty
old
load
over
is
a
tool
that
it's
kind
of
a
favorite
pet
project.
A
Four
four
six
scalability
and
we've
been
working
on
this
for
while
it's
finally
in
really
good
shape
now
so
I'll
go
ahead
and
ask
you
guys
to
try
it
out,
it's
a
tool
to
very
simply
define
scalability
test
like
declare
scalable
scalability
tests
in
a
declarative
way
and
run
them
against
your
cluster.
So
I
regard
going
too
much
into
the
details
of
the
API,
but
at
a
high
level
you
can
see
the
the
this
is
how
simple
your
complication
can
actually
look
like.
So
there
are
some
constants.
A
That's
how
we
want
to
generate
the
load,
and
then
there
are
these
steps,
which
are
basically
a
series
of
series
of
steps
which,
which
every
step
is
either
of
measurement
or
a
phase.
So
measurement
is
where
we
actually
go
ahead
and
measure
some
start,
some
measurement
or
actually
go
ahead
and
try
to
obtain
some
metric
and
phase
is
where
we
actually
declare
where
we
actually
create
the
load
according
to
the
tuning
sets
that
we
specify
above
so
in
this
particular
case.
We
just
start
a
simple
measurement
that
looks
for
your
pod,
startup
latency.
A
It
calculates
your
pod
startup
latencies
and
then
we
go
ahead
and
actually
create
create.
So
many
replicas
replica
sets
with
so
many
number
of
replicas
across
your
namespace
and
then
finally,
you
go
ahead
and
again
have
another
measurement
where
you
stop
the
measurement
that
you
earlier
started
and
summarize
the
results.
So
so
yes,
this
is
a
really
cool
way
and
we
really
encourage
you
guys,
I
mean
you
might
be
contributors
or
you
might
be
end-users.
A
We
just
want
to
try
out
running
scale
tests
against
your
cluster,
so
you
you
can
use
this
tool
to
to
run
this
yeah
this.
This
is
a
very
latest
improvement,
I
think
in
the
last
week,
or
so
where
we
are
actually
adding
metrics
from
our
scalability
tests
from
the
duration
of
the
test
itself.
So
earlier
the
thing
that
you
saw
in
performance
dashboards,
that
is
how
these
metrics
are
changing
over
test,
runs
over
time.
A
This
is
how
the
individual
metrics,
like
ApS
or
memory
usage
or
API,
call
it
in
C's
and
stuff
like
that,
is
changing
along
the
duration
of
the
test.
So
if
you
use
this
in
combination
with
cluster
loader,
that
can
give
you
pretty
good
insight
into
the
performance
of
your
cluster
and
what
sort
of
bottlenecks
you
might
be
running
into.
A
A
Okay,
so
this
is
the
third
part
where,
after
doing
the
measurements
and
monitoring
we,
we
identify
some
of
the
areas
that
we
want
to
improve
upon.
This
is
driven
both
by
the
metrics
that
we
got
from
the
previous
step
and
also
based
on
user
feedback
like
what
you,
what
users
of
coupon
ideas
are
running
into
and
scalability
is
a
very
horizontal
effort,
because
it
is
something
which
is
defined
by
many
different
verticals
networking,
storage,
API
missionary
and
a
lot
of
stuff,
and
so
this
inevitably
means
that
we
need
to
operate
in
two
modes.
A
One
is
where
we
are
making
the
improvements
ourselves,
but
because
the
domain
is
so
wide
and
there's
so
many
verticals
included,
we
often
also
collaborate
with
other
six
and
get
some
of
these
improvements
done.
This
is
really
a
from
the
very
from
really
small
pictures
to
pay
the
architectural
improvements
yeah.
So
that's.
A
So
some
of
some
of
the
recent
improvements
that
we
we
have
been
making
in
six
scalability
and,
along
with
other
six
I
mentioned
your
all-
that
I
could
remember.
There
is
a
talk
tomorrow
that
I'm
giving
at
3:55,
which
is
going
to
discuss
these
in
more
detail,
so
I'm
not
going
to
give
much
spoilers,
but
I
can
maybe
just
explain
at
a
high
level
what
these
are.
So
we
made
node
heartbeats
cheaper.
So
basically
we
are
reducing
the
amount.
This
amount
of
space
usage
and
HCD
that
is
used
up
just
by
known
heartbeats
by.
A
A
The
next
one
is
we
moved.
This
is
a
little
bit
older,
I
think
two
or
three
releases
ago
again.
I
won't
give
too
many
details
about
this,
but
we
moved
cubelet
from
watching
for
secrets
and
from
polling
for
secrets
and
conflict
maps
by
making
periodic
gate
calls
to
watching
them,
which
is
which
basically
reduces
a
huge
number
of
API
calls
and
large
clusters.
A
Another
one
is
new
API
for
events
at
some
point,
I'm
sure
many
of
you
would
have
run
into
issues
with
events
API
and
how
they're,
firstly
like
not
super
useful
for
any
automated
consumption
and
also,
like
general,
spamming
of
the
control
plane
with
events.
So
we
are
introducing
a
new
API
for
that
there
have
been
some
improvements
with
scheduling,
algorithm.
A
Yeah,
so
the
main
one
is
basically
that
we
now
allow
for
selecting
only
as
ranking
only
a
subset
of
nodes,
instead
of
like
evaluating
the
priorities
for
every
single
node
that
passed
the
predicates.
This
helps
in
large
clusters
with
respect
to
the
scheduling
throughput
and
last
one
is
IP
vs
going
G
a
which
was
also
a
while
ago,
but
this
helps
us
basically
improve
our
networking
performance.
Yeah.
B
B
Yes,
so
protecting
from
regressions,
like
obviously
one
thing
is
to
get
to
a
given
scale.
The
second
thing
like,
but
it's
one
thing,
but
the
second
thing
is
basically
to
to
not
regress
at
effect
from
a
given
scalability
that
we
already
achieved
and
the
the
most
important
part
in
that
are
obviously
tests.
B
We
have
basically
two
main
tests,
not
that
we
are
running,
which
is
far
away
from
where
we
would
like
to
be
parsed.
Part
of
the
reason
for
that
is
that
our
tests
were
super
complicated.
It
was
like
super
hard
to
understand
it,
even
even
though
we
actually
wrote
them
or
maybe,
because
we
actually
wrote
them
so
I
was
kind
of
resistant
to
extending
them
farther
before
we
had
cluster
louder.
B
Those
tests
are
similar
already
mentioned,
like
we
have
Prasad
meets
current.
Currently
existing
persimmons
are
like
100
node
tests
and
500
note
Q,
mark
test,
which
is
like
this
simulated
cluster.
Those
are
running
on
every
single
PR
and
those
are
basically
responsible
for
caching,
probably
something
like
60
to
70%
of
regressions,
based
on,
like
the
investigation
that
we
did
were
from
the
times
were
when
we
didn't
have
any
Prasad
mates,
but
still
like
the
some
regressions
are
happening
and
those
are
being
uncovered
by
the
post.
Sub
means
that
we
are
running.
B
112
zero
had
a
significant
performance
degradation
correct
connected
to
how
they
were
allocating
the
memory.
It's
actually
not
the
first
time
when
go
broke
us
I.
Remember
that
at
least
like
go
1.8
and
go
1.5
broke
us
to
in
the
past.
The
difference
is
now
is
that
when
they
broke
us
with
1.5,
which
happened
in
2015,
they
said
we
don't
care,
and
now
they
really
like
dropped
many
things
and
and
were
working
very
hard
with
us
to
ensure
that
we
understand
what
is
happening.
B
Exactly
and
that
the
problem
is
fixed-
and
this
is
now
fixed
so
yet
so
that
the
regressions
that
like
are
happening
pretty
much
everywhere,
like
from
go
that
I
mentioned
via
of
yeah,
also
operating
system,
I,
remember
a
back
when
we
bumped
the
kernel
and
it
started
using
a
different
IO
scheduler,
and
because
of
that
like
when
we
are
using
two
disks
for
one
for
logs
and
the
second
for
at
CD
data
like
they
started
interacting
strangely
and
like
it
completely
broke.
Also
the.
B
B
There
are
a
couple
things
that,
like
we
probably
would
like
to
do
here,
also
very
soon,
one
of
the
things
that
we
had
some
documentation
for
best
practices.
How
what
you
should
do
to
avoid
regret
like
introducing
regressions.
This
is
not.
This
is
like
very
old.
This
is
not
widely
known,
I
would
like
to,
or
we
would
like
to
make
that
like
refresh
that
significantly
ensure
that
it
is
like
much
better
known
among
all
community
members
that
are
performing
and
contributing
to
kubernetes
and
yeah.
That
I
think
that's
what
we
had
for
today.
C
B
B
B
D
B
D
Yeah,
apart
from
that
I
remember
in
previous
versions,
they
were
the
bottleneck,
was
etcd
and
so
the
Watchers
and
stuff.
What
do
you
see
now?
That
is
the
bottleneck
for
any
component
that
you
you
see,
that
might
become
that
I
know
that
HC
has
been
involved,
that
in
performance
lately,
so
I
wanted
to
know
that,
and
maybe
I
do
already
another
question
or
you
want
you
can,
which
kind
of
of
pots
do
you
run
on
the
test?
B
To
the
first
question,
like
Etsy,
be
improved
a
lot,
there
are
a
couple
things
that
are:
we
are
planning
or
our
plans
for
free
for
at
CD
version
coming
up
in
June
or
July.
The
biggest
one
I
am
aware
of
is
like
concurrent
reads
from
at
CD,
so
@cd
is
not
that
significant
bottleneck
that
it
used
to
be
in
the
past.
B
It's
hard
to
point
one
exact
thing
that
is
the
biggest
biggest
bottlenecks,
because
it
like
highly
depends
on
what
exactly
you
are
doing
like
I
can
imagine
that
being
a
scheduler,
if
you
want
to
like
schedule
very
small
short
jobs,
not
require
and
not
requiring
any
any
volumes
and
accept
any
specific
networking,
and
everything
like
that,
I
can
see
that
being
like
API
server.
I
can
see
that
being
like
a
specific
controllers
like
it
really
depends,
like
I
think
we
have.
B
This
is
something
we
are
slowly
starting
to
do.
We
are
not
there
yet
like
one
of
the
things
that
is
like
in
progress
is
like
evaluating
performance
of
pasture
autoscaler,
which
kind
of
it's
not
exactly
this,
but
it's
like
kind
of
step
towards
that
direction,
but
we
also
like
we
also
introduced
a
concept
of
like
cows
monkey
to
cluster
louder,
which
is
exactly
doing
which
is
exactly
randomly
killing
some
notes.
It's
I
think
it's
enabled
in
the
smallest
test
now,
but
I
can't.
A
B
We
wanted
to
enable
it
or
so
and
by
default,
but
maybe
that
didn't
yet
happen.
But,
yes,
we
are
aware
of
this
problem
and
yes,
this
is
going
to
happen
and
I
think
you
had
another
question
that
you
already
asked
and
I
forgot
what
it
was
Hey,
oh,
the
type
of
a
pot
yeah.
Yes,
so
basically
in
cluster
louder,
you
can
run
whatever
you
want.
B
B
It's
basically
what
I
mentioned
before,
like
we
were
kind
of
not
doing,
and
the
significant
changes
to
our
test
and
not
extending
them,
because
I
I
almost
was
not
able
to
follow
to
understand
what
the
our
tests
are
doing
like
in
the
previous
versions,
oh
yeah.
So
this
is
definitely
happy
like
going
to
happen.
E
A
B
There
is
a
page
in
community
repo
in
six
scalability
directory
where
we
started
documenting
it
some
time
ago.
To
be
honest,
it's
pretty
outdated,
but
it's
like
many
things
are
not
filled
fully
filled
there,
but
it's
it's
probably
like
a
place
where
you
may
also
want
to
look
at
and
like
we
definitely
would
like
to.
Finally
have
it
updated.
We
like
this
is
this
is
the
question
that
I'm
hearing
a
lot-
and
this
is
something
we
definitely
would
like
to
address
soon.
Ish.