►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Intro: SIG Scalability - Wojciech Tyczynski & Matt Matejczyk, Google
This session will focus on the different efforts that SIG Scalability is involved in: defining what scalability means for Kubernetes, driving performance improvements, maintaining infrastructure for scalability testing, guarding Kubernetes against performance regressions. Time for Q&A will be reserved at the end of the session to understand how the SIG can better engage with the community as well as to allow the audience to provide the input about the roadmap.
https://sched.co/Zx5w
A
B
A
I'm
a
software
engineer
working
at
google
at
work.
I
work
on
google
container
engine
and
I
am
6k
teacher
wojtek
who
couldn't
be
here
today,
unfortunately
also
works
with
me
at
google
and
he
is
our
sixth
dtl
and
he
should
be
during
the
q
a
session.
So
we
can
ask
him
all
those
difficult
questions
that
I'm
not
able
to
answer
all
right
so
tldr
about
six
callability.
What
do
we
do?
First
of
all,
we
define
what
is
fibrous
scalability.
A
We
protect
from
scalability
regressions
and
mostly
by
maintaining
scalability
and
performance
test
frameworks
and
based
on
that
cicd
tests
and
other
than
that.
We
detect
scalability
bottlenecks
and
then
drive
performance
improvements
to
get
rid
of
them.
And
last
but
not
least,
we
keep
kubernetes
scalable
by
coaching
and
consulting
kubernetes
community
about
scalability
and
one
important
thing.
Please,
please
do
not
confuse
us
with
seek
auto
scaling.
A
A
So
if
you
ask
an
average
cubantes
user,
what
do
you
want
in
terms
of
stability?
The
obvious
answer
is
scalable
clusters,
but
when
you
ask
what
does
it
really
mean,
then?
Usually
people
don't
know
what
to
say
and
that's
understandable,
because
scalability
is
not
an
easy
thing.
Luckily,
it's
not
that
we
are
inventing
a
will.
B
A
B
A
A
B
A
A
It's
just
one
of
the
dimensions,
it's
very
important
one,
but
just
one
of
the
dimensions.
Other
examples
are
namespaces
number
of
secrets
number
of
services.
I
know
size
of
secrets
or
or
config
maps
number
of
buckets
per
service.
There's
some
virtual
dimensions
like
potty
turn
or
like
some
other
example
of
dimensions
that
that
attached
to
other
dimensions
like
a
number
of
pots
per
node
and
many
many
many
more,
and
this.
B
A
It's
capability
is
difficult
and
hard,
especially
kubernetes
community.
We
need
to
analyze
this
multi-dimensional
space
and
to
make
things
even
worse,
those
dimensions
they
interact
with
each
other.
So,
for
example,
a
number
of
namespaces
eat
a
number
of
pots.
There
is
a
dependence
between
them.
If
you
create
too
many
pots-
and
you
have
only
one
main
space
most
likely,
we
run
into
some
issues.
A
Luckily,
there
is
something
that
we
call
scalability
envelope
is
a
subspace
here
and
it's
a
safe
zone.
It
means,
if
you
are
within
this
scalability
envelope,
your
cluster
is
happy,
so
our
task
has
six
scalability
is
to
make
sure
we
are
aware
where
exactly
this
safe
zone
is
this
discovery
envelope
and
we
basically
provide
this
information
to
the
users,
so
they
know
how
much
they
can
scale
their
clusters
and
all
right.
So
we
more
or
less
know
the
top
level
overview
of
the
scalability
definition.
A
A
A
A
Like
the
shape
of
this
envelope,
when
it's
approximated
by
a
set
of
limits,
so
what
we
say
is
that
as
long
as
your
cluster
will
stay
below
5000
nodes
and
number
of
pods
in
the
cluster
will
be
less
than
30
times
number
of
nodes.
You
will
have
less
than
10
000
services.
You
will
have
less
than
20
or
250
back-ends
per
service
and
etc,
etc.
Then
your
cluster
should
be
okay,
it
should
work
fine,
all
right
bye,
but
what
does
it
mean
that
your
cluster
will
will
work?
Fine?
A
Well,
if
he's
happy
or
skills,
and
we
also
need
to
be
a
bit
more
precise
here,
especially
if
we
want
to
have
a
framework
where
we
can
experiment
and
try
pushing
some
limits.
If
we
push
one
limit,
we
need
to
still
be
able
to
say
whether
cluster
is
okay
or
not
right.
If
cluster
goes
down,
that's
an
easy
case,
obviously
cluster,
it
means
it
doesn't
scale,
but
if
it
it
didn't
go
down.
Basically,
just
maybe
some,
we
noticed
some
performance
degradation
by
10.
Is
it
still
happy
hard
to
say
right?
A
B
A
A
Apa
call
latency,
we
also
have
pod
startup
latency.
We
have
a
bunch
of
network
related
slos
like
network
programming
or
network
latency
or
dns
related
latencies
all
right.
So
that
brings
us
to
this
definition
like
we
can,
like
formally
say
what
does
it
mean
that
cluster
scales
cluster
scales,
if
all
scalability
slos
are
satisfied
and
to
put
everything
together?
And
now
we
have
this
framework?
It's
you
promise.
We
promise
kind
of
framework.
A
Two,
a
single
sentence
description
is
cluster
order.
Two
is
a
bring
your
own
yam
cluster
load
test
tool.
The
motivation
to
create
faster
order.
Two
was
that
before
our
tests
were
written
in
in
collect
in
an
imperative
way,
so
basically
we
had
all
this.
We
need
to
code.
All
these
operations
for
create
deployment
then
create
config
maps,
then
scale
deployment
up
scale,
deployment
down,
create
demonstration,
etc,
etc,
and,
like
once
it
reached
certain
scale
or
like
complexity,
that
has
reached
certain
complexity.
A
It
became
really
hard
to
maintain
and
really
hard
to
actually
extend
the
test,
and
so
we
had
this
idea
that
scale
like
load
testing
could
be
could
be
much
easier
and
ideally,
you
would
just
want
to
declaratively
provide
the
states
the
cluster
should
be
in
and
then
something
should
bring
cluster
to
those
states
and
that's
what
the
cluster
order
does.
So,
basically,
an
input
to
cluster
order
is
a
yum
file
with
a
test
and
test
can
be
set
of
states.
So,
for
example,
one
state
can
be
okay.
A
In
addition,
class
roller
provides
extra
observability
of
the
cluster
for
debugging
purposes.
This
is
really
important
if
something
stops
working
because
in
general,
the
axis
cavity
is
not
easy.
So
we
need
all
those
tools
like
dashboards,
graphs,
metrics
profiles
and
cluster
order
provides
everything
and,
in
addition,
it
provides
a
bunch
of
extra
features.
We
don't
have
time
to
describe.
All
of
them
describe
all
of
them,
but
if
you
are
interested,
then
there
is
a
link
at
the
bottom
of
this
slide,
where
you
can
learn
more
about
cluster
builder.
A
A
Tool
that
we
have
is
cubemark
tldr
about
qmark
is
a
cluster
simulation
tool
created
to
make
scallop,
makes
call
bd
test
cheaper
and
and,
as
you
can
imagine,
to
run
like
five
thousand
node
end
to
end
test.
You
need
at
least
5000
vms
for
the
nodes,
then
some
vms
for
the
master,
and
then
such
tests
can
take
like
the
fully
load
cluster
to
to
to
certain
pod
density.
A
A
Qmark
is
that
when
we
are
testing
scalability
of
kubernetes,
we
are
mostly
interested
in
control
plane
and
we
don't
really
care
what
happens
on
the
on
the
nodes
locally.
Obviously,
we
are
interested
in
all
the
interaction
between
nodes
and
and
masters.
So,
for
example,
all
the
requests
that
cubelet
sends
to
cube
api
server
are
all
the
watches
that
q
proxy
opens
to
go
to
api
server,
etc,
etc.
A
But
we
don't
really
care
whether
iptv
was
written
or
not
on
the
node
or
whether
a
certain
container
was
run
or
not
on
the
node
and
that's
what
basically
cube
mark
does
so
in
cube.
Mark
kubernetes
is
based
on
the
concept
of
hollow
node.
A
hollow
node
is
a
set
of
three
containers:
a
hollow
cubelet,
hollow
cube
proxy
and
hello.
Node
problem
detector,
and,
as
you
can
imagine,
I
will
explain
using
cubelet
so
hollow
cubemat
is
a
cubelet
is
a
regular
cubelet,
but
it
has
some
parts
locked.
A
So,
for
example,
there
is
no
containing
runtime
interface.
Instead,
there
is
a
mock
which
just
pretends
that
container
was
run.
Similarly
for
cube
proxy,
we
don't.
We
have
mocked
all
the
logic
for
writing.
Ap
tables,
because
we
don't
really
care
about
it
and
all
these
hollow
nodes
they
connect
to
a
real
master,
called
cubemark
master
and
together
it
forms
a
q
mark
cluster
and
using
it
using
this
approach,
we
can
basically,
we
are
making
basically
our
tests
about
10
times
cheaper,
which
is
a
really
great
result
and
interesting.
A
One
one
note
here
an
inception
like
thing
so,
as
you
can
imagine
to
run,
for
example,
a
five
thousand
note
cuber
cluster,
you
need
to
run
a
lot
of
containers.
In
this
case
it
will
be
about
15
000
containers.
So
how
do
we
do
that
right?
If
we
only
had
a
tool
that
would
allow
us
to
run
a
lot
of
containers
and
obviously
we
have
such
tool?
It's
called
kubernetes.
A
A
All
right-
and
I
already
mentioned
observability
and
debugability-
so
either
cluster
order
or
other
tools
they
they
provided
and,
for
example,
cluster
builder,
provides
features
like
profile
gathering
from
important
kubernetes
components
like
api
server,
etc,
controller
manager,
and
we
have
built-in
prompt
users
integration.
So
cluster
loader
is
able
to
set
up
a
prometheus
stack
inside
a
cluster
that
is
under
test
and
monitors.
A
So
on
top
of
this
cabinet
infrastructure-
and
we
we
own
a
lot
of
scalability
tests
using
this
infrastructure
and
again,
we
could
like
spend
at
least
20
minutes
describing
all
tests
that
we
own
it's
a
it's
a
lot
of
tests.
We
don't
have
that
much
time,
so
I
will
keep
it
short.
I
think
the
main
two
groups
of
tests
are
pre-submits
and
periodic
tests
pre-submits.
Every
time
you
send
a
pr
against
kubernetes,
kubernetes
or
kubernetes
per
test.
A
You
will
see
some
of
the
scalability
drops
running
basically
to
make
sure
this
pr
doesn't
introduce
any
scalability
regression.
Obviously,
we
cannot
have
all
the
tests
as
pre-submits,
for
example,
these
large
tests
are
too
expensive,
too
long
to
be
run
as
per
submit.
So
that's
why
we
also
need
periodic
tests
and-
and
we
have
a
lot
of
periodic
tests-
kinds
of
tests
we
have.
A
So
we
really
like
obviously
end-to-end
tests
because,
like
end-to-end
tests
are
the
best
way
to
actually
evaluate
performance
of
of
such
complicated
system
as
kubernetes
and
the
performance
tests
are
the
cluster
order.
Two
tests
and
the
correctness
tests
are
the
regular
kubernetes,
end-to-end
correctness.
B
A
Like
that
check,
whether
all
like
main
functionalities
work-
and
we
just
run
them-
I
really
high
scale,
for
example,
five
thousand
nodes
to
see
whether
to
make
sure
that
all
features
that
work
on
the
smaller
screen.
They
also
are
gonna
on
the
hard
scales
cubemark
that
we
already
described.
Then
we
have
a
lot
of
benchmarks,
for
example,
some
scheduler
benchmarks,
where
we
just
set
up
scheduler,
cube
api
server
and
lcd
without
creating
a
real
cluster
and
then
just
simulating
some
load
or
micro
benchmarks.
A
We
basically
test
there
are
some
methods
or
like
these
important
pieces
of
code
inside
kubernetes
called
base,
and
one
of
the
very
important
group
of
tests
that
we
maintain
are
best
callability
release
blocking
tests.
So,
basically,
if
they
tests
are
not
green,
if
they
don't
pass,
it
means
that
no
new
kubernetes
release
can
be
released
and
these
tests
are
hundred
and
five
thousand
node
performance
tests
and
five
thousand
node
correctness.
A
A
We
usually
debug
it
ourselves,
one
once
we
know
more
or
less
what
is
going
on.
We
either
try
to
fix,
also
ourselves
or
we
or
we
delegate
to
relevant,
seeks
a
few
examples
of
our
recent
regressions
to
give
you
some
intuition
about
the
scalability
regressions
that
we
are
we
are
facing,
so
we
had
like
two
really
large
column
regressions
in
113,
that
was
changes
to
memory
allocation
and
that
basically
completely
broke
the
scalability
of
kubernetes
in
golly
114.
A
If
I
remember
correctly,
it
was
some
timer
changes,
so
core
golem
library-
and
they
also
like,
took
us
quite
a
long
time
working
there
with
golem
team
to
to
debug
and
fix
that,
and
another
example
is
in
118
kubernetes.
We
realized
that
cluster
started
to
fail
during
master
rolling,
upgrade
all
right
and
protecting
from
scalability
regressions,
just
by
looking
at
the
tests
and
reacting
to
that,
it's
kind
of
a
reactive
thing.
Of
course
we
want
to
do
more
than
that.
We
do
more
than
that.
A
We
also
try
being
proactive
and
being
proactive
in
six
calibrity,
meaning
detecting
the
scalability
bottlenecks
and
then
driving
improvements
to
get
rid
of
them
and
again
some
examples
here
too,
to
give
you
intuition.
So
recently
we
made
some
changes
to
watch
cache.
We
added
indexes
there.
We
we,
we
changes
the
worksheet
size
to
be
dynamic,
really
watch
bookmarks,
where
basically,
we
built
on
the
watch
bookmarks
api.
A
A
Okay
and
last
but
not
least,
we
are
also
involved
in
kubernetes
community,
consulting
coaching
about
scalability
and
performance.
One
of
the
most
important
things
here
is
the
scalability
approval
process.
This
is
part
of
the
production,
readiness
review
and
so
basically
starting
in
118
plus
every
cap
will
have
this
mandatory
section
with
questions
about
scalability
and
performance.
This
is
to
make
sure
that
outer
of
the
cap
is
aware
of
potential
scalability
or
performance
issues
at
the
design
phase,
and
we
do
a
lot
of
ad-hoc
skill
testing
example.
A
A
As
I
said
being
present
in
discussions
about
performance
about
scalability,
an
example
that
I
really
like,
the
idea
that
I
really
like
is
consistent
reads
from
cash
cap
listed
here
all
right,
and
that
was
pretty
much
everything
just
one
more
slide
about
how
to
find
us
here
are
links
to
our
homepage,
our
slack
channel
our
mailing
list.
We
also
have
these
public
meetings
every
second
thursday,
by
weekly
meetings.
A
I
encourage
you
to
join
and
at
the
bottom
of
the
slide,
there
are
like
two
issue
lists
that
we
tag
as
help
wanted
and
so
feel
free
to
to
check
them.
If
you
see
anything
interesting,
there
feel
free
to
assign
and
do
it.
We
all
really
appreciate
all
right,
and
that
was
everything
so
now
it's
time
for
the
questions.
Thank.
A
A
B
So
so,
basically,
the
the
idea
behind
cubemark
is
not
really
to
mock
the
api
calls,
but
rather
mock
the
underlying
behavior
of
some
components,
so
in
particular,
like
as
an
example,
we
are
mocking
like
the
calls
to
to
container
runtime.
So
so,
instead
of
like
calling
docker
to
explicitly
do
something
with
the
containers
like
starting
them
or
stopping
or
whatever
we
are,
we
are,
we
are
replacing
the
that
with
like
a
mock
implementation
of
of
docker
or
container
runtime
interface
in
general.
B
That
just
claims
it's
it
does
that,
but
didn't
really
do
anything
didn't
3d
run,
didn't
really
stop
anything
or
anything
like
that.
We
are
not
really
mocking
this.
The
api
calls
per
se
like
we,
we
are
using
the
the
regular
api
server
like
the
the
master
part
of
the
control
plane,
so
in
particular,
api
server
and
lcd
that
are
involved
in
in
in
answering
api
call
or
like
re
replying
to
api
call
are
regular
ones.
B
A
Or
yeah,
why
not
so
technically
it's
possible
hollow
cubelet
is
just
a
container,
so
you
can
run
it.
You
can
configure
it
like.
If
you
have
all
the
like
secrets
and
keys
to
connect
to
your
to
your
your
master,
then
there
is
like
nothing
wrong
with
like
it's
possible
to
connect
cubelet,
hello,
cubelet.
The
question
is
like:
why
do
you
want
to
do
that
right?
B
So
so
I
can
add
one
more
thing
to
it:
I've
seen
examples
of
people
doing
that
actually,
and
one
of
the
examples
is
is
when
you
want
to
when
you
want
generally
want
to
run
cube
mark,
but
you
also
want
to
run
real
nodes
where
you
would
like
real
add-ons
like
dns
or
or
something
else
that
is,
that
should
be.
That
is
not
running
on
the
master,
but
it's
running
like
as
as
a
pod
in
the
cluster.
So
if
you
want
that
those
spots
to
be
really
running,
they
need
to
be
running
on.