►
From YouTube: Kubernetes Scalability Definition Evolution - Wojciech Tyczynski & Andrzej Wasylkowski, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes Scalability Definition Evolution - Wojciech Tyczynski & Andrzej Wasylkowski, Google
When Kubernetes 1.0 was announced in 2015, Kubernetes was claimed to support 100-node clusters. In order to prove that, we made the first attempt to define what does it mean that “Kubernetes scales to X-node clusters”. The SLOs making that definition were described in more detail in the first scalability-related Kubernetes blog post. However, as Kubernetes was maturing and new features were being added to it, it turned out that the initial definition is becoming meaningless, especially for larger users. In this presentation, we will explain how the definition was evolving over time and where we are heading, what are our principles driving that evolution and what efforts were spawned as a result of that (such as new SLIs/SLOs effort, testing tooling etc.)
https://sched.co/MPck
A
Okay,
thank
you
so
yeah.
So
thanks
for
introduction,
I
invite
back
to
chance
key
I
work
in
Google
for
the
last
four
and
a
half
years,
working
on
kubernetes,
focusing
mostly
on
scalability
and
performance,
and
this
is
what
it's
the
stuck
going
to
be
about.
So,
let's
start
with
thinking
what
color
bility
really
means.
So
let's
take
a
look
into
as
a
first
like
approximation.
A
What
wikipedia
says
so
it's
saying
that
like
scalability,
is
the
property
of
the
system
to
handle
a
growing
amount
of
work
by
adding
resource
to
the
system
in
computing
scalability
as
a
characteristic
of
computers,
networks
and
and
so
on
and
so
on.
Well,
an
example
is
a
search
engine
which
must
support
increasing
numbers
of
users,
and
there
are
a
number
of
topics
it
indexes.
So
it's
not
super
useful.
Like
imagine
that
you
are,
we
are
giving
this
definition
to
our
users.
It
will
mean
nothing
basically
to
them,
so
we
need
something
else.
A
What
do
we
have
on
our
minds
where,
where
we
are
saying
that
we
are
working
on
scalability
and
there
are
basically
four
main
pillars
of
scalability
work,
that
I
think
there
there
exist
and
those
are
like
defining
what
scalability
really
is
in
our
particular
system?
What
does
it
means
that
our
system
scales?
The
second,
is
improving,
improving
the
system
to
match
users
expectations.
A
The
third
one
is
testing
infrastructure,
I.
Think
all
of
youth
who
were
working
on
scalability
or
early
were
trying
to
prove
prove
that
your
systems
case,
like
you
all
know
that,
like
it's
on
one
hand
it's
hard,
but
on
the
second
hand,
it's
like
super
expensive,
so
we
need
some
frameworks
to
to
make
it
to
make
scalability
testing
more
approachable
and
the
fourth
Peter
is
like
tests
and
guarding
our
system
against
regressions.
A
A
So
it's
pretty
clear
for
users
that
they
want
the
the
system
that
they
are
working
with.
They
want
them
to
scale.
So,
for
example,
if
they
know
that
at
any
point
in
time
they
will
need,
they
will
not
need
more
than
10,000
machines.
They
want
to
know
that
kubernetes
scales
to
let's
say
$5,000,
there's
1,000
nodes,
but
what
does
it
really
mean
that
kubernetes
supports
or
scales
to
1,000
nodes?
And
the
answer
isn't
really
clear
for
that?
A
A
They
just
don't
what
you
just
don't
want
to
know
what
what
what
the
tree
means
like.
They
don't
want
to
understand
all
the
infrastructure
in
majority
of
cases,
how
it
works
exactly
which
in
our
case
is
kubernetes.
They
just
want
the
cluster
to
to
really
work
at
that
scale.
So
how
do
we
approach
defining
scalability
in
particular
and
kubernetes
like
I'm,
going
to
like
all
the
examples
here
will
be
based
on
kubernetes,
but
many
of
the
things
here
are
pretty
generic
and
apply
to
any
system.
So
how
do
we
approach
that
in?
A
How
did
we
approach
that?
And
the
definition
of
scalability
is
relying
on
two
main
concepts.
The
first
one
is
SLI,
which
is
service
level
indicator.
You
can
think
about
it
as
some
metric
that
we
are
gathering
from
the
system,
and
the
second
thing
is
SLO,
which
is
service
level
objective
that
we
built
on
top
of
that
SLI
and
you
can
kind
of
think
about
it
as
a
metric
plus
some
threshold,
so
with
DOS
having
those
concepts.
We
are
saying
that
a
cluster
works
or
scales
like
this
like
this.
A
A
Okay,
so
we
roughly
understand
like
how
the
high-level
idea
looks
like
so,
let's
take
a
quick
journey
into
like
what
was
happening
four
years
ago
and
how
we
started
defining
scalability
back
then
so
in
2015
we
said
that
we
introduced
basically
to
s,
allows
one.
It
was
API
responsiveness
when
we
reset,
like
99
percent
of
API,
calls
returned
and
ones
returned
in
less
than
one
second.
A
A
It's
like
any
inbound
traffic
to
the
cluster
is
it's
not
working.
Yeah
networking
is
completely
broken,
so
if
that
is
because
of
scalability,
then
obviously
your
cluster
doesn't
scale,
but
our
s
allows
that
I
mentioned
on
the
previous
slide.
Like
can
easily
be
satisfied
in
that
case.
So
it's
pretty
pretty
clear
here
that
we
need
something
more,
but
there
is
also
another
problem
which,
in
my
opinion,
is
actually
at
least
as
important,
maybe
even
more
important,
which
is
lack
of
precision.
A
So,
let's
take
a
look
as
an
example
to
the
API
responsiveness
like
how
do
we
aggregate
different
types
of
API
calls
how,
over
what
time
period
we
are
aggregating.
So
I
bet
that
if
I
take
one
of
you
and
myself
and
we
will
get
two
different
interpretations
of
those
like-
you
may
probably
say
that
I'm
working
on
that
for
like
four
years.
So
maybe
it's
because
of
that,
but
I
bet
if
I
took
like
if
I
would
take
like
any
two
of
you.
A
We
would
also
get
two
different
interpretations
like
there
are
so
many
of
those
that
it's
it's
highly
probable
and
why
is
it
actually
dangerous?
Why
it's
it's
a
problem,
so
imagine
a
situation
where,
in
my
interpretation
the
s
allows
are
satisfied
in
your
cluster
and
in
your
interpretation
they
are
not.
So
you,
as
a
user
are
saying,
like
you
promised
me
that
this
will
be
satisfied
and
it's
not
so
it
doesn't
work
for
me
and
I'm
saying
it
works,
and
the
problem
is
that
we
are
both
right
here.
A
So
it's
pretty
clear
that
we
need
to
something
something
better
and
we
basically
were
realizing
those
problems
from
pretty
much
early
beginning.
So,
let's
see
how
we
try
to
approach
those
problems
and
let's
focus
on
the
coverage
problem
first.
So
the
first
real
attempt
to
address
this
problem,
our
first
significant
one
was
in
April
2017.
A
A
They'd
still
required
a
lot
of
work
to
make
that
really
happen,
and,
as
you
can
imagine
like
it,
didn't
bring
us
that
much
farther
and
the
problem
was
obviously
a
scope.
So,
even
though,
like
I
was
explicitly
excluded,
a
bunch
of
things
from
the
from
that.
From
that
initial
attempt,
like
we
explicitly
excluded
things
like
stateful
applications,
we
explicitly
excluded
batch.
A
It
still
have
like
significant
interactions
with
like
at
least
nine
six,
including
API
machinery,
note
apps
out
of
scaling
off
network,
instrumentation
storage
and
obviously
scalability.
So
looking
a
little
bit
deeper
into
that
like
we
would
probably
need,
like
20-ish
app
reverse
for
that
dog
and
like
50
or
more
reviewers,
so
I
bet
it
wouldn't
be
approved
even
by
now,
and
probably
not
by
the
end
of
2020.
A
Even
so,
we
obviously
needed
something
more
shorter
term
and
I
guess
you
might
be
thinking
that
I'm
trying
to
show
that
we
need
more
iterative
approach.
Yes,
that's
true.
We
need
more
iterative
approach,
but
doesn't
that's
not
the
main
thing
that
I'm
going
to
focus
on,
because
in
fact
this
document
gave
us
something
that
is
super
important?
It
gave
us
basically
a
product
surface.
So
what
is
product
surface?
A
So
it's
basically
having
a
product
surface
is
very
important
for
two
reasons.
One
is
quite
obvious:
it's
it
lets
us
to
to
more
have
a
clear,
clear
set
of
things
that
we
believe
should
be
somewhat
covered
with
the
SLI
senesce
allows,
but
there
is
also
like
a
second
thing
that
is
very
important,
which
is
it
completely
changes
the
way
how
we
can
talk
to
our
users
now
so,
instead
of
asking
a
question
like
what
needs
to
scale,
what
do
you
want
to
scale?
Which
users
don't
really
know
answers
for,
and
they
cannot
really
answer
it.
A
A
A
But
what
about
other
issues?
I
mentioned,
also
lack
of
precision.
There
are
a
couple
other
issues
too,
so
how
did
we
proceed
it
with
with
addressing
those?
So
basically,
what
we
needed
was
a
framework
to
define
a
slice,
and
this
allows
on
top
of
those,
and
there
are
a
couple
of
principles
that
we
that
we
that
we
came
up
with
in
ordered,
so
that
we
would
be
able
to
to
create
those
as
allies
in
SLO.
A
So,
let's
take
a
look
into
those,
so
first
of
all
something
that
we
already
discussed,
that
they
need
to
be
very
precise
and
well-defined,
so
that
everyone
has
same
understanding
of
of
those
SL
of
those
as
those
as
allies
and
as
Laos.
So
the
second
thing
is
consistency
like
they
need
to
be
consistent
with
each
other
in
systems
like
us,
huge
or
us
wide
kubernetes,
where
we
expose
like
hundreds
or
maybe
thousands
of
features
or
api's
to
users.
A
We
can
easily
imagine
having
at
some
point
tens
of
SL
eyes
and
if
each
of
them
will
be
will
be
using
different
terminology,
different
concepts,
different
different
phrasing
and
stuff,
like
that
users
will
never
get
to
that.
They
will
never
understand
it
and
because
they
will
be
bored
by
or
they
will
just
stop
reading
that
at
some
point
and
if
we,
if
we
use
like
a
common
terminology,
common
concept,
common
formulas
in
all
of
those
like
it's
significantly
helps
with
understanding
those.
A
The
first
thing
is,
which
is,
is
something
that
I
already
was
kind
of
alluding
to
is
that
they
really
need
to
be
user
oriented.
So
the
last
thing
that
we
would
like
to
have
is
having
an
SLI
or
SLO
that
users
don't
really
care
about.
Why?
Because
it's
very
non
negligible
overhead
of
having
an
SLI,
you
need
to
earn
this.
Allow
you
need
to
maintain
it.
You
need
to
ensure
that
it's
still
like
up-to-date,
you
need
to
adjust
it
to
changing
products.
You
need
to
ensure
that
you
don't
regress
it.
A
You
need
to
debug
and
fix
regressions
and
stuff
like
that.
If
no
one
is
really
caring
about
that,
like
that's
a
waste
at
work
and
finally,
the
SLS
has
to
be
testable,
so
there
are
basically
two
ways
of
measuring
like.
Ideally,
we
would
like
all
the
SL
ice
to
be
measured
in
any
life
cluster
that
that
can
exist,
but
in
some
cases
it's
just
either
very
hard
or
very,
very
expensive.
To
do
that.
A
So
imagine
that
you,
for
example,
would
like
to
measure
and
network
light
and
see
between
any
two
parts
like
that's
something
that
can
be
measured,
but
potentially
the
overhead
and
the
cost
that
users
have
to
pay
for
that
buy
more
used.
Resources
is
very
huge
and
for
many
users,
none
acceptable.
So
while
we
are
trying
to
make
all
our
s
allows,
S
allows
to
be
measurable
in
any
cluster.
We
allow
for
cases
where
some
of
the
SL
else
can
only
be
measured
in
some
synthetic
benchmarks,
so
yeah.
A
So
we
have
those
principles,
but
that's
still
not
enough
for
for
providing
SLS.
So
let's
take
a
look
into
those
couple
couple
things.
So
what?
If
the
cluster
is
misconfigured
if,
for
example,
your
control
plane
components
are
Mis
are
under
provisioned,
we
know
that
the
cells
probably
want
to
be
satisfied.
A
Second
thing
is
like
extensibility:
it's
something
that
a
lot
of
users
really
takes
advantage
for
from
its
it's
something
that
you
see
a
lot
of
users,
really
love,
and
it's
why
they
are
using
kubernetes.
But
it's
also
like
a
way
that
you
can
shoot
into
your
food.
So
basically,
let's
think
that
let's
say
that
you
are
installing
a
web
hook
that
is
doing
sleep
for
5
seconds.
A
That
basically
means
that
the
API
responsiveness
SLO
cannot
be
satisfied.
So
what
can
we
do
with
that?
And
there's
also
like
a
third
thing
here?
That
is
like
very,
very
interesting,
which
is
like
the
performance
characteristic
of
your
cluster.
Very
heavily
depends
on
the
loads
and
the
cluster,
so
there's
significant
difference
when
you
are,
for
example,
starting
one
pot
per
second
versus
Lykes,
13
100
pots
per
second-
and
this
aspect
is
actually
very
interesting.
So
we
came
up
with
the
with
the
concept
of
scalability
dimension.
A
But
going
back
to
to
to
our
main
topic
like
this
concept
of
scalability
and
velov,
is
one
of
the
things
that
allowed
us
to
really
build
a
final
framework
for
defining
our
as
allows.
So
what
how
do
we
do
that?
So
what
we
are
basically
doing
is
we
are
saying
that
if
you
promise
that
you
correctly
configure
your
cluster,
that
you
keep
your
load
within
the
limits
in
all
the
dimensions
that
we
have
and
stuff
like
that,
then
we
promise
that
the
SAS
allows
will
be
satisfied.
A
A
We
already
seen
how
it
looked
like
in
2015.
This
is
how
it
to
what
we
refined
it
or
actually
half
of
it
in
2017.
So
let's
take
a
look
into
a
little
bit
details
into
what
exactly
happened
here.
So
first
thing
is
that
we
explicitly
split
it
as
a
live
from
the
SLO.
This
is
like
very
important,
because
SLI
is
is
a
way
of
measuring
things
like
it
defines.
How
do
we
measure
stuff?
Slo
is
only
like
saying
what
do
we
guarantee
for
those
metrics
and
when
do
we
guarantee
and
stuff
like
that?
A
So,
let's
take
a
look
like:
how
do
we
define
those?
So,
first
of
all,
we
explicitly
say
what
is
being
measured.
So
in
this
case,
it's
like
latency
of
mutating
API
calls
for
single
objects
in
the
exact
definition
of
the
SLO,
and
our
page
with
us
allows
it's
like
it.
The
mutating
is
also
defined.
It's
like
post
put
patch
delete,
I,
guess
I
missed
something
but
anyway,
so
so
we
are
explicitly
saying
what
exactly
we
are
we
are.
A
We
will
be
measuring,
there's
also
a
sibling
SLI
with
SLO
for
read
on
the
API
calls,
but
that's
that's
not
like.
The
purpose
of
this
talk
is
not
to
like
familiarize
with
you
with
all
the
SLI
senesce
allows,
but
to
give
you
an
idea
like
how
you
should
be
doing
that
if
you
be
doing
that
in
in
the
future,
either
for
kubernetes
or
for
any
other
system
so
yeah.
So
we
are
saying
what
is
measured.
A
A
And
finally,
we
are
saying
that,
like
how
do
we
aggregate
stuff
over
time,
so
we
are
saying
that
it's
like
99th
percentile
over
last
5
minutes
in
general.
We
we
don't
want
to.
We,
we
recommend,
looking
into
some
percentiles,
not
into
about
nothing
to
me
average
or
like
max,
or
something
like
that.
Those
are
generally
bad
metrics
to
provide
guarantees
for
so
yeah.
This
is
about
SLI.
What
about
SLO?
So,
first
of
all
we
are.
A
A
It's
because,
for
example,
of
the
problem
of
that
I
described
with
extensibility,
so
we
would
like
to
avoid
situation
when
someone
installs,
for
example,
a
web,
could
be
that
completely
breaks
performance
of
kubernetes
or
if
your
kubernetes
cluster
is
like
misconfigured
in
some
way,
and
then
we
are
saying
that
it's,
we
are
providing
guarantees
for
every
resource
vapor,
but
we
are
excluding
something.
We
are
explicitly
saying
that
we
are
not
providing
any
guarantees
for
any
virtual
and
aggregated
resources
and
see.
Are
these
like?
There
is
one
caveat
with
CR
DS?
A
A
We
are
trying
to
push
according
to
our
laws
to
make
the
s
allows
consistent
with
each
other
we
would
like
to
just.
We
would
like
to
just
get
to
the
point
where
we
simply
like
remove
the
exclusion
of
service
CR
this
here,
but
we
will
see
like
where
that
goes,
but
I
really
hope
it.
It
will
end
up
with
that.
A
So
hopefully,
like
within
the
next
month,
ish
or
so
the
SL
SL
will
evolve
to
2
by
just
removing
the
CR
DS
from
from
from
here
and
and
finally,
what
we
are
saying
is
like
what
exactly
we
guarantee.
So
we
are
saying
that
99%
I
have
a
clustered
a
is
like
less
than
one.
Second,
so
are
we
done
with
that,
as
I
mentioned
in
the
previous
slide?
A
Not
really
because
we,
because
of
the
changing
world
and
the
fact
that
CR
these
are
going
to
GA
will
probably
be
changing
it
to
also
include
the
Sienese,
but
that's
still
not
there.
That's
still,
not
not
everything
like
based
on
different
discussions
that
we
that
we
have
is
with
users
and
among
developers
it's.
This
SLI
is
still
not
exactly
reflecting
what
our
users
really
care
about.
So
imagine
a
situation
that
99.9%
of
your
calls
for
for
your
calls.
A
We
are
returning
like
four
to
nine,
which
is
like
we
try
after
and
that
finishes
superfast
and
for
the
remaining
0.1%
we
are
all
the
IP
echoes
are
taking
like
10
seconds.
Is
that
really?
Okay?
It's
probably
not,
even
though
the
the
SLO
actually
will
be
satisfied
in
that
case.
So
one
thing
that
we
would
like
to
do
here
is
to
explicitly
exclude
requests
that
kind
of
weren't
processed
and
to
include
things
to
include
only
requests
that
had
reasonable
response
for
users.
A
It's
there
are
still
some
details
that
has
to
be
figured
out
here,
but
it's
probably
the
way
we
are
we
will
be.
We
will
be
evolving
that
it
it's
become
more
and
more
important,
especially
in
the
context
of
what
is
happening
now,
which
is.
There
is
a
like
a
whole
effort
of
of
providing
a
proper,
like
request,
prioritization
right,
limiting,
load,
shedding
and
stuff
like
that,
an
API
server,
so
that
has
become
more
and
more
important.
A
So,
even
though
we
we
have
something
that
looks
pretty
reasonable,
we
are
still
not
there
yet.
So
this
is
just
to
point
you
that,
like
the
defining
SLI
senesce
allows
is
also
a
very
iterative
approach.
Similarly
to
how
you
how
we
are
like
making
software
in
general,
so
how
exactly
the
process
of
defining
a
slice
unless
allows
look.
A
Were
things
like
requests
that
we
are
saying
which
I
after
shouldn't
be
counted
towards
the
SLI
and
based
on
those
discussions
with
users?
We
are
going
back
to
changing
the
set
of
our
SLI
senesce
allows
so
to
be
more
precise,
like
this
doesn't
exactly
looks
like
a
circle
that
can
there
can
be
arrows
between
any
other
any
any
to
write-ins.
Here.
A
But
there
are
also
like
five
more
what
I
call
like
working
progress
as
a
license
allows.
Those
are
allies
that
are
already
precisely
defined.
They
are
approved
by
all
the
interested
parties.
They
were
discussed
with
users
as
things
that
they,
those
users,
really
care
about
them.
We
are
working
on
having
a
proper
implementation
of
how
we
do
how
we
measure
those
working
on
having
tests
that
are
exercising
those
and
stuff.
Like
that,
but
I
really
expect
those,
or
at
least
some
of
those
to
be
to
becoming
like
official
SLS.
A
A
A
B
A
A
We
are
working
like
as
part
of
like
extending
our
testing
infrastructure
and
stuff,
like
that.
We
are
working
on
on
being
able
to
monitor
the
test
runs
themselves,
not
only
to
summarize
the
test
to
build
graph,
how
we
can
where
we
can,
where
we
can
look
into
how
it
how
different
things
were
changing
over
times
over
run.
We
don't
have
any
alerts
yet
or
anything
like
that.
A
C
Hi,
hello,
thank
you
for
the
talk.
I
have
a
question
so
those
as
a
lie.
This
seems
to
me
like
they
are
the
kubernetes
for
the
from
the
kubernetes
for
the
platform.
Users.
I
have
a
lot
of
developers,
and
often
they
ask
me:
do
we
have
any
sli
SLO
for
kubernetes,
because
what
they
care
is
like
I
want
to
deploy.
I
know
that
I
can
deploy
on
the
clusters
when
I
do
a
network
call
it's
gonna
work.
My
dns
request
is
gonna
be
solved
if
I
need
to
scale
up
the
cluster
autoscaler
will
work.
C
A
This
is
exactly
for
end-users.
Ok,
this
is
what
is
targeting
for
end
users
like,
for
example,
like
pot
startup
time,
I,
guess
as
being
one
example,
is
exactly
what
what
you
are
working
for.
So
yes,
there
is
some
work
happening,
for
example,
for
like
how
fast
cluster
autoscaler
will
react
like
this
is
exactly
this
effort,
so
we
are
locking
a
bunch
of
stuff
here,
but,
yes,
we
are
making
some
progress.
At
least
thank.
C
A
E
A
B
A
D
Yes,
I
want
to
just
comment
on
this
question,
so
I
think.
The
point
is
that,
while
the
number
of
moles
does
influence
the
latency,
for
example,
so
you
basically
have
a
function
from
the
number
of
nodes
like.
If
you
keep
everything
else
constant,
the
number
of
nodes
is
going
to
influence.
Let's
say
the
pot
startup
time.
D
What
would
what
we're
trying
to
do
is
not
to
enforce
a
particular
function
underneath
for
the
shape
of
the
function
or
the
curve,
whatever
we're
just
trying
to
make
sure
that,
no
matter
what
the
node
size
as
long
as
it's
within
the
limits,
so
in
the
promise,
it's
not
going
to
exceed
us
a
lot.
So,
of
course
there
will
be
some
change
right
will
be
lower,
for
let's
say
a
10
10
notes
faster.