►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro + DeepDive: SIG Scalability - Wojciech Tyczynski & Mateusz Matejczyk, Google
This session will focus on the different efforts that SIG Scalability is involved in: defining what scalability means for Kubernetes, driving improvements, infrastructure for scalability testing, tests and guarding Kubernetes against performance regressions. Each of those areas will first be described at the high level, followed up with deeper insight into concrete aspects and summarized with most recent achievements and a roadmap for future work. Time for Q&A will be reserved at the end of the session to understand how the SIG can better engage with the community as well as to allow the audience to provide the input about the roadmap.
https://sched.co/UakV
A
B
So
what
do
we
work
on
in
scalability?
So
there
are
basically
five
main
areas
that
we
are
focusing
on.
First,
one
is
like
defining
what
scalability
really
means
and
where
we
should
be
heading
towards
like
what
are
the
aspects
of
the
system
that
should
scale
and
and
what
what
targets
those
should
scale.
The
second
thing
I
will,
we
will
be
talking
more
about
more.
We
will
be
giving
more
details
about
each
of
those
like
in
the
later
part
of
the
presentation.
B
So
this
is
just
like
the
summary,
so
the
second
part
is
basically
monitoring
and
measuring
where
we
currently
are
with
the
system.
Third,
one
is
is
about
proof,
is
about
contributing
or
coordinating
the
actual
improvements
to
the
kubernetes.
Fourth,
one
is
ensuring
that
we
don't
regress
in
terms
of
performance
and
scalability,
and
the
fifth
one
is
coaching
and
consulting
community
like
people
from
other
six
or
other
contributors
that
want
to
ensure
that
their
feature
scale.
They
want
to
do
some
tests
and
they
don't
know
how
to
do
it
and
stuff
like
that.
B
B
Okay,
so
let's
go
into
more
details.
So
if
you
ask
users
or
customers
today,
what
do
they
really
want
and
if
they
want
to
do
if
they
want
to
use
if
they
want
kubernetes
to
actually
scale
they
for
sure
answer?
Yes,
we
obviously
want
the
scalable
clusters
or
large
clusters,
or
or
something
like
that,
but
like
if
you
try
going
into
a
little
bit
more
deeper,
deeper
and
details
and
and
ask
them
what
what
exactly
they
want
to
really
what
exactly
should
scale
in
kubernetes.
B
Everybody
generally
wants
to
you
to
assume
that
the
infrastructure
they
are
using,
underneath
which,
in
this
kind,
the
wind
in
this
situation,
is
kubernetes,
really
works,
and
they
don't
want
to
understand
all
the
gory
details
of
the
system.
So
so,
in
order
to
to
define
what
scalability
really
means
for
kubernetes,
we
need
to
go,
take
a
little
bit
different
path.
So,
let's
maybe
look
what
wikipedia
says
about
scalability,
so
I'm,
probably
not
going
to
read
all
of
that-
maybe
just
the
first
one.
B
B
Support
like
cluster,
currently
support
clusters
with
like
up
to
5,000
nodes,
but
but
that's
not
all
like
scalability
like
there
are
many
aspects
and
many
dimensions
of
scalability,
and
if
you
thing
about
scalability
for
kubernetes,
particularly,
you
need
to
think
about
many
other
aspects,
so
not
only
the
cluster
size.
So
what
we
did
is
like
we
define
this
concept
of
like
scalability
envelope
and
what
what
what
it
basically
means
is
that,
like
it's,
a
I,
don't
want
to
call
it
cube.
B
Unfortunately,
like
computing,
how
this
envelope
looks
exactly
is
really
complex
and
are
you
aware
of
like
this
conniving
framework,
in
your
view,
so
basically
it's
a
it
was
defined
like
in
like
tens
of
years
ago,
but
it
basically
divides
the
problems
that
exist
into
like
five
different
categories.
So
simple
problems,
complicated
complex
and
two
more,
which
I
actually
forgot
now,
but.
B
Chaotic
and
one
more
but
basically
like
there
is
like
a
difference
between
complicated
problem
and
complex
problem.
So
complicated.
Is
it's
basically
a
problem
that,
based
on
like
experience
and
knowledge,
and
some
extended
analysis
like
can
be
solved?
And
there
is
like
a
right
answer
to
that.
Whereas
complex
is
basically
a
problem
that
have
there
are
a
bunch
of
unknown
unknowns
and
like
the
cause-effect.
B
B
The
first
one
is
like
SLI,
which
is
service
level
indicator,
which
you
can
think
as
some
metric
that
is
measured
within
your
cluster
and
SLO
is
like
service
level
objective,
which
you
can
basically
mean
but
think
about
us
that
metric
plus
some
threshold,
and
what
we
are
saying
is
that,
like
we
are
saying
that
the
cluster
scales
within
a
given
conditions,
if
all
the
scalability
s
allows
are,
are
actually
satisfied.
So
if
the,
if
all
those
metrics
that
we
measured
across
this
run,
are
within
the
thresholds
that
that
we
defined
okay.
B
So
how
should
we
choose
the
appropriate?
How
structures
would
we
define
appropriate
as
allies
and
s
allows?
We
are
I?
Think
I
already
mentioned
it
like.
If
we
ask
the
customers,
what
are
the
things
that
they
really
want
to
be
within
some
some
some
boundaries
like
they
probably
are
in
majority
cases,
they
won't
be
able
to
do
any
reasonable
answer,
or
they
don't
even
want
to
give
that
answer,
because
they
don't
want
to
know
these
details.
B
So
what
we
want
to
do
is
basically
we
want
to
come
up
with
those
proposals
for
four
metrics
ourself
and
just
use
users
to
define
the
thresholds
that
will
be
satisfying
for
them,
so
how
to
choose
those
slis
and
those
metrics
and
what
we
do.
What
were
we
are
getting?
Is
this
concept
of
like
product
surface,
so
product
surface
is
basically
a
feature
set
and
API
in
surface
and
stuff
like
that
of
the
product
in
this
case
in
this,
in
this
case
it's
kubernetes.
B
B
Okay,
so
so
so
what
how
we
are
defining
this
allows
or
there's
a
rise
for
sorry.
So
there
are
like
couple
principles
that
we
need
to
keep
in
mind
to
make
the
SLI
and
SLO
like
usable.
So
the
first
one
is.
They
need
to
be
very
precise
and
well-defined
why
it
is
so
important.
It's
like
imagine,
or
maybe
let's
use
the
example
that
we
from
the
real
life.
B
So
in
the
when
we
were
announcing
like
1.0
version,
we
said
like
that
one
one
of
the
sls
for
kubernetes
is
actually
that
api
calls
like
99%
of
api
calls,
will
finish
within
within
one
second.
But
what
that
really
means,
like
it's
so
ambiguous
that
I
bet
that
if
I
take
any
one
of
you
and
your
you
will
provide
like
some
under
your
understanding
of
that
and
we
do
we'll
take
we'll
take
my
understanding.
Those
will
be
different.
B
So
you
may
probably
think
that
I'm
working
on
that,
like
four
or
five
years
and
and
thinking
about
that.
So
it's
because
of
that,
but
I
actually
bet
that
if
I
take
any
two
of
you
and
take
your
definitions
like
your
interpretations
and
that
it
would
be
exactly
the
same,
there
will
be
different
and
why
it
is
so
important.
It's
so
imagine
that,
as
a
user
using
your
interpretation
of
this
like
badly
behaved,
it's
a
lie
or
slow.
B
The
cluster
doesn't
scale,
doesn't
like
it's.
It
doesn't
meet
that
slow
and
in
my
my
interpretation,
which
is
different,
so
the
situation
like
can
happen,
it
actually
scales.
So
we
are
getting
to
the
point
that
you
are
saying
this
cluster
doesn't
work
because
it
doesn't
meet
as
I'm
saying
that
it
does,
and
we
are
both
right.
So
that
is
obviously
pretty
bad
situation.
B
So
that's
why
we
need
to
have
them
very
precisely
define
the
second
fingers,
or
second
principle
is-
is
that
they
have
to
be
user
oriented,
and
that
is
I.
Think
that's
pretty
obvious.
So
the
last
thing
that
you
would
like
to
have
is
like
having
an
SLO
which
requires
a
bunch
of
maintenance
ensuring
that
we
are
not
regressing,
refining
it
because,
like
the
system
is
changing
underneath
and
stuff
like
that,
so
it
requires
a
bunch
of
effort
and
work
and
know
what
like
no
user,
really
caring
about
that.
B
So
that's
really
bad
situation
to
be
at.
Maybe
it
doesn't
heart
the
problem,
but
it
it
doesn't
heart
the
product,
but
it
actually
means
that
we
are
wasting
a
bunch
of
work.
The
third
principle
is
that
we
want
to
make
them
consistent
with
each
other.
I
think
that
is
also
pretty
obvious.
We
don't
want
every
user
to
like
really
need
to
understand,
like
the
everything
from
scratch
for
every
single
SLO
that
we
have.
So
that's
also
like
making
them
consistent.
It's.
B
It
basically
means
that,
like
they
need
to
understand
kind
of
a
framework
once
and
then
understanding
every
single
one
of
those
is,
it
should
be
relatively
easy
and
the
fourth
one
is
that
they
need
to
be
testable.
So,
ideally,
we
actually
want
them
to
be
to
be
measurable
in
the
in
a
single
life
cluster.
No
matter
if
it's
like,
we
are
doing
tests
or
running
production
work
all
day
there
and
we
are
trying
to
to
be
to
build
the
SLO
see
exactly
and
SLI
sand
in
in
in
that,
but
sometimes
it's
like
very
tricky.
B
Yes,
so
what
are
the
existing
kubernetes
approved?
Scalability
slices
allow
so
so
those
that
there
are
two
of
them
currently
one
is
the
API
called
latency
I.
Think
it's
actually
worth
mentioning
that,
even
though
it's
like
pretty
long
definition
here,
it's
not
exactly
the
whole
definition.
It's
still
like
approximation.
We
didn't
want
to
put
like
couple
lines
of
text
in
the
slide,
because
that
would
wouldn't
be
very
readable.
B
There
is
a
link
to
to
the
page
of
SLO
Slyke
later
in
this
presentation,
but
basically
what
we
are
trying
to
say
is
that,
like
it's
roughly
what
I
mentioned
before
so
like
99%
of
like
API,
calls
finished
within
some
seconds
and
some
some
time.
But
basically
there
are
a
bunch
of
details
that
are
not
really
mentioned
here,
like
what
about
web
hooks.
For
example,
like,
obviously
you
can
install
who
have
hooks
in
kubernetes
and
if
your
web
hook
will
be
sleeping
for
5
seconds,
then
it's
obviously
any.
B
A
B
Second,
was
the
second
disallow
that
we
have
is
like
for
what
startup
time,
so
we
are
saying
that
$19.99
percentage,
it
should
finish
within
5
seconds,
but
again
it
excludes
things
like
time
to
run
init
containers
which
can
be
pretty
much
unbounded.
It
excludes
time
to
pull
the
image
because
it's
something
we
completely
this
completely
out
of
our
control
and
and
stuff
like
that.
So
so
those
are
approximations
like
the
exact
definitions
are
on
the
page.
B
We
have
five
more
a
slice,
and
s
allows
that
we
work
pretty
hard
with,
like
other
six,
mostly
with
networking
and
storage
they
are
defined.
We
are
already
measuring.
All
of
them
in
our
test
may
be
without
the
DNS
programming
latency,
or
we
are
almost
that
we
will
be
there,
probably
within
the
next
like
two
three
weeks
or
something
act
like
they
are
not
yet
approved,
because
first
we
didn't
do
a
full.
B
B
This
is
the
link
that
I
mentioned,
where
all
those
things
are
defined
are
written
down
like
all
those
SSL
iced
or
those
principles
that
I
mentioned
before
and
and
stuff
like
that,
okay,
but
but
it's
still
we're
still,
it's
still
not
exactly
what
we
want,
because,
obviously
you
it's
not
that
we
can
guarantee
those
sls
in
every
single,
no
matter
what
you
do.
So
it's
pretty
obvious
that
if
you
fro
billion
of
plots
into
your
cluster,
it
won't
work
and
we
won't
be
able
to
satisfy
those
as
a
loss.
B
B
So
so,
as
an
example,
if
like,
if
you,
your
cluster,
has
like
less
than
five
thousand
or
at
most
five
thousand
nodes
less
than
thirty
times
number
of
note
spots
and
stuff
like
that,
if
your
master
is
provisioned
correctly,
has
enough
resources
and
stuff
like
that,
only
then
we
actually
can
promise
you
that
the
scalability
s
allows
that
that
I
mentioned
before
will
be
satisfied.
So
this
is
this
is
the
framework
this.
This
is
how
this
is,
how
exactly
we
are
defining
it
and
we
are
providing.
What
is
what
we
are
providing
trick
to
users.
B
Yeah
so
computing
like,
as
I
mentioned
computing
this,
how
exactly
this
envelope
looks
like
it's
very
complex,
but
fortunately
we
don't
really
need
that.
So
it's
it's
generally
enough
to
have
some
approximation
for
that,
and
to
do
that
we
are
basically
decomposing
the
scalability
envelope
when
into
a
number
of
somewhat
independent.
B
Dimensions
so,
for
example,
we
are
saying
that
like
number
of
nodes
has
to
be
at
most
like
five
thousand,
that
number
total
number
of
paths
has
to
be
at
most
like
30,
x,
times
number
of
nodes,
and
so
and
so,
and
we
kind
of
decomposed
them.
It
doesn't
mean
that
this
is
exactly
how
the
scalability
and
scalability
and
Vallabh
looks
like,
but
it's
good
enough
approximation
and
it's
what
is
much
more
understandable
and
for
users
and
much
more
easier
to
talk
about
and
there's.
B
This
link
contains,
like
think
that
this
here
contains
like
what
are
the
dimensions
that
we
are
currently
thinking
on
and
what
are
the
the
limits
that
we
got
our
seem
to
be
safe
to
use
and
according
to
our
our
test,
yeah.
So
so
one
more
thing
to
mention
here
is
that,
like
it
doesn't
like
the
fact
that
we
define
some
sls
doesn't
mean
that
we're
done
so.
First
of
all,
like
that,
the
product
or
the
kubernetes
is
evolving.
B
We
are
adding
new
features,
so
probably
my
jority
of
them
should
be
covered
by
some
s
ellos,
because
if
we
are
creating
those
features,
probably
someone
really
cares
about
them.
Second,
the
fact
that
we
are
adding
new
features
in
many
cases
like
effects
the
existing
s
ellos,
like
the
big
funder
examples
is
what
I
already
mentioned.
This
is
the
webhooks
when
we
creating
when
we
were
when
we
defined,
like
API,
call
latencies
for
the
first
time.
No
one
even
was
thinking
about
web
hooks,
but
obviously
they
are
affecting
like
this.
B
The
the
latency
of
of
API
calls
so
so
there
it.
There
should
be
like
an
constant
loop
so
that
we
are
defining
as
allows.
We
are
running
scalability
tests.
We
are
improving
some
performance
of
some
components.
We
are
talking
to
users
what
they
think
about
it.
What
are
the
other
things
new
things
that
they
really
care
about
it
and-
and
we
should
go
back
to
like
and
refine
existing
SLO
refine,
existing
SL
Isis
allows
or
add
new
and
stuff
like
that,
or
maybe
even
remove
some
of
those.
B
If,
if
no
one
really
cares
about
them
any
time
longer,
yes,
so
that's
roughly
what
I
had
about
scalability
worried
about
defining
scalability
one
thing
to
mention:
if
you
are
interested
a
little
bit
more
in
a
little
bit
more
details,
I
was
giving
a
talk
on
previous
cube
con
in
Barcelona.
The
whole
talk
about
like
this
aspect,
so
so
it's
you
can
probably
find
the
recording
on
YouTube
or
oh
yeah
and
I'm
good
yeah
mic
to
to
mark
no
thank.
A
A
It's
really
surprise
to
you
that
we
do
run
tests
to
make
sure
that
converter
scales
and
as
everything
in
kubernetes
we
use
that
has
influenced
our
structure
for
that,
so
the
pro
for
running,
continuous
jobs
or
any
pre
submits
and
using
per
using
test
grid
to
present
the
results.
This
is
a
screenshot
of
our
six
scalability
test
grid.
So
what
kinds
of
tests
do
we
have?
A
We
can
like
define
them
into
into
some
groups?
I
will
start
with
correctness.
I
think
this
one
is
the
less
interesting.
So
basically
in
kubernetes
there
is
a
concept
of
correctness
test.
These
are
trees,
blocking
tests
running
on
the
orluk
end-to-end
test
running
on
for
node
cluster,
if
I
remember
correctly
and
what
they
do.
They
basically
check
that
every
make
every
like
most
current
functionalities
work
as
they
should
write.
A
Like
ant,
went
in
and
and
wait
so,
for
example,
either
deployment
is
created,
the
pods
for
that
deployment
should
be
created
right
and
around
if
I
create
a
service.
I
can
like
ping,
a
pot
which
is
part
of
the
service
and
stuff
like
that.
So
what
we
do
we
take
these
tests
and
we
run
them
at
scale
of
5000
notes
to
make
sure
that
everything
works
and
nothing
breaks
at
this
officially
supported
scale.
A
The
more
interesting
part
is
actually
performance
tests
which
are
using
the
framework
that
Wojtek
described.
So
what
we
do,
we
we
stay
within
the
scalability
envelope,
which
is
basically
we
stand
within
the
the
official
scalability
limits.
So,
for
example,
we
test
5k
now
cluster
we
stay
bid
out
the
30
pods
per
node
average
density.
We
don't
create
like
too
many
names
faces
to
stay
within
the
limits,
and
then
we
test
that
all
scale
VTS
arrows
are
are
satisfied.
Currently
we
have
two
tests,
one
is
called
density
and
the
other
is
called
load
test.
A
We
are
working
actually
on
merging
them
together.
What
they
do
briefly,
so
density
test
was
created
to
measure
put
startup
latency,
what
it
does
it
first
load
cluster
with
30
pots
per
node.
On
average,
it
waits
for
all
the
pods
to
be
created,
and
then
it
creates
additional
one
pot
per
node
on
average
and
measure
how
long
it
took
to
create
this
pot.
A
So
we
measure
like
99%
I,
love
that
what
start
of
a
nice
index-
and
we
expect
this
time
to
be
below
five
seconds
node
test-
is
similar
in
a
way
because
it
also
load
cluster
up
to
30
pots
per
node,
but
it
does
it
in
a
much
more
sophisticated
way.
So,
in
addition
we're
creating
services,
we
are
not
only
using
deployments
we
are
using
demon
set,
stateful
sets
jobs,
we
are
creating
some
secrets,
config
maps
and
the
addition
to
just
creating.
Then
we
have
a
scale
up
scale
down
face.
A
We
are
like
random,
been
scaling,
big
deployments,
jobs
and
stuff
up
and
down
to
generate
some
pod
churn,
and
after
that
we
also
add
metal
and
the
scalability
a
service
and
verify
that
there
are
within
the
de
limits
and
like
the
third
category
order.
So,
basically,
everything
else,
so
we
have
some
storage
tests.
We
have
some
benchmarks,
for
example
scheduler
benchmarks,
where
we
just
on
an
API
server
and
scheduler
and
and
verify
various
things
like
throughput.
A
A
One
of
the
repeaters
is
performance
test,
so
load
plus
density,
running
on
hundred
node
cluster.
These
tests
run
every
hour,
I
believe
or
so,
and
it's
a
very
important
signal,
especially
now
like
during
freeze
and
to
make
sure
that
everything
works,
but
like
hundred
node,
it's
not
actually
it's
far
from
the
officially
supported
5000,
not
scale.
That's
why
we
also
run
to
test
correctness
and
performance
at
this
five
thousand
of
scale
we
random
daily.
Why?
A
Because
the
performance
tests
so
to
create
30
pots
per
node
insert
cluster,
that
is
like
150,000
pots
in
total
and
all
these
operations
so
to
scale
them
up
down,
and
indeed
it
takes
about
12
hours,
right
correctness,
tests.
They
also
take
about
four
hours.
So
in
total,
that's
plus,
if
you
add
some
some
Headroom
for
some
time
out
that
basically
they
take
whole
day
and
when
it
comes
to
none,
we
is
blocking.
We
have
some
qumar
tests,
I'll
be
talking
about
humor
later.
A
This
is
our
flame
framework
for
simulating
large
clusters
for
performance
test
purposes
and,
like
all
the
other
kinds
of
tests
of
storage
benchmarks,
micro
benchmarks
are
non
release
blocking
you
can
find
them
all
in
our
scalability
test.
Suite
right
and
other
categories
category
is
pre
submit.
So
whenever
you
send
a
PR
to
P,
Burnett,
askew
Burnett
is
or
kubernetes
pair
of
tests,
which
is
our
repo
every
host.
A
Almost
all
of
our
tools,
part
of
the
cube
Marcus
custody
in
Connecticut-
is
these
two
tests
will
run
as
a
pre
submit,
so
we
run
performance
tests.
100
notes,
oh
I
forgot
to
mention
that.
So
what
we
do
we
run
kubernetes
on
GC
in
our
test
or
cube
mark,
which
I
will
describe
in
a
moment,
and
we
also,
we
also
test
performance,
cue,
mark
500
notes
and
similarly
for
the
perfect,
a
stripper
all
right
next
tool
is
our
performance.
A
Dashboard
called
perf
is
a
custom,
a
tool
that
basically
it
here
so
long
story
short,
is
a
dartboard
that
visualized
various
metrics
from
different,
continuous
integration
test
invocations
and
the
idea
is
that
basically
can
compare
how
some
metric
changes
across
different
runs.
It's
a
primitive
tool
like
if
you
check
code,
like
a
perfect
our
code,
you
will
know
why
that
is
very
powerful.
I
can
show
you
some
examples,
so
it
can
answer
your
question
about
either.
A
It
is
very
helpful
when
it
comes
to
performance
regression,
debugging
or
if
you
want
to
find
something
about
various
perforated
communities
characteristic.
So,
for
example,
what
is
the
current
scheduler
throughput
in
the
5,000
node
cluster
right?
You
can
check
that
in
perf,
so
here
we
have
tests
that
we
see
5,000
nodes
and,
like
each
point,
hero
represents
one
around
right.
A
So
we
get
a
point
every
day,
as
we
can
see
it's
around
100
pots
per
second,
except
for
this
part
here
here
we
actually
had
a
regression
and
text
to
prove
we
were
able
to
identified
and
folks
from
six
scheduling
fix
that
and
we
are
back
to
100
per
second
alright
other
example.
What
is
the
API
server
memory
usage
right
Oh?
So,
as
you
can
see,
it
went
up
recently,
we
should
look
into
that
or,
for
example,
for
performance
regression.
A
A
A
These
are
useful
for
sometimes
for
debugging
regressions,
especially
if
I
give
someone
change
code
and
something
now
started.
Taking
like
much
longer
than
you
can
take
a
look
at
the
CPU
profile,
and
now
you
see,
for
example,
that
hey
now
we
are
spending
80%
of
the
time
in
this
single
method.
Right
so
most
likely
something
is
wrong
there
and
we
should
optimize
that
and
yeah
the
flagship
tool
of
6
scalability
cluster
older.
It's
a
ringer
on
the
ammo
load
test
tool
is
key
for
cluster
order
to
our
and
to
escovitch.
A
Tests
were
written
in
go,
and
that
was
hard
to
maintain
her
to
extent.
For
many
reasons,
also,
it
was
impossible
for,
like
people
outside
of
six
scalability
to
contribute
or
like
to
create
their
own
scale
test.
So
we
had
this
idea
that
it
should
be
much
simpler.
Basically,
you
should
just
you
should
just
like
define
your
workload.
So
what
do
you
want
to?
You
should
define
a
state
that
the
cluster
should
reach
and
this
tool
should
bring
the
cluster
that
state
right.
So
that's
the
declarative
paradigm.
That
was
like
main
assumption
be
behind
cluster
loader.
A
I
will
explain
that
in
more
details
in
a
moment,
in
addition
to
this
decorative
part,
of
course,
there's
a
bit
of
imperative
part
because
it's
necessary
for
load
testing.
So
we
need
to
specify
how
things
should
happen.
For
example,
what
is
the
true
Polar's,
the
QPS
to
which
we
are
creating
the
objects
and
also
like
what
should
be
measured
during
the
execution?
So,
for
example,
that's
gotta
be
here
any
addition.
It
provides
crops
of
ability
for
free.
A
It's
actually
very
simple,
so
the
main
resource
is
a
test
and
test
consists
of
steps
right
steps
are
speed,
is
really
unlike
what
is
step
so
step,
for
example,
can
be
create
their
teapots
per
node
or
start
measuring,
API
call
latency,
or
stop
measuring
API
echo
latency
and
verify
that
it
was
below
this
threshold
right
or
create
this
other
object
and
stuff
like
that
right,
so
yeah,
that's
a
step
and
how
is
it
implemented?
So
a
step
is
either
collection
of
phases.
A
This
is
this
declarative
part,
so,
basically,
phase
defines
a
state,
after
should
reach
so,
let's
say
30
watts,
per
node,
10
services
and
stuff,
like
that.
Our
measurements
with
the
measurement
will
just
basically
anything
anything
else,
and
usually
we
use
measurements
to
measure
something
or
wait
for
something
and
I
will
show
you
an
example.
I
created
from
my
other
presentation.
A
Right,
so
it's
not
only
amyl,
it's
also
like
got
templates
which
allows
you
to
define
some
variables
and
do
some
crazy
stuff
in
you
like
loops
and
stuff,
not
in
this
example,
but
it's
doable
right.
So
the
leg
is
a
very
simple
example
that
will
create
30
pots.
A
sari
will
create
5
pots,
burn
out
and
then
scale
up
deployments
so
crazy
spots
using
deployment
tool.
We
can
do
like
step
by
step.
Maybe
so.
A
First,
we
are
starting
some
measurements,
we'll
be
measuring,
API
call
latency,
and
we
have
this
measurement
for
waiting
for
pots,
and
it
will
be
targeting
all
pots
with
this
app
is
my
deployment
label.
So
then
we
are
creating
deployments
tuning
set.
Is
this
like
imperative
part?
So,
basically,
how
fast
it
should
happen
will
be
using
uniform,
uniform,
5
GPS
and
we
are
pointing
to
the
deployment
Yamla.
We
is
a
very
humble.
A
So
if
I
get
regular
llamo
deployment
YUM,
but
again
you
can
like
use
some
some
templates
write.
Some
name
will
be
created
for
you
and
you
can
also
like
pass
your
own
parameters
like
number
of
replicas
or
requests.
You
can
do
like
anything.
You
want
to
do
right.
So
that's
very
generic,
and
here
we
are
Pacific
number
of
replicas
and
requests
CPU
memory
requests.
A
Then
we
are
waiting
for
posts
to
be
created.
Then
we'll
scale
deployments
up
by
two
parts
per
deployment:
again
we'll
wait,
and
then
we
will
delete
and
at
the
end
we
will
check
whether
API
call
agency
was
within
threshold
right,
that's
very
powerful
tool
to
to
do
load
testing.
You
can
easily
like
reproduce
your
clothes
here
and
test
it
and
we'd
like
encourage
you
to
do
that
and
share
these
CMS
with
us.
We
are
looking
for
inspirations
to
extend
coverage
of
our
our
performance
testing.
A
Other
features
of
cluster
load,
loader,
including
but
not
limited
to
automation,
is
faces,
so
basically
cluster
order
will
create
namespaces,
for
you
will
also
delete
them
after
the
test.
This
is
like
one
reason
like,
for
these
feature
is
to
make
sure
that
tests
are
hermetic,
so
we
can
run
the
same
test
multiple
times
on
the
same
cluster
and
they
won't
be
interacting
to
each
other
and
chaos
monkey.
So
it's
a
it's
a
feature
that
randomly
kills
nodes
is,
if
you
are
simulating
node
failures,
that's
useful
sometime,
because
that
happens
at
large
schedule
right
so
cross.
A
Looking
component
detection
is
a
new
one,
so,
in
addition
to
like
validating
some
services
allows,
we
are
also
checking
whether
run
any
system.
Pods
that
started
crash
looping
right
sometimes
it
happens.
Even
though
all
sm
eyes
are
satisfied
probes.
They
were
mostly
created
to
implement
the
new
network
network
as
allies.
So,
for
example,
we
have
these
probes
that
are
like
pinging
each
other
and
measuring
the
for
latency
and
stuff,
like
that,
Prometheus
monitoring
we'll
be
talking
about
that
in
a
minute
templating.
A
We
already
seen
a
bit
of
that
tuning
set
so
basically
saying
how
fast
an
object
should
be
created.
All
right-
and
here
are
some
links.
So
all
code
and
documentation
is
in
our
birthdays,
repo
under
the
cluster
loader
to
directory,
and
if
you
are
looking
for
the
load
and
then
seeds
definition
there
also
there.
A
Right
this
is
an
example
of
one
of
our
5k
note
performance
test
run
and
is
how
the
our
graph
another
scores
look
like.
It
looks
like
look
like,
so
we
have
like
requests
coming
from
cluster
loader
a.p,
echo
latency
and
many
many
more
things
like
unhealthy
notes.
We
have
like
some
girl
routines
in
cube,
API
server,
a
lot
of
stuff
about
like
watch
and
performance
info
requests,
requests
rates.
A
It's
super
useful
when
you
to
debug
like
when
something
goes
wrong
with
cluster
yeah,
some
memory,
usage,
CPU
usage
and
stuff,
like
that
all
right,
all
right
so
cube
mark
as
I
said
already,
is
a
tool
for
simulating
large
kubernetes
clusters
for
scale
testing
purposes.
So
the
idea
is
that
it's
actually
expensive
to
run
this
large-scale
test
right
and
in
scalability.
What
really
care
about
mostly,
is
the
control
plane
right,
so
so
the
master
node.
We
don't
really
care
like
a
but
like
what
happens
inside
docker
yeah,
some
extent
but
yeah.
That's.
A
That
was
the
idea
that
maybe,
if
we
like
kept
the
control
play
as
it
is,
and
look
for
some
saving
on
the
notes
I'd
like
started,
simulating
them.
We
could
do
it
much
cheaper
and
that's
true.
For
example,
as
you
can
see,
to
run,
5k
note
scale
test.
You
need
around
5,000
BC
peels
right
and
it
comes
mostly
from
the
notes
and
to
run
cube
mark.
You
only
need
seven
hundred
right.
A
So
we
have
Colo
cubelet
and
is
a
real
couplet
with
and
we
are
faking
some
of
the
giblet
components
so,
for
example,
self
docker
we
are
using
fake
docker
right
and
what
fake
doctor
does
is
whenever
you
ask
it
to
run
a
container.
It
will
just
reply
around
this
container
right,
but
it
will
do
nothing
more
holoku
proxy,
which
can
run
like
a
different
modes.
A
I
will
explain
that
in
a
moment
and
no
problem
detector,
which
is
like
a
standard
container
running
on
every
node
in
kubernetes,
and
when
it
comes
to
cube
mark
modes,
everything
boils
down
to
the
way
the
cube
proc,
the
hollow
cube
proxy
works.
So
one
way
is,
you
can
run
Colin
on
without
you
proxy,
but
that's
not
a
good
idea.
If
you
want
to
scale
test
queue.
A
Burnett
is
right
because
cube
proxy
actually
contributes
a
lot
to
the
to
the
load
on
the
API
server
in
general
right
because
for
every
endpoints
change
for
every
actually,
it's
like
starting
with
pod
changes
that
are
like
part
of
some
service
and
point
control,
will
update
the
endpoint
object
and
then
this
update
is
sent
to
every
node
right
to
keep
proxy
to
process
and
to
program
iptables.
So
if
you
get
rid
of
that,
then
actually
will
simplify
the
to
much
and
that
there
wouldn't
there
won't
be
a
much
gaining
like
doing
such
tests.
A
So
what
we
do
we
either
run
queue
proxy
with
real
proxy
or
we've
paid
proxy.
All
right
so
process
is
the
fact.
This
part
of
queue
proxy.
That
is
responsible
for
because
it
programming
the
network,
so,
for
example,
there
is
like
IP
table
fracture
that
processes
all
the
endpoints
object,
updates
and
services
updates
and
writes
IP
tables.
So
what
we
do
is
similar
to
to
follow
cube
left
it's
a
real
producer,
but
we
are
faking
everything.
What
is
everything
else
right?
So
we
are
mocking
IP
tables,
this
cattle
and
stuff,
like
that.
A
So
it's
still
like
computes
all
the
changes
and
all
the
IP
tell
groups
it
would
normally
written,
but
basically
just
sends
them
to
some
multi
interface
of
IP
tables,
so
they're
like
not
written
down
and
the
other
options
to
use
fake
rocks
are
which
basically
so
there
are
like
still
watches
for
for
endpoints
and
services,
but
we
are
not
computing
anything
in
in
the
coop
rocks
and
right.
So
that's
that's
the
idea
right.
So
we'll
have
real
control
plane.
We
are
a
master.
We
will
have
these
containers
for
simulating
node.
A
So
now
we
have
this
problem.
We
need
to
run
a
lot
of
containers
right.
What
do
we
do
if
we
only
have
the
tool
for
that
right?
I
can
see
that
some
of
you
get
my
joke,
that's
good!
Of
course.
We
use
given
IDs
for
that
right.
So
so
that's
actually
confusing
very
often
for
a
first-time
qumar
users.
Q
mark
actually
two
clusters
right.
You
have
like
you
can
start
either
way.
But
let's
say
you
started
with
the
empty
cluster
just
master
and
that
will
be
cute
McMaster
and
then
you
have
another
cluster.
A
Well,
we
will
run
all
these
hollow
nodes
right
and
you
configure
the
hollow
notes
to
connect
to
this
cue
mark
master
and
that's
it.
You
have
fully
functional
cube
McMaster
so
like
using,
let's
say
ten
nodes:
you
can
simulate
a
few
hundred
node
cluster
right
and
from
the
client
perspective.
Everything
is
transparent
because
client
talks
to
this
cube
mark
master
and
it
sees
a
cluster
with
with,
let's
say,
five
thousand
nodes,
even
though
there
are
only
like
hundred
in
this
real
cluster.
A
A
So
scholar,
it
is
sensitive
right,
it's
very
fragile
sometimes
are
like
changes
that
seem
like
totally
innocent.
For
example,
I
know
bumping
some
bumping,
K
log,
two
to
some
minor
version
right.
What
can
go
wrong
right
and
then
we
discover
that
it
totally
breaks
our
our
skaters
and
we've
seen
regressions
coming
from
pretty
much
everywhere
from
like
go
lonk
operating
systems,
controllers,
API
machinery,
scheduler
at
CD,
cubelet
and
basically
everywhere.
So
what
do
we
do?
We
often
like?
A
A
B
A
A
Gives
like
the
single
view
scheduled
single
API,
several
gets
like
up
to
600,000
goroutines
right
like
and
they're
like
allocating
a
lot,
and
that's
that's
when,
like
even
like,
tiny,
changing,
clunk
and
change
a
lot
overall,
all
right,
some
other
regressions,
so
operating
system.
They
were
like
a
few
in
costs.
I
think
this
one
listed
here
was
that
in
our
like
master
setup,
we
have
two
disks.
A
One
is
the
boot
disk
for,
like
all
the
logs
and
system
stuff,
and
we
have
a
separate
SSD
disk
for
for
HDD
and
usually
like
they
should
be
independent
of
each
other
right.
So
writing
to
one
disk
shouldn't
affect
operations,
the
other
risk,
but
sometimes
like
at
the
operating
system
level.
Something
can
go
wrong,
and
now
the
issue
was
that
during
the
lock
rotate.
A
Some
change
in
Cardenas
was
triggered
by
something
else
resulted
in
cordera,
started
using
more
memory
and
then
like
crushed,
looping
or
stuff
like
that,
and
then,
when
we
have,
you
have
companies
like
Cardenas
so
running
a
lot
of
nodes
which
are
crushed
looping
like
to
restart
a
watch,
as
they
start
sending
very
heavy
least
requests
to
a
server,
and
that
can
cause
a
lot
of
problems.
K
lock,
which
I
mentioned
even
like
something
that
Caleb
cannon
can
break.
Scalability
right,
I
think
that
back
there
was
that
they
started
producing
more
lines.
A
A
First
right,
the
resources
you
are
interested
in,
and
this
will
give
you
the
resource
version
and
starting
from
this
reservation,
we'll
be
watching
all
new
events
right
and
then
we
will
like
the
way
it's
implemented.
You
are
basically
restarting
the
watch
every
from
five
to
ten
minutes
and
if
there
is
like
a
lot
of
happening
on
the
API
server
or
if
your
client
is
slow,
it
may
have
been
done
if
you
want
to
restart
the
watch
with
the
recent
research
on
your
gut.
Api
server
may
reply
that
the
research
version
is
too
out
right.
A
I,
don't
know
about
it
anymore,
like
in
my
watch,
so
then
you
need
to
list
again
and
now.
The
question
is
like
what
resource
version
should
be
used
for
this
released
in,
like
I,
won't
be
getting
to
details,
but
there
was
the
fact
for
the
H
I
set
up
with
three
master:
it's
possible
that
one
master
was
lagging
and
basically
Eureka's
got
to
that
master,
and
then
you
would
get
a
like
old,
like
you
will
basically
go
back
in
time
right.
A
B
Okay,
yes,
so
the
other
thing
that
we
are
working
on
is
either
driving
scalability
improvements
or
coordinating
things
that
that
other
six
are
doing
to
improve
scalability
or
providing
them
requirements
to
to
today,
improvements
like
to
what
extent
to
what
like
level
of
scale
or
performance
they
should
they
should
improve
their
components
or
daily
features,
so
I,
probably
don't
not
going
to
like
go
into
details
into
all
of
them.
It's
like
if
you
are
interested
in
any
particular
one
I'm
happy
to
talk
more
about
any
of
those.
B
B
That's
that's
so
the
things
that
I'm
mentioning
are
from
from
ours,
pretty
much
what
happened
since
last
cube
gone
so
somewhere
since
left,
since
that's
last
cube
con
that
fixed
that
improved
situation
significantly,
but
we
are
not
we're
still
where
we
would
like
to
be
so.
We
are
looking
into
this
problem.
B
We
are
still
looking
into
this
problem,
how
to
make
it
work
even
better
the
new
events
API.
This
is
an
effort
that
is
already
lasting
for
two
hour
to
two
years
at
least,
and
it's
still
not
finished
like.
Basically,
we
have
the
API
the
new
API.
We
have
the
client
implementation
now
that
is
reducing
the
load
on
the
API
server
if
there
are
a
bunch
of
same
events
being
sent
from
from
the
same
component.
B
So,
for
example,
if
the
pod
is
crash
looping
we
are
sending
an
even
every
single
time,
so
so
those
can
kind
of
situations
we
we
change
the
API
so
that,
like
we
just
send
the
update,
send
update
the
event
like
every
five
minutes
or
every
ten
minutes
or
whatever
something
like
that.
Instead
of
like
writing
it
every
time
we
introduce
this
concept
of
like
even
serious
serious,
but
so
far
we
migrate
it
like
only
scheduler
to
the
new
events
API,
it's
it's
like
and
the
API
is
still
like
any
better.
B
So
there
is
still
a
bunch
of
work
to
be
done
here:
cheaper,
not
har
beats.
This
is
pretty
significant
improvement.
It's
probably
the
most
important
from
all
of
those
mentions
here,
so
so,
basically
previously
Cuba,
like
the
only
heartbeat
signal
that
cublas
was
sending.
This
was
this.
It
was
sending
like
the
note
status
updates.
Every
10
seconds
and
note
note
note
status
or
not
object
in
general
can
be
huge.
B
It
can
be
used
for
like
neither
election,
but
it
also
can
be
used
to
like
send
those
kind
of
heartbeats
and
we
changed
the
cubelets
so
that,
like
it's
periodically
changing
periodically
sending
those
those
those
small,
very
small,
tiny
little
little,
these
objects
and
and
sending
note
status,
is
only
if
something
changes
or
like
every
five
minutes
or
something
like
that.
Just
for
kind
of
backward,
compatible,
Heda
it
will
be
sent
and
that,
like
significantly
have
this,
this
is
going
to
GAE
like
in
117
release,
endpoint
slice,
api,
it's
it's
it's!
B
B
So
endpoint
slice
allows
you
from
performance
perspective,
to
shard
the
Charles,
sharp
the
service
into
or
sharp
the
endpoints
from
the
service
into
multiple
objects,
but
it
also
allows
to
effectively
or
effectively
effectively
implement
other
features
like
external
endpoints,
for
example,
for
services.
So
so,
while
the
motivation
like
the
the
most
important
motivation
at
this
point,
was
like
performance
and
scalability
like
we
spend
a
bunch
of
time
ensuring
that
it
will
also
help.
B
It
will
also
be
useful
for
other
networking
features
that,
like
they
want
it
to
do
at
some
point
anyway,
a
bunch
of
like
more
local
improvements,
in
particular
controllers,
a
bunch
of
improvements
in
API
server
itself
or
API
machinery
like
how
we
serialize
object
and
to
avoid
serializing
them
multiple
times
for
every
watch
independently.
For
example,
that
was
a
very
recent
change.
B
What
bookmarks
is
this
concept
of
of
a
special
event,
type
that
we
are
sending
over?
What
that
can
tell
you
like
everything
up
to
this
resource
version
we
already
processed
for
you,
the
fact
that
you
didn't
get
any
any
object
over
the
last
minute.
Five
minutes
over.
There
means
not
that
we
forgot
about
you,
but
it's
just
nothing
that
you
are
actually
requested.
B
Errors
were
watch
was
like
that
the
last
resource
were
the
last
object
or
the
publishers
version
of
a
flask
object
that
we
actually
observe
was
like
too
far
in
the
past
and
the
the
the
next.
The
last
thing
from
the
most
important
improvements
is
like
introducing
concurrent
weeks
in
at
CB
three
point
four.
B
So
so,
basically
previously
when,
whenever
you
are,
we
are,
we
are
sent,
we
are
doing
a
read
or
actually
I
should
say,
like
expensive,
read
in
a
CD
to
the
ED
CD
so
like
listening,
for
example,
listing
all
pots
or
listing
all
notes,
or
something
that
was
actually
blocking
any
other
operations
in
that
city
and
what
what
they
did
in
at
CD.
Three
point
four
is
that
we
are
blocking
only
for
very
short
amount
of
time
to
do
pretty
much
do
some
copy
on
on
the
write
copy
for
for
the
index.
B
Like
one
thing
that
is
probably
very
well
mentioning
here
is
that
we
are
working
on
like
scalability
approval
process,
I'm,
not
sure
if
you
like
heard
about
it
or
or
not
so
so,
as
part
like
we
kicked
in,
kicked
out,
kicked
in
this
process
of
like
production,
readiness
review
so
as
part
of
every
proposal
or
kubernetes,
the
Coburn
proper,
any
proposal
of
for
any
feature
to
kubernetes.
We
are
going
to
require,
probably
in
the
quarter
ish
from
now.
B
We
are
going
to
require
the
outer
of
this
proposal
to
also
fill
this
production
readiness
questionnaire,
where
they
will
have
to
answer
questions
like
how
how
your,
how
how
to
make
your
feature
like
production
ready.
How
should
it
be
monitored?
What
what
are
the
metrics
that
should
be
out
it,
how
it
should
be
upgraded?
B
Can
it
be
disabled
and
stuff
like
that,
and
as
part
of
that,
we
are
also
be,
will
also
have
a
section
about
scalability,
so
the
outers
of
the
feature
will
have
to
think
about
how
a
feature
will
influence
scalability
to
what
target
that
particular
feature
should
scale
wanted
to
break
the
existing
features
in
terms
of
scalability
and
stuff
like
that,
so
that
is.
That
is
something
that
we
are
currently
working
on.
B
There
is
like
we
are
in
the
pilot
phase,
where
we
are
when
we,
where
we
took
couple
existing
proposals
and
working
with
the
routers
to
validate.
If
our
proposal
for
the
questions
actually
make
sense
if
they
are
useful
and
if
they
are
not
like
too
hard
to
understand,
if
they
even
at
the
same
time
being
meaningful
for
for
us
to
catch
something.
B
Yeah
and
I
think
that's
mostly
it
like
how
to
get
involved.
If
you
are
interested
there
are,
we
are
trying
to
keep
at
least
a
couple
opened
issues
for
in
both
like
kaykai,
repo
and
perf,
this
repo,
if
there
isn't
anything
that
you
are
interested,
but
you
want
to
work
on
something
else,
like
police
pingas,
on
slack,
we
have
a
bunch
of
other
ideas.
We
just
don't
have
enough
time
to
file
issues
for
every
single
one
of
them.
So
we
are
more
than
happy
to
to
talk
about
you.
A
B
And
also
we
have
those
like
bi-weekly
public
meetings
were,
if
you
have
any
questions
other
or
you
want
to
contribute
or
you
don't
know
how
to
use
something
or
whatever,
like
feel
free
to
to
come
and
and
ask
those
questions
yeah
do
you
have
any
questions
like
I
think
we
have
a
lot
of
time,
but
I
think
that's
probably
the
most
important
part
because,
like
the
goal
was
to
like
ensure
that
you
actually
leave
this
with
what
you
wanted
to
hear,
not
that
we
thought
what
we
wanted
to
represent.
Yeah.