►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Scalability Intro – Shyam Jeedigunta, Google & Bob Wise, AWS (Any Skill Level)
About Bob
Bob is the Chief Cloud Technologist for Samsung SDS Research America, and opened the Seattle office for Samsung SDS as home base for the SDS Cloud Native Computing Team. Previously he was the CIO at MTN Satellite Communications, where he was responsible for product strategy and development, including global cloud computing and data center deployments. Prior to MTN, Bob was a VP Engineering in HP's public cloud effort, and led transition to Openstack. He made his way to HP via the acquisition of Seattle startup Melodeo, an early mover in cloud computing.
About Shiyam
Google Software Engineer
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
So
we
we
did
a
session
like
this
at
coop
con
austin,
the
last
one,
and
we
came
in
prepared
to
do
more
of
a
developer
discussion
and
we
had
a
whole
deck
and
just
stuff
we
were
gonna
talk
about
and
what
happened
was
a
lot
of
people
who
were
not
really
like
core
developers.
Wanting
to
talk
about
scalability
showed
up
a
lot
of
people
interested
in
building
and
running
large
clusters
showed
up.
So
we
decided
that
we
would
take
the
lesson
and
kind
of
shift
the
discussion
today
to
be
more
in
that
direction.
A
But
let
me
go
ahead
here,
so
the
idea
for
the
session
was
to
kind
of
introduce
SIG's
scalability,
provide
some
general
education
around
how
the
project
thinks
about
big
clusters.
How
we
do
testing
really
want
to
kind
of
be
oriented
towards
operator
concerns
and
questions
we'll
try
to
keep
it
a
little
bit
on
the
short
side.
A
So
if
there's
QA
last
time,
we
discovered
there
was
lots
of
QA
people
wanted
to
talk
about,
then
we
ended
up
getting
thrown
out
of
the
room
because
people
wanted
to
hang
around
and
talk
about
big
cluster
operations
experience
so
with
that
we'll
move
along
okay,
so
just
a
update
on
six
scalability.
So
if
you're
interested
in
big
clusters,
six
scalability
is
a
great
place
to
show
up
and
talk
about
things
and
ask.
So
this
is
the
working
home
by
the
way
the
six
scalability
is.
The
first
was
the
first
grenade'
sig
formed
has.
A
A
So
you
know
one
of
the
really
key
things
is
setting
SLO
s
like
well,
of
course,
as
a
cluster
gets
bigger
and
more
stuff
on
it,
it
may
get
a
bit
slower
and
slower
like
what's
what's
to
slow,
so
I
think
Shawn
was
going
to
talk
about
that
in
a
bit
primarily
concerned
with
control,
plane,
design
configuration
so
not
nodes
right.
So
there's
different,
you
could
think
about
really
big
nodes
and
nodes.
Starting
containers
is
a
performance
issue,
but
six
scalability
is
more
concerned
with.
A
How
do
you
build
the
control
plane
to
run
these
kinds
of
clusters?
So
the
concerns
are
number
of
nodes.
Pod
services
really
not
concerns
or
things
like
nodes
and
multi
cluster,
as
you
might
expect,
there's
a
very
heavy
crossover
between
six
scalability
and
sync
testing
and,
of
course,
practically
speaking,
there's
some.
You
know
very
critical
systems,
design
things
within
a
kubernetes
system
like
how
is
your
at
CD
cluster
set
up
and
configured
so.
A
Just
logistics,
if
you're
interested
in
joining
the
meetings,
so
we
meet
bi-weekly
thursdays.
This
is
essentially
just
before
every
two
weeks,
just
before
the
main
community
meeting
links
here,
you
can
always
find
me
afterwards
as
well.
This
is
all
in
the
kubernetes
community
project
as
well.
So
if
you
want
to
go
and
look
at
this
list
of
SIG's
and
also
there
is
a
group
but
most
of
the
actions
on
slack
in
terms
of
discussion
between
people
so.
B
So
hi
I'm,
Sean
and
I'm
one
of
the
developers
very
closely
working
with
six
scalability
and
I'm
a
engineer
at
Google
before
going.
This
actually
I
wanted
to
add
a
couple
of
points
here,
like
one
of
the
things
which
often
happens
and
like
I
see
happening
is
like
auto.
Scaling
is
also
not
a
concern
like
auto
scaling,
doesn't
really
fall
into
the
purview
of
scalability
and
there
a
lot
of
questions
and
of
people
and
people
looking
for
auto
scaling
related
questions.
So
this
is
not
really
going
to
address
those
that
stuff
around.
B
That
and
I
also
want
to
highlight
the
fact
that
scalability
as
I
see
it
is
more
of
a
horizontal
effort.
So
we
are
having
a
very
big
crossover
with
other
things,
because
we
are
trying
to
like
monitor
and
improve
performance
of
various
components
and
like
test
this
stuff
out
across
different
companies
in
the
cluster.
B
There
are
verticals,
and
so,
for
example,
if
you're
looking
for
something
like
performance
of
different
Network,
plugins
or
trying
out
or
trying
to
understand
the
performance
of,
let's
say,
custom,
schedulers
or
trying
out
different
container
runtimes,
and
these
kind
of
fall
under
separate
verticals
and
usually
different
SIG's.
We
are
trying
to
facilitate
them
with
scale
testing
and
often
consult
them
with
performance
related
stuff.
But
we
are
not
actively
testing
out
different
configurations.
We
are
usually
testing
out
a
kind
of
a
core
plain,
vanilla,
Kuban.
B
It
is
offering
so
I,
probably
start
by
explaining
what
is
our
role
like
one
of
our
roles
is
basically
to
perform
scalability
testing
more
so
as
part
of
CI
in
the
Kuban,
this
project
and
every
release.
We
are
running
a
set
of
tests
and
we
run
tests
at
different
scales,
starting
from
hundred
nodes,
all
the
way
till
five
thousand
nodes
and
across
different
providers.
B
It's
not
a
very
exhaustive
list,
but
we
are
right
now
running
our
tests,
mostly
on
GCE
and
some
1gk
and
there's
also
a
framework
called
cube
mark
that
I
will
come
come
back
to
later.
It's
kind
of
a
cloud
provider
agnostic
way
of
let's
say
running
scale
tests
using
simulated
clusters.
They
are
lightweight,
but
they
offer
a
lot
of
signal
on
the
control,
plane,
scalability
and
yeah.
So
these
tests
are
really
important.
They
are
blocking
the
release,
the
Kuban.
B
It
is
release
and
usually
we
kind
of
face
some
major
issues,
every
release
with
some
some
aspect
of
scalability
screwed
up
and
we
are
trying
to
fix
it
and
make
things
really
production-ready
in
the
project.
So
yesterday,
while
I
was
talking
to
Bob
of
I
I
kind
of
was
talking
about
this
mental
model,
the
name
performance
envelope
was
coined
by
Bob.
I
was
thinking
more
along,
something
like
a
scalability
space
or
something.
So
there
are
different
dimensions
in
which
you
can
try
to
run
the
class
like
try
to
push
the
cluster
to
towards
its
limits.
B
So,
for
example,
you
have
you
have
the
number
of
namespaces
the
amount
of
churn
you
can
create
in
the
cluster,
the
density
of
parts
you
can
run
in
and
and
different
stuff.
So
the
way
it
works
is
like
a
lot
of
these
limits.
Are
you
can
see?
Think
of
them
as
absolute
limits
in
terms
of
like
like,
if
you
had
to
keep
kind
of
like.
If
you
had
to
keep
other
things
under
control,
then
you
can
tweak
this
to
its
maximum,
but
you
cannot
have
all
the
maximum
at
the
same
time.
B
For
example,
we
say
we
support
5000
node
clusters
and
we
also
say
we
support
30
parts
per
no
100
pots
per
node
like
officially,
but
we
don't
support
both
of
them
simultaneously,
which
is
like
5000
nodes
and
100
pods.
So,
for
example,
the
photonic
no
stress
tests
that
we
are
running
have
like
30
pots,
running
per
node
and
which-
and
the
reason
is
that,
like
simultaneously
stressing
out
on
all
the
different
aspects
can
like
effect,
the
let's
say,
the
performance
and
the
control
plane
significantly.
B
Another
example
is
like
we
say
that
we
support
close
to
10,000
services
and
then
we
also
say
we
support
1,000
endpoints,
our
back
ends
in
a
single
service,
but
if
you
try
to
run
like
110
thousand
services,
each
with
1000
backends,
the
control
plane
will
most
likely
not
be
able
to
take
this
load
so
yeah.
So
these
are
a
few
examples,
so.
B
Yeah,
that's
pretty
much
what
I
wanted
to
say
here,
and
this
is
what
I
was
saying
saying
earlier-
that
if
you
want
to
like
really
skew
towards
one
particular
limit,
you
probably
will
be
more
limited
on
other
friends.
So
you
heard
when
you're
designing
your
own
cluster
as
a
user.
If
you
want
to
like
really
start
a
big
cluster,
you
need
to
carefully
consider
these
factors
like
what
are
the
things
that
you're
going
to
most
use.
What
is
the,
what
is
the
kind
of
churn
you're
going
to
create?
What
are
the
most?
B
How
big
are
your
nodes
and
like
what
is
the
density
you
want
to
have
pots
with,
and
things
like
these
and
we
are
usually,
we
are
usually
consulting
people
on
like
what
are
the
things
that
they
will
need
to
keep
in
mind
when
doing
such
things
and
the
limits
and
stuff.
And
if,
if
you
were
looking
for
creating
large
clusters,
you
probably
and
and
have
some
questions,
you
probably
should
contact
us
on
slack
channel.
We
have
a
six
scalable
distraction,
yeah
and
yeah.
This
is
more
about
the
testing
that
we
are
performing.
B
We
have
a
test
grid
which
is
kind
of
like
for
now.
It's
mostly
like
a
very
limited
knowledge.
Only
limited
set
of
people
know
what
exactly
we
are
testing
and
we
probably
should
one
of
the
things
that
we
want
to
do
is
like
make
this
more
open
and
make
this
more
visible
to
people
like
what
are
the
configurations.
B
We
are
testing
and
what
are
the
different
things
that
we're
that
you're,
measuring
and
and
part
of
that
is,
like
the
scalability
release,
validation
process
that
you
introduced
like
a
couple
of
releases
back
where
we
are
for
each
release.
We
are
giving
a
spec
of
what
is
the
configuration
we
test
it
out
and
it's
like
I
was
saying
earlier.
It's
mostly
a
plain
vanilla
configuration
because
you
cannot
really
test
out
on
all
combinations
across
each
verticals,
because
then
you'll
have
a
combinatorial
explosion
of
cases
to
test.
B
But
if
you
are
interested
in
testing
some
particular
feature
or
like
you're
interested
in
looking
like
how
does
a
particular
configuration
perform,
we
might
be
able
to
assist
you
on
the
slack
channel
or
or
mailing
list
yeah.
So
the
testicle
has
it's
mostly
border
tests,
but
maybe
I
can
quickly
show
you
the
performance
dashboard.
We
display
all
the
results
of
our
performance
runs.
B
So
this
is
just
an
example
graph,
where
we
are
kind
of
running
benchmark
tests
against
the
shed
Euler
and
and
like
seeing
what
is
the
latency
of
the
shedule
operations,
and
there
is
a
array
of
different
combinations
like
of
tests.
Are
you
running
like
like
I
was
Aria
saying
we
run
on
GC,
mostly
on
Google
Cloud,
actually
pretty
much
almost
completely
on
Google
Cloud
right
now,
all
the
tests-
and
we
hopefully
will
be
having
these
soon
on
other
clouds
as
well
so
yeah.
So
this
is,
for
example,
let
me
show
maybe
yeah.
B
So
this
is,
for
example,
the
port
necessary
yeah.
This
is
the
hundred
node
cluster
test.
Where
we're
running
our
density
test,
we
primarily
have
two
tests.
One
is
the
density
and
the
other
is
the
load
which
is
not
so
exhaustive,
as
you
can
imagine,
and
there
are
probably
more
stuff
that
we
can
test
out.
B
B
Maybe
it's
City
yeah,
that's
the
wrong
it
city
yeah.
So,
for
example,
you
can
look
at
the
resource
usage
of
control
in
components,
for
example.
This
is
how
the
HCV
memory
and
CPU
looks
like.
So,
if
you
plan
to
start
100,
not
cluster
and
kind
of
how
this
much
of
density-
and
then
you
probably
can
expect
the
memory
usage
of
HDD
to
be
within
this
range.
Of
course,
it
can
vary
on
several
other
factors
like
what
is
the
kind
of
requests
you
make
and
like
how
frequently
you
do
stuff
and
things
like
these.
B
B
Yeah
and,
and
and
the
other
thing
is,
we
are
also
collecting
performance
profiles
of
various
kind
of
things,
and
this
is
this
is
part
of
our
bigger
effort
to
develop
more
tools
to
be
able
to
measure
and
understand
performance
in,
like
we
have
a
better
understanding
of
how
these
components
are
working
and
like
part
of
that
is
collecting.
Let's
say
we're
collecting
the
the
CPU
and
memory
profiles
of
the
API
server
and
we
some
plan
to
have
just
more
extensively
covered.
B
Yeah,
so
this
is
the
these
are
tips
for
cluster
of.
So,
if
you're
actually
planning
to
start
a
really
big
cluster,
then
there
are
a
few
things
that
you
need
to
keep
in
mind.
This
is
not
close
to
being
exhaust
exhaustive,
but
these
are
just
few
things
that
we
came
up
with
while
creating
the
slides-
and
you
probably
would
want
to
read
scalability
good
practices,
talk
which
has
more
details
about
what
are
the
things
you
should
keep
in
mind.
So
a
few
examples
are,
let's
say
this
is
actually
a
this.
B
The
first
one
is
based
on
a
real-life
incident,
I
think
which
we
saw
a
few
days
earlier.
So
if
you're,
creating
a
lot
of
jobs
and
the
jobs
are
like
frequently
getting
finished,
and
there
is
the
huge
throughput
of
jobs
creating
and
they're
finishing
the
these
job
objects,
don't
naturally
get
deleted,
which
is
kind
of
a
problem
that
is
specific
to
jobs,
because
they
usually
run
to
completion
and
all
time
these
jobs
start
occupying
a
space
of
your
HCD,
and
this
can
eventually
cause
memory
outage
issues
for
HCD.
B
Yeah
and
and
another
thing
which
is,
which
is
also
kind
of
happens
from
time
to
time,
is
like
spamming,
api
server
with
a
lot
of
calls
or
some
particularly
heavy
calls.
So,
for
example,
if
you're
trying
to
list
all
the
events
from
a
cluster
events
are
like
in
huge
numbers
and
they're
committed
by
almost
all
the
control
being
component
component
set
at
a
reasonable
rate.
B
It
has
a
limited
buffer
and
it
can
only
take
have
as
many
as
those
requests
in
flight
at
any
given
point.
So
such
long
calls
can
actually
take
up
the
buffer.
So,
as
a
user,
when
you're
operating
the
cluster,
or
even
you
have
some
components
which
are
talking
to
the
api
server,
you
usually
want
to
be
careful
about
not
creating
too
much
pressure
on
the
api
server
with
calls.
So
these
are
things
that
are
potential
areas
of
works.
B
For
example,
one
of
the
things
is,
we
would
probably
want
to
have
some
kind
of
rate
limiting
for
calls
for
api
calls
so
that
people
are
not
able
to
like
one
single
client
or
maybe,
as
a
set
of
clients
cannot
alone
overwhelm
the
api
server
and,
for
example,
the
first
one
they
were
like.
We
are
creating
a
lot
of
objects
of
a
particular
type.
One
of
the
things
we
probably
would
one
of
the
ways
to
solve.
B
It
is
have
proper
quotas
in
place
to
be
not
able
to
create
too
many
objects
in
your
cluster,
so
yeah
and
yeah.
The
the
more
important
part
in
this
slide
that
I
want
to
talk
about
is
cue
mark.
So
we
have
this
framework
called
cue
mark,
which
is
basically
our
way
to
perform
large
cluster
tests
without
actually
starting
large
clusters.
So
this
this
is
like
a
simulated
cluster
where
you
start
a
cuban,
it
is
master
separately,
but
you
don't
really
create
real
nodes,
but
you
actually
fake
them.
B
So
you
create
what
we
call
as
hollow
nodes,
which
are
basically
pods
on
top
of
another
cluster,
and
each
pod
has
a
hollow
version
of
cube,
light
cube
proxy
candy
and
basically
the
node
agents,
hollow
versions
of
those
which
actually
don't
do
real
work,
but
they
still
talk
to
the
API
server
as
if
as
if
they're,
real
components
doing
real
work.
So
so
by
this,
we're
able
to
actually
start
really
big
clusters,
which
is
as
much
as
five
5,000
node
clusters
on
like
any
an
8080.
It's
something
like
on
700
course.
B
So,
if
you're,
a
user
trying
to
test
performance
of
something-
and
you
don't
have
access
to
too
many
too
much
code-
are
too
many
computer
resources.
This
is
something
you
will
probably
want
to.
Try
out
and
people
are
I
have
seen
people
increasingly
showing
interest
in
this
there
there.
So
initially
we
had
this
just
it
was
possible
to
run
this
only
on
GC,
and
then
we
made
it
kind
of
cloud
provider
independent
and
then
there
are.
There
are
a
couple
of
people
who
even
try
to
do
this
on
on
Prem.
A
B
I
recently
wrote
it
is
basically
kind
of
composition
of
small
stories
around
whatever
the
kind
of
scalability
regressions
that
we
saw
in
the
past
in
the
project
and
scalability
like
you're
saying,
is
a
is
a
very
cross-cutting
area
and
a
change
in
any
particular
component
of
the
system
can
potentially
break
the
scalability
of
the
of
the
entire
cluster
and
scalability
in
that
sense
is
actually
a
very
sensitive
characteristic.
So,
for
example,
let's
say
if
if
the
cubelet
starts
making
one
extra
call,
let's
say
every
one
minute
on
a
five
thousand
node
cluster.
B
This
is
like
five
thousand
calls
every
one
when
it
which
is
like
you
know,
close
to
180,
QPS
or
or
whatever.
So
you
are
adding
a
whole
lot
of
QPS
on
the
ApS
order
and
yeah.
So
in
general
there
are
very
interesting
cases
there
that
you
might
want
to
look
at
and
that
that
also
kind
of
demonstrates
like
how
six
scalability
is
kind
of
collaborating
with
other
six
on
on
areas
of
work,
especially
around
performance
and
like
trying
to
point
out
to
other
six
are
okay.
This
is
causing
an
issue.
B
So
all
these
all
so
99
percentile
of
each
kind
of
call
should
be
less
than
one
second,
and
this
is
the
other
test
that
you
are
performing,
which
is
called
the
load
test
and
yeah.
So
those
are
basically
the
two
sls
we
have
in
general.
We
realized
that
these
are
solos
not
completely
informative
and,
in
most
cases,
are
not
enough,
because
there
are
some
aspects
of
the
system
which
are
not
completely
covered
or
that
are
left
out
by
these
s.
B
Ellos
and
it's
possible
there,
it's
possible
to
have
scenarios
where
the
oscilloscope
tup,
but
still
there
is
something
going
bad
in
the
cluster
and
something
really
bad.
So
one
of
the
efforts,
one
of
the
fronts
that
we
are
working
on
is
to
trying
to
define
new
SL
s
and
s
ellos,
which
are
more
fine-grained
and
cover
different
interactions
in
the
system
and
voytek,
who
is
our
other
sig
lead
beside
bob
is
like
doing
quite
some
work
on
this
and
he's
come
up
with
a
proposal
for
a
set
of
a
slice
and
this
dog.
B
So
if
you're
actually
trying
to
know
what
are
the
limits
of
various
kinds
of
objects
we
can
have
in
in
Cuba
notice,
then
this
talk
is
kind
of
as
far
as
I
know,
the
one
main
source
of
truth
so,
like
I've,
said
this
again
is
kind
of
like
this
basically
shows
the
limits
along
each
front,
but
not
like
a
combination
of
all
the
limits
so
yeah.
You
might
want
to
take
a
look
at
this
if
you're
trying
to
like
use
really
big
clusters.
B
Yeah,
so
this
is
about
what
do
we
plan
to
do
in
future?
Some
high
level
areas
of
work,
so
one
of
one
of
the
things
that
we
are
keen
to
look
at
is
like
having
tests
that
are
running
real
workloads,
as
opposed
to
like
simple
tests
that
you
are
running
right
now,
which
is
just
starting
passports
and
and
not
really
like
measuring
performance
within
in
more
realistic
production
like
setups,
we're
actually
trying
to
up
the
game
by
not
by
by
not
doing
this
ourselves,
but
actually
by
trying
to
create
a
tool.
B
There's
this
tool
called
cluster
loader
that
we
are
trying
to
work
on,
by
which
we
want
to
like
empower
normal
developers
to
be
able
to
start
there
on
workload
to
perform
their
own
load
tests,
be
able
to
configure
what
is
the
kind
of
things
that
you
like?
What
is
the
kind
of
tests
you
want
to
do?
What
exactly
do
you
want
to
load
tests?
So
if,
if
you're
really
interested,
then
interested
in
load
testing,
some
particular
setups
watch
out
for
this?
B
Yeah-
and
the
other
thing
I've
already
mentioned,
is
having
more
improved
and
better
SLI
xerneas
loss
at
the
third
main
area
is
developing
tools
and
frameworks
which
basically
help
us
understand
how
understanded
a
better
detail.
The
tiny
aspects
of
performance
and
like
be
able
to
understand
what
exactly
is
happening
at
different
places
and
the
and.
B
A
A
You
could
do
the
same
for
a
performance
result,
and
the
idea
here
would
be.
This
is
kind
of
along
the
line
of
tools
and
frameworks
for
empowering
users
to
figure
out
what's
going
on
in
their
own
clusters.
With
regard
specifically
to
these
kinds
of
scaling
tests,
so
I
think
that
was
a
our
last
slide.
I
think
so.
Okay,.
B
C
C
A
C
Isn't
really
good
story
for
rightly
meeting
now
the
only
right,
Freight
limited
right,
limiting
that
we
are
currently
doing
is
like
client
site
right,
GPS
limits
and
number
of
in-flight
requests
in
API
server.
We
would
like
to
do
something
like
rightly
meeting
based
on
like
use
it
or
amount
of
resources
that
API
server
has,
but
that's
not
even
yet
designed
so
I.
Don't
expect
it
will
happen
anytime
soon,
ish,
that's
the
food!
That's
the
plan
that
we
have
like
medium
long
term,
but
I
can't
promise.
C
Yes,
I
think
that
at
least
short-term
it's
it
sounds
reasonable.
More
longer-term
we
would.
We
would
like
to
have
at
least
things
like
fairness
with
respect
to
API
calls.
We
would
like
to
have
priorities
of
API
calls,
for
example,
like
note
heartbeats
should
be,
should
should
have
some
higher
priority
because
like
if
they,
if
we,
if
note
our
cubelets,
are
not
able
to
send
heartbeats,
then
it
can
for
a
longer
time.
It
may
result
in
pot
evictions,
which
is
like
obviously
a
bad
experience.
C
B
If
you
have
a
really
huge
churn,
then
there
can
be
like
many
cubelets
from
across
the
cluster.
They
were
trying
to
simultaneously
send
a
lot
of
this,
and
actually
like
from
my
experience,
I
I've
seen
a
case
where,
like
someone
was
trying
to
like
upgrade
a
deployment
and
a
really
huge
deployment
and
they
kind
of
upgraded
it
without
having
any
limits
on
max
available
or
max
unavailable
overall,
which
basically
led
to
a
situation
that
a
huge
number
of
those
pods
are
being
deleted
and
created,
which
kind
of
like
hurt
the
API,
server
and
I.
B
C
B
Are
pushing
that
200
in
our
5,000
node
tests,
but
for
production
there
are.
There
may
be
some
cases
that
we
still
have
not
explored
enough
to
be
able
to
confidently
promise
that
100
will
work
always
so
we
are
kind
of
bumping,
the
ups
200
in
both
controller
manager,
and
she
do
that.
Basically,
we
quickly
create
parts
and
quickly
schedule
them,
but
yeah,
but
that's
a
because.
A
Those
are
paused
containers
as
well.
It's
kind
of
a
it's
a
synthetic
case,
so
in
every
in
a
real
system,
you
would
expect
that
you're
not
gonna
actually
be
able
to
start
the
containers
just
like
if
you're
gonna
pull
down
beep.
If
you're
gonna
be
pulling
down
container
images
that
you,
your
total
system,
is
probably
gonna
be
dominated
by
that,
maybe
not.
C
But
it's
odd
to
do
this
like
there
is
there
isn't
any
one
single
universal
answer
to
this,
because
it
really
depends
like
on
how
big
the
cluster
is
like
how
big
your
master
machine
is,
or
whatever
machine
where
your
control
plane
is
running.
If
throwing
more
resources
helps
with
many
different
components
here
so
I
it
I
would
I
would
say
that
proudly
for,
like
relatively
small
clusters,
we
can
go
even
higher
than
100.
Even
now,
if
you
throw
enough
resources
to
control
plane.
B
A
The
5,000
ode
cluster
level,
you're
you're,
starting
to
trip
across
to
think
to
factors.
So
one
factor
is
the
number
of
companies
or
groups
that
are
actually
interested
in
running
at
that
scale,
and
the
other
is
you're
starting
to
trip
across.
Like
failure
domain
considerations,
you
know
you're
talking
about
a
lot
of
eggs
in
one
basket
and
you're,
probably
starting
to
get
into
the
zone
where
you
should
be
thinking
about
a
Multi
cluster
design
and
really
this
like,
though,
this
whole
conversation
really
at
that
kind
of
scale.
A
You
should
probably
be
testing
like
understand
exactly
what
your
profile
is
in
terms
of
container
starts
and
notes,
and
these
other
characteristics
so
that
you're
able
to
qualify
the
configuration
you're
using
to
ensure
that
your
production
system
is
working
properly,
but
this
comes
up.
I
mean
we
have
to
be
I.
Think
in
the
last
session
and
the
last
session
in
Austin,
we
had
people
asking
about.
You
know:
25,
30
thousand
node
kinds
of
system.
A
Thousands
like
that,
well,
let's
strike,
but
let's
go
to
a
thousand
thousand
nodes,
looks
like
750
hundred
nodes.
How
many
people
yeah
so
we've
got
I
mean
even
in
this
room
right
I
mean
we
have
very
steep
dropoff,
but
that
said,
it
is
a
good
point.
Right,
I
mean
there
are
definitely
operational
advantages
you
get
by
running
fewer
bigger
clusters,
so
that
does
tend
to
skew
things,
maybe
in
the
bigger
direction.
Over
time.
A
B
Yes,
that's
a
very
good
question,
and
this
is
a
discussion
we've
been
having
like,
but
maybe
we
can
take
this
offline
Josh
if
this
is
like
not
if
this
is
like
query
developer
centric,
because
this
is
a
long
discussion
in
there
is,
there
is
no
well-defined
like
a
to
well-defined
process
right
now.
We
just
have
like
scalable
to
release
blockers
and
just
some
time
closed
before
the
release
with
like
trying
to
like
put
all
our
energy
into
it
and
fix
stuff,
which
is
not
the
best
way
to
do
it.
B
C
C
C
Maybe
in
general
we
are
running
everything
on
a
single
machine
yeah
or
like
either
like
single
master
machine
or
like
couple
masters,
but
then
still
all
the
master.
All
the
control,
plane
components
are
are
running
on
this
single
machine
I
mean
we
are
not
splitting,
for
example,
at
CD
to
separate
machines
or
like
schedulers
or
or
other
things
like.
C
It
really
depends
on
the
environment
like
if,
if,
for
example,
your
eye
ops
limits
are
bounded
to
to
the
machine
itself,
then
potentially,
yes
like
we
have,
since
we
are
doing
this
on
GCE,
we
we
can
tweak
some
of
those
at
least
independently,
like,
for
example,
I
ops
are
strictly
correlated
with
the
disk
size
which
is
independent
from
the
CPUs.
So
we
can
throw
more
I
ops
by
increasing
disk
size,
and
we
we
did.
We
we
do
such
stuff
and,
like
so,
and.
B
C
A
You
look
at
the
provider
dock
that
we
linked
to
there
a
lot
of
the
construction
of
the
provider.
Dock
was
based
on
getting
a
bunch
of
feedback
from
people
wanting
big
clusters
and
in
that
case
the
imitation
is
to
run
a
separate,
a
TD
plane,
but
some
of
that
is
really
when
you're
starting
to
run
clusters
that
size.
Some
of
your
considerations
are,
you
know,
fault
tolerance,
as
opposed
to
necessarily
strict
performance,
so
yeah.