►
From YouTube: 2017-02-23 Kubernetes SIG Scaling - Weekly Meeting
Description
2017-02-23 Kubernetes SIG Scaling - Weekly Meeting
A
B
A
A
A
Start
with
that,
I
think
that
I
mean
there
was
a
lot
there's
a
lot
going
on,
but
I
think
the
thing
that's
of
the
most
relevant
interest
is.
There
was
a
lot
of
discussion
at
the
board
meeting
and
then
also
at
the
also
at
the
TOC
meetings
around
the
use
of
the
CNCs
bare
metal
cluster,
and
if
the
utilization
on
it's
been
very
low,
I
think
the
Red
Hat
folks,
notably,
have
done
probably
the
most
with
it.
There's
a
bunch
of
issues
with
how
to
how
to
provision
it.
A
So
things
like
having
a
goober
Nettie's
on
bare
metal,
Cabrini's
on
OpenStack,
etc.
So
the
it
seems
like
there's
going
to
be
a
lot
more
energy
on
that,
but
I'm
now
trying
to
just
send
a
long
email
to
the
Intel
folks.
I
looped
in
Chris
Wright
from
red
hat
as
well
and
I
think
we're
going
to
try
to
try
to
see
if
we
can
improve
the
tooling
or
the
governance
around
the
big
big
bare
metal
cluster.
So,
if
anyone's
interested
in
that,
we
can't
let
me
know
and.
A
Or
the
the
one
of
the
original
goals
was
to
be
able
to
scale
test
bare
metal.
However,
there
is
certainly
a
contingent
and
I'd
say
I'm
supportive
of
this,
the
the
notion,
the
notion
that
it
that
at
least
some
portions
of
it
should
be
carved
off
for
CI
for
CI
for
various
projects,
which
I
think
is
thus
I-
think
that's
a
sensible
compromise.
So
that
means
yet.
You
have
I,
don't
know
800
nodes
available
for
big
scale
testing
and
a
couple
hundred
had
been
carved
off
into
smaller,
see
I
clusters.
D
D
A
That
there
are
two
issues:
one
is
the
amount
of
paperwork,
but
the
other
is
the
amount
of
paperwork
kind
of
per
server.
So
the
the
anecdotal
reports
were
that
if
you
that,
because
you're
essentially
asking
a
manual
group
to
go
provision
servers
for
you
by
hand,
you
would
say:
hey
I
need
X,
and
then
they
would
run
off
for
days
of
days
days
to
prep
it
so
it
which
is,
you
know,
for
a
cloud
native
group
a
little
silly,
so
yeah
anyway.
I
certainly
didn't
want
to
take
up
the
whole
meeting
with
that.
E
Echo
so
there
is
a
couple
of
checklist
items
to
be
taken
care
of,
but
it's
my
understanding
that
Matt
from
google
harrah's
already
running
through
those
checklist
items
we
do
need
to
go
through
the
feature
of
auditory.
That's
my
bad
they're
just
doing
checking
off
all
the
bits
so
that
the
PNG
is
happy
for
this
release,
but
for
the
most
part
from
the
checklist
items
that
I'm
aware
of
are
pretty
much
just
the
manual
roll
back
and
we'll
forward
tests.
D
D
E
E
A
C
Ideas
also
extended
life,
not
sex,
hope
you
will
be
here.
So
I
would
have
one
thing
that
we
unlike
said
last
week
that
GM
as
like
as
a
starter
project,
was
working
on
a
tool
to
compare
like
environments
like
tests
accidentally
can
load
test,
run
on
different
environments
to
do
and
compare
the
results
and
see
how
those
environments
are
to
each
other
so
that
we
had
working
panda
and
I
would
see
em
to
the
little
bit
like
okay.
G
So
this
this
I,
don't
think,
is
something
that
everybody
would
be
interested
in,
but
I
did
want
to
raise
it
to
see.
Who
would
so
as
as
we've
done
more
to
automate
running
open
shift
on
GCE
we've
had
some
discussion
about
like
what
the
ideal
architecture
would
be
so
open
shift
added,
out-of-the-box
very
early
on
XFS
project
quota
for
md
ders,
mainly
because
it
was
a
point
of
attack
for
multi-site
cluster
and
right
now.
There's
a
couple
proposals
being
fish.
G
Has
a
proposal
open
that
will
probably
be
worked
on
in
17
from
the
note
sig,
which
is
about
local
volume
access
and
it
sort
of
people
who
work
out
and
touched
on
things
like
we
want
I,
oh
isolation
on
nodes
between
docker
cubelets
and
the
operating
system,
and
then
tenant
workloads,
I
always
and
come
from
yeah
well
in
the
challenge,
ultimately
comes
down
to
there's
several
different
classes
of
actual
disc
workload
and
just
like
CPU
memory
quoz
cheering
in
practice.
We
have
this
today,
but
people
really
are
necessary.
G
I
was
trying
to
get
answered,
which
was
like
just
starting
from
the
gcb
perspective.
When
you
have
these
kind
of
three
classes
of
workload,
you
know
special
disc,
which
is
PVD.
You
attach
the
system
kameez
where
you're
essentially
bringing
in
a
holder
disc,
and
you
know
you
might
have
a
different
sort
of
network
bandwidth
for
them,
but
you
also
don't
want
the
operating
system
choking
on
you
know
having
the
same
I
oak
use
for
those
operation
or
for
those
disks.
G
G
We
generally
recommend
it
be
on
a
separate
disk
or
device
hold
on
metal
and
on
cloud,
but
effectively
empty
der
for
the
workloads
on
the
note
being
treated
as
a
best
effort
from
an
IO
perspective,
which
is
we
set
a
hard
limit
by
a
xss
quota
for
every
unique
user
ID
and
because
we
force
all
containers
to
run
as
different,
unique
user.
Ids,
can
these
leverage
that
ambitious
proposal
there's
more
work
to
be
done
for
XSS
project
quote
and
all
that
I
did
one
dive
too
much
an
FS
group.
G
The
group
ID
can
be
made
unique
across
all
namespaces
on
a
cluster,
for
instance.
Today,
and
so
that's
another
angle
or
when
you
look
at
what
gets
written
I,
don't
think
the
point
of
them
either
and
the
proposal
talks
about
this
is
to
be
totally
isolated,
is
essentially
it's
a
slush
capacity
or
I'm
scratch
storage
and
there
might
be
higher
levels
of
local
storage
in
the
future.
Fish
has
a
number
of
proposals.
My
question
has
basically
boiled
down
to
I
was
trying
to
figure
this
up
in
GTE
experts
was
on
something
like
GCE.
G
D
G
D
G
Id,
the
interesting
thing
is
like
I
think
this
fits
a
little
bit
of
leg
fees
if
you're
not
running
a
lot
of
really
dense
workloads
and
like
I'm,
trying
to
look
I'm
kind
of
teasing
around,
like
as
the
question
of
is
there
a
sweet
spot
for
reasonably
sized
nodes
for
reasonable
app
workloads?
Because
I
would
say
you
know
the
vast
majority
of
work.
Those
people
are
running
a
cute
24
and
three
core
tight
CPU
workloads
that
aren't
you
know
they
might
have
some
tenancy
but
they're,
just
not.
G
E
Think,
right
now,
too,
is
that
we're
running
into
fundamental
limits
that
we're
not
tracking
records
you
go
ahead
turns
is
for
right.
We
don't
have
senses
for
I
ops,
we
don't
have
sensors
for
network
bandwidth,
you
know
and
we're
fundamentally
limited
on
both
front
when
you're
starting
to
run
it
providers
one
different
levels.
G
And
I-
maybe
this
isn't
even
oppression
for
this
group,
but
just
while
I
had
folks
who
knew
a
little
bit
about
the
juicy
stuff
like
if
we
did
single
a
single
PD
with
logical
volumes,
can
we
get
any
kind
of
reasonable,
os-level,
I,
ops,
control,
at
least
in
terms
of
Q
deaths
and
dispatched
I
ops
over
a
window?
Or
do
we
really
have
to
rely
on
PDS
going
forward?
It
was
in
GC
environment
to
get
aisle
isolation
like
that
I.
C
Pulled
out
on
the
level
question
for
me,
toss
away
by
this
is
I
I,
think
he
dumped
it
and
I
think
I,
just
an
abstraction,
so
I
like
to
just
would
scale
exactly
what
size
I.
Never
think
this
should
scale
the
size
of
the
PD.
So
having
blonde
like
my
multiplicative,
one
should
will
make
a
difference
so.
D
D
Tell
you
claiming
that
as
the
the
folks
who
did
the
virtualization
stuff
for
GCE
like
made
local
SSD
work,
they
had
to
tune
the
hell
out
of
the
the
para
virtualized
I'll
have
em,
and
so
there's
no
bottlenecks
there
right,
and
so
it
would
be.
My
guess
happy,
you
know,
I'm
talking
sort
of
you
know
in
guess
amiss
here
and
I
think
it
really
just
comes
down
to
the
disc
and
the
limits
that
are
exposed
sort
of
outside
of
it
and
those
things
are
totally
blind
in
terms
of
which
across
right
inside,
though
yeah.
G
I
kind
of
had
suspected
that
an
inch
of
Tim's
point
I
kind
of
gotten
the
feeling
that
of
the
last
I,
let's
say:
10
major
production
level,
performance
problems,
you
know,
half
of
them
were
stupid,
cpu,
fast
loops
and
the
other
half
were
io
problems
and
the
echo
problems
are
basic
story
is
well.
Maybe
we
can
do
less
I/o
operations,
sometimes
without
really
getting
any
better
at
it
like.
So
it
we
end
up
being
very
the
other
than
doing
less
I.
Oh,
we
don't
have
a
lot
of
jobs.
The.
E
Super
small
I
Oh
bandwidth
limits,
but
it's
not
enforced
and
not
enabled,
and
that
last
time
I
checked
and
trying
to
system
was
down
below.
Well,
they
had
all
kinds
of
issues:
I'm
trying
to
recall
what
the
problems
were,
but
we
just
probably
use
these
back
way
back
in
grid
days
to
limit
the
number
of
people
who
could
do
sort
of
local
storage
when
they're
trying
to
do
like
HDFS
peck
things.
D
E
D
H
D
G
The
current
steal
from
sig
note
is
that-
and
this
is
mostly
vicious
perspective
of
some
facts
from
Dawn-
is
that
the
only
real
isolation
is
device
and
everything
else
ultimately
doesn't
work
and
oh
yeah
block
aye
yo
is
trash.
That
was
the
other
one.
A
black
iose
groups
are
trash
or
iose
groups
in
general
or
trash,
and
so
I
do
think
that
then,
from
to
the
scale
team,
I
guess
my
question
would
be:
how
do
we
better
figure
out
like
against
turns
point?
G
E
Some
some
other
systems
have
more
deep
level
of
introspection
on
the
current
usage
statistics
and
they
view
things
like
they
publish
for
cpu.
They
publish
like
a
load
average
right,
and
you
know
your
base
about
Pierre
and
scheduling
back
amberwood
adverti.
They
do
do
similar
things
and
I
up
track.
So
that
way
you
can
prevent.
If
somebody
is
going
to
be
a
heavy
mist,
loser
they
would
take
into
account
and
what
one
thing
they
typically
do
will
be.
They
rank
so
they
ranked
by
losing
the
most
mobile
apps.
E
D
E
D
So
the
big
thing
is
that,
if
you're
running,
if
you're
running
big
machines
on
G
see
you,
then
then
the
limits
end
up
being
per
device
and
you
can
actually
use
the
device
as
the
limit
there.
If
you're
running
small
machines,
you
know
you're
just
you're
going
to
be
I/o
limited,
regardless
of
what
you
do,
and
so
now
you
essentially
just
like
how
do
you
actually
give
you
that
up,
right
and
and
just
you
know,
see
groups
and
I
ops,
limiting
and
cross
on
disk?
B
D
D
So
you
know
Google
all
like
here's.
One
of
the
truisms
is
that
when
you're
doing
like
a
network
disk
system-
and
you
have
a
lot
of
clients
all
calling
in
essentially,
there
is
no
sequential
writes
right.
Everything
ends
up
being
random.
But
no
there's
no
sequential
reads
right,
because
everything
ends
up
being
random
right,
which.
D
Best
case
with
has
been
the
average
cases,
it's
more
real,
it's
more
predictable
right.
So
that's
you
know,
I
mean
the
long
way
around
is
to
say
you
know,
take
all
your
disk
in
your
cluster
and
then
run
some
sort
of
distributed
storage
system.
On
top
of
that,
and
then
we
expose
that
to
things
using.
You
know
higher
level
things
that
can
cause
yeah.
You
know
statistical
averaging
across
stuff,
but
we're
not
going
to
get
anything.
You.
G
E
Didn't
ask
America
like
this
is
a
signal
problem.
There
is
one
type
of
signal,
at
least
from
the
node,
that's
giving
me
some
type
of
updated
blowing
average.
They
couldn't
want
to
load
average
for
what's
going
on
and
just
publish
that
as
part
of
its
10.
Second
updates
yeah,
because
right
now
right
now
is
writing
literally
8k
every
time
it
is
the
same
data
except
for
that
time
stamp.
So
if
we
add
an
extra
bit
that
changes
along
with
that
a
cake,
I,
don't
think
it's
it's.
Okay,.
H
Stuff,
ok,
so
this
tool
is
pretty
much
for
comparing
or
two
tests
together.
It's
running
very
similar
environments
and,
for
example,
for
example,
is
Q
mark
and
real
clusters.
So
with
the
word
we
do
basically
is
take.
The
last
few
runs
from
both
tests
and
kind
of
aggregate,
the
matrix
take
average
across
all
the
runs
on
kinematics,
like
so
api
call
it
in
peace
and
the
porn
star,
dirk
latency,
and
let
me
actually
starts
with
showing.
H
H
So
you
pretty
much
have
these
options
for
the
tool,
which
is
basically
the
comparison
scheme
that
you
are
going
to
use
and
the
left
job
and
the
right
jobs
that
we
want
to
compare
and
the
the
wrong
selection
scheme,
which
is
pretty
much.
Do
you
want
to
select
the
last
all
the
runs
from
the
last
n
hours
or
all
the
last
n
runs
and
n
itself
is
the
value
of
it.
So
I
will
start
with
running
this
against.
H
H
You
forget
a
given
process:
you're
horrible
dump
of
the
metric
comparisons,
and
sorry
for
that.
But
yes,
so
like
this
is
a
very
sad
thump,
I
she'll,
probably
formatted,
but
this
way
too
much
data
yep.
So
let's
take
some
metric,
for
example,
Kate
endpoint.
So
this
either
it's
pretty
hard
to
understand,
but
yeah.
This
is
pretty
much.
H
So
that
is
followed
by
a
comment
which
kills,
which
tells
the
number
of
sample
points
we
got
from
the
left
test
and
brightest.
There
means
the
standard
deviations
and
the
maximum
values.
So
we
pretty
much
kind
of
get
a
super
high
value
works
to
mean
which
is
which
is
like
798.
It
is
of
the
order
of
flick
of
seven
kind
of
pretty
close
yeah.
F
H
C
H
Not
not
such
a
topic
yeah
and
so
don't
care
for
getting
spit
as
we.
You
can
see
that
they're
pretty
close,
so
they
are.
This
is
still
a
prototype.
Click
for
now
I
ran
a
few
experiments
on
these,
and
it
turns
out
that,
like
close
to,
sixty
percent
of
the
metrics
are
actually
similar
when
we
give
a
slap
of
thirty
percent
and
I
came
into
twenty
percent
now,
so
probably
it
as
much
lesser
I.
If
I
remember
correctly,
it's
like
around
fifty
percent
of
the
matrix
mad
as.
B
C
Fishing,
tough
guy
act,
but
that's
just
like
that.
Such
a
process
like
I,
think
it
is
right.
We
have
gotten
a
soul
that
will
allow
to
compare
environments
right,
and
this
is
the
DB
TT
value
we
see
here
like
for
us.
It's
q
mart
and
the
rest
like
any
normal
Custer's,
but
like
we
can
also
rely
as
long
as
we
have
that
identity
test
results.
We
can
compare
anything
then
take
your
load
test
result
inserting
the
Clayton's
question,
if
I
understood
correctly,
but
I
don't
really
get
it.
C
So
if
anyone
will
just
run
the
test
which
will
automatically
embed
those
days,
all
those
things
you
can
just
compare,
we
can
compare
environments
between
each
other.
I
will
will
be
able
to
compare
environments
with
each
other,
which
is
like
the
first
step
to
actually
I'm
doing.
What
you've
always
wanted
to
do
so
like
see
where
the
difference
is?
I.
B
C
But
I,
if
anyone
wants
to
actually
make
sure
that
there
they
are
I
can
head
like,
is
not
acute
variance
I
any
statistical.
Statistical
significance
is
all
they
need
to
run.
A
number
of
tests
join
single
test
can
be
anything
it
can
be
really
badly
go
to
meet.
One
month
we
mean
anything
yeah
so
that,
like
a
sad
reality
of
mathematics,
sure.
E
F
A
C
E
Think
at
this
point,
given
everything
I
think
just
having
one
extra
metric
that
we
can
agree
it
on
and
start
to
define
and
track
over
time.
That
was
your
original
proposal
right,
yeah,
that's
totally
legit
with
me.
I
think
that's
the
right
way
to
go
the
question:
what
metric
do
you
want
to
want
it
to
be
and
how
to
measurements.
E
E
C
So
what
I
wanted
to
do
to
actually
know
like
this
is
one
thing,
and
second
thing
is
that
we
want
to
start
simple,
but
what
questions,
what
what
does
it
mean
exactly
like?
How
do
we
want
to
describe
it
Ashley,
just
a
number
and
the
definition
as
we
bid
already
or
do
we
want
to
change
the
view
we
have
towards
the
d1
that
you
proposed
here.
E
But
we
did
that
on
density,
which
is
a
replication
controller
right,
so
you
could
say,
like
you
could
do
one
for
each
one
where
you
could
do
start
up
late
and
see
if
this
is
what
happens
when
I
just
try
to
start
up
a
thousand
pots
right.
This
is,
is
the
submission.
This
is
what
happens
when
I
try
to
start
up
a
thousand
pods,
be
a
replication
controller,
which
is
the.
C
E
We
have
there's
been
fixes
in
160
and
hawkins,
I
believe,
has
most
of
those
fixes
in
place,
so
it
should
prevent
the
the
proxy
IP
tables
think
from
going
bananas,
but
I
do
think
there
correct
me
if
I'm
wrong
the
j,
you
had
a
service
load
test
right.
You
can
measure
the
amount
of
services
per
second.
If
you
wanted
to.
C
E
F
C
A
E
Falling
down
so
we
used
to
have
this
thing
like
an
example
here
to
test
would
be
like
what
happens
if
you
start
removing
those
caps
on
communication
right
I
keep
the
S
limits.
Like
you
see
the
curve,
let
me
let
me
kick
off
my
screen,
so
you
can
so
before
so
before
the
ramp
curve
would
be
like
the
smoke
would
be
like
this
right.
You
take
off
QPS
limits
and
all
of
a
sudden,
the
smoke
goes
like
this,
but
then
it
becomes
unbounded.
E
A
E
Like
how
fast
can
we
get
this
without
breaking
everything,
because,
right
now
the
whole
system
is
very
under
dampens
like
the
whole
system's
response?
Time
is
like
this:
when,
in
the
beginning
it
actually
was
faster,
but
we
put
all
those
limits
in
place
because
what
was
happening
is
it
would
be
faster,
but
it
also
start
losing
its
mines
and
breaking,
but
now
I
think,
as
things
have
progressed,
and
we
have
you
know,
they
sort
of
kind
of
other
scalability
fixes
folks
have
fix
the
caching
issues.
Folks,
it
looks
like
a
number
of
other
performance
issues.
E
C
E
Maybe
it
would
make
sense
is
like
start
start
with
your
take
your
original
doc
at
the
break
down
some
other
pieces.
Don't
do
too
heavy
weight
and
you
know
start
to
break
down
what
end-to-end
system
throughput
needs
for
the
different
types
of
resource
objects
right
and
which
ones
we
want
to
work
on.
First,
because
we're
not
going
to
do
all
of
them
right.
Well,.
E
E
So,
there's
still
many
things
going
on
that
we
don't
have
all
control
over
we're,
just
measuring
like
here's,
your
impulse
and
here's
your
response
curve.
So
it's
just
like
trying
to
remember
the
class,
but
like
signals
glass
back
in
college
right,
we're
like
you're,
measuring
the
impulse
response
curve
of
the
black
box
and
to
figure
out
what
the
function
is.
D
A
I
think
there
I
think
there
are
tuning
the
system,
but
I
think
the
other
thing
that
we're
interested
in
with
the
sort
of
thing
is
trying
to
track
regressions
rate
where
we
don't
think
anything's
changed,
but
the
curve
has
managed
to
like
get
whack
has
gone
different
for
some
reason
than
we
want
to
know.
Why
catch
that?
Yes,.
C
A
E
C
C
C
E
See
you're
saying
the
I
think
start
simple:
let's
just
start
simple
and
start
having
a
simple
test
in
place
and
then
eventually
we
can
expand
upon
it
right
now.
I
think
it's
you
know.
As
a
cig,
we
should
probably
be
adding
more
and
more
tests
to
measure
the
different
matrix
of
the
system
over
time
and
right
now,
we've
kind
of
only
have
a
couple
of
those
core
metrics,
but,
as
we
mentioned
before,
they're
kind
of
filthy
lies
right.
If
you
start
loading
up
actual
other
things,
you
start
seeing
abnormal
behavior
right.