►
From YouTube: Everything You Ever Wanted to Know About Resource Scheduling, But Were Afraid to Ask by Tim Hockin
Description
Everything You Ever Wanted to Know About Resource Scheduling, But Were Afraid to Ask - Tim Hockin, Google
"Effective management of shared resources (CPU, memory, etc) is critical to achieving higher utilization and stronger isolation in shared-machine systems like Kubernetes. As the industry evolves from single-purpose machines toward shared machines running containers, we must begin to internalize and embrace the ideas of resource-based scheduling. There are several facets of resource management to consider, but it's clear that many people are still asking the wrong questions.
The truth is that managing resources is HARD. This talk will try to demystify some of the thinking around this topic, share some techniques for better cluster management, and present some ideas for making Kubernetes an overall more robust system."
About Tim Hockin
Tim was one of the founding members of the Kubernetes project, and has overseen large parts of the system, including storage and networking. Before Kubernetes, he worked on Google's Borg and Omega systems, and was responsible for most of the node-side functionality. Before that he fiddled with the BIOS, OS, and other low-level software.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
So
I
went
through
sort
of
the
history
of
some
of
the
interactions
we've
had
with
customers
through
Google
container
engine
and
through
kubernetes,
open
source
and
I,
and
I
picked
up
on
a
theme
of
things
that
people
were
asking
that
I
thought
they
were
asking
the
wrong
questions.
I
was
gonna
title
this.
This
talk,
you're
asking
the
wrong
questions,
but
I
thought
it
was
a
little
too
negative
so
and
then,
in
preparing
for
this
I
thought,
there's
actually
some
background
that
I
need
to
bring
people
up
to
speed
on
before.
A
I
can
actually
answer
these
questions
correctly.
So
I
have
73
slides
in
40
minutes,
there's
no
way.
I'm
gonna
get
to
cover
it
all
so
I'm
gonna
rush
through
some
of
it
I'm
gonna.
Try
to
give
you
a
little
bit
of
a
background
on
scheduling
and
the
way
we
think
about
resources
in
Google
and
in
kubernetes,
and
and
how
are
you
go
through
that?
A
If
you
have
questions,
please
ask
questions
right
away
if
it's,
if
it's
relevant,
we'll
try
to
answer
it
in
line,
if
not
we'll
save
it
for
the
end,
so
I
generally
hate
these,
like
Who,
am
I,
slides
I
put
this
up
just
so
you
can
see
the
physical
me
and
the
github
me
if
you
get
on
github,
and
you
see
that
guy
that
that's
me
so
fair
warning.
Some
of
what
we're
gonna
do
today
is
aspirational.
Actually,
most
of
it
is
aspirational.
A
I'm
gonna
talk
a
lot
about
what
we've
done
in
Bourg
inside
Google
and
what
we
want
to
do
in
kubernetes
what
we
should
do
in
communities.
Not
all
of
this
is
done.
Not
all
of
this
has
even
started.
There's
an
opportunity
for
people
to
help
so
starting
in
I
posit
that
kubernetes
is
fundamentally
underneath.
Everything
is
about
managing
resources,
resources.
A
What
I
mean
by
resources.
Resources
are
those
things
that
are
in
your
computer,
that
you
use
to
run
your
jobs,
there's
CPU
and
its
memory
everybody's
familiar
with
these
right,
but
it's
also
disk
space.
Ok,
we
sort
of
get
that
one.
It's
also
a
disk
time
right.
It's
also
disk
spindles
right,
ok,
I
get
it
I
get
discs.
Let's
talk
about
network
bandwidth.
Let's
talk
about
ports
on
the
machine.
Let's
talk
about
cache.
Let's
talk
about
memory.
Bandwidth
IP
addresses
is
a
very
interesting
problem
in
kubernetes
and
in
containers
in
general
storage
attachments.
A
We
talk
about
things
like
process,
IDs,
GPUs
power,
right,
Google
ghosts
as
far
as
we
manage
power
as
a
contained
resource
and
we
charge
back
for
it.
Remember
I,
said
aspirational
right,
we're
not
there
yet
and
then
there's
this
whole
other
category
of
arbitrary,
opaque
things
that
I
don't
know
about
that.
You
probably
have
in
some
of
your
hardware
that
you
want
to
schedule
jobs
against,
but
you
don't
want
to
patch
kubernetes
to
understand
that.
It's
really
special
to
you.
A
A
Now
we've
got
a
14
dimensional
problem
so
as
I
want
to
build
a
chi
mental
model
of
kubernetes
first
so
nodes.
These
are
us
our
word
for
machines
once
upon
a
time
called
minions
nodes
produce
capacity.
They
offer
it
into
the
cluster
right.
Here's
a
little
snippet
of
the
amol
that
node
object
would
produce.
I
have
capacity
of
4
CPUs
and
I
have
32
gigs
of
memory
pods,
that's
our
icon
for
pod,
get
used
to
it.
A
Pods
consume
resources,
they
say,
I
need,
CPU,
I
need
memory
and
then
the
scheduler
is
going
to
map
those
things
together.
It's
going
to
build
a
binding
between
a
pod
and
a
node,
and
that
pot
is
then
going
to
run
on
that
node,
simple
stuff
right.
So,
let's
build
out
the
mental
model
of
how
do
we
represent
resources,
it's
attractive
to
think
about
resources
in
a
sort
of
a
multi-dimensional
box,
I'm
only
gonna
focus
on
CPU
and
memory
today,
because
I
can't
do
14
at
once.
A
So
it's
it's
attractive
to
think
about
memory
and
and
CPU
as
a
two
dimensional
box
right
and
you're
gonna
pack
stuff
into
the
box,
so
I've
got
my
blue
job
and
it
fits
into
the
box.
Awesome.
Cool
I
can
continue
to
pack
stuff
into
my
box
until
my
box
is
full
right.
No,
this
is
not
the
right
way
to
think
about
resources
when,
in
this
case,
I've
double
used.
Everything
when
you
think
about
resources
that
blue
thing
that
we
schedule
first,
it
casts
a
shadow
in
every
dimension,
so
I
now
have
only
this
much
available.
A
This
is
a
really
weird
way
to
think
about
resources
and
to
represent
them.
It's
really
attractive.
Sorry,
so
we
have
a
set
of
mike
says
it's
a
vector,
not
a
box.
It
is,
in
fact
a
set
of
parallel
vectors.
This
is
how
we
think
about
resources.
This
is
how
I'm
gonna
represent
them.
When
I
talk
about
them
in
this,
in
this
way,
this
is
I
think
much
clearer
to
to
visualize.
It's
just
not,
as
is
immediately
obvious
to
you
so
now.
A
I
want
to
talk
about
scheduling,
scheduling
is
sort
of
the
heart
of
what
qu
Brandis
does
so
imagine
we
have
our
hypothetical
kubernetes
cluster.
We
have
two
nodes:
a
and
B
they
equal,
CPU
and
equal
ram.
They
have
ten
units
of
cpu
and
ten
units
of
ram
okay,
so
I've
got
a
blue
job
and
this
blue
job
comes
in.
It
says:
I
need
six
units
of
CPU
and
four
units
of
ram
no
problem.
The
scheduler
can
stick
it
on
their
first
prop
on
the
first
node
first
fit
now.
A
A
Now,
if
one
node
fails,
at
least
you
don't
bring
out
all
of
your
jobs
with
you,
I
come
along
with
my
yellow
job
and
it
tries
to
fit
on
node
a
that
doesn't
work
so
that
no
day
is
removed
from
the
the
pool
of
eligible
nodes,
but
it
fits
fine
on
node
B
cool
we've
still
got
some
leftover
resources.
Let's
keep
scheduling,
so
we
bring
along
this
red
pod
and
the
red
pod.
We
stick
it
on
no
Dame
that
doesn't
work,
so
we
remove
node
a
from
the
eligibility
pool.
A
A
So
oh
damn,
my
pod
just
went
pending
right
and
you'll,
see
this
in
the
cube
cuddle
state
for
a
pod
that
can't
be
scheduled
pending.
If
you
look
at
the
events,
you'll
see
no
available
in
sufficient
resources
to
run
this.
This
is
a
bad
place
to
be
because
you
look
at
this
cluster
as
a
person.
You
like
both
clearly
there's
enough
resources
in
this
cluster.
If
you
look
at
your
aggregate
dashboard,
there's
more
than
enough
resources
to
fit
in
this
cluster,
what
we
need
is
an
optimizing
reschedule
er.
A
This
is
that
aspirational
part
right,
I
need
something
to
come
along
and
say:
hey,
you
know
what
I
can
VIN
pack,
this
better
I
can
move
the
green
up
to
the
first
node
and
then
I
can
actually
fit
the
red
on
the
second
node.
That
would
be
really
cool
right.
Patches.
Welcome
the
problem
with
all
this
is
we
know
how
to
do
it.
We
know
what
to
do.
We
just
don't
have
enough
hands
on
enough
keyboards
to
actually
achieve
it,
all
in
the
timeframe
that
we
want
to
achieve.
A
B
A
B
A
For
the
sake
of
the
the
recording,
the
question
is
that
there
are
some
resources
that
exist
that
are
not
particular
to
anyone,
know
that
exists
sort
of
in
the
abstract
in
the
in
the
network
as
addresses
and
and
those
sorts
of
things
yes
and
I
can
talk
at
length
about
how
Google
manages
say,
aggregate
network
bandwidth
or
top-of-rack
bandwidth.
I
do
not
have
enough
time
today
to
talk
about.
B
B
A
I,
do
not
pretend
to
have
a
full
list
of
the
resources
here.
I'm
actually
really
gonna
go
a
little
bit
myopic
today
and
we're
gonna
just
really
focus
on
an
unknown.
We're
gonna
talk
a
little
bit
about
cluster
at
the
end,
but
really
we're
talking
about
node
eccentric
resources,
you're
right
that
there's
a
whole
other
category
of
stuff.
A
So
now
that
we've
rescheduled
this
this
red-pawed,
we
noticed
that
the
node
B
has
all
ten
units
of
CPU
consumed,
but
not
all
ten
units
of
memory
consumed.
We
call
this
stranded.
This
is
resources
that
are
just
they're
stuck
they're
isolated.
You
can't
use
them
in
the
current
state
of
things.
It
is
a
real
problem,
there's
not
a
really
good
answer
to
it,
except
that
you
need
to
be
aware
of
this,
as
you
think
about
your
jobs.
A
This
especially
applies
when
you
get
to
these
bigger
clusters
right
we're
in
aggregate
you
start
to
see
sort
of
a
ratio
of
CPU
to
memory
emerge
that
makes
sense
and
there's
the
machines
that
you
build.
Google
has
a
sort
of
a
golden
ratio
of
how
build
we
bit,
how
big
we
build
our
standard
machines
and
we
expect
jobs
to
sort
of
fit
within
spitting
distance
of
that
golden
ratio
and
if
they're
not,
they
actually
get
charged
more
because
they're
going
to
strand
resources
right,
you
don't
necessarily
pay
for
what
you
use.
A
You
pay
for
what
you
prevent
other
people
from
using
right.
So
in
this
case
we
might
charge
back
to
each
of
those,
the
red
and
the
yellow
jobs,
some
portion
of
the
stranded
resources,
because
they're
they're
out
of
ratio
right
so
now
I
get
to
the
fun
part.
So
I
think
a
lot
of
people
are
just
still
looking
at
kubernetes
in
there
they're
coming
at
it
from
the
history
of
what
they've
done
before
and
they're
asking
the
questions
that
they
know
to
ask
but
I
think
they're
asking
the
wrong
questions.
A
A
A
How
do
I
save
some
machines
for
important
work
like
this
is
important
stuff
and
the
other
rest
is
batch.
It's
MapReduce,
it's
whatever
I,
don't
want
those
things
to
interfere
with
each
other.
So
how
do
I
partition
my
cluster
this
way,
so
I
can't
obviously
present
questions
without
rewriting
them.
So
how
do
I
make
sure
that
my
compute,
you,
like
the
pictures,
those
old
machines
sexy
new?
How
do
I
make
sure
that
my
compute
can't
hurt
my
database?
A
What
you
should
be
talking
about
is,
oh
sorry,
the
the
first
question
of
how
do
I
make
sure
my
computer
doesn't
hurt
my
databases.
How
do
I
make
sure
that
they
can't
hurt
it?
It's
not.
How
do
I
make
sure
they
land
on
different
machines?
It's
how
do
they?
How
do
I
make
sure
that
they
can't
actually
impact
each
other?
You
shouldn't
care
about
which
machines
they
land
on.
We
call
this
isolation.
A
A
So
isolation
right.
This
is
fundamentally
about
preventing
applications
from
hurting
each
other.
This
is
the
the
the
cornerstone
of
building
a
system
like
kubernetes.
You
have
to
have
isolation
and
it
has
to
be
reliable.
You
have
to
make
sure
that
if
you
bought
some
resources,
if
you
bought
four
gigs
of
memory
that
you're
gonna
get
that
four
gigs
of
memory,
if
you
don't
you're
going
to
do
bad
things,
you're
gonna,
over-provision
you're,
going
to
not
trust
the
system,
you're
gonna
be
less
efficient.
A
We
want
to
drive
efficiency
up
so
kubernetes
and
docker
already
isolates
CPU
and
memory,
and
they
do
a
reasonable
job
of
it.
Using
the
the
kernel
subsystems
that
exist
today,
we
don't
even
try
to
handle
things
like
memory,
bandwidth
or
disk
time
or
cache
or
bandwidth.
These
are.
These
are
much
harder
problems
and
they
just
haven't
really
been
solved
very
well
in
Linux.
Yet
Google
has
a
bunch
of
patches
that
we're
gonna
continue
to
try
to
push
upstream.
That
will
make
these
things
better.
A
We
need
to
work
with
docker
to
define
the
api's
to
manage
these
sorts
of
things
on
machines.
The
key
here
is
that
when
you
get
to
the
end
game
at
the
very
extremes
of
the
operation
of
the
system,
it
has
to
be
stable
and
it
has
to
be
predictable
and
I
will
take
predictable
over
performant
every
day
of
the
week.
So
you
know
we
have
a
sort
of
running
joke
that
every
summer
at
Google
we
get
a
full
on
stress
test
of
our
isolation
systems.
A
A
So
when
does
isolation,
when
does
it
kick
in?
When
does
it
really
matter?
Anybody
ever
written
an
infinite
loop,
not
I
mean
not
me,
but
maybe
you
did
and
we
have
written
a
memory
leak
again
totally.
Not
me
Discogs
right.
Sometimes
you
just
run
away
with
the
disk
and
you
need
to
forget
to
clean
up
oops
fork
bombs.
I
almost
got
kicked
out
of
school
for
that
and
and
the
most
nefarious
of
them
all
is
cache
thrashing
right,
because
it's
actually
really
not
visible
to
you
when
your
cache
thrashing
you're
hurting
everybody
else.
A
So
how
do
we?
How
do
we
defend
against
these
things?
Infinite
loops
I
can
I
can
limit
the
amount
of
CPU
that
you
use
so
Linux
kernel
gives
us
things
like
shares
and
quota,
which
are
two
different
ways
of
managing
CPU
for
memory.
Leaks
I
can
give
you
memory
limits
so
that
when
you
run
out
of
memory
you
die
not
everybody
else.
Anybody
ever
run
afoul
of
the
Linux
out
of
memory
killer
right.
It's
like
firing
a
shotgun
into
a
crowd
right.
Somebody
gonna
die
I,
don't
know
who
it
is.
A
Usually
it's
the
guy
who
uses
the
most
memory,
which
is
not
necessarily
the
same
as
the
guy
who's,
actually
causing
the
problem
right.
So
with
the
advent
of
cgroups,
this
is
getting
a
little
bit
more
rigid
google's,
put
a
ton
of
work
into
making
this
very
deterministic.
Not
all
those
patches
have
been
accepted
upstream.
A
10,000
should
be
enough
for
anybody
and
when
it
comes
to
cache
thrashing
again
the
most
nefarious
you
can
do
things
like,
say:
I've
detected,
you're,
a
bad
actor
and
I
keep
picking
on
you
because,
right
in
front
of
me,
you're,
a
bad
actor
and
I'm
gonna
put
you
and
all
the
other
bad
actors
or
all
the
other
low
priority
jobs
on
a
single
last
level,
cache
thrash
all
you
want.
You
can't
hurt
everybody
else
right
now.
A
Intel
has
brought
out
new
stuff
with
the
cache
allocation
technology,
which
is
thrashing
in
the
kernel
trying
to
figure
out
how
to
patch
it
in.
But
when
you
apply
things
like
that,
you
can
actually
get
really
awesome
isolation
where
it
doesn't
matter
how
hard
you
thrash
the
cache.
Everybody
else
is
gonna,
be
fine.
A
Now
the
state
of
these
things,
infinite
loops
and
memory
leaks,
CPU
and
memory.
Those
are
pretty
well
understood
at
this
point.
Disk
quota
stuff
is
in
progress
in
current
kubernetes
and
the
other
stuff.
At
the
moment,
nobody
is
working
on
patches,
welcome
proposals.
First,
please
so
I
want
to
introduce
a
quick
resource
taxonomy,
because
I'm
gonna
use
these
words.
So
I
want
to
explain
them.
A
First,
there's
two
fundamental
classes
of
resources:
compressible
resources,
compressible
resources
have
no
state
and
they're
the
main
characteristics
they
can
be
taken
away
very
quickly
without
actually
hurting
the
application,
except
just
performance
right.
These
are
things
that
tend
to
be
measured
in
time.
So,
like
you,
measure,
CPU
allocations,
people
think
of
it
and
how
many
cores
do
you
get?
What
you're
really
getting
is
how
many
core
seconds
per
second
right.
A
A
The
classic
example
here
is
memory.
You
use
up
a
whole
bunch
of
memory
and
I
need
to
take
it
back
from
you.
How
do
I
do
that?
I
can't
just
take
it
away.
You
actually
have
stuff
in
there
maybe
have
pointers
that
are
cross-referencing.
It
right,
maybe
they're
dirty
pages
I
can
do
things
like
write.
You
back
to
disk
I
can
do
things
like
throw
out
all
your
clean
pages
and
force
you
to
fault
them
back
in
or
I
can
just
straight
out
and
kill
you
if
it
takes
too
long.
A
So
let's
then
talk
about
how
we
apply
limits
to
these
things.
So
kubernetes
has
a
two-tiered
concept
of
what
you're
asking
for
with
respect
to
resources.
The
first
tier
is
a
request.
The
request
is
what
we
use
in
the
scheduler.
This
is
your
guaranteed
amount.
If
you
ask
for
four
gigs
you're
gonna
get
four
gigs
we're
gonna.
Do
everything
we
can
to
make
sure
that
you
have
four
gigs
available
to
you.
Should
you
need
four
gigs?
A
We
do
not
over
commit
at
this
level
at
EDD
requests.
The
second
line
is
limits.
Limit
says:
I
might
use
more
than
my
request
if
it's
available
to
me
and
I'm,
okay
with
it
being
at
a
lower
quality
of
service
right.
So
it's
sort
of
opportunistic
CPU.
If
there's
extra
CPU
on
the
machine,
I'll
use
it
to
complete
my
thing
faster,
but
if
it's
not
there,
I'll
be
fine
right.
A
This
gets
really
interesting
when
you
start
talking
about
non
compressible,
read
sources
and
being
over
request,
but
under
limit
right,
if
I'm
using
more
memory
than
I'm
technically
allowed
to
use-
and
somebody
else
needs
it-
you
have
to
reclaim
it.
So
this
is
what
we
used
to
define
our
quality
of
service
bands.
So
quality
of
service
guaranteed,
it
says
I'm
not
going
to
use
more
than
I've
requested
set
request
equals
limit,
we're
happy.
You
should
always
get
what
you
asked
for
in
that
case,
in
a
fully
packed
machine.
A
If
everybody
is
running
full
tilt,
they
will
all
get
their
CPU
request
right.
That
is
the
way
things
are
supposed
to
work.
If
you've
asked
for
a
request
and
an
optional
limit
above
it,
you're
called
first
of
all,
and
you
have
a
slightly
lower
quality
of
service
and
as
long
as
you're
within
that
band,
if
you
hit
the
limit,
you're
dead
or
you'll
get
throttled.
So
we
use
this
again.
A
So
again,
the
the
the
behavior
at
the
limits
is
really
what's
interesting
so
and
it
depends
on
the
resources.
So
when
you
run
out
of
CPU,
if
you've
set
your
limit,
you
just
stop
getting
scheduled
right.
So
this
is
a
it's
a
really
interesting
problem.
That's
not
exactly
obvious!
Imagine
you're
running
on
a
32
core
machine,
you're,
writing
and
go
because
why
wouldn't
you
and
go
as
a
highly
concurrent
language,
it
encourages
concurrency.
So
you've
written
your
your
sieve.
A
Your
prime
number
sieve
to
be
highly
parallel
and
you're
gonna
use
up
as
many
courses
you
can
use
you've
been
allocated.
Eight
cores,
you
bought
eight
cores
for
this
job
you're
running
at
a
32
core
machine
cool.
In
the
first
scheduling
pass,
you
can
actually
have
30
to
go
routines
scheduled
all
at
once.
They
will
all
consume
a
quarter
of
a
CPU
second,
and
then
you
are
stuck
for
the
remainder
of
that
scheduling
window.
A
If
it's
as
if
it's
a
one-second
window,
you
get
nothing
because
you've
used
all
eight
of
your
CPU
seconds
in
that
quarter
of
a
second
right.
This
is
something
that
people
need
to
think
about
as
they
build
applications
and
as
they
work
with
systems
like
kubernetes,
all
the
other
CPU
s/s
were
given
to
somebody
else.
A
When
it
comes
to
things
like
memory,
we
talked
about
reclaim.
We
can
talk
about
soft
limits
when
the
kernel
starts
actively
trying
to
reclaim
pages
from
your
process,
and
we
have
a
time
limit
like
there's
only
so
long
I'm
willing
to
wait
for
you
to
free
up
pages
before
I
end
up
killing
you,
because
somebody
else
is
waiting
for
that
memory.
A
In
this
case,
we
talked
about
correctness
and
sort
of
optimal
miss.
We
need
to
focus
on
being
correct,
first,
an
optimal
second,
it's
easy
to
forget
that,
and
try
to
optimize
things,
I
will
take
predictable
and
I
will
take
correct
over
optimal
every
every
day
of
the
week.
So
what
happens
now
when
you
end
up
with
coupled
resources
right
and
we
try
to
back
up
a
truck
it's
pretty
complicated,
because
it's
coupled
in
an
interesting
way
right
so
in
a
great
example
of
coupled
resources
is
memory.
A
My
good
old
friend
memory,
if
I,
try
to
allocate
memory
and
it
fails
right
but
I'm
below
my
Rome
I
request
so
I'm
supposed
to
be
guaranteed.
What
does
the
system
going
to
do?
It's
gonna
run
around
the
Machine
trying
to
find
somebody
who's
over.
There
request
somebody
who
has
clean
pages
that
they
can
release
first
thing
like
code
pages,
you
can
fault
that
back
in
next
time
you
need
it
right.
Dump
those
pages
out,
give
them
to
the
first
guy.
If
I
can't
do
that,
and
first
of
all
that
consumes
CPU.
A
So
now
I've
got
a
coupled
resource
here.
Then,
if
I
can't
find
that,
then
I'm
gonna
take
some
of
your
dirty
pages.
I'm
gonna
write
your
dirty
pages
back
to
disk
right
and
that
consumes
disk
time
and
CPU,
and
if
that
isn't
sufficient
and
I
have
to
go
and
repeat
this
on
another
container.
This
takes
time
right.
Every
time
you
you
fire
up,
Firefox
or
something,
and
you
hear
the
disk
launch,
it's
cuz,
it's
thrashing
dumping
pages
back
to
disk.
Maybe
this
doesn't
matter
anymore.
Now
we
all
have
SSDs.
We
don't
hear
anything
I'm
old.
A
So,
if
that's
not
enough,
you
do
it
on
another
container
and
you
do
another
container
until
you've
found
enough
memory
to
give
back
to
this
other
guy.
And
how
long
should
we
allow
this
to
take?
Should
it
if
it
is
a
minute
acceptable,
probably
not
like.
If
you're
writing
a
program,
you
call
malloc
and
it
takes
a
minute
to
return.
A
You're,
probably
not
gonna,
be
real
happy
right,
so
we
have
some
heuristics
that
are
built
around
how
long
this
thing
should
take
and
it
you
know
it's
order,
milliseconds
before
we
end
up
saying
screw
it
and
I'm
killing
somebody
and
that's
like
so
the
ultimate
big
hammer
right
in
reality.
We
should
be
constantly
doing
this.
We
should
be
constant
pressure
on
the
kernel
to
free
up
unused
pages,
but
that's
a
two-edged
sword,
because
you're
going
to
use
those
pages,
we
don't
actually
know
which
pages
you
do
and
don't
need.
A
So
one
of
the
things
we
did
inside
Google
is
we
built
a
like
a
two
minute
hot
list
and
we
say
look.
These
are
all
the
pages
you've
used
in
the
last
two
minutes
and
we're
gonna
try
to
protect
those
and
everything
else:
fair
game
right.
We
still
run
into
this
problem
and
we
still
end
up
killing
people.
So
what
happens
now?
If
you
don't
specify
how
much
resources
you
need
right,
just
leave
it
blank,
it's
all
Yambol!
So
it's
all!
It's
all
default
able,
if
you
don't
specify
you
get
0.
A
What
you
get
is
is
best-effort
isolation,
no
promises
to
you.
You
might
get
defaulted
values
depending
on
how
your
clusters
been
configured
or
how
your
namespace
has
been
configured.
We
have
thing
called
limit
range
which
will
help
default
your
values
for
these
things
you
might
just
get
killed
randomly
you
might
get
CPU,
starved
and
not
scheduled
for
five
minutes
or
you
might
just
get
no
isolation.
All
I
all
bets
are
off
if
you
don't
specify.
A
So,
if
you're
running
like
a
MapReduce
or
something,
maybe
that's,
okay
with
you,
if
you're
running
a
production
serving
job,
you
need
to
think
about
what
resources
you
need.
It's
it's
it's
a
really
difficult
problem
to
think
about
how
much
resources
you
need.
So
how
many
replicas
do
I
need?
What
is
my?
What
is
my
scaling
factor?
A
What
do
I
scale
on
I'm
having
a
interactions
with
customers
on
this
all
the
time
where
they
think
they
can
sort
of
wave
a
magic
wand
and
the
auto
scalars
will
just
handle
stuff
right
and
the
auto
scalars
have
very
concrete
signals
on
what
they
use
to
scale
for
you,
and
you
have
to
understand
to
be
able
to
characterize
your
application
or
you're
just
gonna
get
creamed
when
the
traffic
shows
up.
How
many
replicas
is
an
easy
question:
how
much
CPU
and
memory
does
my
job
need
right?
These
things
are
changing
all
the
time.
A
There's
no
such
thing
as
how
much
CPU
are
you
using
it's?
How
much
CPU
are
you
using
over
a
period
X
of
time
right
and
now
you're
back
into
calculus,
so
you
have
to
figure
sort
of
how
do
I?
How
do
I
characterize
these
things?
For
my
applications?
Do
I
provision
for
the
worst
case,
like
okay,
suppose
Michael
Jackson
dies
or
comes
back
from
the
dead?
What
is
what's
gonna
happen
to
my
application.
All
right!
That's
a
spike
about
this
high,
but
my
average
usage
is
actually
down
here.
A
If
you
provision
for
that,
it's
massively
wasteful
right.
Well,
what?
If
I
provision
for
the
average
case
down
here
and
then
Michael
Jackson
comes
back
from
the
dead.
Well,
my
shits
gonna
fall
over
sorry.
My
my
jobs
are
gonna
fall
over
I'm
gonna
get
killed,
I'm
gonna
be
CPU,
starved,
I'm
gonna
have
a
bad
time
right.
The
clear
answer,
of
course,
is
write
a
benchmark
right.
It's
easy,
no
benchmarks
are
incredibly
hard
right.
Even
inside
Google,
almost
nobody
writes
benchmarks
that
work
properly.
Accurate
benchmarks
are
damn
near
impossible.
A
Very
few
teams,
even
inside
Google,
have
the
bandwidth
to
write
an
accurate
benchmark
that
models
a
complicated
system
that
lets
them
predict
with
any
accuracy
what
they're
going
to
need
over
time,
and
so
so,
let's
talk
real
quickly,
then
about
the
horizontal
scaling.
So
we've
got
auto
scaling.
It's
easy
to
reason
about
horizontal
scaling.
How
do
you
adapt
to
load
this
way?
You
just
add
more
copies
of
your
job
right
to
go
back
to
the
user.
A
Question
of
you
know
what
do
I
just
wanna
use
all
the
memory,
but
once
you
think
about
resources,
once
you
draw
a
box
around
your
application,
it
makes
perfect
sense,
then,
to
just
take
multiple
copies
of
that
thing
and
stamp
it
out
and
say.
Well,
I
only
need
one
gig
of
memory
in
the
average
case,
but
now
sometimes
I
need
10
gigs
of
memory,
so
I
can
scale
it
on
a
granule
of
one
gig
at
a
time,
and
I
will
get
better
efficiency
right.
A
You
ever
do
the
calculus,
where
you
draw
the
area
under
the
curve
and
you
get
the
little
boxes,
the
smaller
the
boxes.
The
more
accurate!
Your
estimation
is
right,
it's
the
same
basic
idea.
So
then
it
starts
to
make
sense,
we'll
all
understand
why
I
would
have
multiple
replicas,
but
it's
not
always
applicable.
Sometimes
there
are
just
things
that
don't
scale
horizontally
right.
A
classic
example
that
we're
experiencing
right
now
with
kubernetes
is
DNS
DNS
scales
with
the
size
of
the
cluster.
A
A
A
We
need
to
collect
stats
on
what
you're
using
now.
We
need
to
be
able
to
build
a
model
of
what
we
think
you're
going
to
need,
and
we
need
to
be
able
to
predict
and
we
need
to
be
able
to
react.
This
is
where
bursts
of
all
actually
comes
in,
like
it's
really
cool,
if
you
say
well,
I
predicted
wrong,
so
you
went
over
the
baseline
limit
into
the
optional
space.
I
can
then
react
and
say
I
see
that
you
need
more.
A
Let
me
react
by
raising
the
limit
for
you
now
you're
back
into
safe
territory.
This
can
be
used
to
manage
pods
and
deployments
and
jobs.
We
can
race
to
stay
ahead
of
the
spikes.
It's
still
going
to
be
correct
in
all
cases,
because
worst
cases
you
get
owned
best
cases.
It
works
fine
for
you,
so
we
actually
we
have
this
in
Borg,
it's
called
autopilot
and
the
vast
majority
of
Borg
users,
something
like
2/3
of
board
users,
use
autopilot.
A
They
put
their
job
into
Borg
and
they
say
I
don't
want
to
know
how
much
resources
I
have
here's
that
here's
an
upper-bound
don't
like
build
me,
infinite
money
and
here's
my
credit
card
go.
Nuts
benchmarks
are
really
hard
like
in
the
end,
it
turned
out
to
be
more
efficient
and
more
effective
to
write
an
autopilot
system
than
it
was
to
actually
get
people
to
write
benchmarks.
So
conveniently
the
kubernetes
api
is
purpose-built
for
this
use
case.
We've
thought
about
this
from
the
very
beginning.
How
do
we
enable
us
to
build
a
system?
A
This
is
why
pods
are
unique
entities.
You
can
actually
manage
individual
pods
on
individual
resources
and
they
can
be
different
from
each
other
or
you
can
manage
deployments
and
you
can
make
them
unique
across
the
entire
deployment,
but
we
need
a
vertical
pod
autoscaler
as
if
that
wasn't
a
horizontal
pod.
Wasn't
a
mouthful
enough,
we
need
a
vertical
pod.
Autoscaler.
Remember
that
aspirational
part
patches.
Welcome!
A
A
You
can
show
your
CIO
or
CTO
that
I've
got
a
hundred
percent
bookings
on
my
cluster,
but
utilization
is
still
four
percent
you're
still
not
going
to
be
happy
right,
so
real,
quick
show
of
hands
who,
in
this
room,
has
utilization
above,
say
five
percent.
On
average,
just
five
percent
come
on
hands
up.
Alright,
that's
that's
pretty
good!
Actually,
how
about
over
ten
percent
I'm
impressed
over
twenty
yeah
there
we
go
well,
I
can't
say
what
Google
has,
but
I'll
say
it's
more
than
that.
So
how
can
we
do
better
at
this?
A
First
of
all,
I.
So
I
put
isolation
at
the
beginning
of
this
list,
because
isolation
is
the
most
important
part.
Isolation,
yeah
utilization
demands
isolation.
If
you
want
to
push
the
limits
you
have
to
be
safe.
When
you
get
to
the
edges,
you
need
a
fence
around
that
cliff
people
are
inherently
cautious,
even
if
you
tell
them
there's
a
20
foot
tall
fence
that
somebody
else
paid
for
at
the
edge
of
the
cliff
they're,
still
gonna
be
careful
about
it
and
they're
still
gonna
provision
for
their
90th
percentile
or
their
99th
percentile
case
right.
A
So
let
then
take
away
some
other
lessons
from
the
Borg
system
that
things
we've
built
in
the
past
that
have
worked
reasonably
well.
For
us.
We
need
priority.
We
need
something
that
says
this
job
more
important
than
that
job
right
and
to
borrow
from
Animal
Farm.
If
everybody's
equal
then
nobody's
equal
right,
we
need
to
be
able
to
say
the
low
priority
jobs
in
the
case
of
an
emergency
they
get
paused
or
they
get
killed
right.
A
We
need
to
be
able
to
build
this
into
the
scheduling
system,
so
if
this
high-protein
Lee
needs
its
memory,
I
can
look
around
the
machine
and
say
low
priority
job
out
of
here.
Hyper
a
job
gets
your
stuff
right
and
when
we've
built
it
inside
Google
is
we
have
a
chargeback
model
and
the
high
priority
jobs
they
get
charged.
Prime
rate
and
the
low
priority
job
is
get
charged
next
to
nothing
and
most
people
look
at
that,
and
they
say
you
know,
the
kill
rate
for
the
low
party
jobs
in
practice
is
pretty
good.
A
I'm
just
gonna
use
that,
because
my
bill
is
1/10
as
much
right
and
this
search
and
the
Gmail
and
the
ads
they
buy
the
the
prime
USDA
grade
a
beef
because
they
need
it
now,
once
you've
got
this,
this
distinction
between
sort
of
tiers
of
work
you
can
actually
over
commit
your
systems.
Remember
I
said
we
don't
over
commit
totally
lied.
A
A
If
you
pay
a
tenth
as
much
as
he
did
for
that
other
four
gigs
of
memory
I
can
now
drive
my
utilization
way
up,
knowing
that
I
put
all
those
safeguards
in
place
that
should
he
need
his
memory
you're
out
of
there
you're.
Ok
with
that,
because
you
got
a
lower
SLA
for
the
lower
price.
You're.
Ok
with
that,
because
I
gave
you
a
higher
SLA
for
the
higher
price
late,
a
memory
allocation
might
not
be
a
ten
nanoseconds
might
be
one
millisecond,
but
on
average
you're
gonna
get
really
solid
performance
question.
A
We
do
by
fiddling
with
things
like
soft
limits
and
hard
limits
within
the
cgroups
I'm,
not
gonna,
run
it
I'm,
not
gonna
have
enough
time
to
talk
about
sort
of
the
Google
theory
of
layering
of
containers,
I'm
happy
to
talk
at
length
about
it.
Maybe
it's
another
talk,
but
yes,
we
go
through
a
lot
of
sort
of
soft
real-time
manipulation
of
the
underlying
resources
to
make
sure
that
the
higher
party
jobs
are
protected
more
so
now
we
build
this
overcommit
model,
I,
just
I
just
ruin
my
slide.
A
We
use
a
lot
of
stats
in
a
lot
of
history
to
figure
out
what's
safe
and
then
we
essentially
have
a
knob
that
you
can
turn
that
says
how
aggressive
do
you
want
to
be
with
your
over
commitment?
How
much
utilization
do
you
want?
The
more
utilization
you
get,
the
more
churn
you're
gonna
get
in
your
cluster
sort
of
the
more
volatile
the
market
becomes,
and
then
we
just
let
the
priority
system
and
the
reschedule
or
deal
with
the
fallout
right.
A
Things
are
gonna,
get
killed
and
moved
in
shuffled
around,
and
that's
fine
we've
already
built
the
system
for
this
right.
In
practice,
it
works
really
really
well
I'll.
Tell
you
one
of
the
hardest
things
I
did
too
when
I
came
to
Google
I
came
in
as
a
sort
of
junior
ish
software
engineer
where,
in
my
mind,
things
either
work
or
they
don't
right
and
it's
a
really
attractive
model,
but
it's
totally
wrong
in
reality.
A
When
you
start
talking
about
running
clusters
of
thousands
of
machines,
you
talk
about
the
percentage
of
times
that
things
work
and
you
talk
about
distributions
and
percentiles,
and
this
I
can't
guarantee
that
this
will
work
right
most
of
the
time.
In
practice,
it
works
great
right
and
we
set
the
knob
at
a
place
where
we're
happy
with
the
trade-off
between
the
volatility
and
the
utilization.
A
So
there's
this
this
very
attractive
nuisance,
which
is
I,
want
to
pack
my
cluster
as
full
as
I
can
get,
and
this
comes
out
of
real
customer
experiences.
Right
customer
had
an
outage
where
they
had
packed
their
cluster
to
ninety-nine
point
something
percent
containers
on
every
machine
and
one
of
their
machines
died,
and
it
happened
to
be
the
machine
that
was
running
their
DNS
server.
Oh,
damn
everything
went
sideways,
their
DNS
couldn't
get
rescheduled
because
it
went
pending.
A
Remember
that
pending
slide,
the
cluster
went
went
south
because
everything
depends
on
DNS
at
this
point
and
and
they
were
really
unhappy
and
they
called
us
and
they
yelled
at
us.
So
you
have
to
you,
have
to
think
about
this
right
nodes
fail
the
reality
check
nodes
fail
all
the
time
nodes,
get
upgraded,
I
mean
upgrades
in
general,
involve
taking
some
amount
of
unavailability
on
some
percentage
of
your
cluster,
and
you
have
to
be
able
to
handle
this
think
about
a
tube
of
toothpaste.
A
But
if
you,
if
you
squeeze
out
some
of
that
toothpaste
and
you
leave
a
little
bit
of
room
in
that
toothpaste
and
you
squeeze
it
all
your
toothpaste
shuffles
over
here-
and
you
can
then
fix
the
nodes
over
at
this
end
of
the
toothpaste
to
mix
analogies
and
you
squeeze
the
toothpaste
back
and
you
can
fix
all
the
nodes
over
at
this
side
of
the
toothpaste,
tube
and
everybody's
happy.
This
shows
up
in
your
utilization,
though
right.
Do
you
leave
5%
of
your
cluster
1%
of
your
cluster.
A
You've
got
to
decide
right
and
it
comes
down
to
an
organizational
thing
of
how
much
waste
are
you
willing
to
have
in
your
cluster
for
the
safety
right?
Google
has
numbers
but
they're
specific.
The
way
we
do
our
machine
repairs
and
the
way
we
do
our
scheduling
in
these
sorts
of
things.
The
main
point
here
is:
please
do
not
think
about
your
cluster
as
being
good
at
a
hundred
percent,
because
it's
not
right.
You
are
standing
on
a
time
bomb.
A
If
you
set
your
cluster
up,
200
percent,
so
I
just
got
totally
out
the
5-minute
window,
so
I'm
gonna
wrap
it
up.
Hey,
look
at
that
I
hit
70,
slides
reminder.
Some
of
this
was
aspirational.
A
lot
of
what
I'm
talking
about
here
is
not
implemented
yet,
but
this
is
where
we
want
to
go
with
kubernetes
and
I.
Really
I
thought
it
was
important
to
bring
it
out
and
talk
about
these
things,
because
I
think
people
who
are
having
a
rough
time
using
these
new
cloud
native
techniques
are
running
afoul
of
some
of
these
assumptions.
A
We
still
have
a
long
way
to
go.
We
have
a
lot
of
work
to
do.
800
men
years,
maybe
just
became
1,600
patches,
are
welcome.
Kubernetes
is
an
open
project.
It's
an
open
community
I
love
to
wrap
with
this
slide,
because
this
is
it
like
the
reason
that
communities
is
is
successful
as
the
people
who
are
here
in
this
room,
the
people
who
use
it
that
people
who
contribute
to
it.
This
is
a
chance
for
you
to
come
and
say
what
you
need
out
of
the
system.
So
with
that
I'll
turn
it
over
to
questions.
A
A
I,
don't
know
I'm
actually
not.
This
is
not
my
area
of
expertise
when
it
comes
to
scheduling
and
David
in
the
back
there.
We
need
something
around
priority.
We
already
have
actually
a
sort
of
a
very
binary
form
of
priority
that
we've
implemented
just
to
protect
critical
cluster
services
like
DNS,
as
this
is
a
very
real
concern,
so
we
have
a
binary
flag.
This
is
a
critical
thing
and
the
critical
things
will
be
able
to
sort
of
take
over
and
preempt
other
things.
D
A
There
questions
for
designing
applications,
we
try
not
to
I,
don't
want
to
just
to
prescribe
how
people
should
write
applications
right.
I,
don't
want
people
to
write
applications
for
kubernetes
I,
think
that
would
be
a
bad
thing
for
the
ecosystem
overall,
but
I
think
the
problems
that
we're
talking
about
here
are
not
unique
to
kubernetes
they're
they're
incumbent
in
any
sort
of
scheduling
system,
whether
that's
meso
source,
warm
or
kubernetes,
are
nomads
or
something
else
I
mean
you
have
the
same
problem
with
VMs
right
at
some
level.
A
You
have
to
right-size
your
VMs,
though
that's
a
lot
less
granular
generally,
and
so
you
have
a
lot
less
choices,
and
so
you
pick
something
that's
sort
of
the
next
largest
unit
than
what
you
think.
You're
gonna
need
people.
People
do
think
about
it.
In
that
sense,
we're
giving
people
a
much
more
granular
way
to
think
about
things
and
I
think
that
we
need
to
solve
that
in
general.
I,
don't
think
it's
specific
to
kubernetes
and
no
I
don't
have
any
hard
guidelines
for
you
question
just
yell
I'll
repeat
it.
A
Is
the
scheduler
going
to
take
into
account
real-time
resource
consumption
in
Borg?
We
call
that
reservation,
which
is
a
sort
of
modeled
recent
usage
and
prediction,
and
we
use
that
to
inform
how
much
we
overcommit
also
I,
don't
know
that
we
have
any
immediate
plans
to
do
the
exact
same
thing,
but
certainly
the
topic
comes
up
and
I
think
that,
given
the
success
of
it
within
board,
we'll
see
something
like
that.
A
So
the
question
is:
are
any
of
these
aspirational
features
in
Borg?
Most
of
them
are
most
of
what
I'm
talking
about
here
comes
directly
from
what
we've
done
in
Borg,
some
of
it
is
simplified,
and
some
of
it
is
tweaked
a
little
bit.
Would
we
consider
porting
it?
Porting
is
a
bad
choice
of
words,
because
Borg
is
300
million
lines
of
C++
and
kubernetes
is
not,
and
so
there's
really
no
port.
There's
a
reinvent,
and
yes,
so
I've
already
got
I've
got
people
inside
Google
who
are
interested
in
working
on
some
of
these
things.
A
If
I
can
shake
them
loose
from
their
current
projects,
but
I
don't
want.
The
craze
cannot
continue
to
be
a
Google
project,
Google
Google,
centric
project
right
and
we've
already
haven't.
If
you
saw
the
keynote
yesterday,
Ken's
slide
like
where
Google
is
already
less
than
50
percent
of
the
net
contributions
to
kubernetes,
which
is
amazing,
so
it
needs
to
be
a
community
driven
effort.
The
things
that
we've
done
inside
Google
do
not
always
apply
outside
of
Google.
A
Using
node
pools
to
partition
a
cluster,
so
node
pools
is
a
container
engine
centric
construct,
but
the
idea
is
that
you
have
a
set
of
different
templates
for
different
sizes
of
nodes,
meaning
that
you
end
up
with
a
heterogeneous
cluster
heterogeneous
cluster
is
a
fact
of
life.
I
mean
we
deal
with
it
in
board,
because
we
have
eight
generations
of
hardware
that
we
support
in
a
given
cluster.
A
A
So
if
you
have
a
32
core
machine-
and
you
have
a
four
core
mission-
you
have
some
room-
we
should
actually
save
the
32
core
machine
for
when
that
32
core
request
comes
along
and
not
burn
all
a
bunch
of
four
core
of
jobs
on
those
32
core
machines.
I
think
that
we
have
work
to
do
sort
of
in
that
in
that
vein,
but
I,
don't
I,
don't
want
to
encourage
people
to
say
well.
A
This
is
my
data
specific
pool,
and
this
is
my
compute
specific
pool
because
I
think
that
you're
going
to
end
up
with
lower
utilization.
That
said
far
be
it
for
me
to
say
who's
right
or
wrong
and
using
this,
if
it
works
for
you,
then
it's
right,
but
I
think
that
the
model
that
we
want
to
push
people
towards
is
more
unified.
Higher
can.
E
A
The
funny
thing
is
this
is
not
actually
my
area
of
expertise
at
all,
but
because
I
ended
up
talking
to
a
lot
of
customers
and
and
I
think
it's
interesting
I
thought
it
would
be
a
good
topic
to
talk
about
today.
David,
who
just
asked
the
or
just
pointed
that
out
is,
is
actually
a
scheduling,
expert
I,
just
pretend
to
be
one
and
so
I
think
we'd
be
happy
to
take
more
questions
and
we
kicked
out
of
the
room
because
there's
another
talk,
but
we'll
do
the
usual
thing
of
hanging
out
outside
and
talking.