►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: SIG Autoscaling - Marcin Wielgus & Maciej Pytel, Google
Come join members of SIG autoscaling for a deep dive into recent developments on the cluster, horizontal and vertical pod autoscaler. See the autoscalers in actions, learn how they make decisions, and more.
To learn more click here: https://sched.co/FuLf
B
Guess
we
can
start
this
talk?
My
name
is
Mark
Eva
booths
I'm,
an
engineer
at
Google
and
today
I'm
here
with
my
colleague
magic
to
give
you
a
deep
dive
into
auto-scaling
in
kubernetes,
and
this
is
the
continuation
of
yesterday
talks
and
yesterday
we
on
sticky
auto-scaling
introduction.
We
discuss
what
outer
scale
inquiry
is.
Why
do
you
need
outer
scaling?
How
can
you
auto
scale
your
application?
More
specifically
for
application
of
the
scaling
we
mentioned
about
two
options
you
may
use
the
first
one
was
horizontal,
auto
scaling
in
horizontal,
auto
scaling.
B
B
Of
course
you
don't
want
to
do
it
manually
and
for
that
purpose,
in
kubernetes
we
have
horizontal
pot.
Autoscaler
does
does
this
job
for
you,
horizontal
pot,
autoscaler
were
was
covered
in
details
yesterday,
so
we
will
not
talk
more
about
it.
The
other
option
to
scale
your
application
is
to
scale
your
application
vertically,
with
vertical
auto
scaling.
You
increase
our
decrease
the
size
of
instances
based
on
the
current
and
past
load
for
quite
a
long
time.
B
There
was
no
automatic
support
for
this
type
of
activity
in
kubernetes,
and
so
the
user
had
to
manually
specify
the
pot
sizes
and
come
with
these
good
pod
sizes
in
the
first
place.
Luckily,
this
spring
we
launched
vertical
pot
autoscaler,
which
is
hope
to
set
to
make
size
setting
automatically
for
you
vertical
pot.
Autoscaler
is
not
about
a
reaction
for
today
the
changes
in
in
traffic.
B
One
of
the
reasons
why
we
started
this
project
was
that
many
users
had
no
clue
how
to
set
pot
size
right
or
didn't,
set
it
at
all
or
set
it
partial
or
set
it
too
big,
too
small,
and
that
was
a
huge
problem
for
them.
So
one
of
the
vp8
goals
is
to
help
in
this
case
as
well
to
explain
how
VPI
exactly
does
the
job.
Let's
start
with
the
API,
the
API
is
currently
available
as
a
customer
source
definition
and
will
probably
stay
as
such
for
foreseeable
future.
B
This
summer
we
had
a
quite
extensive
discussion
with
SiC
architecture
about
whether
to
include
vertical
path.
Autoscaler
in
the
core
kubernetes,
along
with
HP
api,
this
decision
was
made
not
that
due
to
more
complex
nature
of
vertical
path,
autoscaler,
it
should
remain
to
be
developed
as
a
kind
of
separate
project,
and
for
that
reason
the
API
will
remain
a
CRD.
B
At
the
vertebral
Evan,
half
VPA
looks
more
or
less
like
every
other
kubernetes
object.
It
has
a
name
in
object
matter.
It
has
a
namespace
information,
you
can
add
labels
or
annotation
to
it
exactly
as
sports
or
deployments
or
replica
set
and
other
objects
in
kubernetes,
but,
more
importantly,
in
a
similar
fashion,
as
other
object
it
contains
pack
and
Status
I
will
cover
it
right
now.
So,
let's
start
with
specification
in
specification,
you
have
three
fields.
B
The
next
item
is
the
update
policy.
It
will
tell
what
exactly
you
would
like
VP
to
do.
The
first
option
for
update
policy
is
to
set
it
to
off.
In
that
case,
a
VP
I
will
only
provide
you
with
the
information
what
the
pot
size
should
be.
However,
it
will
execute
no
action.
The
other
option
is
to
set
it
update
mode
to
initial.
In
that
case,
the
recommendation
will
be
applied
only
to
newly
created
pots,
and
the
existing
pots
will
be
left
alone.
B
B
The
first
element
of
spec
allows
you
to
specify
which
containers
should
be
actually
affected
by
VP
a
so
there
is
a
container
name
here,
and
you
can
also
specify
what
should
be
the
minimum
and
maximum
amount
of
resources
that
you
want
to
give
to
individual
containers
and,
of
course,
you
can
specify
whether
you
would
like
s
container
to
be
auto
scale
or
not.
Why
is
it
important
because
you
don't
have
to
specify
it
for
every
single
container
in
your
application?
B
You
may
use
a
wildcard
pattern
asterisk
here,
and
in
that
case
you
will
be
applying
vertical
path,
autoscaler
to
all
containers.
Inside
of
your
pots.
If
you
are
applying
here
or
vpa
to
all
containers
in
your
pots,
you
may
want
to
exclude
some
of
these
containers
from
VPA,
and
for
that
reason
we
have
this
field
here,
which
can
either
be
set
to
out
or
off
and
enough
PA
will
not
provide
recommendation
or
touch
the
containers
I
mentioned
here.
So
we
know
what
we
can
ask
you
a
vertical
photo
scaler
for
now.
C
B
B
The
other
field
is
the
recommendation
itself.
It
starts
with
the
container
name,
which
tells
you
to
which
container
this
recommendation
can
be
applied.
Then
we
have
four
sets
of
number.
The
first
one
is
relatively
easy.
It
gives
you
information
what
should
be
the
container
size?
The
three
others
are
slightly
more
complex,
so
lower
bound
and
upper
bound.
Is
the
minimum
and
maximum
amount
of
resource
maximum
minimum
and
maximum
size
of
the
pot
that
we
pay
is
still
considered?
Okay.
B
So
if
you
are
running
VP
in
the
fully
automatic
mode
and
the
pot
is
within
this
lower
and
upper
bound,
it
will
not
be
updated
if
it's
outside
of
this
boundaries.
It
means
that
the
size
of
the
pot
is
wrong
and
needs
to
be
updated.
The
last
field
tells
you
what
recommended
what
should
be
the
pot
size
in
the
ideal
case,
if
you
didn't
provide
any
constraints
and
there
were
no
other
obstacles,
how
much
resources
would
VPI
allocate
to
this
particular
pot.
B
B
However,
to
give
you
information
that
the
pot
size
should
be
bigger,
this
5
gigabyte
is
passed
here
in
uncapped
target
and
obviously,
when
VP
will
be
applying
a
recommendation,
3
gigabytes
will
be
used.
Ok
now,
hopefully,
everyone
kind
of
understands
what
the
API
is
all
about.
Let's
see
how
it
works
in
practice.
Probably
the
most
tricky
part
of
this
design
is
how
we
update
part
request,
as
you
may
know,
for
example,
deployment
contain
history
of
the
updates
made
to
them.
B
So
if
you
update
deployment,
you
change
the
image
or
pod
size,
it
creates
a
new
history
in
the
history
which
is
basically
new
replica
set
until
a
VPI
gets
a
good
grasp
on
the
right
size
of
the
autant
containers
in
it
it
may
update
the
pods
multiple
times.
We
don't
consider
this
changes
important
enough
to
go
to
the
deployment
history
which,
for
example,
may
have
limited
capacity.
B
So
we
also
don't
want
to
limit
vp8
on
a
specific
set
of
controllers,
like
only
two
deployments,
or
only
two
State
foot
set.
So
we
decided
to
go
to
slightly
different
paths,
then
horizontal
path,
autoscaler,
and
not
to
update
any
controller
objects
at
all.
Instead,
we
use
admission
plugins
and
the
pod
size
is
basically
override
during
the
admission
pie-faced.
When
the
pot
is
created,
it
goes
through
multiple
plugins,
which
checks
whether
you
have
enough
quota
to
create
the
part
whether
you
have
enough
privileges
to
create
the
part
or
something
else
in
this
change.
B
B
Okay,
now
a
little
bit
about
the
recommendations,
which
are
obviously
very
important
in
this
picture.
The
recommendations
are
calculated
based
on
the
observed
with
real-world
usage
for
that
VP.
A
recommender
component
periodically
checks
the
usage
of
the
pots
that
are
under
VP
a
because
the
data
from
the
metric
server
recalculate
the
recommendations
and
put
these
recommendations
back
into
VP
a
status
object,
which
I
showed
you
a
couple
minutes
ago.
B
If
you
allow
vp8
to
work
in
a
fully
automatic
mode,
it
will
try
to
update
the
pots.
Currently,
the
only
way
to
update
the
pot
that
is
absolutely
running
is,
unfortunately
to
kill
it
and
recreate
it.
So
VP
a
constantly
analyzes,
both
set
of
VPS
and
the
list
of
pots
if
some
of
the
pots
that
are
under
VP
a
have
requests
outside
of
the
allowed
a
lower
and
upper
range.
B
B
Hopefully,
one
day
VPA
will
be
more
gentle
to
the
pods,
and
if
there
is
enough
capacity
on
the
machine
that
is,
the
pot
is
actually
running,
it
may
execute
the
in-place
upgrades
right
now.
There
is
not
an
option.
However,
there
are
a
couple
of
designs:
how
to
do
it,
and
so,
hopefully,
in
the
near
future,
we'll
have
it
okay,
so
now
a
little
bit
about
status
and
we
are
graduating
via
vertical
path,
autoscaler
to
beta
right
now.
What
does
it
mean
for
you?
B
First
of
all,
it
means
that
VP
a
actually
works
and
does
its
job,
while
in
alpha
it
has
been
tested
by
a
bunch
of
clients,
and
their
feedback
has
been
applied
to
the
product.
Being
beta
also
means
that
the
API
is
relatively
stable.
We
may
change
something
in
the
future,
however,
the
changes
will
not
be
drastic
and
they
will
be
preceded
by
with
the
appropriate
and
application
period.
It
also
means
that
you
can
slowly
start
using
VP
in
your
production
environment.
B
If
you
want
to
do
it,
you
should
start
using
VPN
product
by
running
it
in
recommendation
on
remote,
keep
it
in
such
state
for
couple
days
and
check
whether
this
record
its
recommendations,
work
for
you.
It
is
important
that
during
this
run,
your
pods
get
legitimate
real-world
traffic
in
the
appropriate
quantity.
Otherwise,
the
recommendations
of
VP
will
not
be
valid.
B
If
you
are
not
running
on
gke
and
just
set
VP
mode
to
outer,
we
strongly
advise
you
to
put
size
limits
on
your
containers
so
that
they
don't
go
beyond
node
capacity
in
your
cluster
and
in
such
a
way
become
unschedulable,
and
obviously
we
recommend
using
cluster
autoscaler
will
be
PA
because
it
may
give
you
capacity
when
the
pods
grow.
If
you
are
on
Decatur,
is
be
tighter
integration
of
VP
and
cluster
autoscaler
and
scheduler,
and
you
may
execute
slightly
less
caution.
B
D
Yes,
so
let's
talk
about
cluster
to
Sky
data
so
very
quickly,
we
talked
about
it
a
bit
yesterday,
but
it's
the
component
that
makes
sure
that
you
always
have
the
right
number
of
nodes
in
your
cluster.
So
basically,
if
there
is
not
enough
space
to
run
some
of
your
thoughts,
cluster
autoscaler
will
add
modern
nodes.
D
If
there
are
some
nodes
that
are
underutilized
and
can
be
safely
removed,
cluster
to
scale
will
remove
those
to
save
you
some
money,
so
cluster
autoscaler
has
been
in
GA
for
over
a
year
now,
I
think
since
kubernetes
1.8
and
currently
it
supports
for
cloud
providers,
Google
cloud
platform,
AWS,
agile
and
Alibaba
cloud,
and
hopefully
there
are
more
coming
so
yeah.
Let's,
let's
see
how
it
works
and
to
talk
about
crystal
to
scalar,
we
always
have
to
talk,
is
to
start
talking
about
the
scheduler.
D
So
the
scheduler
is
the
kubernetes
component.
That
decides
which
port
should
run
on
which
node
and
for
the
class
notice
Kara
it's
quite
important.
How
scheduler
does
it
so
without
going
too
deeply,
but
a
bit
deeply
scheduler
has
a
process
of
deciding
which
note
is
the
best
fit
for
there
for
each
pot.
So
imagine
we
have
a
new
pot
that
has
just
been
created
and
it
needs
to
be
scheduled.
So
basically
the
two
steps
in
the
schedule.
D
The
first
one
is
the
predicate
functions
and
those
are
sort
of
how
the
requirements
of
the
pot,
basically
things
that
must
be
to
my
folder
pot
to
ever
be
able
to
run
on
a
given
note.
So
some
example
predicates
would
be
added
enough.
Resources
left
on
the
note
to
satisfy
the
pot
resource
requests.
Does
the
note
match
pot
note
selector
of
note
affinity,
I'll,
all
the
port
affinity
and
anti
fi
ET
rules
observed
and
so
on
and
so
forth.
D
This
I
think
between
10
and
20
of
those
and
basically
every
single
predicate
must
pass
from
the
pot
to
be
able
to
schedule
on
a
given
node.
So
once
we
filter
out
the
pots
that
cannot
be
scheduled
at
the
nodes
that
cannot
be
used
for
a
given
pot
and
there
is
the
second
phase
and
it's
about
priority
function.
D
So
those
are
the
functions
that
basically
rate
each
node
to
say
how
desirable
it
is
to
run
a
specific
pot
on
this
node
and
then
they
are
used
to
select
exactly
which
node
will
be
finally
used,
but
I
won't
go
into
more
details
of
priorities
because
we
actually
don't
care
about
those
at
all
in
auto-scaling.
The
important
part
is
the
predicate.
D
So
basically
the
way
that
cast
out
two
scalawags
is
it
guarantees
that
for
every
pot
there
will
be
at
least
one
node
that
passes
all
predicates.
So
there
will
be
a
note
that
makes
it
possible
to
schedule
this
pot.
So
this
one
concept,
that's
going
to
be
very
important
in
our
thinking
about
craft
autoscaler,
and
then
there
is
another
concept,
an
old
group.
So
this
is
specifically
crafted
out
of
schedule
concept
and
the
node
group
is
a
set
of
identical
notes.
D
So
maybe
all
your
two
CPU
nodes
in
the
cluster
will
be
one
node
group
and
then
you'll
follow
CPU
notes,
be
another
notable
and
the
way
crashed
out
of
scale
actually
works.
Is
it
to
the
sizes?
Don't
know
those
note
boobs.
So
if
it
wants
to
add
two
more
to
CPU
notes
will
cluster
it
will
just
resize
the
corresponding
node
group
and
increase
its
size
by
two.
D
So
with
all
those
concepts,
let's
go
into
CA
propel
so
very
quickly.
What
does
cast
out
the
scaler?
Actually
do
so
it
monitors
ilk,
raster
and
it
looks
for
pending
pots,
so
the
pots
that
cannot
be
scheduled
because
there
is
not
a
single
node
that
passes
all
predicates
once
it
finds
such
nodes.
It's
actually
simulates
kubernetes
scheduler
to
see
if
adding
some
new
nodes
would
solve
the
issue.
D
If
it
would,
it
uses
the
node
groups,
it
resizes
the
node
group,
based
on
the
result
of
this
simulation
and
a
lot
of
misconceptions
about
cross
delta
scalar
a
lot
of
things
that
people
expect
it
to
do,
which
actually
doesn't
do
so.
The
node
group
is
basically
implemented
on
the
cloud
provider
level
and
the
resizing
it
just
creates
the
VM
cross.
Delta
scalar
doesn't
register
this
VM
in
kubernetes.
It
doesn't
configure
it,
it
doesn't
do
anything
with
it.
D
Another
common
misconception
is
what
it
looks
at,
because
it's
based
on
scheduling
it
doesn't
actually
care
about
resource
usage.
If
you
set
your
pot
resource
request
very
low
and
it's
booming
or
it's
burning,
the
node
just
using
way
more
cpu
than
it
requested,
crust
out
or
scaleable
will
not
react
because
it's
still
schedulable
it
doesn't
impact
scheduling.
So
for
that
sort
of
thing
you
need
pot
outer
scales
and
because
of
how
tightly
we'll
integrate
it
with
scheduler,
we
don't
support
any
custom
scheduling.
D
So
if
use
that,
then,
unfortunately,
you
cannot
discuss
tile
to
scale
and
okay.
So
now,
let's
proceed
and
sort
of
look
how
it
all
works.
Now
that
we
know
what
it
does
and
the
basically
cross
delta
scalar
is
made
of
three
components:
there
is
the
crowd
provided
module
so
that
basically
needs
to
be
implemented
separately
for
each
cloud
provider,
and
we
currently
have
four
implementations
as
I
said,
and
that
basically
is
the
implementation
of
the
node
group
concept.
D
So
this
is
the
interface
between
class,
Delta,
Skylar
and
cloud
provider,
so
we
have
some
common
abstraction
that
we
can
use.
The
other
part
is
the
imported
scheduler
code.
So
the
way
we
simulate
scheduler
is
we
literally
just
take
and
import
the
predicate
code
from
schedule.
The
good
thing
about
it
is
any
new
feature
that
is
ever
added
to
schedule
is
automatically
supported
by
class
delta
scalar
because
we
will
just
import
it.
The
best
thing
is
is
actually
important
to
learn
the
say,
like
the
version
of
the
cross
delta
scalar.
D
That
is
the
same
as
the
version
of
the
scheduler
that
is
running
in
your
cluster.
So
what
we
do
is
we
release
new
class
delta
scalar
for
each
new
kubernetes
release.
So
if
you're
using
kubernetes
1.12,
you
should
be
using
cross
Delta
scale
at
one
point.
For
12
with
1.15,
we
will
release
class
Delta,
scalar,
one
pinpoint
15
and
so
on
and
so
forth.
D
D
We
just
see
what
would
happen
if
you
added
one
node
and
we
take
the
first
pot
and
we
want
the
scheduler
pedicles
that
we
took
from
the
scheduler
and
we
see
what
will
happen
so
in
this
case
the
Potter
fits
on
the
node,
so
we're
all
good,
but
there's
still
two
more
pending
pots,
so
we
try
to
schedule
them
as
well
and
now
the
problem
happens
because
this
pot
won't
fit
anymore,
because
the
first
pot
basically
took
the
whole
note.
So,
okay,
one
note
is
clearly
not
enough.
D
Let's
add
one
more
and
let's
see
what
happens,
then
the
pot
can
schedule
okay,
so
already
helped
two
of
our
three
pots.
By
adding
two
notes,
we
have
won
multiple
remaining,
we
try
it
with
the
face.
Node
doesn't
work.
Second,
one,
it's
schedules:
okay,
two
notes
are
enough:
we
can
help
all
our
pots
by
adding
two
notes.
Let's
do
that
and
that's
basically
how
crust
outer
scale
works.
D
I
want
to
go
into
as
much
details
on
scale
it
down,
because
it's
slightly
more
complex,
but
the
basic
idea
is.
We
also
monitor
every
single
node
in
the
cluster
and
to
decide
if
we
actually
need
it,
and
there
is
a
bunch
of
criteria
we
use.
So
first
of
all
note
is
unneeded.
If
the
utilization
is
below
50%
well,
it
may
be
an
idiot
if
the
utilization
is
below
50%.
D
We
may
change
this
threshold
in
future,
so
don't
get
to
use
to
the
exact
number,
but
another
important
criteria
is
that
actually
all
the
ports
that
are
running
on
the
node
can
be
moved
to
somewhere
else
in
the
cluster.
We
once
again
use
scheduler
code
and
simulate
scheduling
to
see
if
we
remove
this
nodes
with
all
the
pods
be
schedulable
somewhere
else.
And
finally,
there
are
some
kinds
of
ports
that
we
are
basically
too
afraid
to
touch
and
will
never
remove
a
node
that
has
one
of
those
ports.
D
So
we
don't
touch
the
stem
ports
because
restarting
those
can
cause
some
sort
of
system
outage.
We
don't
touch
ports
with
storage,
because
we
local
storage,
because
we're
afraid
that
we
may
destroy
some
data
and
we
also
have
some
specific
mechanism
that
allow
you
to
say
that
port
should
never
ever
be
restarted
by
class
Delta
scalar,
because
of
whatever
reason
you
have
to
specify
it
and
we
fall
of
that.
It's
actually
quite
a
complex
Fink.
D
So
I
have
a
few
I
have
a
section
about
how
to
see
what's
going
on
with
class
Delta
scalar
and
how
to
debug
it.
And
the
first
thing
is:
we
have
this
class
Delta,
scalars
stateís
config
map
and
it's
basically
automatically
written
and
refreshed
by
class
Delta
scalar,
and
it
gives
you
the
current
status
of
your
cluster
as
perceived
by
class
Delta
scalar.
So
it
describes
how
your
cluster
looks
like
how
every
node
group
looks
like
you
can
actually
find.
What's
the
actual
size
of
the
node
group.
D
As
in
how
many
nodes
that
are
in
kubernetes
versus
how
many
VMs
have
been
requested
for
crowd
provider,
so,
for
example,
you
can
see
that
maybe
some
of
the
VMS
haven't
managed
to
start
up
successfully
and
so
on,
and
it
also
tells
about
crustal
to
scaling
operations.
So,
for
example,
in
this
case
you
can
see
there
is
scale
up
in
progress.
There
are
currently
two
notes
but
crascell
to
scale
it
requested
for
more
nodes.
So
that's
one
thing:
that's
quite
useful
to
see.
D
What's
going
on
with
class
Delta
scalar,
another
things
are
cubed,
netis
events,
we
write
a
lot
of
those.
So
basically,
whenever
crossed
out
the
schedule
does
scale
up
a
scaled
down.
Basically
any
operation
there
will
be
an
event
describing
what
just
happened.
So
you
can
use
those
to
basically
trace
the
history
of
class
delta
scheduled
operations.
D
Another
useful
event
is
basically,
we
write
events
on
ports,
so
if
a
port
was
a
reason
for
scale-up,
there
will
be
an
event
saying
that
if
we
restarted
a
pod
because
we
did
a
scale
down,
there
will
be
an
event.
And
finally,
there
is
an
event
that
will
be
on
the
pot
if
we
see
that
the
pod
is
pending,
but
we
cannot
actually
find
any
skylab
that
would
help
that
pot.
That
also
will
result
in
an
event.
D
So
you
can
look
for
those
if
you
think
it's
not
working
correctly
and
more
info
is
in
our
repository.
There
is
all
the
code
that
is
Vietnamese
separate
for
each
cloud
provider.
That
says
exactly
how
to
run
across
the
autoscaler
on
each
cloud
provider
and
we
have
a
very
extensive
FAQ
that
sort
of
goes
into.
Why
is
crust
autoscaler,
not
removing
some
notes
that
I
think
it
should
remove?
Why
is
it
not
adding
some
notes
and
so
on?
D
D
D
D
E
B
Are
trying
to
do
exactly
the
same
thing?
So
if
you
are
right
now
using
HP
and
VP
at
the
same
moment,
they
will
be
trying
to
both,
for
example,
increase
the
pot
and
increase
the
pot
count
or
decrease
the
pot
count
and
decrease
the
pot
size.
So
at
this
very
moment
we
don't
recommend
you
using
HPA
based
on
CPU
utilization
and
vertical
pot
autoscaler.
E
Second
question:
cysts:
as
far
as
I
know,
then
the
autoscaler
will
always
go
up
one,
not
a
that
IRA,
because
in
my
use
case
then
I
have
an
issue
when
the
horizontal
autoscaler
they
kick
in
for
some
services
and
they
need
to
apply
for
a
symbol
like
100,
mob
hearts
and
then
in
the
case
that,
even
though
cluster
Auto
scalars
come
up
new,
not
it
will
not
be
scheduled.
All
of
the
parts
total
new
parts
on
a
new,
not
against
Costa,
also
scalar,
just
stuck
at
that
site
it.
It
cannot
scale
up
anymore.
So.
B
C
I
have
a
question
for
the
autoscaler,
so
you
mentioned
that
if
I
understand
correctly,
you
put
the
scheduler
as
part
of
the
library
for
the
autoscaler.
You
simulate
the
results
from
as
the
scheduler
and
the
primary
goal
is
to
have
reducers
scheduling
pressure.
Have
you
ever
considered
other
design
where
you
put
these
autoscaler
functionality
into
the
scatter
itself
or
design
a
API
between
consuming
the
updates
from
the
scheduler
trying
to
understand
the
design
reactionary
behind
that?
Thank
you.
Okay.
B
D
Regarding
the
API
part,
why
would?
Because
you
could
think
that,
like
one
possible
design
bit
for
the
scheduler
to
have
some
sort
of
a
delight
and
mode,
so
the
problem
is
with
not
actually
running
against
the
actual
cluster
because
we're
adding
those
nodes
in
our
simulation,
so
the
state
of
the
cluster
changes.
So
we
actually.
What
we
need
is
to
answer
if
the
pot
would
be
schedulable
in
some
sort
of
fictional
scenario.
Well,
we
have
in
memory,
node
objects
and
then
I
mean,
first
of
all,
that's
not
supported
by
scheduler.