►
Description
Kueue: A Kubernetes-native Job Queueing - Abdullah Gharaibeh, Google
Most Kubernetes core components are pod centric, including the scheduler and cluster autoscaler. This works well for service workloads where the pods of a service are mostly independent and all services are expected to be running at all times. However, for batch workloads, it does not make sense to focus only on pods, as the partial execution of pods from multiple parallel batch jobs may lead to deadlocks where many jobs may be simultaneously active while none is able to make sufficient progress to completion or start at all. Even for single-pod batch jobs, whether on-prem or in the cloud with autoscaling capabilities, the reality is that clusters have finite capacity: constraints on resource usage exist for quota and cost management (especially true for GPUs) and so users will want an easy way to fairly and efficiently share the resources. Kueue addresses the above limitations, offering queueing capabilities commonly exist in legacy batch schedulers in the most k8s native way. It is a k8s subproject currently under development at https://github.com/kubernetes-sigs/kueue.
A
A
Just
I
want
to
see
that
this
is
a
community,
a
projects
and
open
source
project,
and
we
hopefully
it
started
like
a
few
months
ago.
It's
it's
new.
We
had
our
first
release
really
like
a
couple
weeks
ago,
we're
here
to
present
the
idea.
The
high
level
idea,
how
we
think
that
curing
should
work
or
potentially
could
work
in
one
way
how
it
could
work
on
top
of
kubernetes.
A
So,
just
being
the
first
talk
in
this
batch
working
group,
maybe
we
should
just
define
what
is
the
job?
The
way
we're
looking
at
the
definition
of
a
job?
Is
that
any
computation
that
has
a
start
and
a
finish?
So
it's
mostly
computations
that
run
to
completion.
Imagine
like
with
service
type
workloads.
You
have
a
process
listening
on
a
port
all
the
time
it
can
be
infinitely
doing
that
receives,
requests
and
processing
them.
A
That's
why
most
applications
we
have
in
kubernetes,
like
deployment
replica
set
state,
will
set
if
your
pod
dies
or
finishes
it
will
directly
recreate,
because
the
assumption
is
that
this
thing
this
deployment
should
always
continue
to
run.
In
contrast,
the
job
api.
It
allows
a
pod
to
run
to
completion,
and
so
it
will
not
restart
the
part,
and
so
we
have
this
concept
of
that.
The
part
runs
the
completion.
A
We
count
how
many
comp,
how
many,
how
many
points
are
going
to
get
completed,
and
this
is
how
we
define
that
the
job
finished
or
not
job
status,
and
so
this
is
a
very
simple
definition
of
a
job,
but
we
find
it
fits
most
types
of
batch
workloads
and
this
kind
of
like
brings
us
to.
The
second
point
is
like
these
types
of
batch
workloads.
They
usually
aren't
not
not
just
a
single
part
like
if
we
cast
these
definitions
into
kubernetes
terminology
and
using
pods
as
a
task.
A
They
either
run
independently,
like
monte,
carlo
simulations
processing,
the
frames
of
an
image
processing
the
trading
days
and,
like
you
know
when
you,
when
you,
if
you
think
about
fintech
or
they
collaboratively
process
a
task,
basically
there's
a
ton
of
communication
that
happens
between
a
group
of
tasks,
think
about
mpi,
doing
and
body
simulation,
for
example,
to,
for
example,
simulate
how
particles
with
gravitational
power
react
over
time.
I
just
want
to
hear
mention
that
the
gray
color
is
lightly
whitish
on
this
on
this
screen.
So
please
use
your
imagination.
A
All
right,
so
a
very
important
characteristic
of
batch
workloads
that
we
can
capitalize
on
as
people
who
are
trying
to
develop
batch,
schedulers
or
batch
orchestrators
is
that
they
are
often
flexible
on
multiple
dimensions.
They
are
flexible
on
time.
They
can
sometimes
they
don't
necessarily
need
to
run
right.
Now,
it's
not
like
a
a
pro.
You
know
a
request
that
you're
serving
to
an
end
user.
Sometimes
they
can,
they
can
wait
and
run
their
start.
Time
can
be
pushed
further
ahead,
they're,
sometimes
flexible,
on
location.
A
If
you
think
about
cloud,
you
have
multiple
regions,
multiple
zones,
they
don't
necessarily
all
the
time
need
to
run
in
a
specific
region.
They
could
run
in
different
in
a
different
zone.
For
example.
Of
course,
there
is
data,
locality
restrictions,
but
giving,
but
taking
that
reconciliation
they
could
still
run
on
different
places.
They
are
also
flexible
in
types
of
resources.
A
B
Thank
you
very
much
the
time
it
takes
to
finish
this
job,
so
especially
in
in
my
experience,
there's
a
huge
difference
between
having
some
kind
of
micro
batches,
which
running
only
seconds
and
those
running,
maybe
for
minutes
hours,
and
hence
I'm
here.
We
have
like
a
big
problem
with
our
micro,
batching
jobs
and
right.
A
Yeah,
definitely
that's
a
very
good
point,
so
you
have
different
type
different
running
time
of
for
for
different
types
of
batch
jobs
that
can
be
taken
into
consideration
when
trying
to
schedule
these
jobs.
This
is
by
no
means
like
summarizing
everything
we're
just
trying
to
highlight
some
of
the
aspects
that
we're
taking
into
account
when
solving
this
problem
in
the
context
of
q.
A
And
why
do
we
need
job
queueing?
Well,
I
mean,
as
I
mentioned
we
in
most
cases
we
have
lots
of
jobs
that
needs
to
be
processed,
but
we
have
limited
amount
of
resources.
So
we
need
to
maximize
utilization
for
these
resources
and
if
some
jobs
can't
fit
right
now,
we
need
to
decide
which
ones
should
start
and
which
ones
should
wait,
and
we
have
two
types
of
infrastructure.
A
A
You
can
infinitely
grow,
but
that's
not
actually
always
the
case
because
you
have
discounts.
For
example,
you
buy
a
discount
on
a
specific
amount
of
cpus
or
core
hours,
for
example,
so
you
want
to
only
fit
your
jobs
and
your
processing
within
that
discount
or
you
could
have
spending
limits.
You
don't
want.
Just
like
you
know,
as
many
jobs
as
you
have,
you
want
to
continue
to
buy
more
and
more.
You
have
you
have
some
restrictions
on
how
much
money
you
want
to
spend.
You
have
also
paired
tenant
limits
like
different
users.
A
Can't
they
just
spend
them
as
much
as
they
want.
Some
users
want.
Some
tenants
could
spend
more
than
than
others,
but
also
cluster
limits
like
in
kubernetes.
We
have
cluster
limits.
You
can't
have,
for
example
like
in
it
by
default.
We
have
probably,
I
think,
5k
in
gk.
You
could
have
like
15k,
but
you
see
at
the
end.
You
still
have
some
limits.
There's
a
limit
to
how
much
the
master,
for
example,
can
scale
how
many
nodes
you
could
have
in
the
in
the
in
the
cluster.
A
And
so
here
I'm
trying
to
summarize
like
what
is
it
that
we
want
to
solve
with
job
queuing?
Obviously,
we
want
a
queueing
jobs
that
don't
fit
should
for
existing
classes
should
wait.
The
other
thing
is
execution
order
who
should
get
to
execute
now
whether
it's
based
on,
for
example,
the
typical
fifo
like
submission
time
or
priority?
You
need
some
these
knobs
that
allows
users
to
to
could
figure
how
ordering
of
the
jobs
should
should
happen.
Also
for
pershing.
A
A
Again
like
it
could
also
be
flexible
on
time
and
location,
and
so
why
are
we
trying
to
propose
a
new
controller?
Well,
with
playing
kubernetes
playing
kubernetes
continuously
tries
to
start
workloads.
It
can
work
itself
today.
So
every
time
you
create
a
job,
the
job
controller
will
create
the
pods,
and
if
you
have
limited
amount
of
resources,
these
pods
will
continue
to
be
unscheduled
and
the
control
plan
will
continuously
try
to
schedule
them
to
no
avail
right.
So
this
is
not
a
desired
behavior.
There's
no
built-in
approach
within
kubernetes.
A
You
say:
okay,
let
me
stop
now
and
try
again
like
later,
and
we
have
one
way
of
doing
that.
It's
called
like
kubernetes
resource
quality,
but
the
the
way
that
resource
courses
are
built
is
not
really
fit
the
job
queueing
paradigm.
It
mostly
tries
to
protect
the
clusters
from
failing
over,
and
so
the
way
that
it
applies
applies
a
resource
creation,
whether
you
get
to
create
the
pod
in
the
first
place,
or
not
so,
there's
no
way
to
say.
A
Okay,
let
me
wait
and
then
I'll
try
again
it's
up
to
some
other
external
control
to
keep
trying
and
even
then
that's,
not
a
good
approach.
You
can
continue
to
hammer
the
api
server,
so
it
doesn't
really
fit
well
the
concept
of
a
queue.
There
are
multiple
existing
customer
schedulers
custom
schedulers.
They
all
fit
really
nice
in
different
use
cases,
but
one
thing
that
we
wanted
to
avoid
is
re-implementing
existing
functionality.
We
didn't
want
to
do
implement
pod
to
node
scheduling,
for
example.
We
want
to
continue
to
have
the
cube
scheduler
to
do
that.
A
We
didn't
want
to
re-implement
the
job
control
drop
lifecycle
manager.
We
wanted
to
continue
to
have
the
core
kubernetes
job
control
to
do
job
management
of
the
of
a
single
job.
The
second
thing
is
that
existing
scheduler
didn't
have
clear
integration
with
auto
scaling,
which
is
really
important
characteristic
on
the
cloud
and
finally,
last
but
not
least,
we
found
that
nobody
actually
saw
the
problem
of
resource
flexibility
fungibility,
neither
at
the
api
level
or
or
mechanically.
A
A
So
what
is
our
was
it
like?
The
main
design
principle
for
queue
is
that
we
don't
want
to
implement
anything
that
already
exists,
nor
implementation
of
auto
scaling,
pod
scheduling,
lifecycle,
job
lifecycle,
management,
even
admission,
control,
auto
scaling.
We
have
cluster
order
scale.
We
have
a
bunch
of
other
order,
scalers
out
there
as
well
other
than
the
one
hosted
by
kubernetes,
cube,
scheduler
hundred
spot
scheduling,
job
lifecycle,
management
handled
by
the
job
controller
admission
control.
A
We
have
many
policy
controllers
out
there
as
well
like
gatekeeper,
for
example,
and
we
want
to
have
native
support
for
the
batch
v1
job
api.
The
idea
here
is
that
we
want
to
have
this
on
ramp,
really
easy
process
of
onboarding
batch
onto
kubernetes.
You
want
to
start
by
I'm
a
user.
I
just
create
a
batch
up
using
v1
job.
Now
my
needs
has
grown.
I
want
queueing,
that's
it!
You
just
install
our
controller
v1
drop.
A
A
So
the
advantages
of
this
again,
as
I
mentioned,
we
reuse
significant,
existing
functionalities,
no
duplication,
there's
no
divergence
in
functionality
and-
and
it
enforces
also
separation
of
concerns.
We
don't
want
to
have
two
schedules
on
this.
On
the
on
the
controller.
We
don't
have
to
have
two
job
controllers
and
what
not?
This
comes
with
some
limitations
that,
like
the
last
column
here,
mentioned
some
some
some
potential
drawbacks
that
it
creates
two
levels
of
resource
management.
A
If
we
have
a
controller
at
the
top,
that
only
does
job
level
management,
it
might
diverge
from,
like
the
lower
level
layer
that
does
pod
level
management,
which
is
most
kubernetes
controllers,
do
right.
They
all
mostly
work
at
the
pod
level.
It
does
have
a
concern
related
to
starting
the
job.
So
if
you
assume
that
I'm
going
to
manage
the
job
once
I
decide
that
the
job
should
start,
I'm
going
to
start
it,
but
then
there
is
the
job
controller
that
have
to
react
to
start
the
pods
et
cetera.
A
So
there
is
that,
but
we
think
that
these
limitations,
our
trade-offs,
are
worth,
are
worth
it
versus
the
advantage
that
we're
gonna
get
and,
as
I
mentioned,
like
we're,
focusing
on
the
job
api,
the
existing
drop
api
is
the
main
vehicle
for
deploying
jobs.
Here,
I'm
just
quickly
mentioned
that
we
have
tried
to
fix
the
job
er.
We
understand
that
the
job
api
did
not
save
all
use
cases.
A
And
so
the
two
percent
is
that
the
queueing
controller
that
we
have
focused
on
is
the
batch
admin,
the
batch
user.
The
batch
admin
is
the
one
that
is
going
to
set
up
the
tenants
set
up
the
queues,
the
usage
limits
and
the
batch
user.
Their
user
gene
should
be
simple,
the
same
thing:
they
use
the
v1
job.
The
only
thing
that
they
have
to
do
is
really
simple:
just
select
the
queue
and
that's
it.
A
Okay,
so
let's
get
to
the
heart
of
it.
I
spoke
a
lot.
Where
is
q?
Show
me
okay,
so
the
resource
model
is
really
introducing
two
main
resources:
the
queue
and
cluster
queue.
Namespace
is
an
already
existing
concept.
As
you
know,
it
is
the
one
that
represents
the
tenant.
Okay,
you,
if
you
have
a
new
tenant,
for
example,
you
have
a
a
new
group
of
a
new
team.
You
create
a
namespace
for
them,
that's
their
working
space.
Basically,
a
queue
is
a
namespace
resource.
A
It
represents
like
a
close-rated
single
tenant
jobs
and
there's
the
cluster
eq.
Think
of
it
as
raw
like
the
cluster
roles
and
roles
like
in
in
in
our
back
cluster
queue
is
a
cluster
resource
cluster
scoped
resource.
It
is
where
we
define
the
core
actual
quotas
for
a
specific
group
of
group
of
tenants,
and
in
this
context
the
queue
is
really.
The
namespace
resource
is
just
simply
a
pointer
to
the
cluster
queue
where
the
resource
is
going
to
be
consumed,
a
user
within
a
namespace.
A
A
They
list
them
and
then
they
just
submit
the
drop
to
that
queue,
and
here
I
want
to
highlight
that
multiple
different
namespaces,
they
can't
point
to
the
same
cluster
queue
and
that's
why,
like
multiple
tenants,
can
share
the
same
amount
of
resources
and
the
actual
queueing
in
order
of
execution
of
the
job
will
happen
at
the
cluster
queue
level.
So
here
you
have
job
1
and
job
3
from
the
from
the
robo
actions
team
from
the
robovirgin
team.
They
have
drop
two
and
four.
A
The
actual
starting
order
is
going
to
be
happening
like
you're
going
to
be
married,
it
is
going
to
be
happening
at
the
cluster
q
level,
for
example,
for
the
robotics
team
pool
and
another
one
last
thing
here
is
that
you
could
have
multiple
cluster
queues,
of
course,
in
the
cluster,
for
example,
one
for
robotics
team
and
the
other
one
for
the
natural
language
processing
team,
each
one
could
have
their
own
resource
pool.
We
have
this
concept
of
a
borrowing
cohort
like
can.
A
If
you,
if
you
see
like
there's
a
cohort
parameter,
there
can
define
that
and
basically
decides
how
far
sharing
happens.
Who
can
borrow
format,
I'm
going
to
highlight
this
in
the
next
few
slides
in
the
apis.
So
again
I
I
spoke
this
about
this
a
lot
we
use
the
normal
job
api
here
and
the
only
thing
that
you
need
to
do
to
use
a
specific
queue
is
just
basically
set
the
queue
annotation
we're
trying
to
make
that
a
parameter
first
class
citizen
of
the
of
the
job
api.
A
So,
as
I
mentioned,
the
queue
api
is
really
simple.
This
is
the
namespace
resource.
The
only
like
in
in
simplest
form,
the
only
thing
that
has
it's
a
pointer
to
the
cluster
queue
where
the
resources
will
be
allocated.
This
is
like
again.
This
model
is
splitting
the
queue
into
two
different
resources:
the
name
space,
one
namespace
one
fits
better,
the
like
the
kubernetes
model
in
general,
the
fact
that
namespace
is
the
organizing
concept
for
teams
and
where
you
do
our
back,
etc,
and
so
this
is
why
we
came
up
with
this.
With
this.
A
A
One
thing
that
we've
introduced
is
called
like
flavors
for
a
specific
resource.
Here,
I'm
only
mentioning
in
one
resource
called
is
the
cpu
you
could
you
could
define
multiple
flavors
of
a
cpu
here,
I'm
defining
two
flavors,
the
on-demand
one
and
the
spot
one
you
can
define
different
quarters
for
each
one.
The
minimum,
for
example,
is
the
minimum
amount
of
resources
you're
going
to
get
the
maximum.
How
much
you
can
you
can
scale
up
to
by
borrowing
from
another
cluster
queue
in
the
same
borrowing
cohort.
A
So,
as
I
mentioned,
dakota
is
basically
defines
the
total
amount
of
resources
that
can
be
requested
from
the
jobs
that
start
from
this.
Through
this
cluster
queue,
it
does
not
really
guarantee
that
these
pods
are
actually
going
to
get
scheduled.
So
I
want
to.
I
want
you
to
think
about
this
as
a
dynamic
content
manager,
not
as
like,
like
really
a
resource
manager
per
se,
it
doesn't
guarantee
that
the
pods
are
actually
going
to
schedule
it,
but
it
just
says
that
similar
to
resource
cortez
we're
going
to
allow
you
to
start.
A
This
is
the
amount
of
which
are
going
to
have
we're
going
to
depend
on
the
low
level
infrastructure
to
actually
provision
the
resources
for
you.
We
are
also
planning
here,
defining
only
quota,
we're
planning
to
support
budgets
in
the
future,
basically
like
over
time,
rather
than
just
on
a
specific
point
of
time.
A
So
again,
flavors
are
associated
with
tins
labels
when,
for
example,
when
I
start
a
job,
I
assign
it
a
specific
flavor,
for
example,
a
specific
job
when
it
starts
I'm
going
to
say,
okay,
I'm
going
to
take
10
cores
from
on
demand.
How
do
I
force
it
to
go
to
the
in
demand?
I'm
going
to
use
that
label,
translate
it
into
a
node,
selector
inject
it
in
the
job
and
that's
how
I
would
force
the
job
to
to
only
use
that
to
you
to
use
that
that
specific
quarter.
A
Teens
is
an
interesting
parameter
here.
It
allows
users
it's
like
it's
it's
an
opt
out
by
default
mechanism,
right
jobs
that
don't
tolerate,
for
example,
taint
they
will
not
if,
for
example,
we're
going
to
start
from
the
top
to
the
bottom,
the
key
controller
will
say:
okay,
let
me
try
to
give
it
some
cores
from
in
demand.
A
If
there
is
no
quota,
I
will
fall
back
to
spot,
but
if
the
job
doesn't
explicitly
tolerate
spot,
then
it
will,
it
will
continue
to
be
pending,
and
so
this
is
the
nice
thing
about
this
approach.
It's
kind
of
a
kubernetes
native
the
way
that
you
try
to
assign
pods
to
nodes
the
same
thing
here
you
use
node,
selector
and
use
tolerations
to
say
whether
I'm
opting
n2,
for
example,
using
spot
or
not,
or
you
can
also
say-
I
always
want
to
use
spot,
and
so
you
could
have.
A
This
is
the
last
point
on
the
job.
If
you
say
in
the
node
selector
I
want
spot,
then
I'm
going
to
skip
the
first
one
and
directly
assigned
to
you,
try
to
assign
to
you
a
quota
from
from
the
spot
flavor
so
that
that
is
the
those
are
the
two
main
apis.
This
is
the
the
main
api
for
defining
quotas.
One
last
thing
here
in
the
api
I
want
to
mention
is
the
borrowing
cohort
I
mentioned
that
before
in
the
resource
model,
different
cluster
cues
can
be
grouped
into
a
borrowing
cohort.
A
So
basically,
this
allows
you
to
do
like
some
sort
of
like
fair
sharing
or
your
unused
using
unused
quota
from
different
cluster
cues.
Imagine
you
have
a
research
department,
you
know
cohort
and
robotics
team
if
they
are
not
using
all
of
their
quota,
a
different
team
within
the
research
department
can
use
it.
A
They
use
the
only
thing
that
it
does,
creates
the
job
and
assign
it
to
a
queue.
One
thing
we're
doing
here
is
that
we
force
the
drop
to
start
in
a
suspended
mode.
No
pods
are
going
to
get
created
at
the
beginning.
We
do
have
a
suspend
of
a
flag
now
in
the
job
api.
We
can
use
a
web
hook
to
force
it
to
do
that.
A
The
second
thing
is
well
our
queuing
controller.
What
it
will
do
is
that
it
will
watch
for
these
suspended
jobs
and,
based
on
I
mean
the
cluster
cue,
that's
assigned
to
the
flavors
that
are
available.
The
code
is
available
its
order
and
whatnot.
It
will
admit
the
job.
What
that
means
that
it
assigns
its
specific
flavor,
give
it
a
quota
and
then
the
way
that
it
does
is.
Basically,
it
injects
the
affinity
into
the
into
the
job
and
unsuspends
the
job.
That's
basically
all
it
does.
It
doesn't
create
pause
it.
A
Doesn't
it
doesn't
do
anything
else
that
already
an
existing
controller?
Do
the
race
is
the
same
as
I
mentioned,
so
these
red
boxes
are
existing
controllers.
The
blue
one
is
the
one
that
we've
introduced
the
job
control
once
the
job
is
unsuspected,
we
create
the
parts
and
the
scheduler
assign
them
two
nodes.
A
So
how
do
we
handle
custom
workloads?
Not
everything
is
a
v1
job
and
not
everything
can
be
modeled
as
a
v1
job,
so
we
do
have
a
hook
to
do
that,
and
the
only
thing
we
we
require
from
a
custom
workload
to
work
in
in
queue
is
that
it
needs
to
be
represented
by
high
level
object
api
like
an
mpi
job
that
controls
creating
the
pods
etc
and
needs
to
implement
the
suspend
semantics,
which
is
the
whole
enabler
for
everything
related
to
queue.
A
We
envisioned
that
for
each
custom
workload
we
need
a
new
workload
controller,
that
worker
controller
does
that
syncing
between
the
actual
custom
workload
and
the
workload
type
that
we
have
in
in
queue,
and
so
the
batch
user
is
going
to
create
the
mpi
job.
For
example.
In
this
we
will
force
it
to
be
start
in
a
suspended
mode.
The
workload
controller
will
should
be
watching
for
these
suspended
mpi
jobs.
It
will
create
the
workload
the
the
key
workload
type
for
us
q
will
be
watching
for
the
workload
type
it
has.
A
It
will
basically
reflect
the
the
the
assignment
on
the
workload
status.
This
is
how,
basically,
we
admitted
the
workload,
then
the
workload
controller
watching
for
that
status
once
it
is
assigned
it
will
inject
the
node
affinity
based
on
the
selected
flavor
and
unsuspend
the
job.
We
have
actually
implemented
the
job
api.
Using
this
model
like
it's
not
really
built
into
queue,
we
have
a
really
tiny
job
controller
that
does
that
syncing
between
the
queue
workload
type
and
the
actual
job
api.
A
One
last
integration
point
that
I
want
to
highlight
is:
is
auto
scaling,
it's
really
important
for
us
to
support
auto
scaling
natively,
because
in
the
cloud
you
know
everything
is
auto
scaled,
and
this
is
the
promise
of
the
cloud.
Is
that
when
you
use
resources,
we're
gonna
scale
it
up
to
you
when
you
don't
we're
gonna
scale
it
down,
but
I
get.
I
guess
I
have
10
more
minutes
right.
A
So
how
do
we
envision
integration
with
with
auto
scalers?
Well,
one
way
is
to
say:
okay,
I'm
going
just
to
create
the
pods.
Some
cluster,
auto
scalers,
like
the
normal
cluster
autoscaler,
basically
reacts
to
unscheduled
posts
and
will
provide
the
resources
when
they
see
an
a
port.
That
is
unscheduled,
but
we
can
do
better
than
that.
We
envision
having
a
research
for
virginia
object
api,
let's
name
it
capacity,
request
that
research
that
cluster
auto
scalers
should
be
supporting.
Instead
of
just
reacting
to
pods.
A
We
want
an
explicit
api
to
request
a
specific
amount
of
resources
from
the
cluster
auto
scaler,
and
so
the
idea
here
is
that
the
cluster
order
scaler,
would
be
watching
for
this
cluster
request
for
capacity
request
and
potentially
scale
up
the
cluster
to
satisfy
that
request,
and
why
do
we
need
to
enable
this?
Well
it
like
this
approach.
It's
it's
an
enabler
for
location
flexibility
before
I,
we
won't
have
the
cluster
or
scale
aware
of
a
group
of
parts
like
when
I
want
to
schedule
resources
for
a
for
for
a
job.
A
I
want
to
say:
okay,
I
want
10
cores
of
each
for
each
for
10
pods
one
core
for
each,
but
I
want
all
of
them
to
be
in
this
region,
but
I
want
all
of
them
to
be
in
one
zone
but
don't
spread
them
across
zones.
I
want
any
single
zone,
type
of
semantics
and
so
well.
I
want
an
api
where
I
can
express
this
to
to
cluster
autoscaler.
A
It
also
allows
us
to
do
better
handling
of
stock
outs.
It
see
that
when
I
before,
I
actually
unsuspend
the
job,
I'm
going
to
create
a
cluster
capacity
request.
Do
I
actually
have
10
cores
to
to
to
provision
or
not
in
cloud
again
it
does
have
this
like
promise
of
infinite
scalability,
but
we
do
face
stock
outs
in
some
cases
on
black
friday
or
cyber
monday,
or
in
specific.
A
A
I
want
cluster
oscar
to
be
aware
of
existing
provision
resources
and
actually
trigger
a
scale
up
all
the
time
if
capacity
is
already
already
provisioned
and
the
way
that
this
we
envision
this
being
integrated
with
q
is
that
after
the
job
is
created,
q
is
not
always
going
to
just
simply
assign
a
flavor
and
unsuspend
the
job
before
it
unsuspends
the
job.
It
will
create
a
capacity
request.
A
We
see
that
cluster
order,
scaler
watching
for
these
capacity
requests
fulfill
it
and
then
queue
watches
for
this
capacity
request.
Once
it's
actually
satisfied,
it
will
inject
the
known
affinity
based
not
only
on
the
flavor,
but
also
the
capacity
requests
we
envision
there
to
be
something
similar.
For
example,
if
the
cluster
or
scalar
selected
a
specific
zone,
we're
going
to
inject
no
dfinity
to
that
specific
zone,
and
then
it
unsuspends
the
job
and
the
rest
is
the
same
jobs.
The
job
control
will
create
the
pods.
A
The
scheduler
will
place
them,
and
here
I
like
one
last
thing
I
wanted
to
highlight
and
stress,
is
that
we
are
trying
to
use
existing
controllers
as
they
are
we're
trying
to
use
the
primitives
that
they
give
us
to
control
when
to
schedule
what
to
schedule
and
where
we're
using
a
lot,
not
affinity,
not
selector
teens.
We
envision,
we
see
those
as
really
powerful
genetic.
You
know,
features
that
allows
us
to
do
higher
level.
A
You
know
types
of
decisions
while
delegating
the
low
level
pod
management
to
the
lower
level
controllers,
and
so
far
so
good
it
worked
us.
It
worked
for
us
really
well
for
the
job
api,
and-
and
this
is
like
my
summary,
so
it's
what
is
q.
So,
as
I
mentioned,
q
is,
is
a
queuing
controller.
You
can
think
of
it
also
as
an
elastic
curtain
manager.
A
A
A
What's
next
again,
we
want
to
prove
that
this
also
will
work
for
custom
workloads.
We
want
to
start
with
integration
with
common
operators
like
spark
kubeflow.
We
want
to
implement
job
preemption,
think
about
the
suspend
again
a
really
powerful,
simple
concept
when
you're
unsuspended
you're
starting
the
job
when
you
suspend
you
think
of
it
as
job
preemption.
So
we
want
to
use
that
to
do
more
advanced.
You
know
job
management,
maybe
based
on
job
runtime
budgets
multi-cluster
as
well
on
our
roadmap.
A
C
A
I'm
guessing
this
is
like
not
really
related
to
the
job
job
curing
per
se,
but
in
general
I
think
the
110
limit
is
not
like.
Is
I
don't
think
it's
fixed
you?
Could
you
could
adjust
it
basically
to
increase
the
number
of
pods
that
you
can
have
in
a
node?
It's
basically
how
many
ips,
I
think,
that's
the
main,
limiting
factor
there,
of
course,
in
addition
to
other
resources,
but
the
main
limiting
factor
is
typically,
we
assign,
I
think,
slash,
24
subnet,
that's
why
you
get
256.
A
E
A
Right
so
right
now
the
assumption
is
that
we
we
don't.
We
don't
really
know
about
the
semantics
of
the
job
how
these
pods
are
being
created.
We
only
manage
the
whole
job
like
as
a
single
unit,
so
it's
pretty
much.
You
can
think
of
it
as
all
or
nothing
scheduling,
but
it's
not
really
scheduling
all
or
nothing
quota
management.
So
I
have
quota
for
the
whole
index
job.
C
A
A
very
good
question:
it
is
on
our
roadmap.
We
don't
right
now
have
that
explicitly,
but
we've
discussed
this
in
the
community
how
to
handle
dynamic
resource
management
we
think
of
it
as
kind
of
similar
to
vertical
part
other
scale.
I
don't
know
if
some
of
you
were
following
that
in
the
community,
how
you
adjust
the
resources
for
a
pod,
we're
kind
of
leaning
towards
a
solution.
In
the
same
point.