►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Scheduling SIG - Bobby (Babak) Salamat, Google
This talk will be useful for cluster operators who are interested in learning about new scheduling features in Kubernetes. The talk will provide insights about ways to run clusters more efficiently and securely. We will give a brief summary of existing Scheduler features and then provide more details about some of the new features of the scheduler, namely priority and preemption, and gang scheduling. We will also provide a summary of Descheduler. Changes in the cluster size and available resources happen continuously. These changes may make some of the scheduling decisions made before sub-optimal under new conditions. Descheduler monitors clusters and helps fixing previously made scheudling decisions by removing (descheduling) already running Pods when there are better placement opportunities in clusters. We will also talk about the new Pod scheduling policies that allow admins to restrict scheduling properties of Pods. Pod scheduling policies help prevent abuse in clusters where not all users are trusted.
To learn more: https://sched.co/GrdO
A
Welcome
to
the
closing
keynote
of
the
company
scheduling
deep
dive.
Actually,
so
my
name
is
Bobby
Solomon
I
work
at
Google,
I
am
the
lead
or
one
of
the
two
leads
of
scheduling
before
kubernetes
I
used
to
work
on
Borg
and
about
two
years
ago,
I
started
working
on
kubernetes
and
I've
been
leading
Sookie
scheduling
for
the
past
one
year
or
so
so
most
of
these
deep
dives
are
meant
to
be
like
he
won
a
as
a
result.
A
I
didn't
add
a
lot
of
slides
to
my
deck,
so
I'm
gonna
go
through
them,
hopefully,
in
about
half
of
the
time
slot
that
it'd
be
happenin
will
open
up
to
questions
that
you
may
have
to
get
us
started.
I
would
like
to
very
quickly
tell
you
what
kubernetes
scheduler
is
so
basically
kubernetes.
The
scheduler
is
a
component
and
kubernetes
ecosystem
that
is
responsible
for
finding
nodes
for
running
the
for
running
pods
in
the
cluster.
A
Kubernetes
is
scheduler,
unlike
some
other
cluster
management
networks
or
crisis
management
systems.
It
is
not
responsible
for
life
cycle
management
of
pods;
it
never
creates
any
pods
and
it
kills
pod
in
very
few
circumstances.
Actually,
this
is
kind
of
a
new
feature
in
kubernetes,
which
is
essentially
preemption.
I'm
gonna
talk
about
that
a
little
bit
later.
Kubernetes
the
scheduler
is
a
feature-rich
scheduler.
It
supports
many
features
among
the
notable
ones.
I
can
point
out,
node
resource
checks
or
when
it,
when
it
schedules
parts
it
is.
A
A
It
supports
trains
and
Toleration
if
you're
not
familiar
with
these
things
on
toleration
as
a
feature
in
kubernetes,
where
you
can
label
some
of
the
nodes
with
a
special
label,
let's
say
which
is
called
taint
and
those
stains
cause
pods
to
not
get
scheduled
on
those
notes
unless
they
have
a
corresponding
toleration
for
those
things,
it
supports
no
affinity.
Basically,
you
can
in
your
pod
spec
you
can
specify
what
kind
of
nodes
or
what
specific
nodes
in
the
cluster
you
want.
This
part
to
be
scheduled
on.
A
It
supports
inter
pod
affinity,
which
essentially
means
where
you
want.
Your
pods
to
be
scheduled
close
to
some
other
pods
in
the
cluster.
This
is
useful,
for
example.
In
case
you
want
to
run,
let's
say,
a
cache
server
next
to
a
web
server
and
stuff,
like
that
also
similar
to
inter
pod
affinity.
It
supports,
inter
pod,
anti
affinity.
Entire
vanity
is
useful
for
a
case
that,
for
example,
you
don't
wanna
on
to
pause
next
to
one
another.
Let's
say
you
don't
wanna
run
multiple
web
servers
on
the
same
node
you
can.
A
You
can
do
that
by
having
anti
affinity.
Kubernetes
scheduler
supports.
Checking
node
conditions
in
kubernetes
nodes
can
have
certain
conditions,
for
example,
if
a
node
is
on
their
memory
pressure,
it
gets
this
memory
pressure
condition
or
if
there
is
a
PID
pressure,
it
gets
that
condition.
So
the
scheduler
tries
to
avoid
those
nodes
which
have
issues
or
under
certain
restrictions
other
than
some
of
these
for
elect
filters.
Kubernetes.
The
scheduler
has
some
preference
as
well
that
sometimes
in
the
code
be
I,
recall
them
priority
functions.
Basically,
these
are.
These
are
preferences
for
scheduling.
A
For
example,
kubernetes
scheduler
can
prefer
choosing
nodes
which
have
lower
resource
utilization
or
higher
resource
utilization
deported
and
depending
on
the
config
that
you
give
to
the
scheduler.
This
is
useful
for
the
case,
for
example,
if
you
want
to
spread
your
pods
among
different
nodes
in
a
cluster
to
hide
them
to
achieve
higher
availability,
you
can
specify
like
choose
nodes
with
lower
resource
utilization
or,
if
you
want
to
reduce
the
cost,
you
don't
care
as
much
about
availability.
You
can
ask
the
scheduler
to
put
as
many
parts
as
pass
but
viewers
on
fewer
notes.
A
So
in
that
case
you
want
to
use
like
choose
the
highest
choose
notes
with
highest
resource
utilization.
You
also
have
a
number
of
other
preference.
For
example,
the
scheduler
can
put
your
pods
on
notes
that
already
have
images
that
those
parts
require
to
reduce
the
amount
of
network
usage
in
your
cluster.
A
So
with
that
I'm
gonna
talk
about
what
are
the
recent
developments
in
the
world
of
scheduling
in
kubernetes?
One
of
the
areas
of
our
focus
have
been
performance,
so
we
wanted
to
improve
performance
of
the
scheduler
as
a
part
of
that
effort
and
other
than
some
of
these
algorithm
algorithmic
improvements.
We
have
done
we've
added
a
new
feature
to
the
kubernetes
schedule
that
they
never
had
before.
A
Borg
and
many
other
similar
schedulers
usually
had
this
feature
where,
once
the
scheduler
finds
a
certain
number
of
nodes
feasible
in
the
cluster,
it
picks
one
of
those
nodes
and
doesn't
continue
scanning
all
the
nodes
in
the
cluster
but
kubernetes
the
scheduler
didn't
have
this
feature.
Basically
in
a
let's
say
in
a
5,000
cluster,
it
would
scan
all
the
nodes
for
every
single
pod
all
the
time
and
as
a
result,
of
course
performance
would
drop.
A
But
in
large
clusters,
like
a
cluster
of
five
thousand
nodes,
we
don't
really
need
to
scan
all
the
nodes
every
time.
For
example,
the
cost
the
scheduler
can
say.
Ok
I
have
already
found
500
feasible
nodes.
Let's
pick
one
of
them
and
move
on.
So
this
feature
basically
helps
you
configure
that
so
there
is
a
flag
or
a
configuration
parameter
that
you
can
specify
what
percentage
of
the
node
the
scheduler
should
find
before
it
stops.
A
In
a
five
thousand
note,
cluster
is
still
significant,
so
basically
it
stops
scanning
for
more
nodes
once
it
finds
250
parts
in
that
cluster
by
default.
This
flag
is
set
to
fifty
fifty
percent,
basically
which
is
still
high
in
large
clusters.
It's
still
very
high,
but
you
can
modify
that
if
you
run
large
development
deployments,
we
also
have
a
plan
to
make
this
semi-automatic
basic
you
can
you
can
still
specify
it
if
you
want,
but
based
on
a
cluster
size?
It's
automatically
adjusted
in
the
next
version,
or
maybe
do
the
version
after
that.
A
Another
area
that
really
required
performance
improvement
is
inter
pod
affinity
and
anti
affinity.
All
of
those
folks
who
have
tried
to
schedule
the
kubernetes
in
the
past
and
have
used
inter
pod
affinity
know
that
it
was
a
very,
very
slow
feature.
Actually,
it
was
three
orders
of
magnitude
slower
than
other
features.
A
So
it's
usable
in
clusters
of
multiple
thousands
of
notes
and
one
other
feature-
is
priority
and
preemption
kubernetes
didn't
have
this
feature
at
all
in
the
past.
Basically,
you
couldn't
specify
importance
of
your
parts
about
with
priority
and
preemption.
You
can
do
so
in
the
past.
If
your
cluster
was
full
and
let's
say,
autoscaler
was
not
allowed
to
add
more
nodes
to
the
cluster
or
you
had
a
cluster
which
was
on-premise,
so
you
couldn't
add
more
nodes
to
the
cluster
and
your
cluster
was
full
a
new
pod.
Probably
the
important
part
was
created.
A
The
pod
couldn't
be
scheduled
because
kubernetes
didn't
have
any
notion
of
importance,
but
with
priority
you
can
provide
that
basically,
the
higher
the
priority,
the
more
important
your
pod
will
be.
Now,
if,
let's
say
the
cluster
is
full
and
a
new
pod
or
high-priority
pod
is
created,
the
scheduler
looks
for
pods
with
lower
priority
and
if
it
finds
them
somewhere
in
your
cluster,
it
removes
those
if
it
can
schedule
this
high-priority
pod.
So
it
deletes
these
like
low
priority
pods
and
schedules
the
high
priority
pod.
A
That's
pretty
much
all
about
the
recent
developments.
I
would
like
to
point
out
the
plant
features
that
we
are
targeting
for,
for
the
scheduler
one
is
gang
scheduling.
This
is
essentially
a
feature
that
tells
the
scheduler
either
schedule
all
the
pods
or
none
of
them.
Well,
a
kind
of
like
an
expansion
of
this
is
to
schedule
at
least
the
number
of
parts
in
a
group
or
none
of
them.
Why
is
this
feature
important,
so
some
workloads
are
designed
to
provide
like
sort
of
interdependency
among
pods,
so
basically
pods
create
input
for
one
another.
A
Some
ml
workloads
are
examples
of
that
or
some
MapReduce
workloads
are
examples
of
that.
What
happens
is
that
if
you
schedule,
instead
of
all
of
them
like
half
of
them
or
a
portion
of
them,
they
will
not
progress,
so
they
basically
get
to
stock
and
whatever
stay
they
are,
and
if
there
is
schedule
they
are
just
using
your
cluster
resources
without
making
any
progress,
so
in
those
cases,
is
better
not
to
schedule
any
of
them.
So
gang
scheduling
helps
with
that.
A
A
Let's
say
that
their
puzzle
is
created
first
in
that
zone
and
is
scheduled
in
that
zone.
That
part
will
prevent
any
other
part
in
a
cluster
from
getting
scheduled
there.
This
is
this
is
an
example
of
a
problem
in
patanti
affinity.
There
are
problems
with
toleration,
for
example,
we
use
taints
and
toleration
for
different
applications.
One
of
them
is
that
you,
for
example,
have
expensive
resources
in
your
clusters.
A
Let's
say
nodes
with
GPU
or
TPU:
you
go
and
take
those
nodes
with,
let's
say
GPU
and
keep
you
taint
to
prevent
pods
that
don't
need
those
resources
from
getting
rescheduled
there.
But
if
anybody
is
allowed
to
put
tolerate
any
kind
of
toleration
on
their
part,
then
a
malicious
user
or
now
or
an
uninformed
user
could
put
those
toleration
zhan
their
parts
causing
parts
that
don't
require
those
resources
to
get
scheduled
on
those
nodes.
A
So
we
think
we
need
to
have
a
way
to
to
give
admin
a
mechanism
to
prevent
these
scenarios.
As
a
result,
we
are
designing
party
scheduling
policies,
these
policies.
There
are
a
few
items
here.
There
are
examples
of
such
policies.
For
example,
you
can
specify
what
is
the
maximum
priority
in
a
namespace
that
users
can
use
or
what
kind
of
toleration
z'
are
allowed,
or
what
kind
of
pot
enter
infinity
is
allowed.
You
can
also
enforce
some
sort
of
node
affinity
and
anti
affinity.
A
Some
users
who
run
security,
sensitive
applications
know
that
they
want
to
put
some
of
the
nodes
of
the
cluster
for
certain
application
in
the
cluster
and
not
share
those
nodes
with
any
other
work
loading
in
the
cluster.
So
with
party
scheduling
policies,
you
can
enforce
that
all
the
parts
that
are
created,
for
example,
from
such-and-such
namespace,
get
this
particular
node
affinity
and
all
the
pods.
All
other
parts
cannot
be
scheduled
on
these
certain
a
number
of
nodes
or
group
of
nodes
that
are
designed
or
that
are
dedicated
to
running
security,
sensitive
applications.
A
B
A
The
best
framework
ever
the
scheduling,
front
work,
so
so,
on
top
of
being
the
best
framework
ever
in
the
history
of
mankind.
The
scheduling
framework
provides
a
certain
number
of
features,
so
it's
essentially
a
it
provides
a
pair
born
of
scheduling,
which
includes
scheduling,
queue
and
a
cache
and
some
event
handlers
for
updating
that
cache
and
then
the
rest
of
the
features
of
the
scheduler
become
plug-ins
for
the
scheduling
framework.
A
If
you
need
we,
we
got
a
lot
of
feedback
from
our
users
that
many
of
them
require
make
sometimes
small
changes
to
this
scheduler
to
do
things
slightly
differently,
but
there
was
no
way
of
doing
it
in
the
past,
so
the
scheduling
framework
is
an
effort
towards
a
direction
of
making
kubernetes
customizing
the
scheduler
easier
and
maintaining
it
also
easier.
So
hopefully,
with
this
way,
this
framework
merging
new
changes
from
upstream
to
your
customized,
scheduled
scheduler,
will
be
very
easy.
It
also
provides
modularity,
so
we
can
wholly
debug.
Our
schedule
is
more
easily.
A
Another
effort
which
doesn't
have
a
timeline
at
the
moment
at
least,
is
the
scheduler.
This
is
another
incubator
and
siggy
scheduling
these
scheduler
responsibilities,
essentially
undoing
what
the
scheduler
does
so
as
clusters
run
certain
things
in
a
cluster
change.
For
example,
we
know
we
talked
about
balancing
resources
in
a
cluster,
so
the
scheduler
can
try
to
balance
amount
of
resource
usage
in
different
nodes,
but
after
pods
are
terminated,
those
balance
will
change
right.
A
The
balance
will
change
or
I,
don't
know,
maybe
at
the
time
that
the
scheduler
was
the
scheduling
replicas
of
a
replica
said
it
couldn't
put
each
replicas
on
a
different
node,
because
some
of
the
notes
didn't
have
enough
resources.
So
it
had
to
put
multiple
of
those
replicas
on
a
single
node,
but,
as
time
has
passed,
some
other
nodes
are
available.
A
So
some
of
these
replicas
can
be
moved
to
other
nodes,
so
the
scheduler
essentially
tries
to
remove
some
of
the
parts
from
the
cluster
that
violate
some
of
the
scheduling
or
they
are
not
satisfying
all
the
preferences
for
scheduling,
with
the
hope
that
they
can
be
moved
to
better
nodes.
After
getting
de
scheduled,
as
I
said,
it's
already
available
as
an
incubator
and
it
actually
kind
of
works.
It's
not
feature-rich
at
the
moment,
but
we're
hoping
that
we
can
add
more
features
and
a
more
simulation
to
it
in
the
near
future.
A
A
A
Electorally
in
all
scenarios,
but
the
the
algorithmic
latency
of
firmament
is
lower
than
our
standard
scheduler
and
how
much
performance
improvement
we
can
get
end-to-end
and
in
real
class
resistant,
not
very
well
known,
but
we're
hoping
that
we
can.
We
can
achieve
better
performance
at
the
end,
with
a
scheduler
for
batch
and
gang
scheduling,
it's
also
available
as
an
incubator.
If
you
want
to
give
it
a
spin
feel
free
to
try
it.
You
know
with
that.
C
D
A
Question
yeah,
so
the
scheduler,
you
know
if
you
need
higher
throughputs,
you
know.
I
saw
you
this
slides
that,
for
example,
right
now
with
default
configuration
of
the
scheduler
in
a
five
thousand
dollar
cluster.
We
can
achieve
about
like
fifty
to
fifty-five
almost
fifty
five
pots
per
second
right.
So
if
you
want
to
have
a
two
hundred
parts
per
second
throughput
of
the
scheduling
and
you
push
your
cluster
to
five
thousand
nodes,
you
will
not
get
it.
E
D
A
F
G
Had
a
question
about
pod
scheduling,
policies,
I
think
they're,
a
really
great
idea,
I
think
there's
been
something
that's
been
lacking
for
a
while,
but
particularly
around
preventing
people
that
can
create
pods
from
creating
pods
on
master
nodes
masters
registers
nodes.
You
can
just
kind
of
tolerate
the
taint
and
get
out
there.
G
We
actually
wrote
something
to
do
something
like
this
to
apply
scheduling
data
to
pods
as
they're
created,
so
we
target
labels,
and
then
we
say:
if
you've
got
these
labels,
we
actually
put
affinity
or
toleration
x'
or
anything
on
them
right
so
I
think
that's
that
was
really
useful.
My
question
is:
where
are
you
doing
this
and
can
I
I
mean
our
projects?
Great,
it's
open
source,
I
love
it
you,
but
it
uses
initializers
and
it's
another
thing
you
have
to
run.
So
where
do
you
plan
on
applying
pod
security
policy?
A
A
good
question,
so
we
are
still
designing
this
feature,
but
the
latest
development-
and
this
is
that
we
are
gonna-
have
to
places
to
enforce
these
policies.
One
is
at
the
admission
time
and
the
other
one
is
at
the
runtime,
basically
dynamically.
The
reason
that
the
scheduler
needs
to
be
involved
is
that
sometimes
they
note
that
is
eventually
chosen
for
running
the
part
should
be
should
be
evaluated
against
those
scheduling
policies.
A
At
the
admission
time
when
a
pod
is
created,
it
doesn't
have
a
note
associated
with
it,
but
the
scheduler
is
the
only
component
that
eventually
picks
a
note
and
after
it
picks
a
note
for
the
part
it
can.
It
can
evaluate
that
note
against
the
policies
and
if
the
note
violates
those
policies
it
can
go
and
ask
for
and
we
can
go
and
scan
for
another
node
in
the
cluster.
H
A
A
E
A
I
J
Hey
I'm
Tomas
I
maintain
one
of
the
maintenance
for
the
diamond
set.
So
the
thing
is
when
to
switch
to
the
new
scheduler.
Oh,
maybe
back
to
the
problem.
The
problem,
usually,
is
you
have
some
admission
that
modifies
the
pots
and
the
demon
said
controller
schedules
all
it
now
doesn't
schedule
them
entirely,
but
it
chooses
the
note
either
way
and
it
does
that
before
the
your
admission
modifies
the
pot.
So
if
you
like
change
the
note
selector,
it
doesn't
see
that
so
the.
G
J
J
Restrictive
node
selected
that
right,
it's
a
lit
after
admission
or
like
during
the
admission
you
can
have
like
thousands
put
in
pending,
but
at
least
now
it
doesn't
go
into
hot
loop
and
doesn't
just
delete
the
port
and
create
it
again
queue
but
failed
it
and
it
was
really
bad
now.
It
basically
just
have
a
big
queue
and
the
purple
solution
is
not
so
simple
and
we.
A
K
A
K
A
Now,
actually,
the
scheduler
never
violates
those
like,
and
if
you
have
and
required
anti
affinity,
the
scheduler
will
not
ever
put
you
only
two
parts
that
violate
that
anti
affinity
on
the
same
node,
for
example.
But
there
are
certain
things:
can
change
in
the
cluster.
For
example,
let's
say
that
you
add
new
labels
to
your
parts
after
they
are
scheduled,
or
you
add,
new
labels
to
some
of
the
nodes
in
the
cluster
that
are
already
running
some
parts
and
those
labels
cause
after
the.
A
A
The
schedule
does
the
run
time
check
if,
after
addition
of
those
labels,
some
of
these
rules
are
violated
in
the
schedule
removes
those
those
parts
but
or
sometimes
you
may
have
preferred
anti
affinity,
so
preferred
anti
affinity
may
not
be
satisfied
at
the
time
that
you
is
scheduling
the
parts
right
so
and
then
later
on,
the
scheduler
finds
out
that
there
are
better
opportunities
for
moving
some
of
these
parts
around
so
that
those
preferences
satisfied.
Then
it
removes
the
parts.
L
Decide
a
question
about
the
scoring
changes.
So
first,
did
you
look
at
efficiency
like
sorry,
efficiency
of
packing,
like
when
you
toggled
how
many
you
were
actually
scoring
like?
Was
there
actually
a
you
know,
just
scoring
fewer
really
meaningfully
actually
impacts?
How
well
you
end
up
packing
the
cluster
so.
A
I
believe
you
are
talking
about
this
number
of
number
of
nodes
that
you
are
considering.
We
don't,
but
the
scheduler
now
has
a
more
advanced
iterator
for
nodes,
so
it
tries
to
basically,
instead
of
just
looking
in
an
array
of
nodes
and
going
from
beginning
to
the
end.
It
goes
through
like
noticing
different
figures,
failure,
domains
and
basically
different
zones,
and
it
also
continues
from
the
point
that
it
like
basically
left
off
when
app
uses
scheduling
cycle.
So
it
tries
to
tries
to
satisfy
some
of
these
preferences.
L
A
A
M
This
mm
probably
isn't
necessarily
your
problem,
but
I
guess
something
that
we
frequently
run
into
is
when
we
have.
You
know
a
massive
failure
notes
of
our
massive
amount
of
rescheduling
in
a
cluster.
It
can
kind
of
overwhelm
the
Kubla
and
container
engine
like
trying
to
put
you
know:
50
60
pods
on
one
node
at
a
time.
I
guess
maybe
like
again
it
when
I
say
it.
A
We
have,
we
have
an
actually
a
feature
to
limit
the
number
of
pots
on
a
on
a
particular
node,
so
there
is
a
max
pot
per
node
which
can
be
changed
if
you,
if
you
want
fewer
pots
on
a
particular
node,
but
not
the
problem,
you're
talking
about
is
a
little
bit
more
than
this.
Probably
if
you
I
guess
this
is
actually.
This
should
be
addressed
at
different
levels.
I
guess,
then
the
scheduler
mm-hmm,
but
I
would
like
to
understand
the
situation
a
little
bit
better.
That
causes
this
to
start.
M
Like,
let's
say
you
know,
we're
have
a
multi
zone
cluster
and
we
have
like
a
stateful
application
and
we
use
preferred
zone,
anti
affinity
right
and
so
just
so
that
we're
not
stuck
in
a
state
where
we
don't
have
at
least
a
little
bit
of
resiliency.
For
that
data
will
often
reschedule
into
another
zone
like
will
autoscaler
zone,
to
provide
the
additional
capacity
and
reschedule
those
pods
from
like
the
third
zone
unto
those
and
it
could
be
thousands
and
thousands
of
pods
right
and
like
the
problem.
Is
that
it's?
M
It's
not
that
so,
like,
let's
say
I
add
you
know,
50
new
nodes
to
his
own
and
I
have
to
schedule
a
thousand
pods
like
it's,
not
that
there's
not
enough
capacity
at
a
steady
state.
It's
that
the
container
engines
and
the
couplet
like
really
are
not
very
good
at
starting
up
a
huge
number
of
processes
in
parallel,
and
so
this
is.
A
Basically,
a
burst
of
pods
all
getting
you
scheduled
on
a
on
a
single.
Now
again,
there's
a
kid.
Well,
you
know
this
is
kind
of
like
an
oxymoron
to
scheduling
throughput
right.
So
if
you
want
to
improve
scheduling,
throughput
scenarios
like
this
could
become
more
common
I
guess
this
should
be
addressed
at
like
cubelets
I'd,
probably
or
maybe
we
can
somehow
rate
limit
the
number
of
path
creation,
yeah.
M
A
N
I
have
a
question
about
the
sort
of
future
landscape
of
schedulers.
So
there's
a
couple
in
Aikido
right
now:
do
you
see
a
world
where
there's
a
year
or
two
down
the
line
where
there's
one
scheduler
to
rule
a
model,
or
do
you
see
that
there's
gonna
be
a
multiple
schedulers
in
the
long
term
and
in
that
world?
What's
the
compatibility
story,
gonna
be
like
yeah.
A
A
If
you
follow
those
practices
and
the
possibility
of
Contin
to
scheduler
competing
with
one
other
and
one
another
well
reduce,
but
otherwise
there
will
be
a
lot
of
competition
between
two
schedulers,
and
sometimes
you
may
see
your
pods
getting
rejected
by
cubelet,
because
two
schedules
will
leave
that
there
are
available
resources
on
that
note,
both
of
them
schedule
their
own
pods.
On
that
now,
then
what
one
of
them
gets
rejected,
so
we
are
going
to
keep
having
kubernetes
multiple
scheduler
friendly,
but
for
standard
components.
A
A
J
A
P
So
some
sometimes
you
are
trying
to
optimize
for
opt
throughput
of
the
scheduler.
In
some
scenario,
maybe
we
want
to
be
it
more
one
way
schedule
apart
will
need
to
be
more
cost-effective
one
efficient.
We
want
to
do
a
better
job
on
bin
packing
for
all
the
parts
and
on
the
other
side,
maybe
we
announce
on
some
other
case.
Maybe
you
will
want
to
emphasize
small
on
part
of
phonetic.
So
how
do
you
balance
between
the
different
goals?
Because
they
can?
They
can
conflict
with
each
other?
They.
A
Don't
necessarily
conflict
all
the
time,
so
if
we,
for
example,
achieve
better
algorithmic
improvements
without
really
compromising
validity
of
the
result
we
can,
we
can
have
higher
performance
without
any
compromise
on
the
end
result.
Basically,
but
in
some
cases
like
what
you
saw
like
scoring
fewer
fewer
nodes
in
the
cluster,
that
can
be
the
case.
So
when
you
score
fewer
nodes
in
the
cluster,
the
quality
of
the
scheduling
could
could
be
reduced,
and
some
of
your
preferences
essentially
may
not
be
satisfied.
This
is
most.
This
is
only
about
preferences.
A
P
P
A
C
C
A
A
good
question
well,
first
of
all,
it's
hard
to
predict,
but
we
have
certain
goals
that
you
want
to
achieve.
So
our
goal
is
to
reach
100
parts
per
second
in
a
5,000
node
cluster,
and
our
hope
is
that
in
114,
which
is
gonna
be
released
in
about
three
months,
we
are
gonna
achieve
that
so
100
parts
per
second
is
the
is
our
goal.
But
one
thing
that
to
note
here
is
that
the
scheduler
has
the
rate
limitation
for
sending
bound
requests
to
the
API
server.
All
right.
A
A
In
fact,
our
scalability
team
looked
at
this
rate
limitation
and
noticed
that
raising
the
rate
limitation
from
like
20
parts
per
second
to
50
parts
per
second
causes
one
whole
CPU,
core,
more
user,
basically
more
usage
on
the
master
node,
so
that's
quite
sensitive,
and
we,
while
we're
trying
to
improve
the
scheduler
throughput
going
further
than
a
certain
number,
is
not
going
to
help
much
because
we
still
need
to
rate
limit
the
number
of
points
that
we
send
to
the
API
server.
Okay,.
H
Q
Component
is
pushing
stuff
to
the
API
server
and
ethnicity,
and
if
at
is
it
is
name,
it
is
there.
What
Hank,
if
it
is
city,
is
not
improved.
What
we
can
improve
is
on
the
other
aspect,
so
as
far
as
I
know
so
right
now
in
Cabin
NES,
there
is
already
someone,
for
example.
If
if
you
know
a
no
hot
bit
so
normally
in
I,
think
in
those
in
current
versions,
every
ten
seconds
you
were
to
try
to
send
a
habit
to
it
has
server
IP,
f,
so
HDD,
so
you
have
four
five
thousand.
Q
That
is
so
right.
Every
10
minutes,
I'm,
sorry,
every
10
seconds
is
just
give
a
happy
to
be
a
server
so
and
right
now,
I
think
in
113
or
114.
They
are
improving.
This
signal
is
kind
of
trying
to
do
a
delta
or
something
habits
kind
of
regularly
they're,
worse
than
one
habit
every
more
minute
and
depends
on
whether
they
have
real
information
update.
Then
there
was
an
update
and
also
I.
Think
there
is
socio.
Improvement
are
other
views,
for
example,
if
I
check
out
the
life
cycle
apart.
Q
There
are
several
phases
like
create
a
pod,
and
the
scheduler
will
try
to
do
the
no
name
binding
right.
That
also
has
the
API
server
history
and
also
in
the
queue
later
side.
There
are
also
some
intermediate
status
updated
to
hit
the
industry
API
server,
so
those
parts
I
think
can
also
be
improved.
So
it's
it's
kind
of
yeah.
Every
part
can
be
improved,
so
right
now,
I
think
for
schedule.
We
are,
we
will
try
to
achieve
100
pass
for
5000
and
for
other
is
improved.
Yeah.
A
Basically,
the
select
short
answer
to
your
question
is
that
scalability
is
a
multi-dimensional
problem
and
it
should
be
addressed
in
many
different
demand
in
order
to
have
a
like
a
global,
higher
scalability.
As
as
use
you
mentioned,
and
we
also
mentioned,
we
will
face
some
issues
on
the
API
server.
If
we
push
the
scheduler
throughput
beyond
a
certain
point
we've.
Similarly,
we
may
face
other
problems
with
respect
to
IP
routing
and
in
our
clusters
there
were
some
issues
with
domain
name
resolution.
A
If
you
ran
many
services
in
your
cluster,
there
were
some
issues
with
a
very
large
number
of
volumes,
dynamic
volumes
in
the
cluster.
So
it's
not
just
something
we
can.
We
can
address
in
one
place
and
say
we
should
resolve
the
problem.
It's
a
multi
team
effort
that
we
are
trying
to
actually
address
yeah
sure.
C
H
G
A
So
I
guess
they
could
answer
that
question
better,
but
anyways
we're
trying
to
to
solve
that
problem.
We
were
thinking
that
maybe
we
can.
We
can
add
that
feature
in
the
next
version,
depending
on
race
availability.
We
may
or
may
not
be
able
to
achieve
that,
but
anyways
it's
in
our
radar,
and
we
are
aware
of
that
problem.
A
We
actually
worked
on
this
problem
in
the
past,
but
since
anti-authority
was
and
generally
inter-party
affinity
was
a
thousand
times
slower,
we
decided
at
that
point
we
shouldn't
add
any
more
features
to
it
until
we
address
the
scalability
and
performance
issues
there.
So
now
that
those
are
kind
of
gone,
we
can
think
about
adding
more
features,
and
this
is
certainly
one
of
those
most
requested
features
for
inter
pod
affinity
and
we're
working
on
it
sure.
G
Q
This
is,
this
is
a
real.
This
is
a
real
problem,
because
I
saw
some
some
users-
yes,
especially
if
you're,
especially
for
anti
affinity,
because
right
now
the
schedule
Adam,
the
minimum
mean
Union
is
apart,
but
I
think
about
the
rolling
update
case
right.
If
you're
rolling
update
existing
one
is
still
there
and
you
feel
you
won't
want
to
replace
that-
not
just
replace
no
real
places
just
hold
one
and
try
to
spin.
Why
and
to
replace
that
well,
but
the
the
new
one
has
no
chance
to
be
to
be
scheduled
at
that
one.
C
A
A
I
misheard,
you
well,
several
lists
is
kind
of
I
mean
there
is
nothing
particularly
different
in
serverless
workloads.
Really.
That
requires
something
new
in
the
scheduler,
but
definitely
we
that's
the
direction
that
you
are
going
to
support,
but
we
don't
know
what
kind
of
particular
scheduling
feature
in
requires.
I.
A
What
supports
the
throughput
is
actually
you're
right.
Throughput
for
service
for
service,
several,
its
application
is
something
that
is
needed
and
actually
the
reason
that
you're
trying
to
get
to
like
a
hundred
policy
and
$5,000
is
exactly
for
to
target
service
applications
better,
but
other
than
throughput.
We
don't
need
to
add
features
or
language
to
do.