►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
SIG Scheduling Deep Dive – Bobby Salamat & Jonathan Basseri, Google (Intermediate Skill Level)
Please join us for in-depth understanding of Kubernetes Scheduler and its advanced features.In this presentation we talk about the internals of Kubernetes Scheduler and how it keeps track of the cluster resources in order to make scheduling decisions. We will cover the scheduling workflow and the life of a Pod during a scheduling cycle. Then we will talk about some of the advanced scheduling features that enable users to customize placement of their workloads in Kubernetes clusters to achieve higher reliability and/or better resource utilization.This session is most useful for cluster admins or those who want to start contributing to Kubernetes scheduler.
About Bobby
Bobby Salamat is co-lead of Kubernetes SIG Scheduling and co-maintainer of Kubernetes default scheduler. He is interested in advanced scheduling techniques to improve resource efficiency of clusters. He is also interested in improving user experience and making default behavior of clusters more intuitive. Before joining Kubernetes, he developed Borg master core at Google.
About Jonathan
He is a software engineer at Google.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
everyone
and
welcome
to
siggy
scheduling,
deep
dive,
I'm
Bobbi
salamat
I'm
here
with
Jonathan
bass.
Sorry,
he
we
are
both
software
engineers
at
Google,
I'm,
also
a
co-lead
of
cig
scheduling.
Today
we
are
going
to
talk
about
some
of
the
features
of
scheduler
and
also
give
you
a
general
idea
of
what
the
scheduler
is.
Hopefully,
these
features
are
gonna
be
useful
for
you,
if
you
plan
to
run
medium
to
large
scale,
workloads
on
kubernetes.
A
So
with
that,
let's
take
a
look
at
what
kubernetes
the
scheduler
is
actually
worth
doing
and
what
it
is
so
kubernetes
is
scheduler
like
many
schedulers
in
many
other
cluster
management
service
service
or
software
is
responsible
for
placing
pods
on
nodes
during
that
process,
it's
actually
uses
various
information,
and
it's
you.
It
uses
a
centralized
architecture
so
in
a
centralized
architecture,
actually
kubernetes.
The
scheduler
must
know
about
everything
in
the
cluster
in
order
to
make
scheduling
decisions.
A
So
one
option
to
know
about
everything
in
a
cluster
is
to
go
and
ask
from
a
like
API
server
to
to
give
it
all
the
information,
but
it
cannot
be
done
for
every
single
part.
Otherwise
performance
would
be
very,
very
low,
so
what
it
does
it,
in
fact
is
is
that
it
caches
a
lot
of
information
about
the
state
of
the
cluster
and
in
order
to
keep
this
cache
up-to-date,
it
relies
heavily
on
the
events
that
it
receives
from
the
API
server.
A
So
it
receives
a
variance
for
new
pods
new
nodes
changes
in
the
pods
or
changes
in
the
nodes
or
resources
and
many
other
things
in
the
cluster,
so
that
those
events
are
a
key
part
of
the
whole
architecture
of
schedule
and
it's
one
of
the
few
components
actually
in
kubernetes
that
relies
heavily
on
on
events
and
never
resyncs.
It's
information
with
the
API
server.
A
Now,
with
this
cache,
when
a
new
pod
arrives,
caterer
has
to
make
decisions,
so
all
these
parts
usually
go
to
the
scheduler
queue
and
schedule
picks
one
part
at
a
time,
at
least
that's
what
it
does
today.
It
might
change
in
the
future,
but
that's
what
it
does
today.
It
only
picks
one
part
at
a
time
and
tries
to
find
a
proper
note
for
that
part
when
it
tries
to
find
out
now
it
goes
through
several
stages
or
similar
steps.
The
first
one
is
to
do
a
feasibility
check
of
those
nodes
in
a
cluster.
A
In
the
feasibility
check,
it
actually
tries
to
filter
out
the
nodes
that
are
not
possible
candidates
for
running
the
pot.
Alright.
So,
for
example,
in
this
slide,
node
1
and
node
n
are
not
possible
candidates.
Node
2,
3,
&
4
are
possible
candidates,
so
it
filters
out
the
rest
of
the
nodes
and
focuses
on
the
nodes
that
are
feasible
for
this
part,
and
then
in
the
next
step
it
tries
to
find
the
best
node
among
these
possible
candidates.
So
this
is
step
in
kubernetes,
the
scheduler
jargon.
We
call
it
like
priority
functions.
A
It
runs
a
bunch
of
priority
functions.
This
is
a
little
bit
of
unfortunate
naming
because
priority
can
be
confused
with
many
other
priority
in
a
system
like
pot
fire.
It
is
another
concept
which
is
completely
different
thing.
It
has
nothing
to
do
with
these
priority
functions.
It's
essentially
the
priority
functions
for
us:
do
ranking
of
these
nodes
and
in
the
process
of
ranking
scheduler
tries
to
find
the
best
node
to
run
the
part.
A
An
example
of
these
ranking
is
that,
for
example,
which
one
of
these
nodes
has
the
largest
number
of
images
that
their
pod
requires
or
depending
on
the
scheduling
policy
it
may
use
like
best
fit
or
worst
fit
to
find
the
node.
That
is
better
for
running
the
pod.
So
among
these
3
nodes
that
we
have
in
in
this
slide
note
3
happens
to
have
the
hi-c
score,
so
scheduler
will
choose
node
3
for
running
the
pot
right
and
then
it
goes
to
the
last
step
of
the
scheduling
which
is
actually
binding
of
a
pod.
A
To
that
note,
in
that
process,
scheduler
doesn't
do
a
whole
lot
other
than
updating
pod
spec.
With
this
note
name,
and
what
happens
at
that
stage,
is
that
API
server
sends
an
event
to
the
corresponding
cubelet
on
that
node
and
tell
that
cubelet
that
you
have
a
new
pod.
Now
cubelet
goes
and
does
its
own
checking,
and
if
it
finds
the
pod
as
a
possible
possible
candidate
for
for
itself,
you
know
it
there
or
there
or
change.
There
are
checks
that
could
fail
and
cubelet
could
reject
this
part.
A
But
if
the
checks
pass,
then
the
cube
later
starts
running
that
part
by
preparing
its
containers
and
stuff
like
that.
This
is
a
pretty
high-level
overview
of
what
the
scheduler
does
when
it
tries
to
schedule
a
part
with
that
I'm
gonna
hand
it
over
to
Jonathan
to
talk
about.
Some
of
the
features
of
kubernetes
is
scheduler.
That
may
be
useful
for
you.
If
you
run
any
production
workload
on
kubernetes
and
then
later
on,
I'm
gonna
talk
about
some
of
the
more
advanced
features
and
give
you
a
an
overview
of
the
roadmap.
B
B
We've
left
links
and
the
slides
or
on
the
schedule
to
all
of
the
documentation
for
these
features
and
we'll
try
to
leave
time
for
questions
at
the
end
or
find
us
and
talk
to
us
in
person
or
on
slack,
so
several
different
scenarios
that
you
can
use
the
scheduler
to
solve
one
of
the
simple
ones
Oh
before
that.
We
need
to
talk.
C
B
How
most
of
these
features
work
so
you're
probably
familiar
with
labels?
These
are
key
value.
Pairs,
arbitrary
key
value,
pairs
that
are
on
any
API
object.
You
can
use
them
to
identify
these
objects
and
then
you
can
query
the
API
server
to
get
a
specific
set
of
those
objects.
So
in
all
of
these
diagrams
we
use
these
circles
with
gears
to
represent
pods.
So
in
this
example,
we
have
four
pods.
All
of
them
have
the
label
app
equals
my
app,
but
they
have
these
phase
and
roll
labels
that
are
different.
B
So
if
you
were
to
query
pods
with
app
equals
my
app,
you
would
get
all
four,
but
if
you
say
app
equals
my
app
and
role
equals
Fe.
You
would
only
get
these
two
so
given
that
how
could
we
target
a
specific
set
of
nodes
for
your
pod?
So
we
have
something
called
node
affinity
which
uses
one
of
these
selectors
to
get
a
specific
set
of
notes.
So
in
this
case
we
require
that
our
pod
will
run
on
a
node
that
matches
this
label
selector.
B
Sometimes
you
have
related
services
that
you
want
to
run
with
each
other,
so
for
that
we
have
pod
affinity.
Pod
affinity
is
a
way
to
say
in
this
case
our
service
a
wants
to
run
close
to
a
pod
with
the
label
service
B.
And
what
do
we
mean
when
we
say
close
to
is
defined
by
this
topology
key
when
the
topology
key
is
own?
We
say
we
want
our
service
to
be
in
the
same
zone
as
any
pod
with
label
service
equals
B.
B
So
the
question
was:
are
there
standard
topology
keys?
The
answer
is
yes,
and
so
the
full
name
is
actually
like.
Kubernetes
do
slash
zone,
they
are
I'm
pretty
sure
in
the
documentation,
but-
and
one
of
the
other
ones
is
in
this
next
example,
you
could
specify
hostname,
which
means
I
want
to
run
on
the
same
node
as
now.
In
this
case,
there's
only
one
service
B
and
that
node
doesn't
have
any
resources.
So
our
pod
cannot
be
scheduled
here.
B
So
another
type
of
affinity
is
pod,
anti
affinity
and
this
causes
pods
to
be
scheduled
away
from
each
other.
The
topology
key
is
the
same
concept,
so
here
we're
saying:
I
want
to
run
my
service
on
a
node
that
does
not
have
any
pod
with
service
equals.
A
and
you'll
notice
in
this
example
that
service
equals
a
is
also
on
our
own
pods
metadata.
So
this
is
a
self
reference
saying.
I,
don't
want
two
instances
of
my
pod
to
run
on
the
same
node,
and
this
is
you
know
useful
for
something
that
you
know.
D
B
And
the
other
important
thing
to
note
is
that
anti
affinity,
pod
anti
affinity
is
symmetric,
so
a
pod
which
is
already
scheduled
on
a
node,
can
affect
the
scheduling
of
other
pods
on
to
that
node.
So
in
this
example,
we
have
a
pod
running
on
node,
two
with
anti
affinity
against
any
other
pod
that
doesn't
have
user
equals
a
so
this
is
kind
of
double
negatives,
but
so
the
the
pod,
the
is
user.
A
here
is
saying:
I,
don't
want
any
pods
on
this
node
with
me,
unless
they
also
have
the
label
user
equals
a.
A
Some
of
the
features
that
you
heard
about
are
useful
in
many
many
scenarios
and
for
many
production
workloads.
Some
of
the
features
that
I'm
gonna
talk
about
are
probably
more
interesting
for
a
smaller
group
of
people,
those
who
may
probably
run
larger
clusters.
One
of
those
features
is
actually
taints
and
toleration,
things
and
toleration
x'.
A
We
use
them
for
various
things
inside
kubernetes,
but
one
of
the
interesting
examples
when
it
comes
to
scheduling
is
actually
to
save
some
of
the
nodes
that
have
a
special
hardware
for
pods
that
actually
need
those
a
special
hardware
in
a
heterogeneous
cluster.
Let's
say
you
have
nodes,
some
of
them
have,
for
example,
GPU,
and
in
this
example,
and
some
of
them
don't
right,
so
a
new
pod
arrives
and
the
pod
only
requires
four
gigabytes
of
RAM.
Let's
say
all
of
the
nodes
in
our
cluster
have
four
gigabytes
of
RAM.
A
Only
the
first
one
node
one
has
GPU
so
scheduler.
If
there
is
no
particular
preference
may
happen
to
choose
a
node
one
for
running
this
pod
right
and
then
a
second
pod
arrives
a
second
later.
This
pod
requires
four
gigabytes
of
RAM,
but
it
also
requires
four
gig
one
GPU.
So
since
the
node
with
GPU
is
already
used
for
running
a
part
that
didn't
actually
require
G,
you,
we
will
might
be
able
to
schedule
this
part
at
all,
so
this
part
is
gonna
remain
pending.
A
In
order
to
avoid
this
scenario,
we
can
taint
that
note
with
GPU
with
with
a
special
change
or
if
it
actually
could
be
any
time
this.
This
is
like
an
arbitrary
key
value
pair
that
you
can
pay.
You
can
put
on
that
note
and
say,
for
example,
key
is
GPU
and
I
and
I
want
this
to
be
to
be
saved,
preferably
only
for
parts
that
require
basically
GPU.
So
here
we
actually
set
the
effect,
as
prefer
block
scheduling.
We
have
different
effects.
A
For
example,
one
can
be
no
scheduling,
which
means
that
only
the
pods
that
we
actually
have
the
proper
Toleration
can
be
scheduled.
But
in
this
case
we
are
just
saying
that
we
be
prefer
no
pods
to
be
scheduled
on
this
node
so
and
note
that
there
are
a
pod
that
arrives
and
needs.
Four
gigabytes
of
RAM
is
preferred
to
be
scheduled
on
node,
2
or
node
3.
If
they
they
are
full,
we
will
schedule
it
still
on
note
1,
but
since
these
nodes
are
available
now
we
scheduled
a
pattern.
A
Let's
say
no
to
and
now
when
that
new
pod
arrives,
which
requires
actually
GPU.
Since
there
is
no
other
choice
for
this
scheduler
to
pick
it
actually
picks
the
first
node.
Sometimes
you
may
want
to
have
higher
guarantees.
In
that
case,
you
may
go
and
actually
put
the
effect
as
block
is
scheduling
or
no
schedule.
In
this
case,
the
scheduler
picks
that
node
only
if
the
pot
has
the
proper
toleration.
A
So
in
this
case
we
put
the
Toleration-
and
this
part
that
requires
the
GPU
and
scheduler
picks
that
part
for
picks
this
node
with
GPU
for
scheduling.
One
thing
that
I
would
like
to
add
here
is
that
there
are
ways
to
add
these
things
or
Toleration
automatically,
so,
for
example,
and
that
you
can
have
an
admission
controller
in
kubernetes
that
looks
at
the
resource
requirements
of
that
pod
and
adds
the
proper
toleration
the
part
automatically
you
don't
need
to
always
do
it
manually,
but
it
also
can
be
manual.
A
A
One
of
the
main
costs
of
a
cluster
is
actually
the
amount
of
resources
that
are
used
in
the
cluster,
and
those
resources
are
usually
directly
directly
correspond
to
the
number
of
nodes
or
amount
of
resources
available
on
the
nodes.
So
a
way
of
saving
money
in
a
large
cluster
is
to
raise
the
resource
utilization
of
each
individual
node
in
the
cluster.
This
feature
that
I'm
going
to
talk
about
was
one
of
the
most
important
features
that
help
Google
raise
resource
utilization
in
in
borg
in
google
internal
clusters.
A
Basically,
in
a
cluster
that
all
the
parts
have
the
same
priority
when
the
nodes
are
full
and
a
new
pod
arrives,
the
pod
cannot
be
scheduled
anywhere.
But
how
do
you
specify
if
some
of
these
parts
are
more
important
than
the
others?
For
example,
you
may
run
your
CI
CD,
along
with
your
production
workloads
in
the
same
cluster.
Why
would
you
do
that?
Because
you
want
to
raise
resource
utilization
in
your
cluster,
but
how
do
you
specify
which
one
is
more
important?
That's
how
you
spell
you
put
actually
priority
on
your
parts.
A
In
this
case,
let's
say
that
the
pods
which,
like
the
red
pods,
have
the
highest
priority
in
your
cluster.
The
orange
parts
are
medium
priority
and
the
green
pods
are
the
lowest
priority
pods
all
right,
so
you
can
put
all
of
them
in
the
same
cluster.
Now
a
part
with
highest
priority
arrives.
The
cluster
is
full
scheduler
will
never
try
node
3,
because
it's
running
also
a
high
priority.
A
Pod
scheduler
never
preempts
a
part
with
the
same
or
higher
priority,
but
no
2
and
no
or
possible
candidates
for
running
this
high-priority
pod
because
they
are
running
pods
with
lower
priority
than
this.
One
scheduler
looks
at
these
two
and
it
has
various
metrics.
But
since
note
one
has
the
lowest
priority.
Pods
that,
if
removed,
can
give
enough
resources
for
to
the
schedule
to
schedule
this
high
priority.
Pod
scheduler
deletes
those
two
pods
or
in
our
technology
preempts
those
pods,
giving
them
their
graceful
termination
period.
And
after
that
graceful
permeation
period
expires.
A
Those
pods
are
actually
removed,
and
now
this
higher
priority
pod
can
be
scheduled,
and
this
is
node.
So
why
does
this
help
with
raising
utilization
in
your
cluster?
So
a
lot
of
users
today,
in
absence
of
a
priority
and
preemption,
use
a
single
workload
or
equally
important
workloads
in
their
cluster
and
rely
on
cluster
autoscaler
to
scale
up
the
cluster
when
there
is
demand
for
more
resources.
A
But
an
alternative
is
to
actually
combine
various
types
of
resources
so
that
when
there
are
available
resources,
you
run
your
lower
priority
tasks
and
when
there
is
a
need
for
more
resources,
for
let's
say
your
production
workloads,
you
remove
some
of
those
lower
priority
tasks
which
sometimes
we
call
them
like
filler
tasks
from
your
nodes
and
use
the
resources
for
running
your
production
workloads.
This
is
also
helpful
if
there
is
a
sudden
spike
in
the
traffic
of
your
cluster,
for
example.
A
Suddenly
there
is
a
news
about
your
website
or
your
service
and
people
jump
on
their
computers
and
want
to
check
out
your
website
or
your
service,
and
suddenly
there
is
a
huge
spike.
So
if
you
have
a
lot
of
available
resources
used
for,
let's
say,
training
your
models
or
I,
don't
know
your
CI
CD
pipeline.
The
new
scheduler
can
react
very
quickly
and
remove
some
of
these
workloads
from
your
cluster
and
use
it
for
running
your
more
reduction.
A
A
A
Yang
scheduling
is
another
feature
that
which
we're
working
on.
We
are
hoping
to
have
an
alpha
version
of
gangue
scheduling
in
112.
If
you
have
feedback
about
how
we
should
do
gang
scheduling,
police
and
send
them
to
us.
We
are
designing
the
feature
these
days
and
we
would
like
to
hear
as
many
feedback
as
possible.
A
There
are
other
performance
improvements
like
equivalence,
cache
that
has
been
in
alpha
for
a
while,
and
we
are
moving
it
to
beta
in
1
111,
which
is
going
to
be
done
on
next
release.
A
fennekin
anti
affinity
has
been
in
a
beta
feature.
Hopefully
we
can
move
it
to
AGI
soon,
painting
note
by
condition
this
is
actually
another
feature
which
require
which
uses
things
and
toleration
we
are
moving.
Those
to
beta
scheduling
framework
is
a
is
a
big
effort
that
we
are
also
working
on.
A
E
Is
there
anything
that
you
can
intelligently
reschedule
all
the
parts
based
on
the
workload
demand
like
because
I
call
it
one
of
the
work
groups
in
East?
You
give
you
an
example
of
Sto.
For
instance,
sto
mixer
takes
more
resources
as
you
load
up
and
the
more
load
come
in.
So
if
there
is
a
feature
in
the
scheduler
that
can
dynamically
risk
Aereo
some
of
the
parts
which
are
overloaded,
that
will
really
help
otherwise
right
now.
E
A
So
it's
not
exactly
scheduler,
but
kubernetes
in
general
is
going
to
have
a
feature
similar
to
what
you
are
asking.
Basically,
autoscaler
is
going
to
actually
I
guess
is
having
this
feature
in
alpha,
which
is
basically
vertical
part,
auto
scaling.
It
increases
the
amount
of
resource
requirements
for
that
part.
That
now
has
a
higher
usage,
and
we
also
have
this
scheduler,
which
is
another
incubator
project.
A
The
scheduler
also
looks
at
the
nodes
which
have,
let's
say
close
to
the
capacity
and
remove
some
of
the
pods
and
when
the
pods
are
removed,
a
scheduler
again
schedules
them
elsewhere.
So,
basically,
a
cooperation
of
various
components
in
a
cluster
will
result
into
the
feature
that
you
you're
asking
yes,
I.
F
Have
a
tree
of
related
questions
about
scheduler
capacity?
Is
there
documented
complexity
of
the
scheduler?
It
looks
like
what
from
what
you
said,
the
cost
of
scheduling
one
pod
is
going
to
be
order
of
n
log
in
and
it
so.
My
question
is:
is
that
right
or
do
you
have
a
different
answer
and
the
related
question
is
I
saw
somewhere
that
there
is
a
goal
of
being
able
to
schedule
I
think
it
was
not
forget
the
goal,
but
is
there
a
stated
you
know
documented
throughput
achieved
through
put
on
the
scheduler?
Yes,.
A
So
in
terms
of
throughput,
we
do
definitely
have
some
consoles
that
actually
you
can
go
and
look.
There
are
like
scalability.
The
team
actually
has
these
graphs
that
shows
the
scheduler
throughput,
but
in
terms
of
complexity,
it's
really
really
hard
to
come
up
with
a
single
like,
oh
of
a
login,
or
anything
like
that.
The
reason
is
that
it
really
depends
on
the
features
that
or
scheduling
features
that
pods
request
from
the
scheduler.
So
a
part
that,
for
example,
has
pod
inter
pod
affinity.
A
A
D
A
A
G
A
Of
course,
yes,
you,
you
will
have
multiple
options
when
you
create,
for
example,
notes
for
your
cluster.
If
your
nodes
have
special
hardware,
you
contain
them
with
various
types
of
teens.
The
teens
are
pretty
generic.
These
are
just
key
value
pairs,
so
kubernetes
is
not
going
to
do
it
automatically
for
you.
It
probably
needs
some
level
of
coding
on
your
side
to
add
properties
for
the
nodes
in
your
cluster,
but
once
you
do
it
yeah,
it
works
pretty
much
for
any
type
of.
A
That
I
can
I,
don't
know
the
answer.
Basically,
what
you
are
you're
asking
is
very
much
node
related.
How
know
basically
assigns
these
two
ports
to
various
pods
in
a
cluster
in
that
particular
node
and
I.
Don't
know
the
answer
I'm,
not
so
sure,
if
actually
support
it
today
or
if
you
can
put
for
that
particular
requirement
in
the
pod
spec
I
am
not
aware
of
anything
like
that.
C
The
question
I
have
is:
is
it
possible
so
currently
the
scheduler
uses
the
number
of
CPUs
in
the
machines
and
the
parts
back.
It
looks
for
the
CPU
requests
to
find
the
available
node
right.
We
wanted
to
basically
make
use
of
you
know,
over-committing
and
not
by
using
a
limits
and
the
request
just
by
in
terms
of
scheduler
bin
packing
more
parts
on
to
the
machine
and
then
the
available
resources.
There's
something
community
is
looking
for.
Yeah.
A
A
That's
we
call
burstable
part
cessation
that
also
achieves
over
commitment
of
the
CPU
or
even
memory,
but
in
case
the
memory
it's
a
little
bit
harder,
because
we
cannot
really
actually
run
more
than
more
positive
the
amount
of
memory
that
we
have
a
node.
So
you
may
end
up
having
some
of
your
parts
in
the
field,
but
CPU
is
a
more
like
compressible
resource,
so
it's
easier
to
achieve
over
commitment
for
CPU.
Yes,.
C
A
A
completely
different
model,
as
we
have
today.
Well,
the
question
is:
are
we
ever
going
to
consider
distributed
scheduling
as
opposed
to
the
centralized
scheduling
that
we
have
today?
So
that's
a
model
effort,
for
example
maysa
users
and
it's
a
different
model
of
scheduling?
We
don't
think
that
we
are
ready
really
to
absorb
all
of
the
changes
there
and
it's
a
little
hard
given
that
the
kubernetes
api
has
evolved
over
the
assumption
that
you
have
a
centralized
scheduler,
it's
a
little
harder
to
come
up
with
them
with
a
central
distributed.
The
scheduler
yes,.
A
So
the
question
is
that
if
there
are
many
pending
pods
when
are
gonna,
if
they're
gonna
get
drops
eventually
or
are
they
gonna
remain
pending
forever
or
what
is
the
behavior?
The
the
behavior
is
actually
kubernetes
schedule
keeps
them
all
in
the
queue
as
long
as
they
exist
in
the
cluster.
So
if,
if
they
are
deleted
at
some
point,
you
are
deleted
from
the
queue
as
well,
but
kubernetes
scheduler
will
keep
retrying
to
schedule
them
until
it
actually
can
order
deleted.
A
A
Right,
so
yes,
so
the
question
is
sometimes
when
a
part
is
getting
retried
a
few
times.
The
scheduler
automatically
automatically
applies
it
back
off
to
that
part,
once
the
part
cannot
be
scheduled
and
as
a
result,
a
part
that
has
been
longer
in
the
queue
probably
can
take
longer
to
schedule
because
of
the
back
of
time
so
kubernetes
the
newer
kubernetes,
the
scheduler,
actually
uses
it.
It's
a
part
of
priority
and
preemption,
which
is
not
enabled
by
default,
but
the
newer
mechanism
actually
is
sorting
pods
based
on
their
priority.
A
So
parts
with
higher
priority
are
in
front
of
the
queue,
and
today
we
don't
have
any
backup
mechanism
for
this
newer
architecture.
We
may
add
actually
a
backup
mechanism
for
that
as
well.
To
make
sure
we
have
a
fair,
we
give
fair
scheduling,
opportunities
to
deposit
in
the
queue,
but
anyway
the
the
idea
here
is
that
the
pods,
with
highest
priority,
get
a
highest
chance
of
getting
scheduled
and
those
with
the
lowest
Peyer
to
go
to
the
end
of
the
queue,
so
that
behavior
is
going
to
change
so
we're.