►
From YouTube: SIG-Scheduling Intro and Deep Dive - Wei Huang, IBM
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
SIG-Scheduling Intro and Deep Dive - Wei Huang, IBM
Speakers: Wei Huang
SIG Scheduling is responsible for kube-scheduler and its related subprojects such as the scheduling-plugins and descheduler. These projects share the common goal of enabling users and developers to ensure that pods are assigned to the appropriate nodes based on various scheduling criteria. We also endeavor to ease customizing and extending scheduler for advanced usage. This talk will introduce attendees to SIG-Scheduling with an overview of the current goals and ongoing work within the SIG, as well as opportunities for new contributors to get involved.
A
Hi
everyone
welcome
to
today's
session
today,
I'm
going
to
to
give
you
a
deep
dive
on
the
sixth
scheduling.
My
name
is
I
work
for
ibm
and
I'm
also
the
current
co-chair
of
the
sikh
scheduling.
So
today's
agenda
contains
three
parts.
First,
I
will
give
you
a
brief
introduction
on
schedule
and
then
we'll
share
with
you
some
best
practice
tips,
and
so
that
you
can
make
the
most
of
your
schedule,
and
the
third
part
is.
A
A
It
also
doesn't
evolve
into
the
spin
of
the
underlying
containers.
You
just
find
a
node
for
the
pending
path
and
technical
terms.
So
basically
scheduler
will
watch
the
pending
path,
which,
which
means
the
path
doesn't
have
the
spectrum
name
set,
and
then
it
will
look
into
the
part.
Explicit
scheduling
constraint,
like
cpu
memory,
requests
another
affinity,
et
cetera
and
also
taking
some
predefined
implicit
scheduling,
preference
into
consideration
and
look
at
the
cluster
resource
usage
and
make
the
decision
to
place
the
path
to
the
optimal
node.
A
So
this
is
what
scheduler
does
and
inside
scheduler
there
are
basically
several
components:
the
first
one
we
call
the
caching
so
to
make
the
most
optimal
decision
schedule
has
to
maintain
a
cache
to
have
a
sort
of
choose
view
of
the
whole
cluster,
like
how
many
nails
are
there,
how
many
tiles
are
there
and
how
many
pv
pvcs,
etc,
etc.
So,
scheduler
we
run
internal
fuel
scheduling
formulas
to
watch
for
the
api
objects
so
that
we
can
maintain
the
internal
cache
properly
and
the
second
part
is
called
queuing.
A
So
if
a
pi
is
coming
to
the
queue
we
schedule
it
with
reasonable
order.
So
internally
we
have
a
priority
queue
and
if
there's
no
note
fit
for
the
path,
we
also
have
some
back
off
mechanism
to
ensure
the
lower
priority.
Part
has
a
chance
to
be
scheduled
and
also
the
path
can
have
a
fair
chance
to
be
re-popped
up
and
be
retried,
and
also
next,
our
main
scheduling
goal
routine
and
binding
routine,
which
is
a
serious
processes
to
do
the
core
scheduling
flow.
A
So
the
core
scheduling
flow
looks
like
this,
so,
firstly,
we
will
pick
apart
from
the
scratching
queue
and
we
will
go
through
a
series.
What
we
call
extension
point
here
so
for
each
extension
point
we
have
a
series
of
plugins
and
we
will
go
through
those
plugins,
either
sequentially
or
in
parallel
depends
on
what
phase
it
is.
A
It
will
first
go
through
the
pre-filter
and
filter
to
check
the
hard
requirements
for
the
part
like
how
much
cpu
it
requires,
detain,
deterioration,
not
affinity,
etc.
So
if
there
are
some
nails
fit
for
the
part,
we
will
do
prioritization
on
the
nails,
which
is
to
give
the
nails
candidate
score
for
each
according
to
predefined,
the
score
plugins
and
then
finally,
the
node
has
the
highest
score.
We
have
chosen
as
the
final
net
and
then
we
go
to
next
cycle
binding
cycle
to
assign
the
node
to
the
path.
So
this
is
the
happy
path.
A
A
So
and
if
it
succeeds
means
we
can
find
a
node
to
serve
the
path
with
the
cost
of
preempting
some
victims,
then
we
will
also
set
the
path
status,
that
nominated
no
name
to
the
path
and
then
put
it
back
to
the
queue,
because
we
cannot
immediately
assign
the
no
name
to
it,
because
the
victims
are
still
being
deleted,
so
we
should
have
a
grace
period
for
those
parts
to
be
deleted.
So
this
is
basic.
A
The
core
scheduling
flow,
and
this
is
the
basic
introduction
of
how
scheduler
does
so
next
I
call
it
make
the
most
of
your
schedule.
The
first
part
I
call
it
policy
customization,
so
schedule
is
a
box.
It
has
a
lot
of
nars
of
a
lot
of
options,
so
if
the
by
this
default
option
doesn't
work
for
you,
you
can
adjust
them.
A
A
And
second
thing
to
keep
in
mind
is
that
by
default,
there's
a
lot
of
scoring
plugins
to
like
to
favor
each
kind
of
preference
and
apply
that
impact
on
your
final
decision
and
by
default
the
weight
is
usually
one.
So
if
you
prefer
one
scoring
policy
over
the
other,
for
example,
in
this
case,
I
put
image
locality.
For
example,
image
locality
means
you
prefer
to
schedule
the
path
to
the
nodes
that
already
has
an
image
downloaded.
A
So
this
is,
I
call
it
a
plugin
specific
weight,
because
this
weight
is
so
plugging
in
the
plugins
configuration
context
and
there's
another
way
to
configure
the
weight
using
the
work
work
cloud
itself
like
in
no
dfinity
part
affinity
about
entire
affinity.
There
is
a
field
called
preferred
during
scheduling
ignore
during
execution.
There
is
also
a
wait
for
your
part
to
show
your
favor
to
the
existing
part.
A
A
And
next
thing
is
that
we
have
a
default
set
of
the
plugins,
but
that
doesn't
prevent
you
to
use
another
plugin
to
replace
the
default
plugins
or
you
just
want
to
make
additional
plugins.
So,
for
example,
by
default,
we
enable
the
list
allocated
plugin,
which
means
we
prefer
to
schedule
the
paths
evenly
across
the
cluster.
But
in
some
use
cases
like
cluster
autoscaler
is
not
the
case.
It's
totally
the
opposite.
A
A
Wait
you're,
maybe
you're
wondering
I
may
don't
have
the
full
knowledge
on
what
kind
of
plugins
are
there
and
what
kind
of
plugins
should
be
enabled.
So
what
if
I
enable
the
most
allocated
plugin
but
forgot
to-
or
I
don't
know
to
disable
the
list
allocated?
What
should
I
do
so
in
this
case,
this
two
plugin
will
work
in
the
opposite
way.
So
this
is
not
the
the
expected
behavior.
A
A
So
this
is
the
example
to
adjust
your
scheduling
behavior
by
modifying
the
default
or
modifying
the
plugin
arguments.
So
here
I
put
a
link
for
you
to
get
familiar
with
all
the
plugins
and
their
arguments
and
what
their
metrics
so,
basically,
that's
the
first
recipe
I
want
to
share
with
you
to
customize
your
schedule,
behavior
by
modifying
the
default
scheduler
config
like
changing
the
plugin
weight
and
changing
the
plugging
arguments,
etc.
A
The
second
part
about
want
to
talk
about
multi
profile
versus
multi-scheduler,
so
my
schedule
you
may
have
know
that
is
that,
in
addition
to
the
default
scheduler,
you
can
run
your
own
schedule
to
cover
some
scheduling
tasks
of
a
particular
path.
So
what's
the
problem
of
running
multiple
scheduler,
the
biggest
problem
is
resource
confliction,
because
these
two
scheduled
banners
doesn't
communicate
with
each
other,
so
they
may
compete
for
the
limited
resources.
A
Suppose
a
node
has
only
one
cpu
left
and
default
scheduler
and
your,
for
example,
full
scheduler
simultaneously
assign
their
path
x
and
y
to
the
node
and
suppose
both
tiles
request,
one
cpu
and
in
this
case
cubelet
will
be
the
eventual
arbitrator.
It
will
reject
the
latter.
Arriving
party
suppose,
in
this
case
it's
part
y,
so
there
will
be
a
very
severe
and
weird
symptom,
which
is
the
letter
arrived.
The
part
will
be
show
up
in
this
case
without
of
cpu
and
in
api's
perspective.
This
part
are
already
finished.
A
The
scheduling
phase
it
has
already
has
the
no
name
set
that
implies
this
part.
We
are
behind
there
forever
until
the
resources
in
this
node
get
released.
That's
a
very
unexpected
behavior
will
sometimes
even
involve
operation
to
manual
resolve
this.
So
this
is
the
biggest
problem
of
multi-schedulers.
A
So
do
you
have
solutions
yeah?
We
do
have
some
solutions
to
mitigate
this,
but
not
perfect.
For
example,
some
users
are
trying
to
divide
the
cluster
into
several
pieces
and
delegate
the
thumb
piece
to
one
scheduler
and
others
to
another.
So
this
can
work
that
will
cause
fragmentation
and
hope
has
caused
a
low
cluster
utilization.
A
Another
solution
is
on
top
of
this
multiple
scheduler.
They
set
up
extra
arbitrator
so
that
each
scheduled
decision
was
reported
to
the
arbitrator
and
the
actuator
works
as
a
central
place
to
do
a
decision
reject
or
accept
so
this
works,
but
as
you'll
see,
will
definitely
cause
extra
workload
and
also
because
you
introduce
another
component,
you
have
to
manage
the
extra
efforts
better
manage
this
kind
of
textures.
A
So
what's
the
community
direction
on
this,
so
the
community
direction
is,
is
to
use
a
multi-profile
schedule.
Much
prefer
schedule
is
on
steel
schedule,
but
inside
the
scheduler
it
can
define
different
profiles
and
each
profile
you
can
think
of
is
as
sub
schedule
so
because
it
has
its
own
set
of
scheduling
policies
to
including
which
kind
of
weights
which
kind
of
plugins
and
which
kinds
of
plugin
arguments
et
cetera.
So
in
this
case
I
provided
a
schedule
config
multi-profile
demo,
and
here
you
can
see
that
we
have
four
kinds
of
scheduled
profiles.
A
One
is
the
first
scheduler
to
keep
it,
as
is
for
the
regular
workloads,
and
the
second
is
image
first
to
favor
the
image
locality
so
that
maybe
some
service
workers
care
about
the
code
start
time.
So
they
want
to
schedule
a
path
to
the
notes,
with
the
image
already
downloaded
and
third
is
called
bin
pack.
So
custom
auto
scaler
may
be
beneficial
for
this
because
you
want
to
save
the
cost
of
running
machines.
A
The
last
is
called
skip
score
because
score
phase
is
the
doing
the
ranking
is
to
to
do
the
preference
for
the
path.
So
in
some
cases
like
the
cluster
is
very
scheduling,
highly
used
and
there's
a
lot
of
cause
queueing
there,
you
maybe
want
to
schedule
the
path
in
most
cases
low
priority
or
offline
workers
as
quick
as
possible.
So
I
don't
care
about
finding
the
best
note.
This
is
the
satisfying
the
heart
requirements
that
works
for
me.
A
So
in
this
case
you
have
four
kind
of
profiles
and
your
workloads
can
specify
the
spec,
the
scheduler
name,
to
map
to
the
profile
name,
so
there's
two
ways
to
specify
secondary
name
once
you
express
them
manually.
The
second
way
is
to
do
that
dynamically,
so
you
can
set
up
a
admission
controller
and
so
that
it
can
intercept
the
creation
of
the
path
right.
A
So
you
can
programmatically
like
assign
the
path
to
the
the
schedule
profile
by,
for
example,
the
path
labels
or
by
even
by
some
external
metrics,
like
how
business
the
current
schedule
is
behaving
right,
like
the.
How
many
paths
are
there
are
being
queued
up
in
the
schedule,
so
it's
very
dynamic.
You
can
work
on
a
lot
of
innovative
ideas
to
do
this
and
in
terms
of
the
scheduled
config,
we
are
also
reading
up
a
cap
to
simplify
the
scheduled
conflict.
A
We,
you
know
we
we
will
introduce
a
multi-point
plugin
so
that
for
a
user
you
don't
need
to
know
whether
the
plugin
implements
score
or
filter
or
postal
filter.
You
just
define
the
name
and
provide
that
as
a
practice.
So
internally
you
will.
It
will
smartly
detect
whether
which
extension
points
it
it
implements
and
then
download
them
properly.
So
this
cab
is
still
being
developed,
so
we'll
be
available
in
123..
A
A
The
third
recipe
is
improve.
Your
throughput
so
scheduler,
especially
on
a
large
cluster,
you
sometimes
don't
need
to
iterate
all
the
cluster
to
find
a
node
to
fit
the
path.
So
there
is
one
option
called
percentage
of
nodes
to
score
and
by
default
is
set
to
50,
but
not
exact
50
is
the
adaptive
algorithm
to
calculate
the
percentage
by
default.
So,
for
example,
suppose
you
have
a
500
nails
you
will.
The
adapted
percentage
default
percentage
will
be
46.
A
A
Instead
of
going
through
all
500
nails.
So
and
then
this
250
30
nails
will
be
passed
down
to
the
scoring
phase
and
give
the
score
to
each
node.
So
in
this
case
you
can
see
that
you
know
200
nails
in
the
most
desired
case.
Unit
only
need
to
filter,
680
nails
and
the
hands
only
score
those
kind
of
females.
A
A
Second,
one
is
also
follow
the
same
idea
to
iterate
a
few
nodes.
Instead
of
going
through
all
of
them
in
the
default
preemption
plugin
there
is
some
parameters
like
minimum
candidates.
No
percentage
or
minimum
candidate
knows
absolute.
So,
by
defining
these
two
parameters,
you
can
define
how
many
nodes
you
want
to
go
through
the
preemption
phase.
Instead,
it
goes
through
all
the
nails.
A
Another
option
is
called
prefer,
nominate,
node,
it's
a
feature:
it's
not
an
option,
it's
just
a
feature
gate
introducing
102
y
and
already
enabled
by
default
in
22.
So
it's
just
a
yes
or
no
option
you
want
to
use
and
we
believe
it's
a
very
useful
feature.
So
we
don't
want
to
provide
as
an
option
it's
a
shortcut
to
leverage
the
status
nominating
node
name
so
that
when
the
path
has
been
already
carried
with
that
field,
we
won't
go
through
all
the
nodes.
A
Instead,
we
will
just
do
once
to
check
whether
the
previous
nominated
node
still
fits
if
it
fits.
I
will
just
assign
the
node
to
it
rather
than
going
through
the
regular
scanning
flow.
If
not
don't
worry,
we
all
fall
back
to
the
regular
screen
flow.
So
that's
why
I
say
this
is
a
feature
I
think
mostly
will
increase
the
scheduling,
especially
the
preemption
throughput.
A
A
So
the
same
thing
sounds
like
to
corresponding
to
the
cpu
cores,
but
underlying
you
will
involve
the
operation
system
as
well
as
go
through
the
go
long
schedule.
So
sometimes
it
cannot
be
hard
coded
to
the
cpu
cores,
so
it
needs
some
tuning
to
get
the
best
of
all
performance.
We
used
to
have
the
pr2
wanted
to
set
this
to
the
cpu
cost
number,
but
it
seems
doesn't
work
properly,
so
we
revert
up
here.
A
A
So
that's
the
third
recipe
I
want.
I
wanted
to
share
to
improve
your
throughput
optimization
and
the
final
one
is,
I
think
it's.
The
knowledge
that
I
just
want
to
highlight
is
that,
in
addition
to
some
regular
resource
core
resource,
like
cpu
memory,
we
can
model
your
customer
resource
in
the
format
of
extended
resource.
A
So
in
this
case
a
lot
I
see
a
lot
of
users
implement
their
own
screen
plugin,
but
it's
not
necessary
a
lot
of
practice
already
taking
the
external
resources
for
citizens
and
supporting
them.
Well,
all
right!
Last
four
recipes.
I
want
to
share
with
you
so
that
I
hope
those
recipes
help,
so
you
can
make
the
most
of
your
schedule
do
more
customized
options
and
without
any
further
coding.
A
A
We
have
been
thinking
about
to
supporting
more
diverse
workers
two
years
ago
or
three
years
ago,
so
we
initialized
the
scheduled
framework
design
to
focus
more
on
expose
the
internal
mechanics
to
the
external
developers
so
that
you
can
use
the
core
scheduling
capabilities
easily
and
then
easily
build
your
own
schedule,
plugins,
and
so
that,
with
your
customize,
the
autofortree
plugins
and
plug
into
the
default
scheduler,
you
will
make
it
a
schedule
plus
plus.
So
it's
still
one
scheduling
binary
with
extra
functionalities.
A
A
So
recently
we
were
thinking
about
that.
So
either
you
can
define
some
your
own
crds
so
that
with
some
extra
controllers
and
the
minutes
animation
web
to
control
the
better
creation
time
of
the
path.
Also
there's
we
have
done
some
upstream
work
like
suspended
job
like
in
the
next
series.
We
were
doing
some
features
like
multiple
scheduling
directory
for
the
suspending
job.
So
this
in
one
way,
we
will
perfect
of
internal
scheduling
mechanics
in
another
way.
A
A
We
remove
some
options
that
are
either
not
used
quite
often
or
doesn't
quite
make
sense,
and
we
will
also,
in
the
next
release,
will
come
up
with
a
simplified
scheduling
conflict
so
that
you
can
have
a
more
easy
to
configure
way
to
configure
this
kind
of
plugins
and
also
well
well,
try
to
make
that
totally
backwards
compatible,
and
there
are
some
features
I've
already
mentioned,
like
prefer.
Normally
now
like
multiple
scheduling
directives
and
also
in
122,
we
add
a
namespace
selector
for
part
affinity
before
in
the
pod
affinity.
A
The
last
thing
we
did
is
the
a
series
of
scheduling,
queueing
enhancement,
to
improve
the
efficiency
like
we
in
the
before
we
re-kill
the
path
in
a
very
brute
force
way
like
any
any
events
comes
in.
We
will
move
the
parts
back
to
the
back
to
the
head
of
the
schedule
queue,
but
now
we
will,
we
defined
the
series
of
rules
for
the
path
only
to
be
moved
to
the
head
on
its
method.
Events
comes
in
like
if
a
part
failed
with
a
pvc
or
pv
restriction.
A
One
for
example,
irrelevant
events
comes
in.
The
part
will
not
be
moved
to
the
head,
so
it
can
save
a
lot
of
scheduling
cycles
and
also
mitigate
the
sort
of
head
of
blocking
problems
in
this
kind
of
queue,
and
also
as
more
and
more
users
are
using
scheduling
framework
to
do
their
customer
plug-in.
So
we
also
did
some
work
to
support
some
dynamic
event
handlers
so
that
you
can
define
your
crds
so
that
the
cd
can
be
also
detected
by
the
scheduling
framework.
A
We
internally
use
some
dynamic
triangle,
mechanics
to
do
this,
and
also
we
provide
the
option
for
the
plug-in
developers
to
proactively
move
keypads
to
active
queue.
It
can
be
sometimes
useful
for,
like
the
co-scheduling
plug-in
to,
for
example,
if
you
have
a
spark
job
when
the
driver
part
comes
in,
you
may
want
at
that
time
to
move
associated
executor
path
back
to
the
xq,
so
the
executor
path
can
be
scheduled
afterwards
immediately.
A
A
We
maybe
want
to
to
spend
more
efforts
on
it
to
make
it
very
have
a
very
solid
sdks,
so
that
the
integrators
can
benefit
from
that
all
right
and
you
can
contact
us
in
the
china's
in
the
mailing
list
and
also
welcome
to
join
our
vehicle
meeting.
You
can
raise
your
gender
item
there.
We
will
discover
your
proposal,
etc
message
all
right.
Thank
you
for
joining
today's
session.