►
From YouTube: Apache YuniKorn A Kubernetes Scheduler Plugin for Batch Work... Wilfred Spiegelenburg & Craig Condit
Description
Apache YuniKorn A Kubernetes Scheduler Plugin for Batch Workloads - Wilfred Spiegelenburg, Cloudera & Craig Condit, Cloudera
Kubernetes has historically focused on service-type workloads. Stateful workloads have also become better supported in recent releases. Batch scheduling continues to lag in Kubernetes core. To better support batch scheduling, several alternative schedulers have been created, including Apache YuniKorn, which has a growing community and is utilised by several large organisations such as Alibaba, Apple, and Cloudera. Over the past few years, Apache YuniKorn has matured into a highly performant, flexible workload scheduler. Recently, we have enhanced Apache YuniKorn with a new execution mode which allows Apache YuniKorn's full power and flexibility to be deployed as a set of plugins to the default Kubernetes scheduler. This allows service and batch workloads to coexist seamlessly. This session will dive into using Apache YuniKorn to schedule batch workloads leveraging the advanced options like workload queueing and quota sharing without affecting the traditional non batch Kubernetes workloads.
A
Understandable
good
afternoon,
wilford
spielenberg
a
patch
unicorn,
a
scheduled
plug-in
for
batch
workloads.
I
was
supposed
to
be
here
with
my
colleague
craig,
but
he
stuck
in
the
us
with
some
personal
things
that
he
needed
to
take
care
of.
So
it's
just
me
here
for
today,
apache
unicorn,
as
this
we've
just
announced
that
we've
graduated
to
a
top
level
project
on
apache
that
came
out
yesterday.
So
that's
a
big
thing
to
hear:
we'll
go
in
and
we'll
dive
into
what
we're
going
to
do
today,
we're
going
to
look
at
why
we're
doing
apache
unicorn?
What's?
A
A
A
So
from
the
perspective
where
we
got
where
we
came
from
we're
coming
out
of
the
yarn
in
the
mesos
world,
we
look
at
pots
or
or
applications
that
get
scheduled
that
have
got
a
number
of
different
pieces
and
they
belong
together.
It's
a
logical
grouping,
it's
not
specific
for
anything
that
we
do
together.
A
Sometimes
you've
got
things
that
you
want
to
be
first
in
first
out
financial
data.
You
first
want
to
process
the
data
from
yesterday
before
you
do
two
days.
That's
the
first
name
first,
but
if
you've
got
interactive
sets
of
data,
you
want
to
do
that
fair
sharing.
You
want
to
make
sure
that
everybody
gets
their
resources
and
everybody
goes
through
the
way
that
they
have
submitted
things.
The
other
thing
is
that
we,
if
you
look
at
batch
workloads,
you
want
to
do
cueing.
It's
it's
been
mentioned
by
everybody.
You
want
to
do
some
cueing.
A
A
Often,
jobs
run
during
the
nights
during
the
evenings
for
for
the
the
standard
batches
that
get
run,
but
the
interactive
workloads
that
will
run
more
during
the
day.
You've
got
completely
different
demands
on
your
cluster
during
the
things
during
the
time
you
want
to
do
this
postpone
certain
workloads
you
want
to
hold
on
to
them,
and
you
want
to
kick
them
off
whenever
you
want
to
so.
A
Why
apache
unicorn,
so
we
looked
at
what
the
default
scheduler
did
and
what
we
had
as
a
demand
from
from
the
customers
and
what
we
were
doing
on
a
yarn
and
a
message
kind
of
thing.
So
in
the
default
schedule
we
don't
have
an
application
concept.
You
can't
just
say
group
all
these
things
together
and
run
this
as
as
one
thing
schedule
them
fit
me.
Resources
do
whatever
you
need
to
do.
A
The
scheduling
algorithm
is
also
limited.
We've
got
one
queue.
We
sort
the
queue
it's
based
on
the
pots
that
are
there
and
then,
if
we've
sorted
all
those
spots,
we're
gonna
run
them,
but
the
queue
that
we've
got
is
one
queue
for
the
whole
cluster
and
I
can
have
multiple
types
of
workloads
running
at
the
same
point
in
time.
So
I
really
would
like
to
have
multiple
types
of
sorting
algorithms.
A
For
instance,
one
namespace
will
run
with
the
first
theme
first
out,
the
other
one
will
do
fair,
sharing
or
priority
based
queuing.
So
that's
that's
the
other
part
limits
and
quotas.
Resource
quotas
are
not
part
of
the
scheduler
on
standard
kubernetes.
You
enforce
them
before
you
submit
things.
So
somebody
comes
up
and
says:
I
want
to
run
this
batch
job.
Oh
there's!
No
resources
available.
Oh
now,
I
need
to
come
back
and
retry
to
submit
that
same
kind
of
job
again
and
again,
up
until
they've
got
the
resources
available.
So
that's
a
problem.
A
A
We
create
a
hierarchical
queue
system,
so
we've
we
set
up
a
queue
on
a
route
queues
underneath
and
etcetera,
etcetera,
and
we
enforce
the
quotas
in
that
hierarchical
hue
system
later
on,
we'll
look
a
bit
more
on
how
we
do
that
and
what
what
that
gives
us
and
what
the
the
real
extras
get
that
we
get
out
of
that,
we
can
do
in
that
hierarchical
uses.
We
can
do
flexible
quota
distribution,
so
I
saw
that
in
one
of
the
other
presentations
coming
back
too
it's
quota
sharing.
A
A
A
How
do
you
distribute
your
quotas
between
the
queues
and
then
we
can
also
look
at
application
sorting.
So
do
you
want
to
do
priorities
first,
in
first
out,
fair
sharing
and
all
these
things
can
be
set
up
per
queue,
so
I've
got
the
possibility
to
have
queues
that
are
doing
fare,
sharing
queues
that
are
doing
fifo
based
on
the
applications
again
and
then
the
last
but
not
least,
we've
got
the
the
node
sorting.
Also
the
area
where
we
come
from
we've
got
customers
that
want
to
run
either
in
the
cloud
or
on-premise
and
on-premise.
A
You've
got
a
steady
cluster.
You
don't
scale
up,
you
don't
scale
down
at
least
most
people
don't
so
that
means
that
you,
you
want
to
be
able
to
sort
your
notes
differently
in
the
cloud
you
want
to
pack.
Everything
together
make
sure
you
get
your
cost
under
control
pack,
all
the
pots
that
you've
got
on
one
note
you
do
bin
packing
and
in
an
on
premise
cluster.
B
A
Bit
more
advanced
scheduling
requirements,
so
apache
spark
has
been
used
a
number
of
times
and
as
an
example.
So
if
you
look
at
advanced
scheduling,
we've
got
the
gang
scheduling.
You
want
to
create
an
application,
but
you
don't
know
beforehand
how
much
it's
going
to
use,
but
you
want
to
give
it
a
certain
set
of
resources,
so
you
specify
on
the
spark
application
or
on
any
application
what
kind
of
gang
resource
you
want
to
use.
A
You
can
have
multiple
gang
definitions,
so
multiple
plot
types,
multiple
combinations
of
things
and
we
only
start
scheduling
based
on
the
resources
that
are
available.
So
the
other
thing
that
comes
up
in
the
gang
scheduling-
and
somebody
mentioned
that
as
sla
scheduling,
I
think
it
was
called-
is
the
soft
and
hard
scheduling
the
for
the
gang.
A
A
In
other
cases,
you
want
to
say:
oh
no,
no,
I
really
need
all
these
resources.
If
I
don't
get
them,
stop
the
application,
just
fail
it
and
let's
go
on,
but
that's
over
all
those
combinations
are
possible
and
we
can
set
up
all
of
that
stuff.
For
you
now
look
at
the
the
application.
Sorting
again
we
sort
perqueue.
A
A
A
spark
application
could
also
be
an
sql
query
that
that
somebody
wants
to
run
and
wants
to
have
a
direct
output
so,
depending
on
where
we
run
depending
what
we
do,
we
can
do
the
different
swords
in
there
and
the
third
point
was
the
the
note
sorting
bin
packing
fair
sharing.
Whatever
you
want
to
do
top
one.
The
bin
packing
private,
is
in
the
public
clouds,
aws,
google,
whatever
you
want
to
run
the
bottom,
you
run
your
private
cloud.
You
share.
You
burst
your
your
containers.
A
All
these
are
available
application,
sorting,
gangster
and
all
that
kind
of
stuff.
You
can
set
up
different
cues
different
things.
You
can
do
whatever
you
want,
but
there's
only
one
node
sorting
policy
allowed
for
the
cluster,
because
there's
there's
no
way
that
you
can
sort
nodes
in
a
different
way,
because
you
share
all
the
nodes
on
on
one
thing.
A
In
the
previous
slides,
I
stepped
really
quickly
over
what
possibilities
we've
had
in
in
the
hierarchical
hue
system.
So,
let's,
let's
dive
in
a
little
bit
more
in
into
the
hierarchical
cue
system.
I'm
going
to
take
a
really
simple
example:
you
can
make
it
as
difficult
as
possible
again
we're
coming
from
the
from
the
yarn
perspective.
A
A
That
one
is
not
going
to
be
scheduled
by
unicorn,
because
we're
running
as
a
plugin
we're
running
as
an
extension
of
the
default
scheduler.
So
what
we
say
is
those
two
namespaces
unlimited
limited
those
will
follow,
unicorn
and
whatever
you
want
to
do,
and
whatever
you
set
up
for
the
non-unicorn
namespace,
we
don't
care,
we
don't
touch
it.
We
just
go
through
the
logic
that
the
default
scheduler
does,
with
whatever
you've
set
up
there.
A
A
A
The
non-unicorn
namespace
that
we've
created
does
not
have
a
quota
within
unicorn,
but
it
still
uses
notes
and
and
and
an
amount
of
resources
in
the
cluster,
so
whatever
is
being
used
in
non-unicorn
namespace
will
be
deducted
from
the
root
quota
that
unicorn
knows
how
to
handle.
So
we
don't
use
the
quota
that
is
set
on
the
non-unicode
names.
We
use
the
real
usage
that
is
there,
so
we
track
the
ports
and
we
deduct
that
from
whatever
is
there
so
again
that
makes
the
quota
at
the
root
level,
dynamically
adjusted
and
then
within
unicorn.
A
We
can
set
different
quotas
on
different
queues.
In
this
case,
we've
set
a
resource
limit
as
an
example
on
the
tenant
queue
of
75
gigabytes,
75
cpus,
whatever
you
want
to
set
is
fine.
We've
limited
the
limited
queue
also
to
a
50
gigabyte
and
a
50
cpu,
so
with
limited,
corresponds
to
a
namespace
that
namespace
or
the
ports
that
will
be
running
in
the
namespace
will
be
limited
by
unicorn
to
the
50
gigabyte.
A
Unlimited
does
not
have
a
quota
set
directly
on
it,
but
the
parent
of
unlimited
is
limit.
Is
the
tenant
queue?
So
what
happens
is
that
when
we
try
to
schedule
things
in
the
unlimited
queue,
we're
not
only
checking
does
this
queue
have
a
limit,
but
we
also
look
up
to
the
parent
and
the
parent
of
the
parent
just
to
make
sure
that
we
keep
track
of
that
quota.
That
is
there.
So
we
do
a
recursive
check
so
effectively
in
the
unlimited
queue
we've
got.
A
A
75
gigabyte,
75
cpu
limit
set,
however,
checking
the
the
parents
and
the
and
the
interaction
at
the
different
levels
is
a
bit
more
complex,
because
what
happens
is
that
the
usage
within
the
tenant
queue
is
really
the
sum
of
all
the
queues
that
are
below
the
tenant
queue.
So
whatever
resources
are
used
in
unlimited
and
limited
get
combined
as
a
resource
usage
in
the
tenant
queue.
So
if
I
use
50
gigabyte
in
limited,
then
really
effectively,
unlimited
can
use
another
25
gigabytes.
A
A
So
again,
looking
at
all
these
different
ways
of
configuring
things,
we've
got
different
cue,
sorting,
algorithms
that
we
can
set
up.
We
can
do
the
pots
the
applications
within
the
unlimited
queue
as
a
fifo
setup.
While
we
do
fair
sharing
in
the
limited
queue,
combinations
are
endless,
you
can
set
up
whatever
you
want.
A
Then
we'll
go
and
have
a
little
bit
of
a
look
at
the
architecture.
So
what
did
we
do?
How
do
we
we
get
where
we
are
here
for
for
this
architecture?
So
when
we
started
the
the
the
system
that
was
available
was
no
extensions
on
the
default
scheduler
we
couldn't
do
anything.
There
was
no
plug-in
framework.
The
extenders
were
probably
still
a
little
bit
in
development,
but
that
was
abandoned
halfway
through,
so
we
really
didn't
have
any
any
options.
A
So
what
we
did
is
we
built
a
simple
architecture
based
on
a
plug,
a
plugin,
a
shim
and
a
core
scheduler.
So
because
we
came
out
of
the
yarn
world,
we
thought
we've
had
multiple
resource
managers,
nasa's
yarn,
kubernetes,
all
these
kind
of
things,
so
we
want
to
be
able
to
run
the
scheduler
on
top
of
whatever
resource
manager.
Is
there?
A
So
what
we
decided
was
we
built
a
core
that
does
all
the
scheduling
stuff,
so
it
handles
the
queues
that
handles
the
quota
checks
that
does
all
the
things
that
we
want
to
do
of
enforcing
all
that
stuff
and
we've
got
a
shim
that
runs
below
that
that
hides
all
the
resource
manager,
specific
things
for
it,
so
the
unicorn
core
doesn't
really
understand
what
a
pot
is.
It
doesn't
know
what
it
is
it
just
knows:
I've
got
a
resource
request.
A
It
has
got
an
allocation
that
I
need
to
schedule,
so
we
can
remove
the
kubernetes
shim
and
we
can
put
a
yanching
in
place
or
a
message
him
in
place.
That
does
the
same
thing.
The
kubernetes
sim
talks
to
the
api
server
and
it
converts
whatever
the
api
server
gives
inputs
into
what
we
understand
in
the
core
as
applications
and
all
that
kind
of
stuff.
A
So
this
was
the
design
that
we
had
when
we
started
about
three
three
and
a
half
years
ago.
This
is
what
we've
implemented
over
the
time
that
we
were
running.
The
plug-in
framework
became
mature
and
we
started.
We
started
using
it.
So
how
did
we
change
our
design
from
what
was
here
implementing
the
whole
scheduler
doing
all
the
things
that
the
default
scheduler
did
to
something
a
little
bit
more
in
line
with
the
plug-in
framework?
A
So
what
we
did
is
we
pulled
apart
the
shim,
so
we
replaced
all
the
things
that
we
did
within
the
shim
with
the
plug-in
framework,
so
we
integrated
instead
of
our
own
code
for
for
handling
the
default
for
the
handling,
the
binding
of
the
ports
and
all
that
kind
of
stuff.
We
replace
that
with
using
the
plug-in
framework
and
letting
the
default
scheduler
handle
all
those
things
for
us.
A
A
A
A
So
we
we've
picked
up
at
the
point
that
we
we
started
and
pick
up
the
pre-filter
in
the
pre-filter.
What
we
do
is
we
pick
out
the
parts
that
unicorn
needs
to
handle,
so
everything
that
belongs
to
either
namespace
or
things
like
that.
We
want
to
manage
as
unicorn
through
the
unicorn
code.
We
pick
those
out
and
we
we
handle
them
anything
else.
We
just
let
go
we
don't
we.
We
don't
bother
with
it.
A
A
Release
whatever
has
passed
so
anything
that
passes
through
the
pre-filter
and
what
needs
to
be
handled
by
unicorn
has
passed
the
internal
quota
checks
for
for
unicorn.
We
know
that
we
are
in
the
in
the
right
state,
so
we've
we
know
that
it's
an
application.
The
application
should
be
running
so
it
runs
within
the
queue.
A
So
after
the
pre-filter
has
passed,
the
default
scheduler
will
start
looking
for
a
node
and
assign
a
node
to
the
to
the
pod
unicorn.
The
core
scheduler
does
the
same.
We
look
at
the
node
and
we
we
also
do
that
in
the
field.
Those
two
things
come
together,
so
the
default
scheduler
says
I've
got
a
node,
it's
for
it
is
running
this
port
and
we
keep
on
rejecting
the
note
up
until
we
say.
Oh
no.
This
is
the
note
that
we
have
now
decided.
A
A
A
C
And
now
for
a
short
demo
of
unicorn's
new
plugin
interface,
we
first
create
two
namespaces
each
having
a
quota
of
one
cpu
and
one
gigabyte
of
ram.
However,
unicorn
has
been
configured
so
that
namespace
2
will
utilize
the
internal
default
scheduling
logic
rather
than
the
queueing
logic
that
unicorn
normally
uses.
B
C
C
C
Now
we
can
clean
up
our
pods
so
that
we
can
create
them
in
our
second
namespace.
Now
we
create
three
pods
in
our
second
namespace.
This
namespace
has
been
configured
to
bypass
unicorn
queuing,
and
so
all
three
pods
will
be
allowed
to
start
at
the
same
time,
and
we
can
see
that
here
we
can
also
see
that
unicorn
has
been
assigned
as
the
scheduler,
so
we
haven't
completely
bypassed
unicorn.
However,.
B
C
A
A
A
A
A
A
A
So,
from
a
from
a
scheduling
perspective
from
the
core
perspective,
we
are
resource
agnostic,
so
whatever
you
define
as
a
resource,
if
even
if
you
say
I
want
to
schedule
pots
that
use
a
resource
called
license,
we
don't
care,
you
can
schedule
whatever
you
want.
So
we
do
if,
if
you
can
define
it
on
the
pot
and
you
can
define
it
anywhere
else,
we
can
schedule
on
that
and
we
can
see
it
as
a
resource.
Yeah.
D
Just
from
your
experience,
do
you
find,
like
you
mentioned,
that
there's
some
customers
using
deep
hierarchies?
Do
you
think
that
was
a
good
idea?
Do
you
think
that
they
are
being
abused?
Maybe
we
should,
for
example,
force
them
not
to
do
that,
because
I
imagine
to
be
really
hard
reasonable,
you're
saying
six,
seven
levels,
that's
just
like.
A
Yes,
yeah
yeah
yeah.
No,
so
yes,
those
q
hierarchies
are
really
difficult
to
talk
about
and
really
difficult
to
understand,
but
the
companies
that
use
them
they,
what
they
do
they've
got
a
logical
grouping
of
that.
So
let's
say
you've
got
a
a
multinational
that
says
I've
got
the
company
as
the
top
level
than
us
as
the
second
level
and
other
regions
around
the
world.