►
Description
This session explores resource management of AI/ML workloads in Kubernetes based environments and shows how two-level resource management using Multi-Cluster App Dispatcher addresses the challenges.
Presenters:
Diana Arroyo Software Engineer @IBM Research
Alaa Youssef, Manager, Container Cloud Platform @IBM Research
A
All
right,
everybody
I'd
like
to
thank
all
everyone
who
is
joining
us
today,
so
welcome
to
today's
cnc
webinar,
the
kubernetes
native
to
level
resource
management
for
aiml
workloads.
My
name
is
danielle,
I'm
working
for
red
head
and
technical
marketing
major
as
well
as
I'm
responsible
csf
ambassador
as
well.
So
luckily
I
will
be
moderating
today's
great
webinar,
so
we
would
like
to
welcome
our
presenter
today,
diane
arroyo
and
software
engineer
at
ibm,
research
and
online
chef,
the
manager
container
cloud
platform
at
ibm
research
as
well.
A
So
there
are
a
few
couple
of
things
about
housekeeping
today
so
probably
started
this
webinar,
so
you
are
not
able
to
talk
as
a
attendee,
so
there
is
a
28
box
at
the
bottom
of
your
screen.
So
please
feel
free
to
drop
your
question
in
there
and
we
will
get
as
many
as
we
can
at
the
end
decision
on
official
webinar
of
cncl.
A
So
this
is
this
is
the
subject
to
csu
code.
We
conduct,
so
please
do
not
add
anything
to
the
chat
or
question.
There
will
be
a
violation
of
the
code
of
conduct,
so
please
be
respectful.
All
of
your
fellow
participants
and
prisoners,
please
also
note
the
recording
and
slidestar
will
be
posted
later
today
at
cnc
webinar,
page
www.cnc.ios
webinars.
You
can
find
it
there
later
today
and
I
will
hand
it
over
to
diane
and
ollar
to
kick
off
today's
presentation.
A
A
B
B
Actually,
we
had
a
recent
cncf
blog
post
with
the
same
title
about
a
couple
of
months
ago
and
basically,
in
that
blog
post,
we
try
to
motivate
for
for
the
need
of
this.
Second
level,
resource
manager
that
we
are
going
to
talk
about
today,
so
some
of
the
characteristics
of
these
ai
workloads
that
we
are
targeting
here.
B
So
basically,
you
all
know
that
you
know
it's.
The
use
of
kubernetes
platform
for
running
these
ai
and
machine
learning
workloads
is
is
on
the
rise
and-
and
these
workloads
have
typically
multiple
concurrent
learners
or
executors
right.
For
example,
if
you
have,
if
you're
using
spark,
you
have
spark
executors
if
you're
using
some,
you
know
doing
some
deep
learning,
you
have
deep
learning
learners
right
and
typically,
they
need
to
run
concurrently
in
in
a
distributed
learning
fashion.
B
They
may
have
some.
You
know
specific
hardware
requirements
such
as
using
gpus,
for
example,
and
a
specific
number
of
gpus
per
learner
also
we're
seeing
nowadays
an
increase
in
the
massively
parallel
jobs,
where
there
is
a
big
number
of
short-running
tasks
that
need
to
be
executed,
like
array,
jobs,
for
example,
and
and
and
these
jobs
are
resource
hungry,
meaning,
if,
if,
if
you
give
the
job
all
the
resources
to
run
the
the
the
thousand,
the
thousand
tasks
that
it's
composed
of
it's
willing
to
take
all
those
resources
and
run
the
thousand
tasks
in
parallel.
B
If
you
give
it
less
resources,
it
has
nice.
You
know
many
of
them
have
nice
elastic
features
where,
if
you
give
it
resources
enough,
for
example,
to
run
a
only
100
tasks
at
the
time,
then
it
will
do
100
tasks
at
the
time
until
it
completes
it
will
take
longer
time
to
complete,
but
the
total
amount
of
consumed
resources
is
going
to
be
the
same,
and
why
would
you
want
to
do
that?
Well,
I
mean
there
are
two
reasons.
B
First,
I
mean
the
resources
are
not
infinite,
but
second,
you
may
want
to
regulate
the
flow,
so
you
allow
for
parallel
jobs
to
proceed,
especially
with
the
increase
in
the
use
of
interactive
mode
like,
for
example,
data
scientists
sitting
in
front
of
a
jupiter
notebook
and
running
some
of
those
machine
learning
jobs
and
want
to
see
results
interactively.
So
they
are
not
all
batch
jobs
that
that
you
know
can
be
run
later
and
and
then,
when
done
somebody's
waiting
for
the
result.
B
So
in
this
kind
of
of
of
environment
right,
so
you
have
jobs
and
tasks
right
and
and
then
there
is
the
the
the
advantages
and
disadvantages
of
managing
at
the
job
level
versus
at
the
task
level.
So,
typically,
a
task
maps
to
a
kubernetes
pod
and
the
job
is
composed
of
multiple
of
those
parts
of
tasks.
B
And
now
the
question
becomes
at
which
level?
Should
you
be
specifying
things
like
priorities
or
classes
of
servers?
Like
you
know:
gold,
silver
bronze,
for
example,
or
quotas
right?
This
is
the
maximum
quota
on
the
memory
resource
or
gpu
resource
for
a
particular
task.
Job
user
organization
I
mean
at
which
level
should
you
set
these
things
and
also
at
which
level
do
you
do
queuing
of
of
these
jobs
or
tasks
right
and
then
and
allocate
resources
and
do
preemption
right.
B
So
these
are
all
the
questions
right
that
that
don't
have
you
know
one
answer,
but
but
you
know
we
are
trying
to
argue
here
that
really
the
the
right
level
of
of
management
that
you
need
to
have
is
at
the
job
level,
and
you
need
to
do
a
lot
of
these
functions
at
the
job
level,
not
at
the
individual
task
level.
Obviously,
the
task
will
inherit
from
its
from
his
job
and
think
about
what
happens:
your
scheduler
and
the
different
controllers
in
your
kubernetes
environment
right
and
in
your
hcd.
B
It
basically
filters
out
the
nodes
that
that
don't
fit
that
that
pod
or
or
for
any
other
constraints
shouldn't
be
used
for
that
pod
and
and
then
for
the
remaining
nodes
in
in
the
cluster.
It's
going
to
prioritize
them
and
and
rank
them
according
to
priority
functions
and
get
the
the
top
candidate
to
put
the
pod
on
right,
and
then
it
binds
the
pot.
So
in
this
example,
here
you
have
a
series
of
pods
that
are
arriving
and
and
queued
at
the
scheduler
to
schedule.
B
B
And
now
imagine
now
that
you
have
thousands
of
these
jobs
and
all
those
pods
are
waiting
for
the
scheduler
to
to
schedule
them,
and
you
obviously
have
limited
capacity
in
your
cluster
and
it
will
keep
trying
to
schedule
them
and
failing
and
schedule
them
and
failing
until
until
once
in
a
while,
it's
able
to
let
in
you
know
one
of
the
parts
so
in
this
example.
This
is
what
happened
right.
B
The
pods
got
scheduled
this
way
and
you
ended
with
two
of
the
three
jobs
not
being
totally
placed
in
the
cluster
and
and
now.
If,
if
these
jobs
require
to
be
the
whole
job
to
be
placed
in
order
for
the
job
to
proceed,
to
do
the
learning
or
whatever
it
needs
to
do,
then,
then
you
have
those
two
jobs:
the
the
purple
and
the
red
here.
In
this
example,
some
of
the
pods
occupying
space
that
is
useless
until
the
remaining
parts
can
be
scheduled
and
you
have
partial
deadlocks.
B
So
these
partial
deadlocks,
obviously,
if
you
have
a
gang
scheduler,
can
help
in
solving
them.
But
just
remember
that
when
you
have
thousands
of
pods,
you
could
have
avoided
that
all
together.
If
you
had
a
second
level
controller
that
is
able
to
look
at
the
cluster
from
a
you
know,
high
granularity
level
and
let
in
to
the
scheduler,
only
jobs
that
are
likely
to
be
placed.
B
B
It's
probably
something
you
want
to
do
at
a
you
know,
little
longer
time
scale
and,
as
I
said,
these,
some
of
these
jobs
are
resource-hungry
and
no
matter
how
much
you
scale
the
cluster
they're
going
to
eat
it
up.
So
practically
speaking
at
every
point
in
time
there
is
a
limit
on
the
available
resources,
and
even
if
that
limit
is
not
permanent
and
it's
going
to
change
and
you
want
to
be
efficiently
using
these
available
resources.
B
B
So
you
may
have,
in
addition
to
on-prem
and
public
clusters,
you
may
have
edge
clusters
as
well,
which
increases
the
the
number
of
clusters
you
know
tremendously
and
and
and
in
that
case
right,
the
static
allocation
of
a
particular
job
or
user.
To
so
a
certain
cluster
is
not
the
best
way
to
go
bursting
scenarios.
B
So,
in
addition
to
what
I
mentioned,
there
are
some
desired
capabilities
that
we're
looking
for.
So
I
alluded
to
the
need
of
having
a
some
form
of
cueing
at
the
job
level
before
admitting
the
pods
to
the
scheduler
to
to
do
the
binding
to
specific
nodes,
and
if
you
are
able
to
do
that,
that
cueing
and
dispatching
to
a
specific
cluster
before
it
reaches
a
scheduler
in
that
cluster.
B
Enforcement
of
of
quotas
like
hierarchical
quota
management,
for
example.
If
you
want
to
say
this
is
the
quota
in
terms
of
resources
that
this
particular
organization
or
department
or
user
is
allowed
to
use
in
my
system,
then
you
can
do
you
can
do
that
at
this
at
this
control
point
in
in
an
easy
global
way,
and
you
can
think
about
all
sorts
of
you
know.
Soft
and
hard
quotas
and
and
of
course,
quotas
that
span
multiple
resource
dimensions.
B
So
what
is
our
solution
to
all
the
shortcomings
that
I
mentioned
and
the
desired
capabilities
that
I
also
talked
about?
Is
the
mcat
or
multi-cluster
app
dispatcher,
which
is
an
open
source
project
that
we
are
going
to
tell
you
all
about
it
and
and
how
it
works.
So
I'm
going
to
hand
it
over
to
my
colleague
diana
and
she
will
take
it
from
here.
Tell
you
a
little
bit
about
how
it
works
and
then
show
you
a
a
a
cool
demo,
so
diana
you
want
to
share.
Or
do
you
want
me
to
flip
the
charts.
B
C
C
Yes,
okay,
great,
okay,
thank
y'all,
so,
as
as
all
I
mentioned,
we've
developed
a
controller
that
tries
to
address
some
of
these
capabilities
and
we
have
an
initial
framework
of
one
and
I'll
talk
to
you
a
bit
about
what
it
does
now
and
then
some
things
we're
looking
at
adding
and
adding
in
the
road
map.
But
essentially,
as
I
mentioned,
we
would
like
to
have
a
way
to
dispatch
to
either
within
one
cluster
or
multiple
clusters,
an
ability
for
cueing.
C
C
So,
as
I
mentioned,
there's
the
mcat
controller,
the
multi-cluster
app
dispatcher,
and
with
that
we
operate
on
the
custom
resource
definition
called
app
wrapper.
The
app
wrapper.
What
it
does
is,
or
what
it
is,
is
essentially
any
and
all
of
your
kubernetes
resources
that
you
create
the
ones
that
are
compute
consuming
resources,
for
instance,
deployments
pods
staple
sets
any
of
the
resources
that
you
define
as
part
of
your
complete
job.
C
Obviously,
if
they're,
not
runnable,
we
put
it
in
the
queue
to
and
re-evaluate
over
time
to
make
sure
that
we
can
dispatch
it
once
resources
become
available,
and
then
another
thing
is
all
I
mentioned
earlier
was:
is
that
we
support
preemption
and
re-queuing,
and
with
that
aspect
we
actually
allow
app
wrappers
to
have
dispatching
priorities.
You
can
define
those
and
I'll
show
how
that
actually
works
in
the
yaml,
as
well
as
a
demo
that
that
we
have.
C
C
It
actually
determines
this
available
capacity
of
the
cluster,
so
it
finds
out
what's
actually
running
what
how
much
resources
are
available
from
by
using
the
normal
kubernetes
model
and
it
tracks
what's
available
over
time,
and
in
addition
to
that,
as
you
bring
in
app
wrappers
into
the
system,
you
submit
them,
the
app
wrappers
will
go
into
a
queue
again.
It's
all
wrapped
in
the
app
wrapper
wraps
all
the
resources
that
you're
trying
to
realize
into
that
cluster,
and
so
we
put
them
into
a
queue
and
this
queue
can
be
fifo.
C
If
you
don't
set
priorities,
that's
how
it
behaves.
If
you
do
set
priorities
on
these
app
wrappers,
then
the
queue
will
be.
The
queue
will
recognize
that
and
recognize
the
put
the
ones
of
the
higher
priorities
at
the
beginning,
and
then
the
goal
here
for
this
mk
controller
is
to
evaluate
taking
the
available
capacity
looking
at
what's
in
the
queue
pulling
jobs
off
the
queue
determining
their
run,
ability,
as
I
mentioned
before,
and
if
they're
runnable
dispatch
them
and
actually
create
the
objects
inside
kubernetes.
C
Here
I'm
showing
just
the
modifications
that
we
have
made
so
that
we
can
support
multiple
clusters
in
this
picture.
Here
you
see
these
big
blue
boxes
here
and
they're
actually
separate
complete
clusters.
So
this
is
a
cl.
These
two
clusters
on
the
right
are
what
we
call
excuse
me
agent
clusters
and
when
you
dispatch
mcad
into
that
cluster,
you
dispatch
it
in
the
configuration
of
an
agent,
and
then
we
have
a
another
cluster
that
acts
as
the
dispatcher
cluster.
C
I
mean
this
is
where
jobs
get
submitted
right
and
what
happens
is
that
the
agent
custer's
agent
controller
collects
the
available
state
or
the
available
capacity
for
the
individual
cluster
and
makes
it
available
to
the
dispatcher
cluster.
The
dispatcher
cluster
or
the
dispatcher
controller
here
keeps
track
of
the
available
resources
on
the
different
cluster.
C
Here's
an
example
of
I
thought
it
would
be
useful
to
have
like
a
use
case,
so
you
can
kind
of
see
how
this
works.
Let
me
see
if
I
can
minimize
this
get
this
out
of
the
way,
so
this
is
kind
of
covering
it
up.
So
I
apologize,
but
this
is
an
app
wrapper
here,
but
before
we
actually
create
the
app
wrapper,
as
I
mentioned
before,
you
bring
these
mcad
controllers
up
on
the
various
system.
C
So
this
is
an
example.
Here
I
have
we
had.
We
worked
with
a
team
who
had
a
bunch
of
resources
that
they
were
trying
to
create
that
represented
one
job,
and
this
is
actually
really
a
subset
of
them
that
I
put
in
here,
but
in
their
job
that
they
submitted.
They
had
multiple
services.
They
had
a
name
space,
they
were
creating
a
network
policy
at
pvc.
C
C
So
that's
what
we
have
as
of
today
and
I
wanted
to
give
you
guys
some
insight
into
kind
of
the
things
we're
working
on
now
as
part
of
our
road
map.
So
this
is
the
current
work
that's
happening
now.
This
is
here
the
box
that
you
just
saw
in
the
previous,
where
we
had
the
mk
controller,
the
queue
and
the
available
capacity
where
object
gets
submitted.
C
So
this
is
the
work
we're
doing
now.
Just
wanted
to
give
you
some
insight
about
how
we
would
be
able
to
take
advantage
of
some
quota
management
evaluation
where,
yes,
we
would
pull
it
into
the
controller
to
determine
available
resources,
but
we
would
also
evaluate
whether
that
job
is
runnable
based
on
quota
management,
in
the
sense
that
you
know
we
may
have
enough
resources,
but
there
may
be
hard
limits
that
you
want
to
restrict
on
some
of
these
jobs,
so
we
would
cue
it
up
and
then
dispatch
it
once
there's
enough
quota,
that's
available.
C
They
are
listed
under
the
item,
so
you
just
have
a
full
list
of
things
that
you
add
here
and
then,
as
I
mentioned
before,
you
can
set
priorities
on
these
jobs
as
well,
right
and
we'll.
This
is
a
dispatching
priority,
so
they're
determining
whether
this
job
can
dispatch
and
giving
higher
priorities
over
other
jobs
and
then,
finally,
just
some
insight
on
where
we
might
be
able
to
express
a
quota
information
right.
C
Okay,
so
I
thought
it
would
be
also
helpful
to
just
get
some
insight
on
how
this
works
with
showing
a
demo.
So
let
me
jump
over
to
the
demo
I
have
here-
and
I
recorded
this
so
I'll
talk
through
this
demo.
It's
a
very
simple
demo,
but
I
wanted
to
kind
of
show
you
guys
how
this
works
initially
and
I'll
show
you
examples
of
cueing
and
then
this
preemption
with
priority.
C
C
It's
the
dispatcher
controller
and
it's
on
a
separate
clusters.
The
green
and
the
blue
box
are
also
separate
clusters
as
well.
I've
made
them
very,
very
tiny,
because
I
wanted
to
be
able
to
fill
up
the
cluster
and
show
you
the
queuing,
so
each
one
of
these
clusters
only
has
one
node
in
there
and
every
node,
I
think,
has
about
eight
cpus
and
available
right
off.
C
C
It's
a
staple
set
and
again
you
would
put
as
many
items
as
you
need
to
represent
your
job
and
then,
of
course,
since
we're
evaluating
whether
it's
runnable
or
not,
here's
the
cpu
that
we're
actually
allocating
and
in
this
example,
we're
having
three
replicas.
So
it's
a
three
replicas
with
that
cpu
and
memory
limit.
C
C
C
C
So
we're
going
to
create
that
again,
it's
the
same
size,
but
since
this
has
a
higher
priority,
we
evaluate
what's
actually
already
running
and
determined
that
everything
else
is
lower
priority.
So
we
will
preempt
a
lower
priority
job
and
actually
put
it
back
in
the
queue
and
allow
the
higher
priority
job
to
be
able
to
get
dispatch.
C
And
this
is
you
want
to
use
this
kind
of
environment.
When
you
have
jobs
that
you
know
takes
time
stamps
it
takes
takes
tracks
the
the
movement
of
the
job,
where
you
can
restart
it
very
easily,
so
that
job
got
preempted.
I
think
it
was
job
number
two
got
preempted
and
then,
of
course,
when
resources
become
free
available
again,
which
I'll
show
here
next
I'll
free
up
job
number
three
and
then
job
number
two
will
get
redispatched
and
again.
C
And
I
think
I'm
going
to
stop
that
now,
because
we're
running
close
to
being
done
here.
I
want
to
give
time
for
questions
as
you
can
see
that
it's
we
got
re-dispatched
so
I'll.
Stop
this
recording
and
I
think
that's
it.
The
last
chart
I
think
I
had
was
really
just
a
call
to
action
where
you
know
feel
free
to
give
this
a
try
out.
That'd
be
great
again.
Here's
the
links
and
the
name
on
the
operator
hub,
give
your
feedback,
we'd,
love
it
and
even
be
even
great.
A
A
A
C
C
Example,
yamo.
So
when
you
submit
your
app
wrapper,
if
you
want
to
take
advantage
of
the
dispatching
priorities,
you
would
just
add
this
stanza
and
set
a
priority
that
you
want
that
job
to
get
assigned
to.
So
this
is
all
configurable
at
the
submission
level
right.
So
whenever
you
submit
a
job,
you
set
the
priority
that
you
would
like
to
have
for
the
job
to
have.
C
A
C
C
One
of
the
things
we're
also
doing
to
address
all
kinds
of
complexities
is
which
is
starvation
right,
so
we
initially
start
out
with
this
user-defined
dispatching
priority,
but
you
may,
as
a
cluster
administrator,
be
able
to
handle
starvation
when
there's
a
low
priority
job,
but
you
still
want
to
get
a
few
of
them
in
so
we're
currently
developing
a
system
priority
that
will
consider
the
user
defined
priority,
but
over
time
those
priorities
will
change,
and
so
we
have.
We
have
that
part
of
our
program.
Mecca
roadmap
as
well.
A
B
I
can
take
that
one
so
in
terms
of
what
we've
shown
today,
we
are
doing
resource
management
for
ai
and
ml
workloads.
So
it's
the
the
opposite
of
of
what
the
question
is
about,
but
of
course,
the
use
of
of
ai
and
machine
learning
in
resource
management
right
to
in
the
resource
management
function
and
its
benefit.
B
There
is
obvious,
and-
and
we
have
other
orthogonal
work,
for
example,
focusing
on
using
ai
in
in
deciding
on
the
scaling
of
elastic
applications
vertical
or
horizontal
scaling,
the
use
of
reinforcement,
learning
to
work
around
failures
and
avoid
failures
so,
but
but
different
orthogonal
pieces
of
work.
The
the
mcat
controller
is,
is
not
really
using
ai
to
to
control
resource
management
for
for
the
jobs.
A
A
All
right
thanks
diana
and
aura,
for
the
great
presentation
and
acquaintance
for
reputation
once
again
and
all
right,
so
so,
and
everybody
thanks
for
joining
us
today
again
and
the
webinar
recording
and
slice
that
will
be
online
later
today,
and
we
are
looking
forward
to
seeing
you
at
the
future
cnc
webinar,
as
well
as
kubecon
and
clown
navy
called
the
next
month
in
november.
Please
register
and
we
will
meet
soon
again
and
have
a
good
rest
of
day.
Thank
you.