►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
KubeFlux: An HPC Scheduler Plugin for Kubernetes - Claudia Misale, IBM T.J. Watson Research Center & Daniel Milroy, Lawrence Livermore National Laboratory
Adoption of cloud technologies by high performance computing (HPC) is accelerating, and HPC users want their applications to perform well everywhere. While container orchestration frameworks provide advantages like resiliency, elasticity, and declarative management, they are not designed to enable application performance to the same degree as HPC workload managers and schedulers. In response to increased interest in scheduling flexibility, the Kubernetes community developed the Scheduling Framework to facilitate integration of new policies and schedulers. We present KubeFlux, a Scheduling Framework plugin based on the Fluxion open-source HPC scheduler developed at the Lawrence Livermore National Laboratory. We discuss uses for KubeFlux and compare the performance of an application scheduled by the Kubernetes default scheduler and KubeFlux. KubeFlux is an example of the rich capability that can be added to Kubernetes and paves the way to democratization of the cloud for HPC workloads.
A
A
My
name
is
daniel
milroy
and
I'm
a
computer
scientist
at
the
center
for
applied
scientific
computing
at
the
lawrence
livermore
national
laboratory
in
california.
In
the
united
states.
I'm
going
to
be
the
first
speaker
and
we'll
be
introducing
some
of
the
background
and
then
hand
off
to
claudia
misali,
who
is
a
re
research
staff
member
at
the
ibm,
tj
watson,
research
center.
We
are
representing
a
larger
collaboration
between
three
organizations
which
are
again
ibm:
tj
watson,
research,
center,
the
lawrence,
livermore,
national
laboratory
and
red
hat.
A
A
Computing
achieves
high
performance
through
three
primary
areas,
which
are
efficiency,
proximity
and
shape.
Now,
hpc
applications
themselves
strive
for
high
parallel
efficiency,
which
we
also
call
scalability,
and
this
basically
means
that,
as
you
may
add,
more
processors
to
an
application.
The
application
performance
improves
more
and
more
now
many
high
performance
computing
applications
have
problems
that
are
subdivided
and
distributed
across
all
of
the
different
processors
in
an
application,
and
then
those
process
processors
cooperate
and
communicate
with
each
other
in
a
pattern
where
they
exchange
subdomain
boundary
data
or
data
within
with
their
neighbors.
A
A
A
Now
cloud
is
really
becoming
a
dominant
market
force
which
is
influencing
high
performance
computing,
and
a
lot
of
this
is
simply
a
question
of
economics
and
community
use.
The
public
cloud
revenue
forecast
is
projected
to
achieve
500
billion
dollars
by
2022
and
up
to
600
billion
dollars
by
the
end
of
next
year.
You
can
contrast
that
with
hpc
spending,
which
should
achieve
approximately
32
billion
dollars
by
the
end
of
this
year
now,
while
more
money
is
certainly
going
into
both
areas
proportionally
a
lot
more
goes
into
cloud.
A
A
A
There
is
the
rapid
covet
19
small
molecule
drug
design,
workflow,
which
is
a
2020
acm,
gordon
bell,
special
prize
finalist,
it's
actually
being
adapted
to
make
use
of
kubernetes
as
well.
We
have
the
new
autonomous
multi-scale
project
at
the
laboratory,
which
is
going
to
take
advantage
of
a
lot
of
the
elasticity
and
automation
that
kubernetes
offers.
A
Despite
all
this
current
interest,
the
2020
laboratory
application
survey
determined
that
fewer
than
10
percent
of
applications
are
currently
using
cloud,
but
there's
a
huge
amount
of
latent
interest.
73
percent
may
adopt
cloud
in
the
future
now
as
a
team,
we're
interested
in
converge
computing,
which
is
really
the
best
of
both
cloud
and
hpc
together
within
the
same
environment
or
system
now
high
performance,
computing,
schedulers
or
the
component.
A
A
The
resource
expression
in
kubernetes
is
limited
in
terms
of
what
hpc
needs,
as
I
mentioned
before,
high
performance
computing
applications
like
to
know
every
single
detail
of
the
hardware
itself
and
its
topology,
the
node,
the
node
feature.
Discovery
project
aims
to
provide
that
level
of
detail
to
the
application,
but
that
is
in
addition
to
kubernetes
rather
than
a
core
component.
A
There
are
significant
trends
toward
complex
workflows,
extreme
resource
heterogeneity
and
converged
computing,
which
are
rendering
traditional
workload
managers
increasingly
ineffective.
There
are
five
principal
challenges.
The
first
is
the
co-scheduling
challenge,
and
this
is
where
complex
workflows
require
component
coupling
and
co-scheduling
of
different
types
of
hardware
such
as
cpus
and
gpus.
A
The
second
is
the
throughput
challenge
where
uncertainty,
quantification
and
other
ensemble-based
techniques
can
submit
tens
of
thousands
of
short-running
jobs
which
overwhelm
traditional
hpc
resource
and
job
management.
Software,
third,
is
the
job
communication
and
coordination
challenge
in
this
case
workflows
depend
on
data
transfer
between
various
components
within
a
framework.
A
Finally,
we're
starting
to
see
extreme
resource,
heterogeneity
and
cloud
integration
which
stretches
or
exceeds
the
resource
model,
capabilities
of
traditional
resource
managers
and
schedulers
flux
or
the
flux
framework
solves
all
five
of
these
key
technical
problems.
Flux
is
an
open
source
project
in
active
development.
A
A
In
this
sense
they
are
monogenous
or
derived
from
a
common
simplistic
node-centric
route.
However,
flux
or
the
flexion
scheduler
uses
a
directed
graph
of
resources,
which
consists
of
a
set
of
vertices,
which
are
the
resources
themselves
and
edges
which
define
relationships
between
these
resources
in
turn,
elevating
resource
relationships
to
an
equal
footing
with
the
resources
themselves.
A
A
Now
flexion
itself
offers
a
rich
api
for
performing
graph,
traversal
and
resource
allocation
specification
and
fulfillment
through
matching
as
an
example,
this
the
flux
framework
solves
the
critical
schedule,
need
for
el
capitan,
rabbit
multi-tier
storage.
In
order
to
schedule
ssds
in
each
rack,
you
can
actually
select
different
components
of
the
graph
itself,
so
you
can
mount
these
components
or
ssds
as
node
local
storage
to
compute
in
the
same
rack,
and
then
that's
used
to
build
an
ephemeral
per
job
luster
parallel
file
system.
A
B
In
this
second
part
of
the
presentation
I
will
discuss
what
are
the
requirements
needed
to
run
a
traditional
hpc
workload
on
kubernetes,
then
cook,
flux,
internals
and
some
preliminary
performance
results
to
run
an
hpc
or
ai
workload.
We
need
to
fulfill
the
following
requirements
that
are
missing
in
the
kubernetes
default
scheduler.
B
We
need
to
first
have
batch
scheduling
and
schedule.
The
application
as
a
whole
be
able
to
express
hardware
topology
and
use
that
to
do
a
better
placement,
allow
application
to
use
resources
exclusively
or
not
and
finally
choose
among
different
placement,
algorithms
that
better
suit
the
application
requirements.
B
Now,
let's
go
over
all
of
these
points
in
detail
and
we'll
see
what
we
can
do
with
upstream
kubernetes
and
what
we
can
enable
with
codeplugs
with
batch
or
group
scheduling.
We
mean
that
an
application
is
composed
of
different
replicas
that
need
to
be
scheduled
as
a
single
entity,
because
the
application
will
not
work
if
any
of
the
replicas
stays
pending
to
solve
this.
B
In
upstream
kubernetes
we
can
use
the
pod
group
crt
and
the
co-scheduling
plugin
scheduler
published
in
the
official
scheduler
plugins
repository
coolflux
reuses,
the
pod
group
crd
to
label
pods
and
gather
them
to
create
the
batch.
Then
it
creates
a
request
to
flagstone
to
schedule
it
in
an
all
or
nothing
fashion.
B
Since
fluxing
is
a
batch
scheduler.
This
was
very
easy
to
enable,
and
for
instance,
if
we
want
to
schedule
an
npi
job
with
four
replicas,
we
get
an
allocation
if
and
only
if
there
is
enough
capacity
to
start
all
of
the
four
replicas
to
enable
topology
awareness
and
scheduling
upstream
kubernetes
provides
the
non-resource
topology
crd
and
the
topology
aware
scheduler
plugin
published
in
the
official
scheduler
plugins
repository
as
well.
B
This
plugin
limits
the
topology
at
node
level
and
defines
the
normal
topology
another
way
to
define
a
topology
is
manually
via
labels
and
then
using
affinity
and
diabetes
and
tolerations
extensively
in
the
pod
spec.
But
this
solution
is
not
scalable
cool
flux,
resource
model
is
graph
based
and
the
topology
is
at
cluster
level,
and
the
topology
can
also
be
extended
with
software
information.
B
Multi-Tenancy
is
a
core
component
in
kubernetes,
but
hpc
applications
are
not
friendly
to
noisy
neighbors
and
would
rather
run
undisturbed.
In
upstream
kubernetes
we
can
use
labels,
affinity,
etc.
But
again
this
is
not
a
scalable
solution.
You
can
also
use
trimaran.
That
is
an
upstream
plugin
scheduler
for
a
lot
of
work.
Placement
trimaran
learns
from
running
application
and
prometheus
metrics,
therefore,
is
not
able
to
solve
the
problem
at
the
source.
B
B
B
Alternatively,
you
can
force
placement
decisions
by
using
the
use
of
label,
but
you
won't
be
able
to
simulate
pack
algorithm,
for
instance,
also.
It
is
not
always
possible
to
spread
pods
using
entire
affinity.
Quickflux
already
implements
different
flavors
and
packing
spread,
which
are
useful
to
schedule
workloads
with
very
different
requirements.
For
instance,
network
intensive
workloads
versus
compute
intensive
points,
also
high
or
low
level
constraint,
can
be
specified
at
the
job
level
in
the
yaml
file,
which
can
also
produce
different
placement
results,
and
it
makes
it
very,
very
flexible.
B
The
coupe
schedule
first
goes
through
the
interface
where
it
filters
out
the
node
that
cannot
cause
the
particular
pod
and
then
the
nodes
that
may
host
that
pods
are
scored
into
the
score
face
here.
The
nodes
are
given
a
score
based
on
different
requirements
and
the
node
with
the
higher
score
will
host
the
bond.
B
The
green
and
yellow
labels
in
the
figure
on
the
left
are
called
the
extension
point
and
those
can
be
customized
with
the
preferred
logic.
Since
the
schedule
framework
is
used
to
extend
the
default
scheduler,
it
exposes
the
same
functions.
The
default
scheduler
implements
in
coolflux.
We
implement
the
filtering
filter
extension
points
avoid
completely
the
scoring
part,
and
then
we
let
kubernetes
do
the
not
reservation
fault
allocation
and
its
execution.
B
B
B
B
B
B
B
Now
there
is
still
a
significant
gap
to
be
closed
between
kubernetes
and
hpc
before
we
have
fully
converse
computing
on
the
top
branch
of
the
diagram.
Hpc
needs
more
cloud
capabilities
to
allow
it
to
integrate
the
bottom
branch
moving
from
slurm
to
lsf
and
then
to
flux.
There
is
increasing
cloud
readiness
on
the
cloud
branch
you
have
progress
in
terms
of
increased
scheduling,
sophistication
and
performance
and
efficiency
moving
from
basic
kubernetes
to
third-party
application.
B
Plugins
such
as
volcano,
which
integrates
batch
capabilities
in
kubernetes
the
kuflux
plug-in
for
kubernetes,
is
an
attempt
in
bringing
these
two
branches
together
into
a
unified,
converged
computing
environment
and
we're
working
towards
the
future,
where
environments
exhibit
the
elasticity
and
automation
of
cloud
and
the
performance
and
efficiency
of
hpc.
That
is
the
best
of
both
worlds.