►
From YouTube: Lightning Talk: Fluence: Approaching a Converged Computing Environm...Daniel Milroy & Claudia Misale
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Fluence: Approaching a Converged Computing Environment - Daniel Milroy, Lawrence Livermore National Laboratory & Claudia Misale, IBM T.J. Watson Research Center
Adoption of cloud technologies by high performance computing (HPC) is accelerating, and HPC users want their applications to perform well everywhere. While container orchestration provides resiliency, elasticity, and declarative management, it is not designed to enable app performance like HPC schedulers. In particular, Kube-scheduler is not suited to scheduling emerging HPC workflows that require pods placed advantageously. In response to interest in scheduling flexibility, the K8s community developed the Scheduling Framework to integrate new policies and schedulers. KubeFlux, a Scheduling Framework plugin based on the Fluxion open-source HPC scheduler, provides HPC scheduling capability in K8s. We detail our improvements to the MPI Operator and demonstrate its scalability to 16,384 ranks. With the improved operator we compare the performance of HPC benchmark apps scheduled by Kube-scheduler and KubeFlux. We conclude that KubeFlux makes pod placements that enable much higher app performance than Kube-scheduler. KubeFlux is an example of the rich capability that can be added to K8s and paves the way to converged computing environments with the best capabilities of HPC and cloud.
A
All
right,
thank
you.
My
name
is
Daniel
Milroy
and
I'm,
a
computer
scientist
at
the
Center
for
Applied
scientific
Computing
at
the
Lawrence
Livermore
National,
Laboratory
and
I'm,
going
to
be
speaking
on
behalf
of
my
collaborators
at
the
laboratory
and
also
Claudia
masale,
who
could
not
be
here
today
from
the
IBM
TJ
Watson,
Research
Center,
so
I'll
be
talking
about
fluence
approaching
a
converged
Computing
environment.
A
So
to
give
everyone
a
little
bit
of
a
background
on
what
I'm
talking
about
here.
I
want
to
give
an
example
of
a
pre-exist
scale,
scientific
workflow
that
really
strained
the
capabilities
of
traditional
HPC
resource
managers
and
schedulers.
So
this
is
the
multi-scale
machine
learned,
modeling
infrastructure,
also
known
as
mumi,
which
won
the
sc19
best
paper
award,
and
this
features
message
passing
interface
or
MPI
based
simulation
it
coupled
with
in-situ
analysis
and
machine
learning
as
well.
A
So
there
are
three
primary
areas
that
really
text
or
strain
the
capabilities
of
resource
managers
and
schedules,
and
the
first
is
the
co-scheduling
challenge.
So
the
coarse
grain
component
of
this
particular
workflow
and
I'm,
just
kind
of
going
to
over,
provide
a
brief
overview
view
of
this,
because
it's
very
complex
required
the
analysis
be
bound
to
cores
that
were
nearest
PCI
Express
buses
on
node,
because
those
particular
cores
needed
to
communicate
and
exchange
data
data
with
the
gpus
on
the
nodes.
So
we
need
to
exploit
very
fine-grained
node
local
topology
information.
A
The
second
was
the
jobs,
communication
and
coordination
challenge,
so
this
particular
workflow
consisted
of
approximately
36
000
individual
concurrent
tasks
using
176
000
CPU
cores
and
over
16
000
gpus
on
the
Lawrence
Livermore
National
Laboratory
Sierra
pre-exascale
cluster.
So
the
coordination
between
a
lot
of
these
tasks
was
very,
very
difficult,
using
traditional
workload,
managers
and
schedulers.
Finally,
the
portability
challenge.
So
given
all
this
coordination
and
given
the
difficulty
of
actually
adapting
this
workflow
to
a
particular
resource
manager
and
schedule,
reporting
that
between
different
resource
managers
and
schedulers
is
tremendously
difficult.
A
So
in
terms
of
Next
Generation,
what
we're
going
to
from
here
is
we're
starting
to
see
cross-cluster
scientific
workflows
which
are
demanding
portability
as
well
as
integration
with
the
cloud.
So
this
diagram
is
an
architectural
diagram
of
the
American
Heart
Association
molecular
screening
workflow,
which
was
the
e-science
22
best
paper,
and
this
combines
more
traditional
MPI
based
simulation
analysis
with
machine
learning
and
now
containerized
components
as
well.
A
So
when
I
mentioned
that
this
is
a
cross
cluster
workflow,
it
features
one
component
that
runs
on
a
dedicated,
CPU
cluster,
one
component
that
runs
on
a
CPU
and
GPU
cluster
and
then
a
kubernetes
cluster
that
provides
a
message
queuing
glue
between
these
two
clusters,
so
we're
seeing
on
this
particular
pattern
at
the
laboratory
and
Beyond.
There
are
other
examples
at
the
laboratory,
such
as
traditional
combining
traditional
HPC
simulations
with
AI
and
machine
learned
surrogate
models,
as
well
as
potentially
orchestrated
databases.
A
So
flux
can
actually
instantiate
itself
inside
of
itself
essentially
add
infinitum,
and
this
allows
for
specialization
each
at
each
level.
So
at
each
level,
in
a
nested
flux
hierarchy,
you
can
change
the
scheduling,
algorithms
and
use
divide
and
conquer
approaches.
The
second
point
is
that
flux
can
actually
manage
resources,
basically
anywhere
from
bare
metal
resources
to
Virtual
machines
in
the
cloud
Etc.
Another
huge
Advantage
is
that
workflows
really
only
need
to
program
to
flux.
I
mentioned
the
portability
challenge
before
well,
flux
can
actually
instantiate
itself
underneath
external
resource
managers
like
slurm
and
lsf.
A
Finally,
you
see
the
direct
graph
diagram
at
the
bottom.
Well,
the
directed
graphs
are
used
as
a
resource
model
to
express
complex
or
potentially
complex
and
changing
resources.
Finally,
flux.
The
flux
framework
provides
rich
and
well-defined
interfaces.
This
facilitates
communication
and
coordination
between
different
tasks
in
a
workflow
flux
framework,
provides
CLI,
python,
C,
C,
plus
plus
bindings
rust,
and
we're
working
on
go
bindings
as
well.
A
So,
given
the
emphasis
and
MPI
and
traditional
HPC
work
workloads,
we
want
to
actually
declaratively
create
kind
of
an
HPC
Cloud
slice,
and
this
is
really
complicated
and
requires
a
couple
of
things.
The
first
is
scalable
MPI,
bootstrapping
and
also
fine
or
high
quality,
pod
placement
and
I'll
talk
about
what
that
means
in
just
a
minute.
But
basically
the
idea
is
that
you
have
more
complexity
than
you
do
in
a
kind
of
bare
bones
or
bare
metal,
HPC
environments.
You
have
mapping
of
MPI
ranks
to
pods
mapping
of
PODS
to
underlying
physical
infrastructure.
A
So
the
first
thing
that
we
did
to
now
enable
this.
This
this
capability
was
to
allow
declarative,
MPI
to
scale
three
orders
of
magnitude
higher
than
it
did
in
our
tests.
So
we
use
the
kubeflow
MPI
operator,
which
initially,
in
our
tests,
only
scaled
to
about
80
ranks,
so
we
identified
and
fixed
some
race
conditions
in
the
process
or
the
flow
diagram
to
to
the
left,
one
of
which
was
that
the
launcher
didn't
wait
for
workers
to
be
ready.
A
A
So
the
second
component
of
this
is
fluence,
which
used
to
be
known
as
kubeflux,
which
brings
HPC
grade
scheduling
and
improved
performance
to
kubernetes,
It's
featured
as
part
of
the
kubernetes
plug-in
schedule
framework.
So
fluent
supports
CPU
memory
and
GPU
resource
requests.
It
allows
for
Zone
awareness
via
locality
information,
that's
actually
embedded
into
the
function
or
scheduler
component
resource
graph.
We
tested
this
using
Department
of
energy,
Coral
2
exascale
pre-exascale
benchmarks
such
as
AMG
lamps
and
qmc
pack.
A
We
found
that
optimized
AMG
ran
well
in
IBM
cloud
and
AWS,
and
we've
got
lamps,
which
is
latency
sensitive
to
strong
scale
in
eks
up
to
3008
ranks,
so
fluence
actually
accelerates
Coral
two
benchmarks
in
openshift
on
IBM
Cloud.
So
this
is
a
rocks
cluster
I'm,
going
to
provide
a
little
bit
of
overview
of
these
performance
results.
You
can
see.
Qmc
pack
is
the
rightmost
plot
and
lamps
or
sorry
the
leftmost
plot
lamps
is
the
rightmost
plot.
These
sets
of
box
plots
in
these
sets
of
box
plots.
The
blue
indicates
default.
A
Cube,
scheduler
and
orange
is
fluent,
so
you
notice
right
away
that,
depending
on
the
rank
size,
the
variability
is
much
higher
with
a
default
scheduler
than
fluent
and
also
the
run
times
are
higher,
so
Coupe
schedule.
Part
of
the
reason
is
that
Coupe
schedule
is
unable
to
pack
to
end
single
nodes,
so
even
when
limiting
or
or
modifying
affinity
and
anti-affinity
Coupe
scheduler
was
not
able
to
able
to
perform
as
desired.
A
We
also
compared
multi-app
simulated
workflow
performance,
which
was
scheduled
by
fluence
versus
Coupe
scheduler.
So
we
simulated
workflows
by
taking
the
three
Coral
two
benchmarks
and
running
them
simultaneously
on
within
a
single
availability
zone
of
an
IBM
eks
cluster,
and
this
heat
map
represents
pod
placement.
So
at
the
top
we
have
Coop
scheduler.
Those
are
the
top
40
jobs
and
the
bottom
40
jobs
are
actually
fluent
so
along.
A
The
horizontal
axis
are
the
unique
node,
IDs
and
again
vertical
accesses
job
IDs,
and
the
color
indicates
the
number
of
PODS
that
were
actually
mapped
to
each
individual
node.
When
we
saw
what
a
pathological
mapping
Behavior
due
to
the
random
tie
breaking
of
the
coop
scheduler,
which
resulted
resulted
in
artificial
resource
starvation
and
delayed
execution
for
some
of
the
constituents
of
the
workflow.
So
you
can
see
this
represented
in
two
buck
plots
to
to
my
right.
A
The
first
is
AMG,
where
you
can
see
that
that
is
heavily
skewed
distribution
for
the
default
scheduler,
which
you
can't
really
see
the
color
there,
but
you
can
take
my
word
for
it
that
the
one
that
is
skewed
is
Coupe,
scheduler
and
and
qmc
pack,
you
see
similar
Behavior.
So
in
general
we
saw
the
fluent
scheduled
workflows
run
up
to
three
times
faster.
You
can
take
a
look
at
this.
This
will
be
published
in
the
Canon
P22
workshop
at
sc22.
A
So
finally,
I
want
to
talk
really
briefly
about
our
latest
work,
which
is
actually
to
replace
the
MPI
operator
using
the
flux
operator.
This
is
going
to
enable
both
HPC
grade
pod
scheduling
and
also
hierarchical,
Management
in
kubernetes.
The
idea
behind
this
is
to
bootstrap
the
flux
tree
base
overlay
Network
across
pods
such
that
complex
scientific
workflows
can
run
and
make
use
of
the
hierarchical
resource,
subdivision
and
scheduling
capabilities
that
they
use
on
bare
metal.
A
C
Oh,
that's
really
interesting!
Thank
you
for
that
I
guess.
One
thing
I'm
wondering
about
is
this
is
mostly
Outsider
looking
in,
but
I
have
have
there
been
talks
in
the
doe
about
like
converging
on
these
different
like
science,
like
scientific
workflow
of
our
presentations
like
I
know,
it
seems
like
every
every
lab
kind
of
has
their
own
and
I
know
the
bioinformatics
community
has
a
bunch
of
their
own
too
and
I
guess
I'm
just
curious
if
I'm
I
just
want
to
know
your
thoughts
on
that.
So.
A
You're
talking
about
kind
of
standardizing,
workflow
components
or
standardizing
workable
architectures
yeah
among
the
doe
yeah,
there
are
there's
an
EXO
Works
project.
That's
part
of
the
exascale
Computing
project,
I'm,
not
part
of
that,
but
I
believe
there
are
several
members
of
the
Lawrence
Livermore
laboratory
that
are
working
on
that
I
believe
Dan.
Laney
is
one
of
them.
So
I'm,
not
an
expert
on
that
particular
portion,
but
I
know
their
efforts
underway
to
do
so.
E
A
F
A
No
not
yet,
but
this
is
something
that
we
want
to
integrate
in
the
future.
So
I
mentioned
that
the
flux
framework
scheduler
is,
is
we
work
on
elasticity
in
it?
So
we
want
to
actually
integrate
that
into
fluence
and
allow
the
allocations
to
change
shape
over
time
so
yeah
that
that
is
definitely
something
we're
working
on.
B
Any
more
questions
I
got
one.
So
with
your
comparison
with
Cube
scheduler
like
it's
clearly,
it
shows
that
Cube
schedule
is
optimized
for
service
type.
Workers
tries
to
spread
Etc.
Did
you
try
at
least
to
tune
it
other
than
using
potent
Affinity
Etc,
but
the
configuration
itself
right
like
changing
the
weights
of
these,
like
you
know,
various
features
that
they
have
inside
yeah.