►
From YouTube: HPC on Openshift Deploying Scientific Workloads on OpenShift with the MPI Operator OpenShift Commons
Description
HPC on Openshift Deploying scientific workloads on Red Hat OpenShift with the MPI Operator
David Gray (Red Hat)
OpenShift Commons Gathering KubeCon NA
November 17 2020
High Performance Computing (HPC) workloads increasingly rely on the use of containers that make applications easier to manage, preserve their dependencies and add portability across different environments. Red Hat OpenShift Container Platform is a Kubernetes-based platform for deploying containerized applications on shared compute resources. In this talk we will show how to effectively deploy scientific applications, GROMACS and SPECFEM3D Globe, on OpenShift using the MPI Operator from the Kubeflow project using two different distributed shared filesystems, Lustre and CephFS.
466756 HPC on Openshift Deploying scientific workloads on Red Hat OpenShift with the MPI Operator
B
Hello,
I'm
kevin
peuge,
I'm
a
senior
software
engineer
in
the
same
team
as
david.
In
this
talk,
we
are
going
to
present
the
toolset
we
use
to
deploy
parallel
scientific
hpc
workload
on
openshift
container
platform,
then
we'll
introduce
the
performance
result.
We
obtain
in
a
proof
of
concept
environment
using
these
tools.
First.
A
Then
we'll
go
over
some
of
the
performance
results
we've
collected
so
far
from
the
scientific
applications,
gromacs
and
specsam
3d
globe
and
some
mpi
micro
benchmarks.
We've
run
as
well
the
first
to
give
some
context.
Why
are
we
testing
high
performance,
scientific
applications
on
openshift,
with
the
emergence
of
big
data
and
machine
learning?
Scientific
computing
is
becoming
a
more
integral
part
of
enterprise.
Computing
organizations
are
running
more
and
more
hpc-like
workloads
to
create
value
for
their
business,
and
this
change
coincides
with
the
rise
of
containers
and
kubernetes
for
deploying
enterprise
applications.
A
A
So
that
they
can
be
run
in
many
different
environments
for
these
containers
to
be
useful,
you
need
a
platform
to
run
them
on
like
kubernetes.
Openshift
is
an
enterprise-ready
kubernetes
platform
for
the
hybrid
cloud,
and
this
talk
aims
to
show
how
hpc
applications
can
be
effectively
deployed
on
openshift.
A
Both
gromacs
and
specfem
are
scientific
workloads
which
support
multi-threading
via
openmp
and
parallel
parallelization
over
multiple
nodes
with
mpi
gromacs
is
a
package
for
running
molecular
dynamics
simulations
with
gromacs
we
benchmarked
with
the
popular
water
dataset,
which
simulates
hundreds
of
thousands
of
water
molecules,
typically
gromacs
with
mpi
scales
well
up
to
about
20
nodes
or
so,
but
beyond
that
high
speed.
Networking
with
technologies
like
infiniband
becomes
a
must
beckfem.
On
the
other
hand,
is
a
geophysics
code
that
simulates
earthquakes
and
seismic
wave
propagation
in
the
earth's
crust.
A
It's
a
reference
benchmark
for
supercomputers,
thanks
to
its
good
scaling
capabilities,
although
it
will
not
be
discussed
much
in
this
lightning
talk.
Both
of
these
workloads
can
be
accelerated
by
gpus
on
openshift.
This
can
be
enabled
using
the
nvidia
gpu
operator,
which
is
based
off
of
the
special
resource
operator,
which
will
be
discussed
later
in
this
talk.
A
A
One
example
in
this
talk
would
be
the
mpi
operator
which
introduces
the
concept
of
an
mpi
job
resource
and
manages
mpi
job
objects
when
they
are
created
by
the
user.
An
operator
works
as
a
software
control
loop
to
set
and
maintain
desired
state
of
certain
kubernetes
objects
for
the
purposes
of
running
parallel.
Multi-Node
hvc
software
on
openshift.
A
Some
of
the
testing
we
have
done
with
gromacs
used
lustre
as
the
shared
file
system
to
store
the
input
data
for
the
gromacs
containers.
For
these
experiments.
We
used
an
openshift
cluster
running
on
amazon
web
services,
because
aws
has
a
lester
file
system
service,
called
amazon
fsx
for
lustre
fsx
for
luster
allows
for
lesser
file
systems
to
be
created
and
linked
to
an
s3
bucket.
A
A
Although
this
csi
driver
is
specifically
written
to
run
on
aws,
a
similar
approach
could
be
used
for
an
on-premise
lester
file
system
instance.
Normally
this
csi
driver
would
require
would
require
every
node
to
have
the
lester
client
kernel
modules
installed
on
the
host,
but
we
are
able
to
deploy
the
kernel
modules
with
driver
containers.
A
Driver
containers
are
being
used
increasingly
in
cloud
native
environments
to
deploy
kernel
modules
or
to
enable
special
hardware.
Drivers.
Driver
containers
are
especially
useful
for
pure
container
operating
systems
like
red
hat
enterprise,
linux,
core
os,
which
is
the
default
os
for
open
shift
nodes.
A
A
The
benchmark
results
we
will
share
at
the
end
of
this
talk
are
from
a
bare
metal
cluster
not
running
on
aws,
and
we
didn't
use
luster
in
this
environment.
But
I
wanted
to
highlight
what
we've
done
to
create
an
sro
recipe
for
luster
to
show
how
lustre
or
a
similar
storage
system
can
be
enabled
in
openshift.
A
A
Running
in
external
mode,
openshift,
container
storage,
the
software
defined
storage
integrated
with
and
optimized
for,
openshift
container
platform,
it
is
built
on
red,
hat,
cef
storage
and
can
be
deployed
on
premise
or
in
the
public
cloud.
In
our
tests
we
ran
ocs
in
external
mode
so
that
we
could
use
a
cfs
file
system
which
was
running
external
to
the
openshift
cluster.
This
way
we're
able
to
use
the
same
file
system
for
bare
metal,
baseline
benchmarks
and
our
benchmarking
on
openshift
to
compare
apples
to
apples.
A
Alternatively,
ocs
can
run
ceph
on
the
openshift
cluster.
I
will
note
here
that
the
performance
of
the
workloads
we
are
running
gromacs
and
specs
have
3d.
Globe
is
not
dependent
on
the
file
system,
performance
and
benchmarking.
The
file
system
itself
was
not
a
part
of
this
effort.
We
aim
to
show
how
lester
can
be
used
on
openshift
because
of
its
popularity
at
hpc
sites.
Additionally,
steph
offers
an
alternative,
scalable,
flexible
and
reliable
storage
solution,
which
is
easy
to
use
on
openshift
thanks
to
ocs.
A
The
message
passing
interface
or
mpi
is
a
specification
for
a
message.
Passing
libraries
and
mpi
libraries
are
the
de
facto
standard
for
writing.
Parallel
message:
passing
programs,
many
hpc
applications,
including
chromax
and
specfm,
are
written
so
that
they
can
be
compiled
to
run
across
multiple
nodes
using
mpi.
A
The
mpi
operator
came
out
of
the
kubeflow
project
and
was
originally
created
for
running
distributed
ml
model
training.
The
operator
defines
the
mpi
job
custom
resource
mpi
jobs,
define
the
mpi
run
command
the
hardware
resources
required
for
the
worker
pods
and
the
number
of
worker
pods
once
an
mpi
job
is
created.
The
operator
creates
the
worker
and
launcher
pods
and
configures
them
to
be
able
to
communicate
using
the
mpi
library
installed
in
the.
A
Containers,
one
of
the
things
we
will
discuss
as
we
go
through
the
performance
results
is
using
maltis
maltis
is
an
open
source
project
that
enables
kubernetes
pods
to
attach
to
multiple
networks.
It
is
a
container
networking
interface
plugin
that
calls
other
cni
plugins
for
running
mpi
applications.
We
use
it
to
bypass
the
software
defined
overlay
network
in
openshift
for
certain
types
of
network
traffic.
The
sdn
can
have
some
performance
overhead
due
to
encapsulation
and
open
flow
rules
in
general.
Throughput
is
not
diminished
much,
but
latency
can
be
affected.
A
Another
way
to
bypass
the
sdn
is
to
give
the
worker
pods
access
to
the
host's
network
namespace.
However,
this
has
security
implications
as
it
could
be
used
to
snoop
on
network
activity
of
other
pods
on
the
same
node.
If
the
pod
were
somehow
compromised
because
of
this
using
multis
is
preferred
and
in
our
tests
gave
us
the
same
performance
of
improvement
as
using
the
host
network.
B
B
The
openshift
cluster
was
composed
of
37
nodes,
three
masters,
two
for
infrastructure
management
like
telemetry,
routing
storage
operators
and
32
worker
nodes,
and
for
getting
the
reference
time.
We
converted
this
32
worker
nodes
to
run
real
8.2
bar
metal
and
each
of
the
nodes
have
two
high-speed
network
cards.
One
used
by
openshift
for
the
cluster
communication
and
one
available
for
our
mpi
communications
on
the
node
have
four
calls.
We
didn't
use
hyper,
threading
and
62
gigs
of
run.
B
B
We
can
see
clearly
that,
when
we
feed
bigger
problems
to
the
cluster,
we
get
back
very
close
to
the
bare
metal
reference
with
less
than
six
percent
of
overhead
or
most
of
the
rents
and
for
problems
big
enough.
This
applies
up
to
32
nodes,
including
the
super
linear
efficiency,
and
I
will
let
david
continue
with
gromac
results.
A
A
The
green
line
here
reflects
the
performance
while
running
the
gromacs
pods
on
worker
nodes,
which
are
also
running
on
some
cluster
infrastructure
pods.
This
includes
pods
such
as
monitoring
the
cluster
ingress
controller
and
logging.
The
rest
of
the
results
we
gathered
are,
with
these
pods
isolated
to
two
specific
infrastructure
nodes
to
prevent
these
pods
from
using
any
cpu
cycles
during
the
benchmark
run,
which
improves
performance
and
reduces
variability.
A
The
graph
on
the
right
shows
all
of
the
openshift
results
compared
with
bare
metal.
As
we
can
see,
the
result
using
the
sdn
for
all
of
the
network
communications
is
the
outlier
here
to
get
good
performance
on
more
than
two
nodes,
bypassing
the
sdn
appears
to
be
crucial
and
using
malthus.
To
do
this
gives
performance
just
about
on
par
with
bare
metal.
A
A
A
Additionally,
isolating
some
of
the
open
shift
infrastructure
pods
to
specific
infrastructure
nodes
can
also
help
to
reduce
overhead.
We
have
more
results
of
running
these
workloads
on
openshift
than
we
were
able
to
fit
into
this
lightning
talk,
but
we
have
a
couple
of
blog
posts
for
the
openshift
blog
on
this
topic.
So
keep
your
eye
out
for
these
on
blog.openshift.com
for
a
deeper
dive.