►
From YouTube: 2020-10-02 - Albert Reuther - Enabling Interactive, On-Demand HPC for Rapid Prototyping and ML
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Abstract: Fordecades, the use of HPC systems was limited to those in the physical scienceswho had mastered their domain in conjunction with a deep understanding of HPCarchitectures and algorithms. During these same decades, consumer computingdevice advances produced laptops, tablets, and smartphones that allow millionsto interactively develop and share code projects using high productivitylanguages and environments. The HPC community faces many challenges associatedwith guiding researchers from disciplines that routinely utilize highproductivity interactive tools to effectively use HPC systems, since it is fruitlessto expect them to give up the interactive, on-demand nature of their workflows.
For over adecade, MIT Lincoln Laboratory has been supporting interactive, on-demand HPCby seamlessly integrating familiar high productivity tools to provide userswith an increased number of design turns, rapid prototyping capability, andfaster time to insight. In this talk, we discuss the lessons learned whilesupporting interactive, on-demand high performance computing from theperspectives of the users and the team supporting the users and the system. Atits core, it involves an expansion of what the HPC ecosystem paradigm entailsincluding expansions in system architecture, scheduler policies, metrics ofsuccess, and supported software development environments and tools. We concludewith how our team supports users and the systems in this paradigm expansion.
A
He
brought
super
computing
to
lincoln
labs
through
the
establishment
of
ll
grid,
which
is
on-demand
grid
computing
with
matlab
he's
a
founder
of
llsc
and
he
leads
their
computational
science
and
engineering
team
he's
also
the
computer
system
architect
of
mit's
super
cloud,
and
for
that
he
won
a
eaton
award
for
design
excellence
from
purdue
university.
Our
shared.
A
His
areas
of
research
include
interactive
hpc,
computer
architectures
for
machine
learning,
graph,
analytics
and
signal
processing
and
computational
engineering,
and
today
he's
going
to
tell
us
about
enabling
interactive,
on-demand
high
performance
computing
for
rapid
prototyping
and
machine
learning
there
at
llsc.
So
albert,
please,
please
go
ahead.
B
B
So
I'm
going
to
start
off
with
in
the
introduction
to
tell
you
a
little
bit
more
about
lincoln
laboratory,
it's
a
common.
I
think
everybody
knows
mit,
but
plenty
of
people
really
don't
know
too
much
about
lincoln
laboratory
and
that
other
part
of
mit
and
then
doing
that.
I
want
to
share
with
you
kind
of
the
the
premise
with
which
we
built
our
interactive
hpc
capability.
B
I'm
then
going
to
share
with
you
four
lessons
learned
three
under
the
three
under
the
interactive
hpc
bullet
and
then
we'll
talk
about
metrics
in
and
of
themselves,
and
then
I
thought
it
would
be
good
to
share
a
couple
of
technologies
that
are
key
to
providing
interactive
capability.
One
is
preemptable
batch
job
executions
and
then
we'll
finish
off
with
jupiter
notebook
portals.
B
So,
let's
talk
a
little
bit
about
lincoln
laboratory.
It
was,
it
came
out
of
mit
campus
out
of
the
radiation
laboratory
or
rad
lab.
The
rad
lab
was
very
active
in
the
30s
and
40s
and
designed
about
half
of
all
u.s
world
war
ii,
radars
for
surveillance,
fire
control,
navigation
systems,
but
at
the
end
of
world
war
ii.
B
B
Magnetic
core
memory
came
out
of
that,
as
well
as
light
pen
crt
interfaces
that
came.
That
was
part
of
that
early
experiment
with
interacting
with
computers
and
reacting
to
bomber
threats
fast
forwarding
now
to
to
2020,
and
the
laboratory
is
a
lot
bigger.
The
original
buildings
are
still
here
on
campus.
B
B
Many
more
employees
we're
up
to
about
4
000
employees,
but
it's
all
in
the
support
of
national
security,
and
we
take
advantage
of
our
ties
with
mit
campus
to
work
both
with
researchers
on
campus
but
actually
also
work
with
researchers
all
over
the
country
in
finding
capabilities
that
can
be
used
for
national
security
to
apply
to
national
security
problems.
Our
key
roles
are
in
system,
architecture,
engineering,
technology,
development
and
system
prototyping,
and
I
I
don't
have
the
update.
Yet
we
actually
added
an
11th
mission
area
in
biosecurity,
bio
sensing
and
security.
B
B
Well,
I
mentioned
the
sage
system,
but
there
is
a
long
history
of
super
computing
at
lincoln.
You
know
over
here
in
1951
I
mentioned
the
whirlwind:
computer
whirlwind
was
that
first
ibm
computer
that
was
integrated
in
what
is
f
building
in
our
complex
that
then
led
to
a
number
of
other
capabilities.
Tx0
and
tx2
came
out
of
that
for
the
transistor
experiment,
computers
tx2
was
actually
then
the
host
for
sketchpad,
which
was
the
first
interactive
system
that
was
fielded
at
lincoln.
B
There's
a
number
of
companies
that
have
spun
out
of
lincoln
among
them
are
digital
equipment
corporation,
as
well
as
to
a
large
degree,
sun,
microsystems
and
actually
ivan
sutherland.
The
one
of
the
founders
of
sun
was
one
of
the
brains
behind
the
sketchpad
and
the
tx2
interactive
computing
capabilities
well
in
ensuing
years
somewhere
somewhere
in
like
the
early
80s
or
so
a
lot
of
the
unclassified
computing
capability,
kind
of
dried
up
and
a
lot
of
the
high-end
computing
capability
that
the
lab
had
when
classified.
B
They
actually
tried
three
times
with
c
plus
c
and
c,
plus
plus
libraries,
including
this
spacetime
adaptive
processing
library
staple
which
was
a
per.
It,
was
a
parallel
native
library
to
enable
sensor
signal
processing
for
radars.
The
trouble
that
they
had
was.
It
was
actually
really
well
received
among
expert
programmers,
but
it
was
not
particularly
accessible
and
it
wasn't
until
the
early
aughts,
when
jeremy
kepner
nadia
bliss
and
I
started
developing
matlab,
mpi
and
pmatlab
that
we
started
getting
some
traction
in
parallel
general,
parallel
library
adoption
within
the
laboratory.
B
My
group
did
not
have
a
cluster
yet
and
so
that
didn't
last
very
long.
My
burrowing
in
and
you
know,
I'd
have
windows
open
all
over
the
place
of
like
how
much
load
was
on
on
these
different
computers.
So
I
wrote
my
own
scheduler
and
that
also
didn't
last
long,
because
the
scheduler
was
pretty
bad.
It
did
not
have
the
features
that
I
really
wanted.
B
So
I
started
integrating
matlab
mpi
and
then
eventually
p
matlab,
with
with
the
condor
scheduler
from
university
of
wisconsin.
Subsequently,
we
have
ported
grid
matlab
to
pbs
grid
engine
lsf
and
now
slurm.
So
I
think
we
have
experience
with
all
five
major
schedulers
and
I
think
we're
gonna
stick
with
slurm
for
a
while.
I
really
like
slurm
a
lot,
but
then
we
also
started
fielding
clusters
because
our
upper
management
started
seeing
that
they
were
opera.
B
We
were
getting
more
and
more
users
using
pmatlab,
matlab,
mpi
and
gridmatlab,
and
so
you
can
see
down
here.
This
was
our
first
very
first
cluster
of
32
nodes
with
a
switch
and
a
small
array
of
of
storage,
and
it's
the
the
capability
really
blossomed,
jeremy
kepner
and
I
and
nadia.
We
all
worked
nadia
bliss.
We
all
worked
in
a
basically
a
parallel
processing,
an
embedded
systems
group-
and
this
was
a
great
way
for
us
to
do
prototyping
of
those
codes
before
actually
having
system
hardware
that
we
would
deploy
upon.
B
B
What
we
deliver
it
for
our
mission
areas
that
have
a
lot
of
vast
vast,
vast
amounts
of
data
from
lots
and
lots
of
sensors
is
that
we
have
a
vertically
integrated
capability
that
matches
their
development,
their
development
workflows.
We
run
our
own
data
centers,
we
have
our
own
hardware,
we
develop
our
own
software
stack.
B
B
We
often
are
more
productive
than
standard
super
supercomputing,
but
you'll
see
that
I
don't
want
to.
You
know,
paint
this
wonderful
picture
that
we've
solved
solved
world
peace
there
are.
There
are
things
that
we
don't
do
well,
but
in
terms
of
our
high
productivity
capabilities
for
our
users,
we're
quite
productive
and
we've
done
a
lot
of
comparisons
to
clouds.
B
So,
given
that,
let's
jump
into
these
lessons
learned
from
from
enabling
interactive
hpc
for
lincoln
lab,
as
we've
been
on
this
18-year
or
so
journey,
we
found
that
we,
it
is
as
much
a
cultural
expectation
as
it
is
a
technology
capability
and
that
cultural
shift
involves
we've
distilled
it
down
into
four
key
lessons
learned.
I
imagine
if
we
thought
about
it
harder.
We
probably
could
make
more
key
lessons.
But
let's
stick
with
four.
B
B
We
are
not
in
we
are
in
the.
I
don't
know
how
to
say
this,
kindly
we're
in
the
unfortunate
situation
but
fortunate
situation
where
we,
our
general
user,
does
not
have
any
hpc
experience,
and
I
think
that's
very
much.
In
contrast
to
what
you
all
experience
where
I
would
imagine,
many
grad
students
and
undergrads
that
come
and
work
at
a
national
lab
have
experience
many
of
our
folks.
B
B
I'll
talk
a
little
bit
more
about
re-architecting
for
interactive
hpcs.
Our
systems
on
the
surface
look
similar
to
conventional
super
computers,
but
there
are
some.
There
are
some
differences
and
I'll
talk
about
that
and
how
we
integrate
them
into
the
into
the
desktop
experience
and
laptop
experience
from
our
users.
B
We
see
this
continuum.
Expanding
the
hpc
ecosystem
is
talking
about
how
how
we
deliver
the
software
as
well
as
how
do
we
deliver
training
and
then,
of
course,
reframing.
The
metrics
is
expanding
the
metrics
from
just
utilization
discussions
to
utilization
and
other
metrics
that
are
important,
especially
important
for
the
user
and
because
it's
important
for
the
user.
It
is
important
for
the
organization.
B
We
were
all
about
matlab,
because
95
of
the
code
that
was
built
was
being
written
at
lincoln,
was
matlab
code,
and
so
early
on
you
know
I
talked
about
integrating
the
scheduler
with
matlab.
That
was
because
I
found
that
the
users
really
were,
and
it
wasn't
just
that
they
were
enamored
with
that
interface.
B
As
you
can
see
here,
we
run
dynamic,
hadoop
environments,
we
run
lots
of
python,
we
run
julia
java
code,
but
that's
not
to
say
that
we
also
we
have
plenty
of
fortran
and
c
plus
compilers
on
board.
We
also,
interestingly,
run
databases,
because
we
find
that
the
file
system
isn't
the
only
thing
that
users
need
to
to
manage
their
data.
They're
often
prototyping
with
databases,
and
so
by
providing
dynamic
databases.
B
They
also
they
are
able
to
use
more
of
our
resources
and
integrate
more
within
our
our
capabilities
and
in
terms
of
the
rapid
prototyping.
It
is
for
algorithm
development
data
analysis
and
machine
learning
training.
As
well
as
inference,
some
also
do
application
steering
from
their
desktop
and
from
login
nodes
and
now
more
even
visualization.
B
Let's
see,
let
me
give
you
a
few
examples
of
what
what
this
looks
like
for
them
along
the
top
are
some
ides
that
we
support
from,
of
course,
parallel
matlab.
I
I
should
take
a
real,
quick,
quick
subroutine
call
for
you
here,
a
method
call
you're,
probably
wondering,
and
I'm
just
gonna
answer
the
question
that
might
be
on
your
mind.
B
Well,
what
about
this?
Matlab
parallel
matlab,
parallel
server
capability
from
matlab,
so
rp
matlab
was
one
of
29,
parallel
matlab
capabilities
that
were
out
in
the
published
and
published
articles,
and
we
got
we
as
well
as
a
few
folks
from
campus
got
the
attention
of
math
works
that
actually
cleve
mohler
had
written
an
article
in
1995
saying
there
will
never
be
a
parallel
matlab.
Well,
he
he.
B
He
admitted
that
that
was
10
years
ago
at
that
time,
and
he
brought
us
in
along
with
some
folks
from
mit
campus
and
jeremy
kepner,
and
I
actually
met
with
folks
at
mathworks
monthly
for
like
a
two
year
plus
period
to
help
them
build
what
became
what
has
become
their
parallel
matlab,
which
is
a
very,
very
capable
interface
as
well
so
end
of
subroutine
call
that's
how
that
fits
into
the
whole
picture.
We
are
capable
of
running
their
parallel
matlab
as
well,
along
with
rpmatlab
on
our
clusters.
B
B
We
also
found
that
users
just
sometimes
want
to
use
a
linux
box
because
they
happen
to
be
on
windows
or
mac
os.
They
just
want
to
kind
of
rent,
a
linux
machine
or
a
fraction
of
a
linux
machine.
So
we
enable
that
for
interactive
hardware,
we
found
a
lot
of
users
doing
bulk
synchronous
processing,
so
mapreduce
style,
processing
and
they'd
come
to
us
saying
we
want
mapreduce
and
we'd
have
to
tell
them.
B
You
know
the
hadoop
map
produce
really
doesn't
solve
your
problem,
but
we've
written
some
wrappers
around
job
arrays
in
the
scheduler
that
just
make
it
very
intuitive
for
you
to
do
mapreduce
and
then.
Finally,
some
of
our
users
use
dynamic
development,
dynamic
web
services,
much
like
they
do
in
the
cloud
and
they'll
prototype
and
integrate
that
with
other
of
our
services.
So
we
enable
a
dynamic
web
service
capability.
B
Well,
let's
talk
a
little
bit
more
about
the
hardware
that
we've
got
and
how
it
differs
from
conventional
supercomputers.
So
we
run
two
unclassified
systems.
Tx
green
is
a
system
of
systems
that
is
available
to
the
lab
to
laboratory
employees
only
and
then
we
run
a
a
collaborative
system
called
txe1.
B
It's
for
engaging
super
computing
one.
You
notice.
We
use
the
tx
in
homage
to
tx0
and
tx2
that
were
on
that
history
chart
and
we
have
every
intention
to
continue
to
use
tx
as
our
our
monikers
are
the
starts
of
our
monikers
for
our
systems
in
terms
of
processing,
capability
and
storage
and
such
they
are
they're
much
the
same
as
any
large
compute
nodes,
and
we
we
house
these
out
in
three
locations.
Actually
one
of
them
is
in
that
f
building.
B
Remember
I
talked
about
the
original
whirlwind
and
tx
computers
being
an
f
building.
Well,
our
laboratory
data
center
continues
to
be
an
f
building,
but
we
also
have
assets
about
90
miles
away
from
lexington
in
holyoke,
right
in
the
center
of
the
state,
the
massachusetts
green
high
performance
computing
center,
the
mgh
pcc
houses
txe1
and
then
about
three
blocks
away.
B
So
we
use,
we
do
use
lustre,
but
we
use
luster
file
systems
based
on
ddn
and
seagate
arrays,
and
they
are
tuned
for
very
fast
metadata
accesses
with
reasonably
fast
data
accesses.
So
we
have
clustered
metadata
servers
taking
the
latest
from
the
the
lustre
file
system
capabilities
and
then
the
other
thing
that
is
rather
unique
about
our
systems
is
that
for
the
most
part
we
only
use
ethernet
and
we
have
had
so
the
storage,
the
two
storage
systems.
I
should
go
a
little
bit
more
into
detail
there.
B
We
have
home
home
accounts
on
one
of
the
storage
systems,
and
then
we
have
a
groups
where
we
do
groups
file
system
where
we
do
collaboration
between
users
and
teams,
the
there's
an
infiniband
network,
that
is
in
the
background
of
those
storage
networks,
but
we
generally
do
not
field
our
systems
with
infiniband
or
omnipath.
We
do
have
omnipath
networks
available
and
if
users
really
need
to
use
omnipass,
we
will
we
will
enable
them,
but
our
default
is
to
use
gigi.
B
Our
current
networks
are
a
10
gig
and
25
gig
networks,
depending
on
the
the
the
processor
types
and
when
we
bought
them,
and
the
other
thing
that's
rather
unique-
is
that
our
topology
continues
to
be
a
basically
a
star
network.
We
use
enormous
thousand
plus
a
thousand
plus
plug
ethernet
switches,
so
core
data
center
core
switches
to
connect
all
of
them.
So
pretty
much.
It's
a
single
hop
between
any
one
compute
node
and
the
storage,
as
well
as
other
compute
nodes.
B
B
There's
trade-offs
between
interactivity
and
optimal
utilization
for
batch
queuing,
and
I
use
this
chart
to
show
that
we
really
are
on
a
continuum
between
doing
all
batch
and
doing
very
optimized
launching
of
jobs
and
and
dispatching
jobs
onto
compute
nodes
and,
at
the
other
extreme
doing
all
immediate
scheduling
where
everything
that
gets
run
gets
run
immediately.
And
if
you
can't
run
immediately,
you
get
denied
at
launch.
B
I
know
that
most
most
centers
are
somewhere
in
the
middle,
with
bachelor's
batches,
with
reservations
and
batches
with
cues
that
do
do
full
interactivity
and
the
way
that
we
handle
it
is
with
interactivity
first,
just
because
of
the
nature
of
lincoln
business,
but
we
do
also
accommodate
batch
queue
jobs.
We
do
limit
users
to
how
many,
how
much,
how
much
of
the
system
that
they
can
occupy
at
a
given
time.
B
We
do
sometimes
have
individuals
who
say:
there's
there's
plenty
of
processing
left.
Why
can't?
I
have
it
we'll
address
that
at
the
end,
with
that
preemptive
capabilities
in
terms
of
our
jobs,
you
know
I
would
kind
of
hit
the
hit
the
hit.
The
point
home
we
came
up
with.
I
came
up
with
this
this
course
versus
duration
plot
that
we've
been
using
for
over
a
decade.
B
Most
of
our
users
are
when
they're
doing
data
analysis,
they're
getting
understanding
of.
What's
in
the
data
as
they're
running,
and
it's
not
uncommon
for
them
to
cut
a
job
off,
10
20
30
minutes
into
the
processing
because
they
know
what
they
want
to
run
next
and
the
rest
of
the
processing
doesn't
matter
anymore.
They
they
they're.
Just
let's
cut
this
one
off.
Let's
change
the
parameters,
let's
change
the
algorithms
and
let's
look
at
what's
where
it
needs
to
go
next,
so
the
more
that
they
can
do.
B
B
B
Furthermore,
we
have
some
python
anaconda
environments
that
we
build
and
configure
for
our
users
with
an
awful
lot
of
mod
packages
in
them
with
the
intent,
and
then
we
we
also
install
those
locally
on
each
node
again
for
runtime.
The
other
thing
that
these
high
productivity
languages
do.
That
is
it's
not
as
well
understood
until
you
run
into
the
problem.
Is
that
because
they
are
a
just-in-time
compilation,
they
they
often
do
stats
on
whether
the
the
libraries,
the
modules
the
packages
are
available,
and
so
you'll
see
this
stat
storm.
B
B
We
naturally
it's
challenging
with
so
much
software
to
do
everything
in
environment
like
bash,
rc's
and
launch
scripts,
but
we
with
the
overhead
for
virtual
machines.
We
really
only
want
to
use
that
when
we
really
really
have
to,
we
did
have
a
virtual
machine
capability
using
qmu
and
kvm,
but
we
have
had
so
little
use
of
that
that
we
have
kind
of
let
that
wither
away
for
the
time
being,
we
can
resurrect
it
if
we
need
to.
B
However,
we
do
also
use
singularity
containers
for
certain
situations,
and
it
usually
means
that
our
users
are
doing
some
sort
of
collaboration
with
other
organizations
and
government
entities
where
those
singularity
containers
become
a
godsend
for
them
to
move
entire
environments
around
but
yeah.
We
love
linux
modules.
A
lot
I'm
going
to
do
a
quick
run-through
of
a
few
of
these
software
technologies
that
we
use.
We
have
an
interactive
parallel
matlab,
as
I
discussed,
and
basically
it's
a
pgas
language
for
matlab.
B
B
So
frequently
you
get
users
that
ask,
for
you
know
I
just
have
a
folder
or
multiple
folders
of
input
data
that
I
want
to
run
through
the
same
executable.
How
do
I
do
it?
So
we
developed
this
capability
to
make
it
very
simple.
It
came
out
of
necessity,
trying
to
explain
job
arrays
to
someone
who
is
completely
new
to
parallel.
B
Another
nice
thing
about
this.
This
wrapper
is
that
when
we
chain
when,
when
we
have
changed
from
one
scheduler
to
another,
we
could
change
the
bindings
to
the
underlying
scheduler
without
changing
the
environment
for
our
users,
then
there's
a
few
other,
a
number
of
other
technologies.
I
want
to
highlight
four
of
them
here,
two
of
them
in
databases
and
then
of
course,
so
big
dog
polystore
is
a
multi-database
framework
for
integrating
data
sets
from
that.
B
Similar
to
that
is
the
the
graphulo
library
for
the
cumulo
database,
which
allows
you
to
do
graph
operations
like
breadth.
First
search
and
oh,
I
don't
know
triangle
counting
within
the
database
itself
that
takes
us
some
of
its
cues
from
d4m,
which
is
an
associative
array,
language
for
matlab,
octave,
julia
and
python.
B
B
It
is
really
easy.
Well,
on
the
surface,
it's
really
easy
to
measure
your
costs.
It's
often
the
software
cost
maintenance
cost
and
system
cost,
but
there's
actually
a
few
other
costs
we'll
get
to
utility
in
a
moment.
But
let's,
let's
do
the
easy
one.
Let's
look
at
cost.
First,
there's
actually
more
than
just
those
three
costs
that
are
involved
in
defining
your
cost.
There
is
actually
coming
from
folks
who
use
a
lot
of
serial
codes
and
want
to
parallelize
them.
There
is
the
time
to
parallel
parallelize
the
code.
B
We
use
the
time
saved
by
users,
because
that
is
ultimately
what
we're
trying
to
do
is
help
them
gain
greater
insights
in
their
data
analysis
and
machine
learning
and
rapid
prototyping,
and
so
time
is
of
the
essence
and
that's
the
key
for
them.
In
the
the
tichenor
and
ruther
article,
the
bottom,
one
from
ct
watch.
B
B
The
the
top
journals
in
the
country,
those
you
could
even
be
a
a
utility
opportunity
for
this,
this
type
of
processing
in
terms
of
that
processing,
we
have
for
them
formulas
on
how
to
capture
each
of
the
components
here
and
it's
detailed
more
in
in
both
of
those
papers
below,
but
what
we
found.
Let's
go
to
some
numbers.
These
numbers
come
from.
Actually
I
have
to
confess
I
haven't
run
these
numbers
in
a
number
of
years.
B
B
When
I
did
the
when
I
did
the
time
saved
by
users
on
a
linear
scale
of
the
number
of
cpus
used,
the
roi
just
blew
up,
it
was
crazy
and
I
started
getting
discredited
of
like
you
know:
you've
got
crazy
numbers.
This
doesn't
really
make
sense,
so
I
then
started
developing
multiple
other
ways
of
looking
at
it.
The
other
comparison
that
I
show
here
is
a
log
2
logarithmic
measure
of
the
the
amount
of
time
saved
and
benefit
to
the
users
based
on
the
number
of
cpus
use
the
amount
of
parallelism
being
used.
B
A
B
I
appreciate
it,
but
I
also
appreciate
the
questions
excellent.
So
do
we
run
rocky
over
ethernet
for
the
thousand
plus
star
networks?
Yes,
yes,
we
do.
We
actually,
just
last
week
at
ieee,
hpec
published
a
paper
doing
distributed
tensorflow
and
pie
torch
training
and
showing
rocky
versus
omnipath
on
our
on
that
tx
gaia
system
on
that
volta
system.
Yes,
we
we
use
rocky
a
lot.
We
have.
B
I'm
going
to
jump
to
the
third
question
I'll
come
back
to
annette's
question
in
just
a
moment.
Is
there
a
significant
amount
of
mpi
work
in
addition
to
other
workload,
types
that
you
mentioned
significant?
There
is
it's
significant
in
terms
of
our
mission,
but
it
is
in
terms
of
the
overall
mix
of
our
jobs.
Mpi
jobs
end
up
be
being
around
10
of
our
workload,
probably
a
little
bit
less,
but
some
of
those
are
the
most
important.
You
know:
radar
cross-section
studies,
computational
fluid
dynamics
studies.
B
Actually,
let
me
tell
you
a
fun
story
with
that:
the
the
mpi
style
jobs
in
terms
of
computational,
fluid
dynamics,
structural
dynamics
and
things
like
that
are
tend
to
be
interactive
in
nature
as
well,
and
this
story
kind
of
paints.
The
picture
we
were
talking
with
our
mechanical
engineering,
our
design
group,
that
does
a
lot
of
the
about
10
of
the
number
of
jobs.
B
I
don't
know
what
it
is
in
terms
of
number
of
hours.
To
answer
that
fourth
question:
we
were
talking
to
our
mechanical
engineering
team
who
use
structural
dynamics,
computational
fluid
dynamics,
codes
extensively,
and
we
were
trying
to
get
a
picture
of
like
what
is
their
workflow.
How
do
they
do
their
work
and
they
said?
Okay,
here's
the
job
that
we
have.
We
have
two
weeks
to
do
all
the
computational
runs
possible.
B
They
were
tasked
with
modifying
adding
a
pod
to
the
side
of
a
two
propeller.
What's
called
a
a
b,
what
is
it
the
beaver
or
something
like
that?
No,
the
the
otter,
it's
this
two
engine
plane
with
a
high
wing,
that's
great
for
mounting
a
whole
bunch
of
sensors
underneath
and
they
were
going
to
mount
another
sensor
on
this
thing
and
they
said.
Basically,
we
have
two
weeks
to
figure
out
what
which
of
the
pod
designs
we
want
to
use
so
that
this
plane
doesn't
fall
out
of
the
air.
B
B
So
that
kind
of
gives
you
a
picture
of
the
kind
of
jobs
that
even
our
mpi
users
do
very
seldomly.
Do
they
go
over
a
week?
Actually,
our
distributed
machine
learning
jobs
which
do
use
mpi
with
horovod
are
tending
to
run
longer
than
our
physical
simulation
jobs.
And
then,
let's
come
back
to
a
net's
question.
Is
there
is
your
parallel
matlab
equivalent
to
the
parallel
toolbox
or
the
parallel
server?
B
It
is
more
like
parallel
server.
We
take
advantage
of
parallel
toolbox
in
using
parallel
toolbox
to
take
advantage
of
the
gpus
on
board,
but
it
is.
It
is
the
launching
capability,
through
the
scheduler
and
the
messaging
capability
that
pmatlab
does
kind
of
as
a
as
a
another
solution
to
parallel
matlab.
B
B
C
B
I
think
it's
probably
10
of
the
the
number
of
jobs.
I
don't
have
a
good
sense
of
the
number
of
hours
I
I
need
to
get
back
into
doing
analysis
of
our
jobs.
I
haven't
done
that
in
a
while.
I
have
other
folks
doing
it
and
they're
looking
at
other
metrics,
but
I
have
I'm
usually
more
interested
in
these
kind
of
things
too.
B
Well,
so
yeah
so
in
terms
of
idle
time,
because
because
of
the
responsiveness
of
the
the
scheduler,
that
actually
is
a
great
lead
into
this
next
section
here
on
preemptable
batch
jobs,
batch
execution,
it's
hard
to
go,
get
away
from
utilization
and
what
we
often
found
was
utilization
in
the
50
to
80
percent
zone,
and
but
we
would
have
times
where
you
know
we
would
be
around
50
utilized
and
it
it
when
you
have
really
cool
hardware
to
use
it's
like.
Where
are
the
users?
B
Why
aren't
they
all
using
it,
and
especially
when
you
know
like,
at
the
end
of
a
fiscal
year,
so
you're
going
to
have
a
lot
of
scrambling
to
get
jobs
running
and
and
get
results
off
the
jobs
like
why?
Why
aren't
folks,
using
this
more
so
we,
our
management,
was
pretty
good
with
having
a
range
between,
like,
let's
say,
50
and
80
utilization
pretty
consistently
even
on
evenings
and
weekends
is
another
story.
B
B
So
we
really
wanted
to
have
all
our
job
types
available,
didn't
matter
to
us,
how
many
partitions
we
really
wanted
to
re-queue
and
as
you'll
see
in
the
next
couple
of
slides,
the
automatic
scheduler,
lua,
subscription
and
manual
were
all
just
taking
far
too
long
for
our
requirements.
So
we
found
a
way
to
do
it
with
job
scripts.
B
B
None
of
the
automatic
preemption
really
worked
for
us,
but
we
found
that
with
this
cron
job
preemption
with
high
water
marks
to
keep
some
nodes
idle
was
really
the
way
to
go
for
us
and
we
are.
We
have
been
rolling
it
out
over
the
past
few
months.
I
don't
have
any
statistics
yet
for
you
on
how
that
how
that
has
been
going.
We
do
have
several
users
using
it,
but
basically
the
way
we
see
it
is
that
we
have
a
few
users
or
a
few
teams
of
users
that
have
access
to
this.
B
This
spot
queue
to
run
speculative
jobs,
that
can
be
preempted
and
they
are
very
much
bulk.
Synchronous
types,
types
of
execution
runs,
and
but
we
can
also
do
it
with
mpi
jobs.
It
is
my
personal
opinion.
It's
not,
I
don't
think
it's
the
best
fit,
but
we
have
some
users
that
may
that
have
expressed
interest
from
the
mpi
ranks.
D
B
That's
right,
that's
right
and
we
do
get
into
oh
another.
Little
tidbit
of
this
is
when
we
do
when
we
do
reach
capacity.
My
entire
team,
including
myself
and
the
other
team,
leads
we
all
get
an
email
saying:
we've
exceeded
capacity
start
looking
at.
What's
going
on,
we
can't
run
any
more
interactive
capabilities,
so
that
really
motivates
us
to
help
shape
the
user's
usage
and
also
motivates
us
to
help
them
use
the
resources
very
efficiently.
B
D
C
B
B
There's
further
complications
for
these
applications,
and
that
is
that
there
isn't
a
one-size-fits-all
one.
Solution
fits
all
for
authentication
methods,
there's
a
whole
bunch
of
different
ways
to
authenticate
they
may
or
may
not
be
using
https
for
a
variety
of
https
protocols
and
ssh
port
forwarding
could
be
a
solution,
but
it's
just
not
something
that
I
want
to
teach
my
user
base.
It's
it's
challenging
to
make
work.
There
are
ways
that
you
can
have
others.
B
Co
use,
use
the
same
tunnel
that
we
just
don't
want
to
get
into.
We
actually
started
working
on
these
portals
about
10
years
ago,
where
we
enabled
actually
full
matlab
parallel
job
launches
through
a
web
portal,
which
we
then
expanded
out
with
authenticating
users
to
web
applications
and
then
also
integrating
with
a
user-based
firewall.
Now
I
should
take
a
quick
couple
sentences
on
the
user-based
firewall
and
what
that
is
for
most
of
us
most
supercomputers.
B
You
know
that
internal
network
between
compute
nodes
and
the
compute
nodes
and
the
service
nodes
are
kind
of
open.
There's
no
encryption,
no
verification
that
it's
the
same
user
or
same
group
accessing
the
open
ports
for
the
communications
and
for
some
of
our
some
of
our
capabilities.
We
actually
needed
to
have
a
verification
that
it
was
the
same
job
or
at
least
the
same
group
that
is
being
that
is
using
all
the
the
ports
and
all
the
ports
between
computers.
So
you'll
see
on
the
next
graphic.
B
What
we
have
that
the
user-based
firewall
plays
in
here.
What
we
built
was
a
way
to
use
multi-processing
modules
in
apache
to
take
the
input
you'll
see
over
here.
Some
urls
on
the
on
the
left,
take
the
input
and
allow
the
apache
web
server
to
do
the
right
thing,
based
on
what
the
incoming
url
components
of
the
incoming
url.
So
when
it
has
grid
san
involved.
B
B
B
B
The
four
lessons
learned
from
environment,
our
system,
architecture,
as
well
as
our
ecosystem
of
usability
and
productivity
and
utilization
metrics,
and
then
a
couple
of
the
tools
and
the
team
is
by
no
means
just
me.
We've
got
some
really
talented
system,
engineers
and
computational
engineers,
as
well
as
a
very
supportive
leadership,
and
I
apologize
for
going
a
few
minutes
over.
A
A
B
A
A
B
Yeah,
so
that
is
actually
the
the
one
above
that
let's
go
well
I'll
start
with
the
job
suspension.
Once
since
that's
already
been
asked,
we
have
looked
at
job
suspension
where
you
you,
basically
let
the
lower
priority
job
continue
to
suspend
on
the
node
in
virtual
memory,
but
we
quickly
ran
into
memory
issues
so
another
little
thing
kind
of
a
tidbit
of
information
about
how
we
run
our
nodes
is
that
we
do
not.
Actually
we
we
have
virtual
memory
enabled,
but
our
virtual
memory
is
exactly
as
large
as
our
physical
memory.
B
The
reason
that
we
do
not
have
any
spillover
into
virtual
memory
into
true
virtual
memory
is
because
of
the
high
productivity
languages
they
really
really
slow
down.
When
you're,
when
you
they're
pages
of
memory
start
going
out
onto
disk,
it
is
dramatic.
It's
our
users
say
that
it's
like
dropping
a
boat
anchor
on
their
job,
and
so
we
found
that
we
really
had
to
contain
the
amount
of
memory
available
and
so
that
ended
up
kind
of
putting
a
a
an
end
to
our
exploration
of
job
suspensions.
B
So
we
usually
kind
of
the
sweet
spot
for,
for
these
spot,
jobs
is
in
the
five
to
ten
minute
region
so
that
you
get
enough
work
done
and
have
made
progress
in
case
you
do
get
preempted,
but
are
also
not
doing
so.
Many
like
python,
relaunches
or
matlab
relaunches
environment
relaunches
that
you
are
filling
up
the
queue
and
spending
a
lot
of
time,
spinning
up
your
jobs
and
spinning
down
your
jobs.
B
So,
let's
see
are
those
preempted
jobs
capable
of
resuming
execution
where
they
left
off?
Yes,
but
only
if
they
it's
up
to
the
user
to
do
checkpointing,
so
if
they
do,
and
some
of
our
users
do
do
checkpointing
even
on
bulk
synchronous
jobs,
they
will,
they
will
keep
track
of
what
work
they
got
done
and
what
work
they
didn't
get
done
and
then
launch
users
again
when
they're
able
to
to
pick
up
where
they
left
off.
B
B
So
when
we
have,
I
believe
we
currently
have
it
that
the
user
also
gets
a
gets,
an
email
saying
that
we
could
not
accommodate
their
interactive
job.
We
are
currently
at
full
capacity
and
I
think
they
get
the
same
email.
We
do.
We've
had
a
few
users
say
man,
that's
a
lot
of
emails
that
I
get
like
yeah
welcome
to
the
team.
E
So
I
guess
I
asked
that
question.
I
was
thinking,
that's
an
interesting
point
you
made
to
it,
but
I
guess
I
was
thinking
on.
What's
the
definition
of
full
capacity
right,
is
it
just
that
there's
something
running
everywhere
or
is
it
that
you
can't
commit
that
there's
not
something
preemptable
available
or
what.
B
B
So
that's
what
I'm
referring
to
as
full
capacity.
There
are
times
when
we
are
nearing
full
capacity
on
all
four
processor
types
at
the
same
time,
and
that's
that's
scramble
time,
I'm
on
the
phone
with
users,
I'm
on
the
throne,
with
my
whole
team's
on
the
phone
with
users,
often
times
when
that
happens.
B
E
B
E
B
A
different
organization
that
isn't
so
rapid
prototyping
oriented,
I
totally
understand
how
you
end
up
needing
to
use
batch
queuing
in
order
to
get
get
all
the
work
done,
get
all
the
allocations
satisfied
in
a
given
year,
but
man
yeah
there.
There
have
been
a
few
years
where
jeremy
and
I
have
really
done
a
lot
of
soul
searching
about.
B
E
B
It's
like
well,
we
have
special
capabilities.
Well,
why
can't
you
do
those
in
the
cloud?
Well,
we
don't
own
that
hardware
and
we
can't
actually
do
as
much
as
much
of
that
fine-tuning
and
because
we're
behold,
we
would
be
beholden
to
other
people's
we'd,
be
renting
other
people's
hardware,
so
lots
of
skepticism-
and
I
think,
from
my
experience,
it's
just
been
explaining
those
subtleties
explaining
those.
What
is
right
for
what
organization
for
certain
parts
of
our
there's?
B
So
it's
a
matter,
you
know
in
our
own
there's
two
other
clusters
in
our
own
organization
that
are
very
batch
oriented,
and
so
it's
really
a
matter
of
being
transparent
and
explaining.
You
know
for
this
type
of
use.
You
want
to
do
this
for
this
type
of
use.
It's
essential
that
you
do
interactive
and
you
know
recently
that
has
really
come
to
the
forefront
with
machine
learning,
especially
machine
training,
machine
learning,
training
and
even
machine
learning
inference
where
those
engineering
terms
are
so
so
vital.
B
Oh
so
I
would
yeah.
I
would
definitely.
I
probably
would
do
to
jump
right
to
the
equation.
I'd
probably
have
something
in
the
denominator
of
that
cost-benefit
equation
that
captures
those
lost
opportunities
of
launching
those
urgent
jobs
that
then
helps
bubble
up
the
the
how
important
those
jobs
are
to
getting
run
when
they
need
to
be
run.
B
But,
more
importantly,
it
becomes
a
conversation
of
organizational
urgency
of
running
those
jobs
and
an
agreement
not
because
every
user's
job
is
urgent
to
some
degree.
They
really
would
love
to
have
the
answers
sooner
rather
than
later,
in
almost
all
instances,
but
then
an
organizational
understanding
of
what
urgent
really
means
is
also
important
in
in
following
that
line
of
reasoning,
I
hope
william.
I
explain.
I
explained
that
well
enough.
B
B
Cool,
let's
go
to
shreya's
question,
then
how
do
you
manage
running
jupiter
or
other
spawned
via
portal
tools
on
different
classes
of
back-end
resources?
So
yeah?
That's!
That's!
Probably
a
good
thing
for
me
to
jump
into.
I
kind
of
glossed
over
that
so
coming
back
to
the
screen
of
what
the
user
looks
at
when
they,
when
they
want
to
launch
a
jupiter
notebook,
they
get
to
choose
what
type
of
hardware
both
in
the
partition
that
they
choose,
the
cpu
type,
the
number
of
gpus
they
the.
B
What
the
what
the
underlying
code
in
apache
then
does
is
it
formulates
a
slurm
job
submission
for
them.
Slurm
launches
the
job
underneath
in
the
meantime,
the
apache
server
creates
the
reverse
proxy
entries
in
its
database
to
look
to
so
that
when
the
job
is
launched
and
the
notebook
is
ready
to
go,
the
user's
desktop
automatically
gets
redirected
to
where
that
jupiter
notebook
is
speaking
and
gets
goes
through
the
reverse
proxy
to
their
desktop,
so
that
so
that
the
right
processor
type
is
brought
to
bear
for
that.
Jupiter
notebook.
A
The
way
I
understood
your
architecture
you
have
so
the
hub
jupiter
hub,
has
a
proxy
and
we
run
jupiter
how
that
has
a
proxy,
but
you
guys
kind
of
have
a
proxy
that's
kind
of
above
that
that
you
described
that
lets
you
get
access
to
jupiter
notebooks
like
through
this
interface
or
the
other
services
that
you're
running
so
you've
kind
of
taken
that
proxy
and
kind
of
elevated
it
another
level.
That's.
B
A
Is
do
you
like
that
design
or
is
there
anything
you'd
like
to
change
about?
I
noticed
it
was
like
this
apache
kind
of
thing
is
that
is,
are
you
is
it
working
really
well?
Is
that
what
you
like,
or
do
you
have
any
kind
of
issues
with
like
forwarding
stuff
through
that,
or
does
it
keep
up
with
the
traffic
pretty
well.
B
So
it
keeps
up
with
the
traffic
awfully
well
and
that
was
kind
of.
Oh,
my
goodness,
is
this
really
going
to
work.
The
the
the
user-based
firewall
can
at
times
be
a
little
slow,
and
so
we
have
tried
it
for
certain
processors
we're
we're
able
to
turn
that
off.
We
don't
have
to
worry
about
it
for
other
clusters,
who
have
that
have
more
more
v
are
more
exposed.
B
We
have
to
keep
that
on.
We
we
like
it
a
lot.
We
what
it
allow.
I
think
the
the
benefit
far
out
outweighs
the
challenges
of
keeping
it
up
to
date,
because
it
is
all
for
us,
it's
all
custom
code
that
is,
that
is
running
this
stuff.
I
think
what
we
really
like
about
it
is
that
we're
not
limited
to
a
small
set
of
nodes
or
a
certain
set
of
nodes
on
which
users
can
run
jupyter
notebooks.
We
literally
could
fill
the
entire
cluster
up
with
only
jupyter
notebooks.
B
If
we
really
wanted
to.
If
that,
if
the
need
would
arise,
I
think
if
we
would,
if
we
were
to
start
today
or
start
have
started
two
years
ago
on
this
endeavor
of
instead
of
ten
years
ago,
when
we
started
on
these
portals,
we
would
probably
take
a
long
and
hard
look
at
open
on
demand
as
being
as
at
least
enabling
this
capability
or,
if
not
other,
capabilities
as
well,
because
it
has
a
really
nice
plug-in
architecture
that
you
can
use
to
enable
any
sort
of
web
web-based
application.
A
Right,
I
think,
that's,
I
think,
that's
a
question
that
we
don't
haven't
really
looked
at.
Do
we
need
that
kind
of
capability?
But
you
know,
I
think
that
we
might
we've
kind
of
taken
the
view
that
we
can
kind
of
enable
a
lot
of
things
through
jupiter
hub
and
through
the
you
know,
the
you
mentioned
tensorboard
and
things
like
that.
A
lot
of
people
are
doing
that,
basically
through
jupiter
notebooks
and
there's
a
server
proxy
that
runs
through
that.
B
Yeah,
but
I
mean
jupiter
hub-
has
a
jupiter
team
and
jupiter
hub
as
a
whole,
has
been
doing
a
really
good
job
of
addressing
all
of
these
scalability
challenges
as
well
and
so
yeah.
I
think
again,
if
we
would,
if
we
had
done
the
analysis
two
years
ago
or
one
year
ago,
instead
of
ten
years
ago,
we
probably
would
have
ended
up
with
something
very
different.