►
From YouTube: 2020-05-15 - Steven Farrell & Mustafa Mustafa - Deep learning production capabilities at NERSC
Description
NERSC Data Seminars: https://github.com/NERSC/data-seminars
Deep learning production capabilities at NERSC
Abstract: Deep Learning is increasingly being used for scientific problems which require large scale computing resources. High performance computing centers are adapting to accommodate these new kinds of workloads which can differ significantly from traditional HPC simulation workloads. NERSC supports and enables deep learning workflows by providing an optimized software stack and by supporting users to deploy their applications effectively and productively. In this presentation we will describe NERSC’s production capabilities for scientific deep learning applications, including details of the software stack, system performance with extensive benchmarking, and workflow solutions to enable productive science. In addition, we will discuss our outlook for the future of AI at NERSC on the upcoming Perlmutter supercomputer and beyond.
A
C
Okay,
I
hope
you
can
all
see
my
screen,
yep,
okay,
so
hi
most
of
all
talk,
I'll,
start
the
talk
and
then
Steve
will
take
it
on
from
halfway
through.
So
as
mentioned,
this
talk
will
cover
the
production
capabilities
that
we
have
nursed
for
deep
running.
So
we'll
talk
a
little
bit
about
deep
yearning
for
science
in
general
and
then
talk
about
the
stack.
What
we
provide
and
also
the
different
frameworks
and
benchmarks
of
performance
benchmarks
for
the
different
frameworks
that
we
have
we'll
also
show
some
results
from
our
latest
machine
learning
at
North
survey.
C
These
are
still
preliminary
results.
I'll
talk
about
that
in
a
minute,
so
we're
trying
we'll
try
to
leave
some
time
for
questions
at
the
very
end.
But
please,
during
the
talk,
if
you
have
any
questions,
just
ask
that
to
probably
make
the
presentations
a
bit
more
interactive,
given
that
we
are
all
on
Zoo,
okay,
so
deep
learning
for
science.
As
you
all
know,
the
turning
now
is
becoming
a
very
big
research
area
in
science.
They
hope
is
that
it
will
help
with
analytics
for
large-scale
data
sets
and.
C
That's
that's
the
major
application
of
of
the
chronic
right
now
and
in
science.
The
other
two
areas
are
for
accelerating
expensive
simulations.
That's
a
very
big
research
topic,
for
example,
for
all
the
chemistry
simulators,
PDE
solvers
for
climate
systems,
fluid
dynamics
and
also
for
high
energy
physics
simulations
it's
it's
it's
one
of
the
opportunities
is
to
use
deep
learning
to
accelerate
this
very
community,
computationally,
expensive
simulators
and
then
in
the
area
of
real-time
control
and
design
of
experiments.
C
That's
an
area
that
we
think
that
deep
learning
will
will
be
very
effective
and
more
ease
were
seeing
more
research
in
that
area.
Deep
learning
for
Sciences
as
you've
seen
there's
a
rapid
growth
in
machine
learning
conferences
in
that
area,
and
we
have
also
seen
major
prizes
granted
deep
learning
funding
agencies
are
focusing
heavily
on
on
AI
for
science.
There
are
big
opportunities
there
we
have
seen
calls
recently
from
Oscar
and
and
ECP
is
already
working
on
deep
learning
programs,
the
AI
town
halls,
the
for
AI
town
halls
that
happened
AI
for
science
town
halls.
C
That
happened
have
produce
a
very
long
report
on
the
different
opportunities
and
different
domains
of
science.
The
report
is
linked
here:
I
think
it
gives
a
good
overview
of
what
are
the
milestones
and
the
that
we
need
to
achieve
to
affect
and
opportunities
for
deep
learning.
And
finally,
we
are
expecting
an
ACP
like
program
on
AI
for
science.
That
will
be
a
major
effort
for
for
the
TV
okay,
so
deep
learning
deep
learning
is
is
a
sub-branch
of
machine
learning.
It
solves
the
same
problems
that
machine
learning
tries
to
solve,
but
using
neural
networks.
C
So
we
will
see.
Currently
we
use
neural
networks
for
the
same
problems
like
for
supervised
learning
and
unsupervised
learning
and
reinforcement
learning,
but
the
same
tasks
that
you
can
solve
with
traditional
machine
learning
are
now
much
more
it's
much
easier,
more
effective
to
solve
them
using
neural
networks.
I
won't
go
into
the
details
of
what
you
neural
networks
are,
but
we
will
talk
about
the
workloads
that
these
neural
networks
need
to
develop
and
and
deploy,
and
also
about
the
different
scales,
the
all
the
computational
needs
related
to
to
building
and
deploying
neural
networks.
C
Before
I
go
on,
so
we
will
be
showing
the
results
from
the
survey.
The
machine
learning
at
North
survey
that
we
have
in
2020.
These
are
still
preliminary
results.
We
are.
The
survey
is
still
open.
So
if
you
haven't
responded
to
the
survey
yet
we
please
respond.
We
data
helps
us
in
so
many
ways,
so
the
survey
tries
at
targets
the
scientific
communities,
both
nurse
and
non
non
North
users,
and
we
mainly
focus
on
communities
that
could
use
HPC
sister
HPC
resources
for
deep
learning.
C
We
try
to
track
different
trends,
so
things
like
problems,
workload,
model,
architectures
framework,
scaling
strategies,
hardware
and
software.
The
hardware
and
software
needs
of
machine
learning
and
many
other
things.
We
try
to
track
them
in
the
survey
and
we
we
often
make
the
the
results
public.
So
it
serves
us
answers
other
people
beyond
nurse.
C
The
survey
is
also
a
very
good
way
for
us
to
understand
what
are
the
use
cases
of
of
the
stack
that
we
have,
and
also
for
identifying
areas
for
to
improve
user
experience
and
performance
in
the
near
future,
and
it
also
helps
us
anticipate
what
kind
of
workloads
you
will
have
in
the
near
future
and
prepare
for
that.
As
you
can
see
in
these
plots,
the
respondents
are
sparing
multiple
domains.
C
Okay,
so
now
back
to
deep
learning,
we'll
be
talking
about
things
that
are
related
to
different
aspects
of
the
deeper
the
deep
learning
workloads,
but
just
to
outline
one
is
going
to
go
on.
So
when
you
are
developing
your
networks,
there
are
different
stages.
One
of
the
stages
is
training
the
model,
so
you
need
to
you
have
a
neural
network
and
you
want
to
train
that
neural
network
which
is
trained
on
a
larger
amount
of
data
and
usually
especially
for
science
as
well
as
I'll
show
in
a
little
bit.
We
have
very
large
scale.
C
C
So
you
take
a
frozen
trained
model,
and
then
you
apply
it
on
on
data,
that's
mainly
for
data
analytics,
but
and
sometimes
it's
used
in
offline
analytics
and
but
it
could
also
be
used
in
online
and
Mattocks
like,
for
example,
for
all
the
they
cloud
providers
or
like
services
like
Google
and
Facebook.
They
would
need
to
use
these
for
the
Train
models
in
real
time.
This
is
not
like
a
clean
separation
between
training
and
entrance.
C
You
could
have,
for
example,
if
you
have
a
set
up
of
active
learning,
you
have
some
model
that
you're
training,
along
with
from
data
that
is
coming
from
your
simulator,
so
you
could
be
running
a
simulator
and
training
your
model
using
and
using
the
model
at
the
same
time.
So
we
are
quite
month
later
using
simulator,
you
improve
the
performance
of
the
model
and
then
use
it
to
you.
Query
the
model
for
as
you're
running
your
simulator
for
to
improve
the
simulator
speed.
C
So
there
are
instances
where
you
would
need
to
do
both
training
and
inference,
and
that
requires
its
own
has
its
own
challenges.
For
example,
if
you
MA,
if
your
simulator
is
running
on
CPUs
and
then
and
your
model
is
running
on
GPUs,
that's
something
that
would
need
to
be
thought
through
carefully.
We
see
from
our
user
surveys
is
the
same
thing
in
2018
and
2020
that
currently
people
are
mostly
focusing
on
developing
models,
but
they
and
that's
what
most
of
the
demand
goes
through
to
and
then
in
the
near
future.
C
So,
as
I
said,
these
models
could
be
very
large.
There
are
so
many
ways
to
show
how
large
these
models
could
be
and
how
much
resources
they
need
to
train,
but
one
of
the
ways
is
just
to
look
at
how
long
it
takes
people
to
train
their
models,
so
you
can
see
to
the
left
here.
This
is
a
quick.
This
is
a
survey
question
of
how
long
it
takes
people
to
train
the
model
and
most
people
takes
hours.
C
Today's
and
15%
in
this
survey
show
that
they
need
weeks
to
train
the
model
and
think
of
this
as
a
compilation
time
for
your
software.
If
it
takes
you
weeks
to
to
compile
your
your
code,
it
would
take
forever
to
actually
develop
something
that
works.
So
this
needs
to
be
this.
This
training
time
is
to
be
reduced
tremendously,
and
for
this
we
need
scale
in
English
PC
systems
to
be
able
to
to
do
this
training
faster.
C
You
can
also
see
it
like
the
trend
in
general
and
in
deep
learning,
since
its
onset
or
its
recent
onset,
we
have
more
and
more
complex
models
and
these
complex
models
they
need
more
compute
to
Train,
and
the
trend
that
we
see
right
now
is
that
the
more
complex
the
task
that
you
need
to
solve.
You
need
to
develop
a
more
complex
model
for
that
and
to
to
train
the
model.
C
You
need
more
data
and
more
compute,
and
this
is
a
logarithmic
growth
in
the
compute
time,
and
we
don't
expect
this
to
come
down
to
slow
down
anytime
soon.
We
also
see
that
in
the
survey
results
you
see
that
people
use
mainly
your
single.
Let's
assume
this
is
mostly
for
development,
but
then,
after
that,
you,
you
also
need
multiple
GPUs,
so
between
a
single
node
that
has
four
to
eight
GPUs
and
then
hundreds
of
your
peers
to
train
models
so
to
to
achieve
that.
How
do
you
actually
achieve
paralyzation
and
practice?
C
There
are
multiple
modes
of
doing
that.
The
most
common
one
is
data
parallelism
where,
for
example,
you
have
a
very
large
amount
of
data
you
need
to
to
pass
through
that
data
multiple
times
much
faster.
So
what
you
do
is,
instead
of
having
a
single
worker
processing
your
data,
you
have
data
parallelism,
so
you
clone
your
model
or
multiple
workers
in
this,
for
example,
GPUs
or
no,
and
then
each
one
of
them
will
process
its
own
batch
and
then
you
do
all
reduce
on
the
gradients
and
propagate
the
back
and
updated
parameters
that.
C
Multiple
motor
parallelism
and
we
will
show
the
results
that
we'll
show
in
this
in
this
talk
are
mostly
are
actually
all
on
data
parallelism
performance
at
norsk,
other
modes
of
parallelism
as
these
models
become
more
complex
and
you
need
to
to
build
bigger
models.
You
need
to
go
into
model
parallelism
model
parallelism.
One
way
of
doing
that,
you
would
take
you,
take
a
single
layer
from
your
neural
network
and
you
spread
split
that
layer
itself
on
multiple
workers
and
then
yeah,
and
then
you
process
process
your
data.
So
this
is.
C
This
splitting
your
large
model
over
workers
is
to
do
pipelining,
which
is
instead
of
splitting
the
layer
itself
over
multiple
workers.
You
split
the
layers
among
so
you
have
the
first
layer
here
now
on
a
GPU
and
then
the
second
layer
on
a
different
GPU,
and
you
have
some
layer
by
dividing
both
of
them
have
their
own
challenge:
communication,
challenges
and
opportunities
for
optimization
and
finally,
actually
in
practice,
you
can
imagine
that
you
actually
need
the
hybrid
hybrid
parallelism.
C
So
if
you
have
very
large
models
to
train
you,
they
don't
need
a
lot
of
data
and
if
they
need
a
lot
of
data,
then
after
you
split
your
model
over
multiple
workers,
you
need
to
do
data
parallelism.
You
have
groups
of
work
workers
for
the
model
and
then
multiple
of
those
groups
to
do
data
parallelism.
So
in
practice.
Eventually,
we
would
need
to
do
mostly
hyper
parallelism.
C
C
To
do
that,
to
accomplish
this,
the
parallelization,
you
need
some
framework
that
helps
you
do
that
so
tensorflow
on
PI
torch,
which,
as
I,
will
talk
later.
There
are
the
main
frameworks,
the
most
popular
frameworks
for
doing
deep
learning,
and
they
both
have
some
parallelism
native
parallelism
support
for
native
parallelism
routines
and
they
are
the
most
common
way
to
actually
distribute
distribute
training,
especially
for
pi
torch.
Most
people
use
their
native
white.
C
Paralyzation
api
next
would
be
harvard
which
is
an
uber
framework
for
doing
data
parallelism
and
it
supports
multiple
backends
so,
for
example,
MPI
or
nickel,
and,
as
I
mentioned
later,
we
use
those
on
CPU
and
GPU
and
yeah.
So
Harvard
is
very
popular
and
it's
very
easy
to
use,
and
we
expect
you
to
have
Harvard
for
a
while,
and
there
are
other
frameworks
that
people
are
experimenting
developing
or
experimenting
with.
So
that's
on
the
introduction
now
to
what
we
actually
have
at
norsk.
C
So
our
strategy
for
for
for
our
stack,
is
to
have
the
most
popular
frameworks
and
to
work
with
the
vendors
to
make
sure
that
the
underlying
libraries,
the
backend
libraries,
are
optimized
on
our
machines,
whether
the
the
CPU
partition
or
the
GPU
partition.
So
we
do
work
with
Intel
and
NVIDIA
on
on
both
and
also
Google
and
Facebook,
for
optimizing
the
frameworks
themselves.
We
we
were.
C
We
also
focus
on
providing
productive
platforms,
not
only
compute
resources,
but
also
ways
to
access
those
compute
resources
conveniently
and
to
especially
for
the
kind
of
workflow
that
goes,
while
you're
developing
deep
learning
models,
and
we
also
provide
training
and
consulting
and
multiple
collaborations,
especially
at
this
stage
where
deep
learning
is
picking
up
in
science.
So
we
make
sure
that
we
work
closely
with
a
lot.
D
C
Most
anticipated
moment
for
computer
resources
is
to
have
our
next
machine
paramotor
machine,
which
we
expect
to
have
the
first
phase
of
that
in
later
this
year
or
late
this
year
we
would
have
12
GPU
cabinets
and
which
will
house
around
more
than
six
thousand
nvidia
and
pair
GPUs,
which
were
just
announced
yesterday.
We
will
have
also
very
large
storage
area
that
is
very
performant
for
especially
for
kind
of
workloads
where
you
need
a
lot
of
I/o
like
deep
learning
and
then
by
mid
next
year.
C
We
expect
to
have
the
second
phase,
which,
which
will
add
CPU
on
any
cabinets,
12
AMD
CPU
on
the
cabinets
so
yeah.
We
expect
to
have
more
deep
learning
on
parameter
than
we
currently
have
now,
okay,
so
for
frameworks
there
are.
There
are
multiple
frameworks
on
the
on
the
market.
They
are
mostly
dominated.
C
People
use
both
Kerris
in
terms
of
flow
and
most
of
the
time
when
they
sakera
season
is
a
high-level
api
description
and
there
are
multiple
implementations
of
that,
but
the
tensorflow
implementation
is
was
mostly
people
used
mostly,
and
that
essentially
has
Kurt's
API
with
philosophical
back-end
and
then
PI
torch
is
under
eyes,
and
it's
now
a
lot
of
people
who
are
doing
deep
learning,
especially
for
research,
are
using
PI
torch
and
there
are
other
frameworks.
This
is
this
column.
Here
is
sensitive
one
versus
two
major
releases.
B
C
C
The
message
here
is
that
90%
of
at
least
of
the
survey
respondents
they
use
the
nernst
modules
directly
and
then
sometimes
they
have
to
either
customize
their
condo
environments
or
build
their
own
condo
environments,
and
that's
the
next
most
available
way
to
or
common
way
to
do
this,
and
then
some
people
build
from
source.
We
do
provide
help
with
all
of
these
a
word
on
the
frameworks
and
their
popularity.
C
There
were
other
frameworks
on
the
market
like
cafe
and
cafe,
and
the
piano
both
of
them
are
dwindling
in
popularity.
This
plot
that
at
the
top,
is
it's
only
it's
the
mentioning
of
these
frameworks
in
machine
learning
papers
and
up
to
2018,
which
is
I,
expect
that
this
is
even
more
the
case
now
that
Cafe
at
Vienna
or
nobody
is
using
or
less
and
less
people
are
using
them
for
the
survey.
C
For
example,
we
didn't
have
anyone
who
and
another
way
to
look
at
this-
is
to
look
at
the
Google
search
trends
and,
as
you
can
see,
I
torch
was
right,
rising
since
2017
and
over
at
least
over
the
past
year.
We
see
at
least
average
of
these
splits
between
terms
of
rockers
and
by
George.
Google
search
trends
are
stable.
C
It's
hard
to
get
more
concrete
statistics
of
what
are
the
popularity
of
these
frameworks,
but
generally
people
use
PI
torch
for
research,
and
the
flow
has
a
lot
of
deployment
ecosystem
that
people
like
to
use.
But
that's
that's
not
a
hard
split,
3d,
ok,
so
more
on
the
popularity
of
different
of
deploring.
So
the
top
right
plot
shows
the
the
increase
in
the
number
of
CPU
and
GPU
users
at
norsk.
C
This
is
only
for
2
months,
April
and
May,
and,
as
you
can
see,
it
has
been
more
than
doubling
every
year
since
2017,
and
we
also
see
a
significant
portion
of
users
now
are
using
pipe
torch
on
the
GPU
partition
that
we
have,
which
is
a
small
test
partition
that
has
18
nodes
with
8
Voltas.
Each
more
than
70%
of
the
workload
is
gone
too
deep
running,
and
it's
not
only
that
people
are
doing
single
GPU
stuff.
There's
deep
learning
workloads
are
spanning
all
sorts
of
scales,
all
the
way
to
the
entire
machine,
almost
okay.
C
So
on
how
to
use
the
software
stack
as
I
mentioned,
we
have
multiple
ways
of
you
have
multiple
ways
of
doing
that
at
norsk.
The
most
straightforward
way
is
just
to
check
their
module.
So,
for
example,
you
can
say
module
available
density
or
torch,
and
you
will
see
the
different
versions
and
then
you
can
load
the
version
that
you
want.
C
These
modules
are
we
we
put
the
most
performant
versions
of
the
libraries
that
are
available
and
we
also
try
to
include
everything
that
all
the
packages
that
you
might
need,
but
in
case,
if
you
want
to
customize
your
installation
further,
you
can
always
say
you
can
always
install
things
in
your
user
area.
So
that's
that's
the
they
the
most
popular
way
of
customizing
environments,
or
at
least
that's
what
we
recommend
right
now.
C
C
So
moving
forward
to
four
currently
for
the
current
quarry,
GPU
partition
and
also
for
parameter,
we
hope
to
provide
the
entire
stack
in
shifter
images
instead
of
modules.
The
reasoning
is
that
Nvidia
provides
highly
optimized
containers
for
performing
performance
on
GPUs,
and
we
currently
have
we're
currently
testing
that
we
think
we
have
something
that
you
can
start
using
and
you
can
use
that
both
in
interactive,
actually
an
interactive
batch
system
and
also
in
through
Jupiter
for
interactive.
C
This
is
how
you
would
loaded
you
look,
for
example,
at
PI
torch,
a
pie,
torch
image,
and
for
if
you
are
submitting
bad
jobs,
please
make
sure
to
put
the
image
not
in
the
command
of
shifter,
but
use
the
slurm
options.
That
will
give
you
much
better
performance,
especially
in
distribu,
if
you're
training
distributed,
and
we
have
also
tensorflow
too
right
now
and
if
you
have,
we
are
looking
forward
for
feedback
on
these
images.
C
So
if
you
test
them
and
there's
something
that
you
don't
mind
cores
not
working
for
you,
please
let
us
know
you
can
also
use
those
from
from
Jupiter
and
we
do
have
jupiter
kernels
to
use
this.
These
same
ships,
commish's,
ok,
so
our
general
guidelines,
I
kind
of
set
a
few
things
here
and
there
of
what
we
recommend,
but
so
generally,
just
to
recap.
We
generate
recommend
that
you
use
the
modules
directly
to
intensify
or
or
PI
torch.
C
For
if
you
are
for
developing
and
testing
your
your
workloads
and
using
the
interactive
QoS
on
Canal
or
Kahneman
as
well.
The
CPU
partition
over
Jupiter,
on-demand
compute
resources
are
very
convenient
and
we
recommend
to
use
10
support
for
visualization
on
our
machines
and
Steve
will
talk
about
this
in
a
bit
for
performance
tuning.
C
There
are
as
if
you
have
trained
a
deep
learning
model
before
you
know
that
there
are
so
many
moving
parts,
and
especially
when
it
comes
to
IO
and
then
performance
on
a
single
GPU,
node
or
multiple
GPUs,
and
there
could
be
a
lot
of
bottlenecks
in
that.
So
at
some
point
you
might
want
to
tune
your
model
for
for
faster
turnaround
in
training
and
the
way
to
do
that.
C
The
first
thing
you
can
do
is
just
use
to
check
the
CPU
and
GPU
utilization,
and
that
also
gives
you
the
memory
usage
just
using
top
or
Nvidia
and
I
never
give
you
an
idea
of.
What's
going
on.
You
can
also
track,
for
example,
if
you're
using
GPUs,
you
can
track
invidious
MI
and
and
plot
that
or
if
you're,
using
any
productivity
framework
monitoring
framework.
Some
of
them
provide
a
way
to
visualize
the
utilization,
and
that's
your
first
hint
of
at
least
how
efficiently
are
using
your
compute.
C
So
there
are
a
lot
of
knobs
that
you
can
use
to
tune
the
performance
and
for,
if
you're,
doing,
I/o
that
specially,
if
directly
from
desk,
then
we
recommend
I
use
the
burst
buffer.
For
for
that,
that
will
give
you
much
better
performance
if
you
need
to
do
more
than
that,
there
are
profilers
that
you
can
use
like
C
profile
or
Nvidia.
Dl
prof,
which
are
on
poor
shape,
use.
C
Tensor
board
is
recently
started,
providing
profiling
tools
in
tensor
board
for
tip
for
tensor
flow,
and
that
is
something
that
you
can
also
try
for
distributed
training
for
on
with
tensor
flow.
We
do
recommend
that
you
use
Harvard.
We
think
Harvard
is,
as
I
said
earlier.
It
it
compiles
with
both
MPI.
It
can
use
API
or
nickel
backends
on
our
GPU
CPU
machines.
Mpi.
Would
we
compiled
with
Cray
MPI,
which
will
give
you
best
performance
there
and
for
there
were
few
machines
and
we
compile
it
with
nickel
back
end
and
that'll?
C
Get
that
gets
much
much
better
performance,
as
you
will
see
later,
XT
will
show,
and
we
do
you
can
use
the
Harvard.
Examples
should
run
out
of
the
box,
no
machines,
but
if
you
have
problems
with
that,
you
can
consult
the
docs
or
check
with
us
and,
as
I
mentioned,
tensorflow
has
profilers
that
you
can
check,
especially
if
you
are
doing
training
on
GPU.
It
has
a
lot
of
features.
C
Okay,
so
for
pi
torch
distributed
training,
we
recommend
I
use
distributed
data
parallel
API.
There
is
another
one
that
is
called
data
parallel,
and
that
is
the
easiest
use.
But
it's
not
the
best
performance
wise,
so
we
recommend
that
you
use
the
distributed
data
parallel.
It
allow.
It
is
very
easy
to
do.
C
Entra
note
and
enter
note
distributed
training
with
that
and
it
works
seamlessly
with
CPUs
and
GPUs
if
your
code
is
optimized
to
to
do
that,
not
optimize
like
you
written
to
to
actually
switch
between
CPU
and
GPU
seamlessly
and
again
the
same
case
here
with
on
our
CPU
partition.
The
PI
torch
runs
with
the
MPI
back-end
that
gives
best
performance
and
on
a
GPU
partition.
Nickel
back-end
gives
the
best
performance.
C
C
B
C
I
think
testing
on
quarry
GPU
has
helped
us
a
lot
to
understand
what
is
what
is
coming.
We
did
sort
out
so
many
issues
Steve
would
show
some
of
these
results.
Currently
we
expect
that
we'll
be
able,
in
terms
of
the
software
running
at
when
we,
hopefully
everything
will
run
time
and
can
run
out
of
the
box.
C
There
are
some
some
things
related
to
the
networking
and
if
we
will
be
able
to
use
nickel
on
the
network
to
have
one
on
paramotor,
that's
something
that
we're
currently
investigating,
but
in
terms
of
the
software
stack
I,
think
it
it
actually
Rob
will
be
at
you
know.
For
the
first
period,
it
will
be
alright,
given.
A
D
C
I
think
what
we
currently
have
is
at
the
first
order
will
work.
Fine,
the
the
few
things
that
we
need
to
sort
out
is:
how
do
you
run?
What
is
the
best
way
to
run
multiple
multi-gpu
multi
node
training
from
jupiter,
which
is
something
that
Steve
has
talked
a
little
bit
about
one
solution
to
do
that,
but
we're
still
not
sure
if
that
would
be
the
best
way
to
do
it,
but
and
that's
something
that
where
is
on
our
less
to
actually
sort
out
so
and
I?
C
Think
I
I
talked
to
you
a
little
bit
and
you
provide
that
the
access
to
multiple,
multiple
GPUs
from
from
Jupiter,
at
least
in
the
test,
testing
Jupiter,
and
we're
still
not
sure
how
to
actually
integrate
to
allow
the
Jupiter.
You
know
job
to
actually
run
on
multiple
GPUs
in
in
the
best
way.
Other
things
are
regulated
to
the
profiling
likely,
as
you
will
see
here,
tensor
board
will
will
be
running
from
from
Jupiter
I'm,
not
sure.
C
If,
at
some
point,
we
will
need
to
have
the
ability
to
to
monitor
to
monitor
Jupiter,
neural
networks,
training
or
profile
profile
that
if
it
would
work
out
of
the
box
or
not
to
it
with
what
we
have
right
now,
that's
a
lot
of
random
things.
But
it's
something
it's
something
that
you
need
to
investigate
other
than
that
I
I,
don't
have
I
I,
don't
think
there
are
other
things
that
are.
You
know,
pressing
issues
with
using
Jupiter
on
promoter.
We.
B
This
work,
yes,
okay,
just
let
me
know
we
don't
see
it.
I
am
assuming
you
do
so,
first
Jupiter,
which
is
what
we
were
just
talking
about,
so
hopefully
probably
we're
all
already
familiar
with
what
you
Pater's
and
how
popular
it
is.
We'd
have
a
Jupiter
hub
service
at
nurse.
Provides
this
really
nice
notebook
ecosystem
to
be
able
to
interact
with
our
systems
on
Cory,
it's
very
popular
among
our
nurse
users,
something
like
700
users,
I
believe
use
Jupiter
at
nurse
and
in
our
our
survey
form
our
survey.
B
It's
clear
that
a
lot
of
use
vomiter
to
actually
do
the
R&D
for
their
machine
learning.
So
for
deep
learning,
users
can
use
Jupiter
to
deploy
their
workloads
onto
our
systems.
They
works
on
both
core
CPU
and
GPU.
Thanks
to
Roland
for
setting
up
the
integration
with
GPU,
we
have
our
provided
software.
Kernels
I
think
I
mentioned
every
flea,
but
you
can
see
the
list
I
think
that's
all
of
them
down
on
the
bottom.
B
It's
basically
a
subset
of
our
module
installations,
but
users
can
also
build
their
own
kernels
or
their
own
kernels,
from,
for
example,
Conda
environments,
and
so
there's
a
link
to
the
docs
page
on
how
to
do
that
with
Jupiter.
There
are
a
lot
of
tools
that
you
can
you
can
access,
so
one
of
these
is
tensor
for
that
we're
stuff
I
mentioned.
So
this
is
really
the
de
facto
way
hate
to
visualize
everything
in
deep
learning.
B
You
can
see
someone's
stuff
on
the
bottom
right
here
from
your
your
models,
computation
graph
to
tracking
the
training,
loss
and
accuracy,
tracking
various
different
experiments
and
even
like
model
model
weights
evolving
over
time.
So
this
started
as
a
tensor
flow
tool,
but
has
since
become
natively
supported
in
pi
torch.
So
there's
a
lot
of
good
community
behind
this
and
you
can
actually
run
tensor
board
in
a
sort
of
manual
way.
B
Instead
of
SSH
tunneling
to
nurse
can
run
the
server,
but
we
recommend
actually
using
Jupiter
to
start
it
up
to
access
it,
and
we
just
there's
just
a
tiny
bit
of
boilerplate.
Basically,
we
provide
a
little
helper
module.
You
can
click
on
this
link
here,
it's
a
github
repo
with
a
single
Python
file
and
your
notebooks,
you
just
import
the
module,
startup
tensor
board
and
with
just
fine
down
here.
You
get
a
URL
that
you
click
on,
so
you
have
to
worry
about
the
the
other
stuff.
B
You
can
also
use
to
Pater's
for
as
a
front-end
for
going
distributive
learning
on
our
systems.
So
generally
Jupiter
notebook
is
running
a
single
kernel
process
like
Python.
That's
executing
your
cells,
but
there
are
these
tools
and
frameworks
out
there
for
distributed
parallel
computing
like
IFI,
parallel
ray
and
desk
that
are
able
to
spin
up
clusters
of
worker
processes,
and
then
you
can
connect
to
these
clusters
and
execute
distributed
workloads
so
for
deep
learning.
B
You
can
use
this
then
to
to
run
distributed,
training
or
distributed
hyper
parameter,
optimization
directly
from
one
notebook
that
might
be
even
running
just
on
CPU.
You
might
be
able
to
submit
it
to
the
GPU
systems
and
and
then
you
can
also
pull
in
all
this
extra
cool
ecosystem
stuff.
That's
in
Jupiter
like
interactive
widgets,
to
do
monitoring
and
and
steering
your
workflows.
So
this
is
a
little
while
ago
now,
I
think
2018,
but
we
put
together
some
some
notebook
examples
that
use
this
and
we
chose
I-5
parallel
and
kale
kale
with
kale.
B
We
worked
with
some
collaborators
in
CRD,
matt,
henderson
and
sri
australia,
but
for
for
a
parallel
first,
we
have
a
couple
notebooks
here,
demonstrating
how
you
can
do
distributed
training
and
HBO
optimization.
So
it's
a
cool
way
to
get
interactive
iterative
distributed
training.
Actually
we
show
that
it
scales
pretty
well
and
then,
with
with
this
stuff
from
koe
these
cool
interactive
monitoring,
which
it's
so
on
the
right
here.
This
is
a
live
widget
or
the
table
here
is
full
of
training
tasks.
B
B
We
support
a
number
of
hyper
parameter,
optimization
solutions,
so
some
of
these
we
prioritize,
but
others
folks-
can
just
sort
of
deploy
more
easily.
In
general,
though,
hyper
parameter,
optimization,
so
selecting
the
right
kind
of
model
architecture
and
tuning
the
hyper
branders,
it
can
be
very
critical
for
getting
the
most
out
of
deep
learning
on
your
problems.
B
You
know
we
don't
really
need
to
do
anything
special
for
we
chosen
to
have
we
chosen
to
look
at
it
too,
a
couple
of
tools
to
begin
with
and
have
prior
to
prioritize
support
for
because
they
map
well
onto
our
systems
and
have
a
nice
HPC
relevant
features
like
HBO
and
retune.
I'll
briefly
talk
about
those
shortly.
You
see
users
also
like
weights
and
biases.
That's
this
one
down
here
in
green.
B
If
there
are
folks
who
have
tried
other
solutions
and
deployed
nice
things
on
our
systems
for
HBO
we'd
like
to
hear
about
it
so
reach
out
to
us
so
first
cray
HBO,
so
crazy,
our
main
system
vendor.
So
they
made
this
tool
and
obviously
it's
designed
specifically
for
HPC
systems.
That
means
a
nurse
get
integrates
seamlessly
with
slurm,
so
it
actually
it's
not
going
to
spin
up
a
cluster
of
processes
that
then
it
sends
work
to
it's.
Just
gonna
use
slurm
directly
to
manage
your
allocations.
B
It
can
request
allocations
and
delete
them
and
it
can
launch
training
tasks
directly
with
the
slurm
commands.
So
I
have
some
popular
HBO
algorithms.
It
doesn't
it's
not
a
very
exhaustive
list,
but
I
know
they're,
adding
more
and
the
way
it
works.
Is
the
user
just
defines
a
training
script?
It
doesn't
need
to
be
even
machine
learning.
It
doesn't
need
to
be
Python
just
some
and
an
executable
that
can
be
steered
by
a
command-line
arguments
and
reports
back
a
figure
of
Merit.
So
it's
very
generic.
B
You
can
use
it
with
any
framework
and
yeah
because
it
uses
learn.
You
can
do
things
like
just
whatever
you
can
do
a
slurm,
so
you
can
say
each
training
cast.
Should
you
eight
GPUs
in
a
distributed
fashion,
or
something
like
this
so
been
Albrecht
was
an
engineer
at
Cray,
put
together
a
nice
lecture
and
some
hands-on
material
that
that
we
showed
at
the
deep
learning
for
science
school
last
year.
There's
some
links
to
the
material
down
below
for
Krejci.
Again,
here's
the
our
nurse
documentation,
the
Cray
documentation.
B
In
a
restricted
computing
budget,
the
plot
on
the
right
shows
one
of
our
nice,
a
for
Learning,
postdocs,
Brandon
wood,
comparing
random
search
to
some
of
these
approaches
that
can
actually
quit
training
models
early
based
on
kind
of
an
outlook
of
how
well
they're
going
to
do,
and
that
can
really
really
save
you
computing
resources.
So
there's
a
lot
of
cool
stuff
in
retune
and
also
integrates
with
other
solutions.
B
This
little
video
here
shows
Rae
tune
in
practice.
So
this
is
like
this
is
a
slurm
script.
We
can
set
up
the
software
that
the
code
below
is
what's
actually
starting
up
the
cluster,
it's
not
too
much
code,
but
then
you
you,
you
call
this
other
script
and
I
think
what
this
gonna
do
is
it's
it's
going
to
run
the
example
and
we
start
to
see
output.
It's
reporting
that
you
know
it
has.
B
Node
example:
so
it's
using
sixteen
GPUs
to
do
high
programmer
optimization
one
GPU
per
training,
starting
we're
gonna,
we
start
to
see
status.
All
these
pending
training
tasks
is
submitting
to
the
sturm
we're
starting
to
get
results,
because
this
is
very
fast
example.
All
these
results
for
training,
different
models
and
he
scrolls
up,
and
we
can
kind
of
see
an
updated
table
here
of
tasks
that
have
actually
been
terminated.
You
see
right
here.
B
These
are
ones
that
were
aborted
early
because
they
were
not
promising
and
we
can
monitor
things
like
the
current
accuracies
and
the
current
epochs.
So
a
lot
of
useful
stuff
there
gotta
go
back
to
full
screen
there.
We
go
okay,
now
I'll
get
into
some
of
the
performance
and
benchmarking
stuff
that
we
have
so
our
general
strategy
here
for
benchmarking
and
characterizing
their
system
and
software
performance
is.
B
We
have
a
mix
of
what
a
basic,
simple
benchmarks
that
might
come
from
like
the
framework,
official
benchmarks
or
official
examples
and
models,
as
well
as
more
holistic
applications
that
are
actually
like
science,
with
real
data
start
with
the
tensorflow
performance
on
CPUs.
So
here
we
take
benchmarks
from
this
repository
called
TF
CNN
benchmark,
so
it
has
some
optimized
vision
models
you
see
here
and
we
supplement
with
a
deep
convolutional
generative
model
here
and
so,
as
you
can
see,
we're
tracking
performance
of
these
models
across
tensorflow
versions.
We
follow
up
with
our
vendors
as
needed.
B
The
ones
on
the
right
show
to
holistic
science
application.
This
is
climate
and
cosmical
applications,
actually
our
benchmark
benchmarks
from
ml
curve,
which
I'll
mention
in
a
little
bit
for
scaling
here.
We're
demonstrating
this
with
hora
vaad,
which
is
the
recommended
solution
using
the
right,
cray
NPI
for
the
best
performance
and
it
scales
pretty
well.
B
Similarly,
on
GPU
we're
tracking
bottles
and
software
versions,
also
comparing
to
a
test
some
test,
GPU
boxes,
we've
had
and
then
on
the
right.
This
is
scaling
specific
model
inception
v3
on
Cori
GPU.
The
yellow
model
is
something
that
we
kind
of
had
at
first,
which
was
aided
setup.
So
we
this
was
actually
a
vanilla,
M
fish
MPI.
So
not
even
CUDA,
aware
I'm
going
to
see
how
well
that
scales,
and
then
we
switch
to
the
recommended
nickel
library
for
communication
from
video
and
the
importance
of
using
the
right
communication
back-end.
B
So
the
green
is
the
scaling
without
any
actual
data
I/o,
and
this
is
really
just
pushing
on
the
nickel
communication.
That's
doing
the
gradient
all
reduces
and
it's
near
perfect
and
then
the
blue
and
red
show
with
a
real
real
data
set.
That's
imagenet.
The
only
difference
is
that
the
blue
is
using
the
Nvidia
container,
so
containers
are
giving
us
basically
close
to
native
performance
and
sorry
to
bare
metal
performance
and
scalability,
which
is
good
to
see
for
pi
torch.
B
We
have
a
custom
repository
here,
that's
kind
of
orchestrating
things,
but
it
does
pull
official
models
from
torch
vision,
these
convolutional
network
models,
and
then
we
supplement
with
a
3d
convolutional
network
and
an
LS
TM
recurrent
Network.
This
code
base
is
using
just
synthetic
data.
It's
similar
to
that
TF,
CN
n
benchmarks
actually,
and
we
can
test
lots
of
things.
Hardware,
communication,
backends,
software
and
and
so
on.
The
table
on
the
right
here
shows
the
hardware
results
that
we
have.
B
Actually
their
software
here
so
first
performance
on
CPU,
where
it's
using
just
like
the
tensorflow,
Intel,
MKL
DNN
and
create
M
pitch
performance,
is
fairly
stable
across
versions
and
scaling
is
quite
good,
at
least
for
models
where
the
you
have
a
favorable
ratio
of
computation
to
communication,
so
Alex
in
that
year
is
bore,
but
that's
because
it's
a
really
big
model
and
doesn't
actually
have
many
layers,
a
ResNet,
and
these
others
are
looking
quite
good.
There
are
notebooks
down
at
the
bottom
to
see
the
full
results
we
doing
here
trying
to
wrap
up
pretty
soon.
B
This
is
the
first
performance
on
on
GPU
we're
here,
we're
using
then
recommended
nickel
back-end.
We
also
have
results
with
glue.
I
didn't
show
them,
but
they
they
don't
scale
as
well,
but
on
GPU
versions.
Again
performance
is
pretty
stable.
The
last
two
things
here
are
actually
using
nvidia
containers.
B
They're
mislabeled
so
ignore
the
label
but
they're
fairly,
recent
versions,
containers
and
then
scalability
here
is-
is
pretty
good
for
for
most
models,
so
nice
to
see
things
are
mostly
working,
so
those
were
what
I
refer
to
as
the
basic
benchmarks,
but
also
we
want
to
do
more
holistic,
real,
like
end
to
end
kinds
of
benchmarks,
so
we're
doing
this
partially
within
and
they'll
perf,
so
ml
birth
is
an
organization
and
set
of
benchmarks.
Sweets
for
machine
learning
basically
has
its
it's
driven
by
all
the
major
tech
companies
is
very
strong
involvement
from
industry.
B
You
know
Google
Intel
and
video
all
them
and
there's
also
a
bunch
of
academic
institutions
and
national
labs
involved.
It
has
industry
standard
benchmarks
for
training
and
inference
so
industry
standard
I
really
mean
like
the
applications
that
are
kind
of
the
canonical
machine
learning
applications
like
image,
classification
and
segmentation,
language
recommendation
and
that
kind
of
thing,
but
they
do
have
a
holistic
approach.
So
it's
really
about
time
to
solution
for
training
or
the
throughput.
B
You
can
get
an
inference
so
in
a
mapper
if
we
started
an
HPC
working
group
to
try
and
develop
and
smart
specifically
for
supercomputers,
we
have
a
Google
Group
here
mailing
list
you
can
join.
It
already
has
over
a
hundred
members
from
industry
and
labs
and
stuff
like
this.
These
are
the
co-chairs
and
deputy
chairs
on
the
bottommost,
so
we
have
an
article
that
Murali
here
wrote
on
inside
HPC
and
give
you
a
lot
more
background
info
and
cover
some
of
the
outreach
and
preliminary
results.
We
have
please
reach
out
to
us.
B
If
you
are
interested
in
getting
involved
in
this
effort,
so
I'll
just
very
quickly
go
through
the
rest.
You
don't
need
to
go
through
all
the
detail,
but
you
know
the
motivation
here
is:
we
need
good
benchmarks
for
HPC
and
we
want
to
capture
the
relevant
characteristics
so
much
marks
that
really
push
on
the
system
in
all
the
ways
that
are
relevant
to
that
big
scale,
computation
and
communication
and
I/o.
B
We
want
to
collect
all
these
relevant
metrics
beyond
just
time
the
solution
there
a
number
of
challenges,
though
it's
designing
a
benchmark
suite
at
this
scale,
they're.
Actually,
not
a
lot
of
standard
applications,
so
we're
kind
of
taking
what
has
been
kind
of
published
and
use
like
SC
large
batch
training
is
in
general,
difficult,
don't
have
time
to
go
into
all
those
details.
We
have
a
whole
tutorial
on
that
and
tuning
these
benchmarks
at
scale,
because
we
have
to
kind
of
restrict
the
rules
and
hyper
parameters
that
submitters
can
use.
B
This
is
very
expensive,
so
this
is
an
on
go
process
and
we'll
go
into
details
about
the
show
we're
using
climate
segmentation,
which
is
a
Gordon
Bell,
prize-winning
application,
SC
18
and
this
cosmic
flow
application,
which
also
came
out
of
nearest
folks
and
was
at
SC
18.
So
those
are
the
two
or
we're
starting
with
you
can
refer
to
the
slides
for
more
details
or
the
papers,
we're
hoping
to
do
an
actual
benchmark
release
this
year.
So
there's
ongoing
work
to
kind
of
finalize
the
rules,
the
benchmark,
details
and
open
up
a
submission
period.
B
So
we
hope
to
get
a
lot
of
HPC
facilities
to
participate
and
run
the
benchmarks
and
submit
results,
and
we
ideally
like
to
have
results
publicly
available
around
the
timeframe
of
SC
20,
but
we
will
have
to
see
and
we're
looking
at
other
other
benchmarks
as
well.
I
want
I,
won't
list
them,
but
things
that
push
on,
for
example,
model
parallelism
and
stuff
like
that.
B
Okay,
shifting
into
the
last
session
of
section
of
the
talk,
I'm
not
going
to
go
through
this
in
detail,
but
just
I
wanted
to
call
out
with
a
big
slide
showing
that
we
are
involved
through
science,
engagements
and
research
projects
with
a
lot
of
different,
basically
scientific,
deep
learning
applications.
So
you
can
you
can
check
this
out,
and
this
is
not
an
exhaustive
list
at
all.
There's
a
lot
of
things
that
we
we
are
involved
in.
B
We
do
training
events,
so
the
big
one
I
want
to
call
out
is
the
deep
learning
for
science
school
at
Berkeley
Lab.
Last
year
we
had
a
very
comprehensive
program,
lectures
demos,
hands-on.
You
can
do
all
the
material
from
last
year
at
this
website,
including
the
videos
we
were
going
to
do
it
again
this
year,
but
due
to
the
situation,
we're
all
aware
of
the
in-person
event
was
canceled,
but
instead
we
do
hope
to
do
a
summer
webinar
series.
B
We
have
this
deep
learning
a
scale
tutorial
that
we've
been
showing
at
a
bunch
of
different
conferences
organized
during
the
Cray
but
also
nvidia
and
intel
has
been
involved,
presented
it
at
multiple
SCS
at
ISC.
The
ECP
meeting
we're
submitting
again
to
see
this
year
and
you
can
check
the
material,
the
latest
material
from
su
19
there
and
then
one
last
advertisement
is
the
this
data
seminar
series.
Basically,
so
this
github
page,
we
should
be,
should
be
kept
up
to
date
on
upcoming
and
things
upcoming
seminars,
and
it
will
have
videos.
B
So
in
conclusion,
we
don't
say
deep:
ler
is
coming,
but
really
we
say
deep
learning
for
sciences
here
and
it's
growing.
So
there
are
clearly
a
lot
of
powerful
capabilities
there.
We
have
a
very
enthusiastic
community
and
growing
interest
in
actual
workloads
targeting
HPC,
which
is
great,
so
hopefully,
we've
convinced
you.
The
nurse
guys
are
productive
and
performance.
Software
stack
for
deep
learning.
We
have
our
optimized
frameworks
and
solutions
that
are
scalable
and
also
these
tools
and
things
that
support
productive
workflows.
Please
join
the
nurse
user
slack.
We
do
have
a
machine
learning
channel
there.
B
We
may
use
that
to
make
announcements.
It's
a
good
place
to
ask
questions
and
stuff
like
that.
Just
build
the
community
and
we've
shown
a
lot
of
things
from
our
user
survey,
but
hasn't
the
stuff
I
said
it's
actually
still
open.
So
please
take
some
time
if
you're
doing
machine
learning
to
fill
this
out.
This
can
help
inform
our
strategy
and
make
sure
we
have
the
best
systems
for
all
of
you
and
with
that
then
I'll
conclude.
Thank
you.