►
From YouTube: 02 Intro to Perlmutter and GPUs
Description
Part of the Migrating from Cori to Perlmutter Training, December 1, 2022.
Please see https://www.nersc.gov/users/training/events/migrating-from-cori-to-perlmutter-training-dec2022/ for the training day agenda and presentation slides.
A
Hi
everybody
I
am
the
application
performance
lead
at
nurse
and
so
I'm,
going
to
kind
of
give
you
an
introduction
to
the
promoter
system
a
little
bit
about
programming
gpus
and
like
the
different
programming
models
and
languages,
libraries
Frameworks
that
are
available
on
promoter
and
I.
Think
we'll
close
here
with
just
a
few
early
science
stories
to
maybe
motivate
the
the
audience
with
what
what
can
and
kind
of
has
been
done
on
the
on
the
system.
A
A
So
this
is
the
nurse
system
roadmap,
and
one
of
the
things
I
want
to
highlight
here
is
that
we're
we're
kind
of
in
this
transition
from
what
was
kind
of
pretty
typical
HPC
systems
a
decade
ago,
with
Edison
being
a
kind
of
a
multi-core
cpu-based
system
replicated
across
you
know
thousands
of
nodes
to
to
make
an
HPC
an
HPC
architecture.
We've
started
this
transition
towards
sort
of
energy
efficient,
exascale
like
architectures.
In
order
to
kind
of
Meet
the
demands
of
the
community.
A
We
started
that
transition
with
Corey,
which
was
powered
by
Intel
Xeon,
5
processors
and
then
with
Pro
Mudder.
We
have
our
first
ever
GPU
accelerated
system
and
we're
thinking
we're
kind
of
already
beginning
the
process
to
procure
the
nurse
10
system,
which
I
think
is
expected
in,
like
the
2025-2026
2026
time
frame.
There's
not
much.
We
can
say
about
that,
but
I
think
we're
expecting
this
trend
towards
these
energy
efficient
architectures
to
continue.
A
So,
if
you
look
at
Pearl
Mudder,
we
have
two
types
of
nodes:
we
have
first,
the
Nvidia
amdia
ampere
I
should
say
GPU
powered
nodes,
Each
of
which
has
four
gpus
and
one
CPU
per
per
node,
and
they
have
a
tremendous
amount
of
performance
in
the
in
the
node.
You
can
see
here
over
75,
I,
guess,
I,
guess
that
says
teraflops
but
I
think
that's
a
lower
number.
A
A
A
If
you
include
the
the
power
and
capability
of
the
tensor
cores
within
the
within
the
gpus
compared
to
about
you,
know
close
to
I,
guess
eight
petaflops
for
the
for
the
CPU
nodes.
This
top
row
here,
where
my
mouse
is,
is
showing
the
performance
of
the
CPUs
that
are
within
the
GPU
node.
So
if
you
ignore
the
gpus,
you
have
another
a
little
bit
of
a
about
about
four,
more
petaflops
of
performance.
A
It's
all
downstairs
in
the
building
that
I'm
talking
to
you
from,
and
you
can
find
even
more
details
about
the
the
architecture
and
the
the
different
components
at
this
URL
here
I'm
going
to
talk
a
little
bit
more
just
about
the
different
about
the
two
types
of
nodes.
So,
as
I
mentioned,
most
of
the
performance
on
the
system
comes
from
the
GPU
notes
and
I
mentioned
that
you
have
one
AMD
Milan
processor.
So
that's
pictured
here
with
four
Nvidia
a100
gpus.
Those
are
the
four
ampere
gpus
pictured
here.
A
Each
one
of
those
is
a
64
core
part.
They
support
AVX,
2
instructions
similar,
but
not
quite
as
high
of
a
of
a
vector
width,
as
you
had
on
the
Intel
Xeon
5
processor
for
Corey,
and
they
do
have
relatively
High
memory
bandwidth
for
C
for
c
for
CPU.
You
see
that
you
have
204
gigabytes
per
second
memory
bandwidth
but
of
course
that's
significantly
lower
than
that
memory,
bandwidth
that
you
saw
on
the
on
the
gpus.
A
So
we
we're
moving
up
significantly
in
the
total
capability
of
the
system.
Significantly
more
memory,
one
of
the
real
big
differences
is
the
performance
per
node
has
gone
up
dramatically
on
promoter
compared
to
Corey.
These
nodes
are
much
much
more
dense
in
terms
of
their
compute
power.
You
see
from
about
three
teraflops
to
70
teraflops,
the
the
no
performance
or
the
the
processors
now
include
the
accelerator.
A
The
number
of
nodes
is
actually
shrunk,
I
think
and
that's
consistent
with
the
the
nodes
themselves
being
more
powerful,
and
the
other
thing
I
would
highlight
is
this
old
flash
file
system
I?
Think,
actually
is
one
of
the
things
that
makes
running
on
promoter.
Even
you
know,
perhaps
eat
a
little
bit
easier
than
than
it
was
on
on
Corey.
A
So
we
we
kind
of
started
this
process
of
when
you,
when
we
were
thinking
about
kind
of
procuring
a
GPU
system.
We
were
looking
at
our
workload
and
and
asking
ourselves
what
fraction
of
the
workload
could
really
take
advantage
of
the
gpus,
and
this
is
where
things
sort
of
stood
in
the
2017-2018
time
frame
and
the
good
news
is
because
gpus
had
been
around
at
a
number
of
different
facilities
and
have
been
used
different
places
throughout
the
world.
A
The
number
of
codes
had
a
GPU
versions
available
already,
but
a
number
of
applications
we're
also
kind
of
only
partly
ported
or
hadn't,
even
even
kind
of
started
in
in
some
cases,
and
so
we've
been
working
with
these
code
teams.
You
know
with
a
number
of
code
teams
over
the
last,
you
know
five
five
five
years
or
so
to
make
sure
that
they
would
be
ready
for
for
promoter,
and
you
know
some
of
what
I
want
to
tell
you
about.
A
One
of
the
ways
to
think
about
this
is
through
this
analogy
that
I'm,
showing
on
this
slide
that
a
you
know
a
CPU
is
something
like
a
Ferrari
I
think
that
I
think
this
is
supposed
to
be
a
Ferrari
on
this
on
this
picture
and
that's
like
it's
a
car
that
it
can
go
really
fast.
Make
really
tight
turns,
but
is
really
good
for
kind
of
doing
one
one
task
at
a
time
or
taking
one
person
to
kind
of
one
place
at
a
time
where
a
GPU
is
something
like.
A
You
know
this
double
decker
bus,
which
is
kind
of
good
at
going
taking
a
lot
a
lot
of
different
people
to
one
place,
not
as
fast
as
the
CPU
would
take
an
individual,
but
with
a
higher
overall
through
throughput,
and
this
is
kind
of
evident
if
you
compare
the
amount
of
parallelism
that
is
available
in
a
CPU
from
our
Quarry
system.
So
this
is
the
the
Corey
Haswell
partition
compared
to
a
GPU
on
Chrome
letter.
A
And
if
you
were
to
use
those
avx2
instructions,
you
can
compute
on
two
by
256
bit
vectors
at
a
time,
so
that
all
adds
up
to
about
2
000
way,
parallelism
that
a
Haswell,
CPU
node
is
sort
of
capable
of.
If
you
compare
that
to
a
single
a100
GPU,
the
equivalent
of
the
64
cores
is
approx
is
is
basically
these
108
SMS
or
what
they
call
streaming
multi-processors
on
a
GPU
and
each
one
of
those
can
support
up
to
64.
A
So
another
big
difference
between
a
the
CPU
and
the
GPU
is
the
memory
bandwidth.
So
if
you,
if
you
look
at
the
Haswell
CPU
you,
we
had
128
gigabytes
of
DDR
per
node
and
approximately
128
gigabytes
per
second
of
memory,
bandwidth
on
that
Haswell
CPU
node.
Now,
if
you
compare
that
to
a
single
a100
GPU
again,
we
have
40
gigabytes
of
high
band
with
memory,
but
significantly
higher
memory.
Bandwidth
so
1500
gigabytes
a
second,
so
an
increase
in
an
order
of
magnitude.
A
Again
now,
as
I
was
highlighting
when
I
was
kind
of
talking
about
the
architecture,
this
should
be
compared
against
the
speed
of
moving
data
between
the
CPU
and
GPU
across
the
PCI
Express
Bus,
and
that
is
about
32
gigabytes
a
second,
and
so
you
can
see
you
can
move
data
within
the
GPU
very,
very
fast,
but
moving
data
across
the
CPU
to
GPU
can
be
very
slow.
So
the
lesson
learned
here
is
to
try
to
avoid
moving
data
back
and
forth
frequently
and
in
general.
A
This
I,
just
wanted
to
kind
of
highlight,
was
an
all
hands
on
deck
activity.
A
number
of
the
people
who
you
see
here
will
be
kind
of
available
in
the
afternoon
to
work
with
you
and
to
kind
of
talk
with
you
about
your
own
applications,
and
one
of
the
things
I
really
want
to
highlight
here
is
that
these
hackathons
have
been
really
effective
in
kind
of
helping
improve
various
code
teams
around
the
country.
A
That
I,
don't
think,
really
could
have
been
done
without
the
without
this
new
system
itself
and
the
work
that
the
team
put
into
the
to
the
to
the
project
so
yeah,
so
the
one
you
know,
one
of
the
other
things
I
wanted
to
highlight
is
that
we've
been
working
with
teams
to
do
really
large
scale.
Science
runs
on
Pro,
Mudder
and
related
GPU
systems.
Over
the
last.
A
So
just
some
observations
about
this
process.
This
is
I
think
that
you
know
many
applications
have
been
successful
in
preparing
for
prom
letter
and
we'd
really
like
to
keep
engaging
with
with
you
all
in
the
community
to
enable
you
all
to
basically
use
the
system
productively,
and
we
really
do
encourage
everyone
to
join
these
Community
hackathons,
GPU,
hackathons.org
I
think
that's
just
a
great
way
to
to
get
a
lot
done
done
quickly,
and
we
do
kind
of
recognize
that
they're,
you
know
moving
you
know
optimizing.
A
The
the
other
thing
that
I
would
just
note
here
at
the
end
is
that
I
I
really
think
that
we've
seen
a
lot
of
energy
coming
from
you
all.
The
community,
who
is
really
excited
about
the
potential
of
of
pro
Mudder
and
I,
think
that's
really
great
to
see,
and
it's
something
that
we
are
really
really
excited
about
as
well,
so
I
want
to
now
change
so
I
think
I'm
kind
of
moving
into
the
second
part
of
this.
A
A
We
also
have
a
pretty
robust
programming
environment
around
data
and
Analytics
Helen
had
highlighted
using
Jupiter
to
log
on
to
the
system
and,
of
course,
Python
and
the
Nvidia
rapid
stack
are
supported,
and
you
know
pytorch
and
tensorflow.
Two
of
the
the
the
more
important
machine
learning
Frameworks
are
also
really
well
optimized
for
the
system.
We
have
a
set
of
debuggers
and
profilers
that
I
highlight
here
in
particular,
including
the
insight
profiler
that
I
that
I
talked
about
earlier
in
regards
to
the
roof
line
performance
modeling.
A
A
You
know
4.5
5.x
support
as
well
as
things
like
Coco
switch,
are
big
Investments
within
the
doe
to
allow
C
plus
applications
to
run
on
CPUs
and
gpus,
from
multiple
vendors
and
and
also
into
the
into
the
future.
So
our
strategy
has
really
been,
as
I
said,
to
kind
of
support,
all
those
major
programming
models
and
languages
to
also
pre-install
optimized
versions
of
many
of
your
kind
of
favorite
applications.
A
You
know,
particularly
in
the
material
science
and
chemistry
space.
There
are
a
lot
of
shared
applications,
codes
like
vasp
and
lamps,
which
we
talked
about
and
Quantum
espresso,
as
well
as
working
with
the
vendors
to
kind
of
make
the
process
of
understanding
your
performance
on
gpus,
something
that
is
a
little
bit
more
tractable,
and
you
know
I
just
want
to
highlight
again
how
I
think
how
useful
these
GPU
hackathons
can
be.
A
If
you
go
to
GPU
hackathons.org
again,
you
can
register
for
an
upcoming
an
upcoming
event,
so
we
in
particular,
have
kind
of
invested
some
of
our
own
time
and
resources
into
enabling
a
couple
performance.
Portable
programming
models,
the
one
that
I
really
want
to
highlight
here
is
openmp
4.5
and
5.x,
which
has
gained
accelerator
support
in
the
Nvidia
HPC
compiler
stack
because
of
nurse
collaboration
with
with
Nvidia.
A
A
This
is
a
material
science
application
where
the
goal
here
is
to
kind
of
help,
design
potential
future
qubits
for
like
a
Quantum
quantum
computer
in
complex
materials
that
have
some
sort
of
defect
in
them.
So
in
this,
in
this
case
the
defect
is,
what's
called
a
die
vacancy,
so
essentially,
two
atoms
are
removed
from
the
the
Crystal
and
to
understand
the
kind
of
quantum
states
around
that
that
defect,
and
so
to
do
that
they
needed
this.
A
You
can
see
kind
of
performance
improvements
as
the
gpus
have
evolved
into
the
final
GPU
part
for
for
promoter,
and
you
can
see
the
that
the
scaling
to
essentially
a
full
GPU
system,
like
like
Pearl
Mudder
in
the
in
in
the
plot
above
another
example,
is
exobiome.
So
this
is
a
metagenomics
code
where
they
basically
take
and
analyze.
A
And
you
can
see
that
even
with
a
set
of
work
that
isn't
kind
of
immediately
amenable
to
gpus.
They've
been
able
to
make
significant
improvements.
And
what
you're
seeing
here
is
a
comparison
of
the
CPU
versus
the
GPU
performance
on
on
promoter
for
for
their
particular
algorithms
and
I.
Think
maybe,
let's
see
if
I'm
just
going
to
highlight,
maybe
one
more
here,
which
is
some
of
the
early
successes
we've
had
working
with
different
facilities,
in
particular
some
of
the
light
sources
and
the
kind
of
astrophysics
or
observational
facilities.