►
From YouTube: Using GPUs as Accelerators
Description
Max Katz from NVIDIA presents a talk on Using GPUs as Accelerators. Recorded live via Zoom at GPUs for Science 2020. https://www.nersc.gov/users/training/gpus-for-science/gpus-for-science-2020/ Session Chair: OisÃn Creaner
A
That's
a
synonym
for
saying
that
they're
optimized
for
serial
tasks
that
they
have
very
large
amounts
of
memory.
They
have
very
high
clock
speeds
and
they
have
very
large
caches
which
help
them
to
reduce
latency,
and
so,
when
you're
trying
to
access
some
data
from
memory,
for
example,
cpus
are
very
good
at
ensuring
that
the
time
it
takes
to
access
that
memory
is
as
small
as
possible,
but
they're
not
good
at
everything,
so
they
have
relatively
low
memory
bandwidth
and
if
the
cpu
gets
it
wrong.
A
If,
if
the
data
that
you
were
trying
to
access
is
not
immediately
available,
then
that
is
costly
and
in
terms
of
energy
efficiency.
They
are
relatively
low
efficiency,
if
you
think
about
it
in
terms
of
a
number
of
floating
point
operations
per
watt,
say,
conversely,
gpus
exist
to
hide
latency,
and
so
they
have
very
high
bandwidth
memory
but
low
capacity
memory.
A
But
on
the
other
hand
they
have
very
high
amount
of
compute
resources
and
they
are
high
latency
in
their
access.
So
when
you
have
one
thread
accessing
a
particular
date
item
of
data
from
memory
that
takes
a
long
time,
but
the
way
that
we
help
avoid
that
problem
is
by
having
many
threads.
And
so
while
one
thread
is
trying
to
do
something,
and
it
takes
some
time
to
fulfill
that
request.
A
We
have
many
threads
that
are
doing
something
in
order
to
hide
the
latency
of
that
request,
and
so
their
individual
performance
of
any
one
thread
is
relatively
low
compared
to
a
cpu
thread.
However,
by
combining
many
threads
in
parallel,
we
can
solve
problems
effectively
and
in
terms
of
energy
efficiency.
Gpus
are
relatively
high
efficiency,
which
kind
of
goes
hand
in
hand
with
the
fact
that
the
individual
cores
that
make
up
a
gpu
are
relatively
streamlined.
They
don't
do
a
whole
lot,
but
the
things
that
they
do
they
do
well,
when
combined
in
parallel.
A
So
gpu
acceleration
is
the
task
of
taking
your
application
code
and
then
identifying
the
parts
of
it
that
can
run
on
a
gpu
and
putting
those
parts
in
the
gpu
and
then
leaving
the
rest
on
the
cpu
and,
in
terms
of
say
the
number
of
lines
of
code
you
might
put
a
relatively
small
fraction
on
the
gpu.
It
really
depends
on
what
your
application
looks
like,
but
it
might
be
just
a
few
percent
of
the
lines
of
code
that
make
up
the
time
that
it
takes
to
dominate
the
runtime
of
your
application.
A
I
want
to
tell
you
a
little
bit
about
the
gpu
architecture,
because
I
think
that
will
help
make
sense
of
this
claim,
I'm
making
about
gpus
being
massively
parallel
devices,
but
also
ones
that
really
require
a
massive
parallelism
to
succeed.
So
jay
showed
you
a
a
an
image
of
what
the
chip
looks
like
from
the
outside.
This
is
kind
of
an
inside
view
of
the
latest
nvidia
gpu,
the
a100
gpu.
A
A
That's
certainly
true
for
the
the
high-end
cpu
nodes
that
we
have
see
running
today.
They
run
like
10
to
100
cores,
by
contrast,
gpus
have
thousands
of
cores
and
that
allows
them
to
solve
problems
massively
in
parallel,
but
the
catch
and
I'll
show
you
this
in
a
moment.
Is
that
again
those
cores
don't
solve
every
problem.
A
A
Gpus
is
that
they
have
a
very
large
amount
of
floating
point
and
integer
units
and
so
they're
very
good
at
floating
point
integer
math,
but
they're,
not
so
good.
At
other
problems
like
that,
don't
involve
math.
They
additionally
have
these
tensor
cores,
which
are
well
optimized
for
solving
matrix
multiplication
problems
and
they
have
a
relatively
small
amount
of
cache
memory,
and
so,
while
gpu
cpus
may
have
something
on
the
order
of
you
know
hundreds
of
megabytes
for
caches,
you
typically
have
much
smaller
amounts
for
gpus.
A
So
if
you
take
one
of
these
sm
which
are
kind
of
the
building
blocks
of
a
gpu
and
then
you
kind
of
lay
them
out
horizontally
like
this,
they
have
different
memory
levels
in
their
hierarchy,
so
they
have
registers
which
are
the
actual
source
and
destination
of
the
computation
on
the
gpu.
These
are
on
the
on
the
actual
chip
and
then
those
these
are
what
the
floating
point
units
and
integer
units
use
to
do
their
math.
So
those
are
the
high
bandwidth
and
very
low
latency
access
memory.
A
You
also
have
on
these
sms
or
or
individual
cores.
If
you
like
to
think
about
them
at
l1
cache,
which
has
the
the
close
cache
to
the
to
the
chip,
then
you
have
a
device-wide,
l2
cache
and
then
a
main
global
memory,
and
so,
if
you
then
lay
those
out
from
closest
to
the
chip
and
therefore
highest
bandwidth,
but
also
smallest
in
capacity,
you
have
registers
which
have
only
a
few
tens
of
kilobytes
per
sm
or
core,
but
can
be
accessed
at
much
greater
than
one
terabyte,
a
second
worth
of
bandwidth.
A
And
then
a
similar
thing
is
true
for
this
l1
cache
that
it
lives
on
each
one
of
these
sms
or
cores.
If
you
want
to
think
about
it
that
way
again
like
100
kilobytes
per
sm,
but
can
be
accessed
at
more
than
10
terabytes.
A
second,
the
l2
cache
has
a
size
of
40
megabytes
on
an
a100
gpu,
an
fba
100
gpu.
A
A
A
So
then.
In
summary,
your
main
priority
is
to
make
the
code
faster
and
that
doesn't
always
mean
making
the
part.
That's
on
the
gpu
run
faster,
really.
Sometimes
it
means
optimizing
data
transfer
back
and
forth
to
the
gpu
because
they
have
their
own
separate
memory
spaces
and
sometimes
it
just
means
refactoring
your
cpu
code
to
expose
parallelism
a
lot
of
codes
that
exist
today
before
they
start
according
to
gpus,
don't
have
their
parallelism
exposed.
A
Naturally,
you
have
to
spend
a
lot
of
time,
or
at
least
some
time
making
that
true,
and
so
that's
a
big
part
of
the
gpu
reporting
process
and
so
don't
go
in
and
start
writing
cuda
code
or
open
ac
code
or
anything
like
that
to
identify
what
are
the
parts
of
your
application.
That
could
benefit
from
parallelism
and
is
it
exposed
in
that
way,
and
you
should
always
use
profiling
tools
to
help
you
with
this.
A
It
is
generally
the
case
that
you
will
hear
lots
of
received
wisdom
about
what
works
and
what
doesn't
on
gpus,
and
I
encourage
you
not
to
to
try
to
live
by
that
wisdom,
but
instead
experiment
with
yourself
and
then
use
tools
like
profiling
tools
to
help
you
understand
whether
you're
succeeding
or
not.
So
with
that
I'd
like
to
thank
you
and
I'm
open
for
any
questions.
B
B
B
A
A
That
means
your
problem
needs
to
be
able
to
expose
hundreds
of
thousands
of
independent
pieces
of
work
that
can
be
solved
simultaneously,
and
so,
if
your
problem
just
doesn't
express
itself
in
that
way,
like
you're
solving,
you
know,
your
main
application
is
solving
matrix
multipliers
that
are
like
10
by
10
and
you're
only
doing
one
of
them
at
a
time.
That's
not
appropriate
for
running
on
a
gpu
it'll
actually
be
faster
to
run
that
on
a
cpu.
A
Now,
sometimes
you
can
express
your
code
in
a
way
where
you
can
batch
operations
so
that
many
are
being
assaulted
once
and
then
maybe
you
have
some
chance
of
using
the
gpu
effectively,
but
really
it's
the.
The
main
pitfall
is
when
your
code
just
doesn't
have
enough
work
to
do
to
saturate
the
gpu
and
then
you'll
actually
end
up
being
worse
than
if
you
just
used
the
cpu
alone.
B
A
Gpu
profilers,
this
will
depend
on
which
profiling
tool
you
use.
The
nvidia
provided
profiling
tools
certainly
have
the
capability
to
work
when
using
multiple
mpi
tasks
per
gpu.
They
don't
the
ones
that
are
currently
available.
Don't
have
the
the
greatest
ability
to
combine
that
in
an
mpi
aware
way
to
show
you,
for
example,
multiple
ranks
running
at
the
same
time,
but
they
certainly
can
run
and
you
can
analyze
e-rank
independently,
and
if
you
look
broader
than
the
nvidia
provided
products,
that's
something
we're
working
on
that.
A
B
B
A
You're
using
and
so,
for
example,
it'll,
look
a
little
bit
different
from
on
the
pro
motor
motherboards
than
on
other
systems.
Where
there's
you
know
more
or
fewer
gpus
per
node,
but
that's
customer
gearbox
is
the
total
bi-directional
throughput
from
any
one
gpu
to
all
of
the
other
gpus
in
the
system
and
then
just
gets
divided
across
the
other
gpus.
A
I
can't
really
talk
about
their
energy
cost.
It's
not
really
something
I'm
prepared
to
meaningfully
speak
about,
but
this
will
be
essentially
the
the
default
way
that
you
communicate
between
gpus
on
on
promutter,
and
so
it's
something
to
be
aware
of
that.
You
have
much
higher
bandwidth
than
you
might
be
familiar
with
using
standard
pcie.