►
From YouTube: 06 Debugging and Profiling Tools
Description
Part of the NERSC New User Training on June 16, 2020.
Please see https://www.nersc.gov/users/training/events/new-user-training-june-16-2020/ for the training day agenda and presentation slides.
A
A
Dislike
the
first
part
is
debugging
tools,
so
why
do
you
use
the
debuggers
because
we
want
to
find
the
other
programming
errors?
You
know
that
can
crash
your
code,
and
sometimes
you
can
make
the
program
hangs
and
get
generally
wrong
results.
How
do
you
use
early,
find
and
I
find
them
and
fix
them?
The
user
way
is
to
use
the
print
statement.
A
In
order
to
guess,
you
know,
extract
some
sort
of
hint,
and
if
that
was
not
successful,
you
would
you
add
another
print
statement,
something
else
and
you
kind
of
get
it
this.
The
problem
with
this
is
even
though
this
is
kind
of
easy
way
to
try
to
solve
the
problem.
It's
very
difficult
to
guess
what
is
wrong
and
the
water
pinch,
so
it
is
really
time
consuming
and
he's
13
years
and
really
severe
in
frustrating.
A
Also,
if
we
print
a
lot
of
lot
of
stuff
out
there-
and
it's
very
easy
to
you-
know
the
to
see
that
what
is
wrong
from
the
printed
text-
values
yeah.
So
this
is
why
we
are
using
debuggers
70
debugging
workflow.
Is
you
kind
of
build
your
program
and
the
side,
your
program
on
the
debugger
and
oh?
And
when
you
started,
you
kind
of
you
know,
set
up
places
where
the
programming
should
stop
for
a
moment.
A
A
Of
investigate
around
that
there
yeah
and
if
we
don't
get
any
any
hint
from
there,
then
you
can
define
a
different
break
points
and
you
can
continue
this
run
on
the
debugger.
So
it's
a
lot
easier
because
you
don't
have
to
compile
every
time
you
just
compile
only
once
and
they
saved
a
lot
of
effort
actually
and
you
can
actually
control
the
program,
executions,
you
can
stop
and
you
can
continue,
etc
and
also
the
tool
has
a
lot
of
you
know
the
features
so
that
you
can.
You
can
print
visualize.
A
You
can
actually
plot
the
values
using
the
visualization
tool
of
the
the
debugger
so
that
you
can
actually
visually
check
quickly
check
whether
something
might
have
gone
wrong
here.
So
on
quarry
we
have
a
several
kinds
of
debuggers,
the
two
types
that
parallel
the
traditional
the
cui
tools
called
the
DDT.
In
the
total
view,
they
are
kind
of
in
a
main
debuggers,
but
also
we
have
a
kind
of
speciality
Pagar
selects
that
ATP
very
drying
and
in
the
internal
specter
and
the
stead
and
ATP
are
white,
sometimes
useful
to
debug
hanging
programs.
A
If
you
program
hands
for
some
reason,
then
you
want
to
find
out
where
the
program
hangs
here
so
for
that
purpose,
stat
and
ATP
are
very
powerful,
very
simple
and
very
fast.
The
background,
as
you
know,
that
that
is
a
suite
of
debugging.
The
profiling
tools,
but
mostly
people
know
know
that
by
is
memory
debugging
too
many
tracks.
So
you
see
you
kind
of
use
it.
You
can
show
that
a
lot
of
the
memory
lately
messages
from
there.
You
can
actually
see
the
where
there
is
a
memory
leaks
over
the
memory
errors
here.
A
So
let's
go
over,
did
it
and
to
review.
That
is
a
as
I
said,
that
is
we
base
traditional
already
bombers.
You
can
use
it
with
the
c
c++
Fortran
mpi
OpenMP
pieces
acerola.
So
we
have
a
quite
a
large
number
of
by
the
licensed
seats
4096
for
DDT
and
for
doctoral
P.
We
have
a
small,
smaller
license
512,
but
you
know
these
tools
are
complimentary,
so
that
if
you
cannot
find
the
error,
neither
one
one
tool
you
can
try
a
different
one
and
that
that
I
was
able
to
do
that.
A
You
know
certain
situation,
so
it's
a
kind
of
you
know
good
to
have
two
kinds
of
better
the
deeper
works
here,
so
the
how
to
build
and
run
with
a
DDP.
So
you
have
to
use
compartments,
G
flag,
to
add
the
debugging
symbols
and
you
do
not
include
any
optimization
in
your
code
so
that
the
the
the
light
come
flow
is
very
similar
to
the
what
you're,
what
you
write
down
in
your
program
so
use
a
minus
or
0
for
no
optimization.
A
You
build
it
and
when
you
study
interactive
patches
are-
and
you
know
that
in
a
linear
for
2
module,
and
then
you
start
your
application
with
activity
like
that
and
there
you
set
the
are
the
wrong
parameters,
and
so
as
Steve
Steve
mentioned
earlier.
So
if
you
run,
if
you
try
to
run
this
GUI
tool,
far
away
from
nurse
there,
so
it's
really
painful.
A
You
know
the
because
when
you
click
it
and
the
response
will
come
out
in
a
few
minutes
later,
so
that
is
because
of
the
intrinsically
high
latency
associated
with
the
x11
client
and
the
Sauber
communication
here.
So,
as
is
that
that
the
one
one
way
to
do
it
to
deal
with
it
is
to
use
no
machine
NX
now
another
way
is
to
use
the
arms
for
Janee
mode
client.
A
So
this
is
kind
of
popular
way,
so
I
kind
of
showed
in
the
web
page
how
to
set
it
up,
how
to
download
it
and
how
to
set
up
the
remote
client.
Some
people
prefer
that
and
basically
the
the
setup
is
explained
in
this
webpage
here.
Basically,
you
need
to
set
up
the
the
hostname
using
the
the
login
node
and
the
map
node
and
the
locations
where
the
program
is
in
Seoul,
and
this
is
probably
not
much
really
yeah.
This
is
kind
of
this
is
the
command
but
to
load
up
the
linear
40
module.
A
And
also,
on
the
right
hand,
side
there's
a
you
show
us
the
the
the
values
of
a
local
variables
here
and,
and
also
you
showed
us
some
sort
of
the
lines
here
which
are
called
the
spark
lines.
This
is
the
the
variable
values
across
the
entire
X
here,
so
this
flat
line
means
that
this
variable
value
is
constant
of
all
the
MK
rank
to
my
ID
sempia
ranks.
Of
course,
you
know
you
see
that
this
kind
of
equivalent
wanna
call
increasing
line,
etc.
So
you
can
have
a
kind
of
digital,
quick
check.
A
You
know
whether
something
like
might
have
gone
wrong.
This
is
the
parallel
stack,
the
course
that
you
were
screwing.
So
this
is
a
Predacon
stack.
This
isn't
to
show
that
we're
in
pay
rings
are
in
this,
the
preload,
the
cold
frame
here,
and
also,
if
you
want
to
print
out
certain
certain
things
you
can
put
an
expression
here
in
this
window,
so
that
we
can
check
the
other
values,
are
correct
or
not.
A
So,
as
I
said
that
the
you
need
to
create
a
certain
special
points,
break
points,
so
this
is
top
where
you
want
the
program
to
stop
when,
when
that
point
is
which
this
is
called
a
break
point,
you
can
create
it
by
using
your
double
click
on
the
line.
I
said
there
are
several
ways.
Another
type
is
called
watch
point,
so
you
kind
of
set
a
watch
point
for
a
certain
variable
so
that
the
when
the
variable
values
changes,
so
the
debugger
will
stop
there
so
that
you
can
check
the
at
that
point.
A
You
know
if
you
don't
expect
that
that
variable
to
change,
but
the
program
somehow
got
changed.
You
know
you
want
to
know
where
that
thing
happens
here.
So
this
is
where
know
why
you'd
want
to
use
the
which
ones
trace
points
is
likely
not
the
kind
of
print
statement
when
when
it
reaches
there,
you
just
painted
that
that
the
bigger
reach
to
that
mind-
and
you
can
also
specify
a
certain
list
of
variables
so
that
you
can
print
the
values
here.
A
A
Just
like
you
know,
so
everything
looks
smooth,
but
if
you
know
when
I
was
working
on
is
the
developing
a
code
I
had
a
lot
of
issues,
a
lot
of
errors
occurring
when
the
vendors
at
the
header
exchange
when
he
at
the
boundary
of
a
okay
on
the
domain.
So
that's
where
the
lot
of
errors
happen.
So
in
that
case
you
will
see
certain
oddities
on
the
edge
of
the
domain.
For
instance,
if
you
don't,
you
know,
this
is
a
kind
of
give
your
kind
of
way
of
a
quick
sanity
check
after
the
header
exchange.
A
A
So
also
Kady,
whether
told
you
we
started,
you
can
father
this
portion
shows
here.
The
source
code
ended
from
there.
You
can
define
the
breakpoints
here.
You
can
actually
click
on
the
variable
to
check
the
value
of
values,
etc,
and
also
this
shows
that
the
values
are
variable
here
as
well.
So
once
again
that
you
know
there
are
many
tools
to
display
but
reject
us
the
status
of
the
other.
Your
program
learning
here
using
this
tool-
and
another
thing-
is
the
the
stat.
This
is
development
lawrence
livermore.
This
is
a
together
stack
back
traces.
A
You
know
you
could
call
from
the
function
f
and
the
future
cost
function
b,
and
then
he
shows
all
these
sequences
and
costs
up
to
that.
The
current
location
they
actually
get
us
all.
The
stack
vectors
information
from
all
n
k,
ranks
and
they'll
show
them
in
a
graphical
way
in
a
single
clock.
You
know
single
tree
like
a
structure
here,
so
it's
kind
of
easy
to
see
that
the
where
each
champion
rankings
are
located,
so
you
can
quickly
check
the
status
of
your
program
using
stat
and
then
for
the
detailed
information
for
detail.
A
The
debugging
or
effort
you
can
use
a
DDT
or
view,
so
there
are
several
stat
commands
here
usually,
but
let
me
show
you
an
example,
quick
example.
So
if
your
code
is
hanging,
for
instance,
you
can
use
the
stat
for
tip
to
start
to
debug
the
area
code.
One
is
to
build
your
code
that
way
this
way
and
the
sati
interactive
batches
are
run
it
and
then
running
in
the
background,
and
it
will
show
you
the
process
ID
and
then
you
can
run
that
test
SEL
with
the
process.
A
A
So
this
is
what
you
see
the
if
you
do
that,
so
he
shows
that
ranks
zero
was
here
rank
one
reg
three
was
here
and
also
here.
So,
as
I
said
that
there
are
tens,
there
are
something
about
ten
times
the
amount
of
do
you
do.
Those
ten
samples
rank
three
was
here,
but
the
other,
the
other
samples
was
here
and
the
distance
rank
one
and
two
were
here.
A
So
if
the
code
is
hanging
is
stuck
somewhere
in
this
stage,
you
can
tell
that
where
is
change
Iraq
years
and
you
can
check
in
using
the
debugger
or
look
at
your
source
code
to
see
why
this
kind
of
work,
the
hanging
happening
at
these
locations,
so
another
tool
that
is
useful
is
the
ATP.
This
is
a
crystal.
So
what
it
does
is
that
when
the
program
fails,
the
ATP
involves
stat,
so
that
you
can
generate
clear
the
previous.
A
A
So
this
is
an
example,
so
you
build
your
code
and
you
enable
the
variety
ATP-
and
this
is
the
Intel
Fortran
related
environment
variable.
If
we,
if
your
code
is
built,
if
your
audience
on
Fortran
code
built
with
the
Intel
compiler,
then
you
said
that
this
invalid
value.
Additionally,
you
run
it
and
then,
if
your
code
is
hanging,
then
you
SSH
to
see
the
man
node
and
then
you
check
you
see
that
the
status
of
your
program
by
using
a
sub
count
command
that
Helen
mentioned.
A
And
then
you
see
that
this
is
the
the
NK
job
that
you
are
running
their
support.
That
is
also
hanging
and
you
cancel
it
using
this
command
here
and
the
get
out
of
the
blue
beam
or
the
mono,
and
then
when
cancel
each
other's
a
DAT
file.
So
at
that
point
you
load
a
step
model
and
then
visual.
The
display
is
one
of
these
two
dot
file
here.
B
A
You
do
that
you
see
that
it
was
showing
that
at
the
time
already
you
brand
as
cancel
the
rank.
0
was
here,
Rank,
1
and
2
was
here
rank
3
was
here
again
we
can
go
back
to
your
source
code
to
see
that
one
might
be
having
happening
that
these
ECMA
iraq
in
these
locations
to
try
to
deduce
unity,
the
cause
of
the
error.
So
there
is
a
pretty
useful,
simple
tools
here,
so
I
want
announcer
theater
tutorial
coming
up
on
July
16th.
B
A
Second
tool
and
thought
to
his
arm
Thomas
reports
to
this
is
to
give
up
the
Puffin
summary.
This
is
that
I
think
that
this
will
be
a
very
good
tutorial
for
the
beginning
and
intermediate
level
users
so,
and
also
will
teach
you
how
to
use
a
map
and
the
performance
reports
to
provide
a
Python
application
performance
here.
It'll
be
very
useful
and
you'll
be
it'll,
be
a
hack
tape.
Tutorial
will
be
given
by
the
engineer,
so
please
go
to
the
eighties
website
that
the
date
is
is
not
correct.
A
A
B
We
having
a
total
view
to
tutorials
sometime
in
the
future
too.
No.
A
B
A
A
Now
this
is
parties
profiling
tools.
This
is
to
measure
the
coded
performance
if
we
want,
if
you
see
that
your
code
is
running
really
poorly,
that
you
want
to
see
that
the
weather
couldn't
taking
in
so
much
time.
You
want
me
to
Dakota
thoughtfulness
here.
So
when
Cory
we
have
a
lot
of
our
tools
were
scrape
pad.
What
again?
The
second
is
our
map
and
this
performance
reports
intel
vtune
popular
one,
interlude,
Weiser,
trace
on
our
connector,
etc.
So
I
will
to
cover
the
crepe
head
and
arm
for
today.
A
So
crepe
pad
is
a
is
a
craze,
mostly
text-based
kind
of
you
know
the
big
it
to
here.
This
is
a
to
craze,
totems
or
less
to
that
has
been
around
for
a
long
time
and
it
is
made
of
several
parts
of
modules
here.
So
what
is
the
this
tough
to
stash
base
model?
This
is
required
to
load
the
actual
photo
moons
Mara
module
here.
So
you
need
to
know
that
isn't
first
before
you
know
that
this
one
of
these
two
module
and
also
this
is
really
quiet.
A
If
we
want
obsess
the
problems,
related,
documentation,
main
phase
etc.
So
and
tough
tools
is
the
front
street
and
tough
to
try.
This
is
kind
of
easy,
easy
to
use
version
so
so
to
use
the
crepe
at
you
have
to
know
that
our
module
doesn't
accept
another
profiling
tool
for
I/o,
so
you
kind
of
interview
each
other.
So
you
want
to
know
this
one,
and
then
you
know
that
the
postal
base
or
postal
tools
before
you
start
to
compile
and
then
the
wonderful
thing
is
that
you
have
to
create
a
after
code
that
all
files.
A
A
So
there
are
two
ways
of
collecting
performance
data
on
a
subject
of
program:
sampling,
experiment.
This
is
to
choose
the
sample
program
location
for
it.
What
equal
to
program
counters?
This
is
the
location
your
program
at
that
enter
certain
regular
time
interval
so
that,
at
the
end,
if
you
see
at
the
us
a
lot
more
samples
in
a
particular
location,
then
it
means
that
the
codes
are
spending
a
lot
more
time
there,
the
by
default,
the
trade
pad
uses
the
time
it'll
interval,
0.01,
second
and
the
tracing
span
is,
is
different.
This
is
a
use.
A
When
you
build
your
code,
you
specify
a
certain
number
of
functions
to
be
traced,
so
the
trace
function
can
be
user
defined
functions,
I
mean
the
one
that
you
define
or
it
can
be
a
library
function.
I
can
carry
functions
there
as
well,
and
what
it
does
is
that
when
you
use
at
this
experiment,
he
echoes
the
when
the
function
is
entered
and
when
the
function
is
accident,
so
it
can
give
you
more
detailed
information,
it's
not
the
sample.
You
know
the
location.
A
You
can
actually
see
that
when
the
program
enter
enters,
he
knows
where
the
matter.
So
he
exactly
knows
that
you're,
the
exact
amount
of
me
less
time
that
the
program,
so
this
one
is
that
it
gives
a
more
detailed
information,
but
already
larger
so
to
build
it.
The
top
two
light:
that's
simpler
one,
you
load
these
two
models
and
build
it.
This
way
this
is
an
instrumented
original
version.
This
is
instrumentally
executable,
and
then
you
run
this
the
instrumented
executable.
A
When
you
do
it,
and
then
you
created
these
that
factory,
where
the
only
collected
talking
data
are
located
the
the
last
letter
s
signifies
this
is
the
resource
of
sampling,
okay
and
then
this
one
shows
the
total
mascara
summary.
Let's
turn
that
out
and
if
you
look
into
this
directory,
there
are
several
files.
One
is
that,
except
for
this
is
Robin's
data.
This
is
process
of
already
talked
to
Lionel
any
process
of
this
excel
file
to
bring
in
P
to
file
this
importable
process
data-
and
this
is
tax
base.
The
year
report.
A
Our
of
this
data
here
so
as
I
said
that
the
slum
cent
allowed
file
contain
the
optimist.
Data
from
this
talk
to
light
run
also,
the
exact
same
thing
can
be
found
in
that
directory.
He
shows
a
lot
of
what
Aeneas
you
know
the
runtime
by
functions
to
like
here
so
just
spent
about
2.4
billion
samples,
one
sample
time
it
0.1
sec.
2001
second
rank,
so
has
this
many
sample
this
many
samples
in,
in
the
other
main
main
routine,
this
many
samples
in
here.
A
A
So
now
this
is
how
to
use
a
portal,
not
really
a
simpler
one.
This
okay,
simply
here
you
loaded
this
module,
I,
build
it,
and
then
you
build
the
instrument
executable
using
pad
bill.
So
you
get
at
you
the
original,
the
funny
instrument
original
executable
and
when
you
run
the
PAP
build
on
it,
you
create
the
instrumented
version
here
and
run
with
this
instrument
version
and
again
he
creates
directory.
As
for
sampling,.
B
A
Then,
after
that,
you
run
the
pad
report
in
that
directory
to
create
a
tax
based
to
deport
here
and
once
again
this
is
content
of
the
directory.
This
is
raw
data.
This
is
the
processed
data
and
this
one
interesting.
This
one
shows
the
proposal
with
the
options
for
the
next
tracing
experiment,
if
that
isn't
necessary,
based
on
the
results
from
the
assembling
run
here.
So
this
is
a
dsk
file
that
showed
that
condense
all
the
bid
build
options
here
again.
Pet
report
shows
the
tokens
data
out
of
here.
This
isn't
like
that
and
a
PA.
A
That
is
what
what
I
was
talking
about.
This
is
a
to
the
crate.
Will
guide
you
to
create
a
next
trace,
trace
experiment
out
of
these
the
sampling
experiment.
This
is
called
a
PA
and
again,
as
I
said
in
the
previous
case,
it
has
created
this
APA
the
adoption
file,
so
you
rerun
Capel
using
this
file.
Then
little
created
this
instrumented
another
you
through
until
it
version,
but
this
time
for
tracing.
So
when
you
run
it,
you
will
see
the
tracing
resource
here.
A
So
again,
you
see
the
various
kind
of
plotting
data
and
therefore
the
tracing
experiment.
If
you
want
to
interesting
this
frame
right
away,
can
loaded
this
one
and
the
pepper
using
the
tracing
of
the
build-up
to
a
production
here.
This
is
a
originally
executable
on
instrumented.
This
is
that
instrument
executable
for
tracing
experiments.
So
if
we
use
a
you
didn't
trace
all
the
functions
defined
in
your
your
user
source
code
and
you
can
select
a
certain
certain
user
functions
also,
you
can
also
specify
certain
library
functions
like
that.
A
A
So
34
shows
me
all
kinds
of
interesting
things
now
that
instead
of
sampling,
this
is
a
tracing
resource
right.
So
instead
of
samples
we
have
an
actual
time
and
in
each
subroutine
is
a
function
here.
So
this
is
a
previously
set
for
sampling.
We
have
a
simply
called
a
sample
columns
here
right.
So
now
we
have
this
more
accurate.
The
information.
B
A
A
So
this
is
an
armed
man.
This
is
unparalleled
profiling
tool.
This
is
basically
sample,
and
the
interesting
is
that
this
source
code
lines
are
annotated
with
the
components
data
actually,
so
that
you
can,
you
can
easily
spot
the
hot
spots,
so
he
they
are
nothing
that
is
different
from
the
crepe
and
is
that
this
is
time
series
of
homeless
beggar,
not
the
summary,
not
the
aggregate
son
right.
We
have
a
puddle
a
lot,
a
lot
of
large
number
of
license.
B
A
Again,
we
have
to
use
an
X
or
remote
client
so
build
the
code.
I
mean
the
digits,
tripled,
I,
think
or
the
the
actual
commands
here.
So
you
can
follow
along.
You
can
start
me
check
bad
job
know
that
this
one
start
your
program
on
the
map.
Then
you
create
this
kind
of
out
here.
You
see
that
this
line
shows
that
the
pig
at
a
certain
this
period.
This
is
a
time
series.
This
is
kind
of
summary.
You
see
that
the
where
in
this
case
is
a
48
percent.
A
This
is
probably
around
40
percent
associate
with
this
loop,
so
you
can
kind
of
shows
that
the
hot
spots
I'm
sorry
about
that
and
the
works
going
on
pass,
pass
right
away
associated
with
each
source.
For
so
you
can
easily
find
you
know
the
lines
that
we
would
spend
a
lot
of
time,
but
one
one
warning
is
that
it
because
yesterday
me
the
time
series
start
for
source
code
right.
The
map
uses
total
1000
samples
for
mk
wreck,
so
he
may
not
be
suitable
for
long
runs.
B
A
Collect
oversee
1000,
even
though
you
can
Christmas
emphasize
this,
but
the
it
is
they
are,
throwing
discourage
that
so
keep
in
mind.
So
again,
this
is
the
hot
spot
based
on
data
from
this
is
a
great
provisions.
For
instance,
this
is
of
functions,
but
interesting
thing
is
again.
This
is
a
time
series
performance
data,
not
the
aggregate.
Some
summary
may
be
n,
so
you
can
add
the
more
display
so
once
again,
because
it
has
a
limited
number
of
samples,
1000
point
K
bring.
If
you
use
the
body
processes,
it
has
to
be
divided.
A
Trust
easier
thread
has
to
use
a
smaller
number
of
samples
here.
So
because
of
that
concern
you
may
want
to.
You
may
want
to
come
profile
only
a
certain
section,
not
the
entire
code,
so
that
you
don't
need
to
expand
on
the
initial
initialization
program,
injection
or
finalization
for
well.
A
lot
of
I/o
is
being
done,
so
you
can
use
this
kind
of
API
call
to
star
and
the
star
sampling
here.
The
final
one
is
that
this
is
kind
of
to
show
that
the
bottom
is
summary.
A
So
what
these
three
does
is
to
kind
of
at
the
end
of
run.
He
gives
a
quick
summary
about
the
your
program,
whether
you
just
compute
by
I'm,
sorry
about
that
compute
bound
and
K
bound
or
I/o
bound,
so
results
in
HTML
file.
You
build
your
code
as
you
do,
with
form
at
the
status
generate
the
HTML
file.
So
in
this
case
the
spend
a
lot
of
computation
here.
So
he
says
that
is
a
complete
bath
right,
so
this
is
kind
of
give.