►
From YouTube: 6. Debugging Tools
Description
Learn about using the debugging tools available on Cori.
Slides for all sessions can be downloaded from here: https://www.nersc.gov/users/training/events/new-user-training-june-21-2019/
A
Hello,
my
name
is
Susan
young
and
I'm
in
the
or
so
in
the
same
group
you
say
engagement,
group
and
I'm
going
to
talk
about
the
the
the
debugging
tools
on
on
query
system.
So
this
usually
a
very
quick
tour.
It's
not
like
a
hands-on
session,
probably
the
best
way
to
learn
how
to
use
the
debugging
tools
is
to
use
I
through
the
hands-on
session.
But
this
is
really
I'll
be
probably
skipping
on
the
surface
year
anyway.
A
So
the
the
intention
is
for
you
to
quickly
introduce
us
okay,
what
kind
of
given
you
tools?
We
have
how
you
can
start
and
what
features
they
have
so
that
and
the
witch
debugging
tool
you
should
use
for
certain
purpose.
Okay,
so
when
you
start
to
modify
your
code,
you
will
see
some
of
these
errors.
You
know
the
program.
A
A
And
how
do
you
fix
fix
it
so
easy
way
is
to
put
a
print
statement
here
and
there
the
easy
way
nothing
to
learn
here,
but
the
problem
here
is
that
it
is
very
difficult
to
guess
where
to
put
these
print
statements
right,
especially
if
it
is
a
you
know,
parallel
code,
there's
a
lot
of
MPI
tasks,
it'll
be
difficult,
it
can
be
from.
You
know,
different
rank,
for
instance,
and
also
what
to
print.
A
So,
if
you
gasp
is
incorrect
about
the
location
or
the
variable
to
print,
then
you
have
to
move
the
print
statement
somewhere
else.
That
means
that
you
have
to
recompile
and
submit
a
job
again,
and
you
have
to
do
this
kind
of
blue
over
and
over
again.
So
this
is
a
very
tedious
and
exhausting
and
very
time-consuming,
and
also
from
this
output
is
very
difficult
to
extract
information
about
it.
The
the
error
that
is
so
here
so
debuggers
are
here
too.
You
know
remedy
this
situation.
A
You
don't
need
to
compile
over
and
over
again
you
just
compile
once
and
you
run
it
under
the
control
of
a
debugger,
and
you
can
make
the
program
stop.
You
can
make
the
debugger
to
display
the
variables
that
you
want
to
check
and,
and
also
you
can
actually
use
the
visualization
tool
to
display
the
variable
to
see
the
weather
that
there's
some.
You
know
very
strange.
You
know,
anomalies
appear
and
that's
where
that
the
problem
might
be
right
and
this
kind
of
thing,
so
you
can
use
this
tool
quite
easily.
A
So
we
have
a
several
kinds
of
debugging
tools
here.
So
the
first
kind
is
the
more
traditional
parallel
graphical
GUI,
based
the
debugging
tools
called
the
DDT
and
the
total
view,
and
most
of
people
are
using
DDT,
but
they
are
kind
of
complementary.
Sometimes
DDT
misses
the
certain
certain
things,
and
in
that
case
you
can
use
a
total
view
to
find
the
bug.
So
if
user
Lee
TD
T
is
very
more
intuitive,
so
I
do
like
to
suggest
you
to
DDT,
but
sometimes
even
misses
certain
debugging
situation.
A
So,
in
that
case
we
have
to
use
auto
review
and
also
more
the
there
are
other
kinds
of
specialized
debugger
so
like
a
stab,
it
doesn't
care
about
it
with
the
values
is
just
which
display
the
course
tech
traces
from
all
the
MPI
ranks
in
a
graphical
form.
So
it's
very
easy
to
see
that
where
the
code
is
having
instance.
So
if
the
code
is
hanging,
the
14
there
first
thing
that
you
can
try
is
stat,
for
instance,
and
the
ATP
this
is
great.
Oh,
it
is,
it
is
basically
pay.
A
It
is
using
the
stat
underneath.
So
what
it
does
is
that
when
the
program
crashes
recalls
that
to
show
that
the
to
capture
the
course
callback
traces
at
the
the
a
failure
point,
so
these
are
very
light
weighted
and
but
it's
very
powerful.
So
whenever
some
people
happy,
they
don't
know
where
the
code
is
crashing.
So
I
usually
suggest
that
these
things
here,
and
so
also
there
are
other
kinds.
You
know
very
ground
which
is
very
popular.
A
It
is
actually
all
sweet,
orbitty
bugging
and
the
profiling
tools
here,
but
people
use
it
mostly
for
these
man
check.
This
is
a
very
powerful
memory,
debugging
tool.
Isabelle
did
it
gives
a
very
detailed
report
about
you
where
the
memory
error
errors
are
occurring
here
and
also
we
have
a
very
nice
Toni
Intel
inspector,
with
the
with
it.
We
can
detect
the
threading,
the
race
condition,
for
instance,
for
different
stress.
If
we
use
the
openmp,
you
can
detect
the
threading
error
or
you
can
detect
the
memory
error
so
DDT
and
to
review.
A
So
as
I
said
that
this
is
a
traditional
parallel
debuggers,
it
can
be
used
as
C
C++,
Fortran
MPI,
OpenMP
P
threads
Azeroth
did
it.
We
can
use
the
DPP
up
to
about
a
thousand
MPI
tasks
so,
but
we
have
a
very
small
licenses
for
Toro
view.
So
for
debugging
or
large
you
know
larger
application.
You
have
to
use
the
DVP,
but
the
a
thousand
the
licenses
are
shared
among
all
users.
If
some
people
are
using
half
of
half
of
them,
you
can
only
run
up
to
the
four
thousand,
for
instance,.
A
A
Once
you
do
that,
then
you
start
the
interactive
batch
job
using
SL
up
as
Helen
foot
is
playing
already
and
then
load
a
linear,
40
module
and
the
start
T
DT
like
that,
and
then
it
will
show
you
the
startup
window,
and
then
you
can
indicate
whether
this
is
openmp
debug
code
or
you.
You
may
want
to
specify,
for
instance,
C
value
test,
the
CPU
pine
equals
course
etc.
That's
Ron
arguments
as
well.
A
So
so,
as
Tim
mentioned
this
morning,
when
you
are
working
not
working
here
but
working
somewhere
else,
the
x11,
the
protocol
user,
is
not
very
efficient
over
network
here
so
because
of
very
high
latency
and
low
bandwidth
is
kind
of
opposite.
So
you
have
to
use
an
axe
basically
or
if
you
use
the
arm
the
the
linear
tool,
then
the
we
can
use
the
is
demo
client
program,
and
then
you
can
run
it
from
your
desktop.
And
then
you
start.
A
You
start
your
job
on
quarry
and
then
you
can
send
that
information
to
your
desktop
and
the
day
to
kind
of
interact
each
other,
and
this
really
fast,
nope,
no
latency.
The
really
fast
actually
is
faster
than
an
axe
so
for
detailed
information
how
to
set
here
for
the
a
demo
client
you
have
to
go
to
here,
but
this
is
the
way,
the
things
that
you
do
basically
use
that
yeah
this.
You
have
to
put
this
exactly
for
the
installation
directory
and
remove
script
path
here,
exactly
and
uncheck
this
for
MFA.
A
So
once
you
start
that
DDT
you
see
that
you
see
this
kind
of
window
and
the
top
the
top
area.
These
are
for
the
Earth.
You
know,
navigation,
select,
good
what
events
online
or
going
through
the
holder
function
and
get
out
from
that
their
function
for
the
calling
function,
for
instance
here
it
shows
the
kind
of
directory
and
all
kind
of
functions
are
available
in
the
source
code,
and
this
is
this
shows
the
source
code
itself,
and
here
it
is
placed
of
values
of
the
variables
that
are
being
currently
using
in
the
routine.
A
So
you
can
check
the
other
values
itself,
so
you
can
check
by
just
quickly
glad
you
know
doing
this
thing
to
see
that
whether
there's
something
wrong
quickly
and
also
this
is
called
a
spot
line.
This
shows
that
the
variation
of
that
particular
variable
over
different
MPI
ranks.
So
you
can
quickly
check
by
cleansing
this.
You
can
tell
whether
this
is
something
unexpected.
A
In
that
case,
you
can
dig
into
that
particular
variable,
so
it
gives
a
very
quick
visual
summary
so
that
so
this
this
is
a
very
good
with
a
DDT
actually,
and
here
it
shows
that
where,
where
the
you
are
basically
in
that
course
stack
for
all
the
entire
ranks,
so
all
the
I,
all
the
16
ranks
I
started
with
16
runs.
All
of
them
are
currently
here
right.
So
if
they
are
spread
in
a
different
location,
it
will
show.
A
We
said
the
two
ranks
I
here
and
the
eighth
rank
see
here
I,
sir,
so
it's
a
kind
of
quick
visual.
You
know
indication
what
is
going
on.
This
is
where
you
can,
you
might
want
to
check
the
certain
variable
certain
expression
you
can
enter
this
expression
to
see
the
value,
whether
that
is
something
unexpected
right.
Yes,.
A
Poor
know
if
Python
people
are
still
trying
to
lately
integrate
into
DDT
and,
of
course,
total
view,
but
the
problem
here
is
that
we
are
using
anaconda
bill
for
Python
and
we
need
a
debugging
symbol
for
that,
a
particular
the
anaconda
Python,
which
we
don't
have
right
now
so
currently
and
nurse
this
is
just
cannot
be
done.
Yeah.
A
Okay,
yeah,
okay,
but
anyway
you
cannot
use
that
the
Python
DDT
and
the
breakpoints,
and
and
this
as
I
said
that
the
debugging
tool,
one
thing
there
is
good-
is
that
you
can't
control
the
pace
and
you
can
make
it
stop.
You
can
run
it
up
to
a
certain
point.
You
can
make
it
stop
there
so
that
you
can
check
the
variables
so
the
way
you
do
it,
you
kind
of
define
the
breakfast.
You
know
I
by
preview.
I
Satya,
where
the
program
should
stop
right.
A
This
is
called
the
breakpoints
or
I
can
define
certain
points
where
that,
when
the
the
variables
change
these
values,
the
program
you
stopped
by
is
equal
to
what
point
trace
point
is
where
you
know
you
you
may
want
to
use
if
you
use
the
debug
using
the
priests
statement,
you
to
say
that
print
I'm
here
right,
something
like
that.
So
this
is
what
what
it
does.
Basically,
when
you
reach
a
trace
point,
he
says
that
you
reach
it
up,
so
you
can
make
you
know
certain
condition
to
that,
so
that
you
can.
A
The
breakpoints
can
be
activated
under
certain
condition.
For
instance,
if
there
is
a
big
loop
you
can
set
that
you
can
make
the
disk
breakpoint
active
when
looky-look
variable
is
10,
for
instance,
so
you
didn't
continue
until
the
living
variable
becomes
10,
for
instance,
so
and
as
I
said,
you
can
check
the
various
quite
easily
right
click
on
the
editing
window.
A
You
can
show
these
values
like
that
and
then,
as
I
said,
that
on
the
right
hand,
side
it
shows
the
local
variables
UNIX
expression
in
the
availab
pane
and
so
that
you
can
check
the
values
of
expression.
The
one
thing
one
nice
thing
is
the
MDL.
This
is
a
visualization
tool
for
array.
So
if
I
click
on
the
array
and
the
selected
functionality,
I
can
display
yes,
so
you
can
clearly
see
the
smooth
kind
of
variation,
which
is
good.
If
something
I
made
a
mistake
here,
you
may
have
a
certain
anomalous
pick.
A
You
know,
for
instance,
dad
indicates
something
so
quickly
visually
indicate
whether
something
is
going
on
right
correctly
or
not.
You
can
actually
see
that
statistics
about
the
values
totally
is
very
similar.
You
started
that
way.
Interactive
job
and
the
loader
total
you
module,
and
then
you
start
toward
using
that
this
kind
of
man
and
then
you
have
to
click
some
more
step.
This
is
a
less
intuitive
than
the
DDT,
but
you
get
basically
two
main
in
those
route
window
that
shows
the
status
of
MPI
and
MPI
rank
and
threat.
A
A
The
old
MPI
rank
whether
that
the
variation
is
something
that
is
expected
or
now
or
you
can
navigate
through
the
nprm
sort
of
address
by
clicking
on
these
buttons,
so
that,
as
I
said,
that
this
is
a
very
light
weighted
tool,
but
it's
very
powerful.
You
can
gather
the
stack
back
traces
from
all
the
entire
ranks.
A
A
You
start
the
interactive
job
and
then
you
run
as
run
either
in
the
background
right:
the
m
%
@
TN
and
then
at
the
point
you
need
to
load
the
stem
module
and
the
run
the
stasi,
L
on
that
the
process
ID
and
then
what
it
does
is
that
says
samples
the
locations,
the
the
stack
back,
traces
for
all
MPI
ranks
and
the
Creator
file.
Here
right.
It
says
that
the
results
are
written
into
that
directory.
So
at
that
point
you
run
this.
That
view
to
display
the
results
graphically
here
is
something
that
you
see.
A
A
Look
inside
the
MPI
rank
0,
so
we
cannot
complete
the
MPI
send
NBC,
because
0
is
kind
of
stuck
some
way
off
like
from
this.
You
can
create
check
that
where
they
are
and
go
back
to
here
or
you
can
use
a
DDT
at
a
point,
because
you
now
know
that
where
the
problem
might
be
right,
so
this
is
a
very
quick
tool.
I
use
a
very
simple
tool
that
can
quickly
tell
you
where
to
look,
but
it's
very,
very
powerful
another.
A
The
last
tool
that
I
want
to
mention
is
the
ATP
of
normal
termination
processing.
This
is
great
tool
if
you
use
the
stat
as
I
mentioned
in
the
in
the
beginning.
So
this
one
calls
that
only
when
the
program
fails
when
the
program
fails,
it
causes
that
to
get
that
kind
of
work,
the
data
so
that
we
can
display
this
kind
of
thing
here.
A
A
At
the
end,
then,
at
that
point
you
know
the
stem
module
with
this.
Previous
result,
you
see
something
like
that.
So
these
are
very
quick.
You
know
introduction
a
quick
tour
about
your
carabiners,
simple
tools.
We
are
not
a
simple,
but
you
know
powerful
tools.
We
have
on
quarry
to
develop
other
colors
here.
That's
what
I
have
any
question.