►
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Speaker:
Sudheer Chunduri, Argonne National Lab
Kevin Harms, Argonne National Lab
Title:
Darshan 3.3.0 & Autoperf 2.0 updates
Abstract:
AutoPerf is a lightweight profiling tool used for Automatic performance collection of MPI (focused on MPI 2.0 operations) usage and hardware performance counter information. AutoPerf 1.0 was deployed on Argonne’s earlier general machine called Mira and was successfully collected logs for over 4-5 years. The analysis from this data was helpful in providing several insights in the MPI space* and beyond. Considering the feedback from this study and to enhance the coverage for more MPI 3.0 operations and other improvements in the recorded and reported summary data, Autoperf 2.0 is designed. AutoPerf2.0 implements two additional Darshan instrumentation modules that can provide details on application MPI communication usage and application performance characteristics on Cray XC platforms. We will describe our plans for future work in this topic and provide a summary of Darshan latest release.
Bio:
Sudheer Chunduri is a member of Performance Engineering team at ALCF working on the interconnects and MPI developments and performance analysis.
Kevin Harms is a Performance Engineering Team Lead at ALCF working on Parallel IO, storage and platform analysis and benchmarking aspects.
Host of Seminar:
Taylor Groves
Advanced Technologies Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
All
right:
well,
I'm
going
to
go
ahead
and
give
an
introduction.
So
today
we
have
kevin,
harms
and
siddhir
chanduri
talking
about
darshan
and
autoperf,
so
both
kevin
and
cydia
work
at
argonne,
national
lab
and
I've
had
the
pleasure
of
working
on
them,
looking
at
network
performance
and
routing
in
the
past
and
congestion.
So
they
have
a
lot
of
experience
looking
at
and
profiling,
the
performance
of
their
systems
at
argonne,
particularly
looking
at
mpi
usage,
numerous
papers
at
sc.
A
C
B
B
So
I'll
talk
about
what
we
have
been
doing
with
auto
perf
and
then
kevin.
He
will
talk
about
the
darshan
part
of
it.
Yeah,
please
feel
free
to
interrupt.
If
you
guys
have
any
questions
so
out
of
the
design
goal
is
for
it
to
be
a
monitoring
tool
so
to
run
on
the
system
for
all
the
applications
collecting
monitoring
data.
So
as
you
could
guess,
it
should
be
with
low
overhead.
So
it's
a
lower
head
profiling
tool,
not.
D
B
Like
pad,
or
this
kind
of
tower,
this
full-fledged
profiling
tools,
we
want
to
kind
of
capture
with
minimum
possible
overhead
some
useful
information,
so
this
was
used
in
production
on
mira,
and
so
it
provides
a
very
coarse
grained
resource
usage
characteristics
like
one
of
the
module.
One
of
the
key
module
in
the
auto
puff
tool
is
the
mpa
component,
so
it
collects
the
mpa
usage
like
what
are
the
different
collective.
B
I
mean
for
each
mpa
operations.
It
collects
few
statistics
and
if
you
kind
of
analyze
the
logs,
auto
perf
logs
across,
say
some
set
of
jobs
collected
over
few
months
or
a
few
years
or
so,
then
you
can
kind
of
extract
some
useful
information
out
of
those
saying
like
okay,
how
is
mpi
being
used?
What
kind
of
operations
are
being
used
predominantly?
B
B
B
B
We
envisioned
that
this
will
be
more
useful
for
the
future
systems
where,
as
you
all
know,
how
this
quality
of
service
classes
etc
to
tune
those
okay
which
application
to
be
placed
in
which
quality
of
service
class.
So
if
you
collect
some
network
counters
and
if
you
can
do
some
characterization
of
the
applications
like
binning
mapping
applications,
we
could
get
some
some
useful
information
from
this
tool
for
the
mapping
and
also
we
have
done
some,
especially
when
we
are
evaluating
some
design
points
for
the
aurora.
C
B
Example
like
okay,
this
in
this
much
injection
bandwidth,
is
that
sufficient
for
the
future
mission
or
so
so
we
try
to
do
you
based
on
the
auto
proof
data.
Some
analytical
modeling
try
to
do.
Okay,
given
this
on
on
the
future
mission.
Would
this
particular
injection
bandwidth
would
that
be
sufficient,
or
so
that
kind.
D
B
D
B
D
B
B
Another
one
is
the
across
all
the
ranks
that
therefore,
in
the
application,
the
rank
with
minimum
mpa
time,
so
it
doesn't
already
call
and
find
out
like
what
is
the,
which
is
the
rank
that
has
the
minimum
mpa
time
and
the
max
the
rank
with
maximum
mpa
time
and
the
rank
with
an
mpa
time
that
is
closer
to
the
average
mpa
time.
So
the
data
from
those
four
ranks
is
what
reported
from
these
logs
and
the
interface
that
used
for
the
intercepting
these
counters
is
through
the
pmpa
mps
transformation
based
standard
pmp,
mba,
profiling
interface.
B
E
As
an
example
cydia,
could
I
ask
you
a
couple
of
quick
questions
here?
Just
help
clarify
things
for
me,
so
this
is
doing
an
aggregation
over
the
whole
run
right,
there's
no
time
series
data,
okay
and
is
it
and
it's
it's
rank
based
right
so
for
like
for
like
memory
usage,
is
it
able
to
assign
some
amount
of
memory
used
per
rank
like
using
alex
or
something
else,
or
is
it
kind
of
a
node
based
memory
usage.
B
D
D
E
B
E
B
Intensity
of
the
so
by
communication
density.
Here
we
mean
like
what
is
the
total?
What
is
the
mpa
time
divided
by
the
like
total
time,
loosely
speaking
and
the
relative
important
importance
of
each
mpa
operations
in
terms
of
the
time,
primarily
by
importance
I
mean,
and
the
message
size
across
the
operations?
Okay,
what
is
the
kind
of
average
message
size
for
these
operations
and
what
are
the
kind
of
different
threading
modes
that
are
being
used
predominantly
so?
Some
of
these
insights
were
like
most
production
applications
tend
to
spend
like.
B
I
mean
okay,
so
this
is
the
general
assumption
we
had,
but
from
this
study
we
see
that
I'll
show
you
one
plot
like.
If
you
see
these,
these
are
the
different
applications
that
we
kind
of
from
the
different
from
the
executables
we
kind
of
extracted
which
application
that
is,
and
the
plot
here
is
showing
a
fraction
of
that
mpa
time
within
the
total
time,
and
the
x
axis
is
the
different
applications
and
the
y
axis
is
the
fraction
of
the
mp
time
as.
B
Like
kind
of
more
than
half
of
the
applications
are
spending
like
more
than
50
percent
of
the
time
in
mpa
again
here
there
are
some
big
caveats
when,
when
we
say
mpa
time,
it
is
the
load.
Imbalance
is
also
included.
It's
not
just
like
purely
communication
time
mpa
time
here
is
not
equal
to
the
communication
time,
so
it
also
includes
if
there
is
load
imbalance
that
is
also
shown
as
in
the
mpa
time.
That
is
one
one.
D
B
B
C
B
C
F
B
D
B
C
B
B
Yeah
this
I
have
already
covered
and
then
the
this
is
elect
across
all
the
applications.
It
was
time
across
all
the
jobs,
whatever
100k
logs
those
jobs
that
we
seen.
If
you
kind
of
take
the
time
in
across
all
the
collective
across
all
the
mpa
operations,
you
could
see
that
all
reduce
the
time
for
all
reduce
is
the
predominant.
B
C
B
B
B
I'll
give
a
reference
to
that
paper
and
all
these
mpa
counters
etc,
and
their
idea
was
they
just
spread.
This
data
on
these
features
into
a
machine
learning
model,
and
this
is
the
tsne
tsme
plot.
They
call
it
in
the
machine
learning
it's
to
visualize
the
high
dimensional
data,
and
you
can
see
that
there
is
some
clustering
of
this
data
and
when
they
kind
of
label
the
the
this
data
with
the
executable
name.
B
You
could
see
that
this
is
some
executable
exhibiting
certain
kind
of
feature,
and
and
also
the
this
of
that
getting
mapped
to
a
particular
user
name,
a
particular
user.
Maybe
this
is
some
some
specific
milk,
milk
application
or
something
and
which
is
exhibiting
a
distinct
feature
from
say
from
lamps
say,
so
they
could
build
that
kind
of
application.
Taxonomy.
B
E
B
C
B
Yeah,
so
we
have
given
you
can
you
can
pitch
in,
but
we
have
worked
on
something
called
like
variability,
so
there
is
this
run
tour
and
variability.
Can
we
use
this
network
counter
data
to
understand
whether
a
particular
run
was
run
was
influenced
by
this
congestion
or
not
so
to
that
study
like
we
call
it
like
we
get
variability
guide.
E
Yeah,
so
I
think
you
know
we
don't
get
things
like.
You
know,
insight
into
the
mpi
calls
and
everything
like
you
do,
which
is
great,
but
it
does
up.
It
does
also
open
the
opportunity
to
be
able
to
take
data,
that's
collected
through
auto,
for
something
or
something
like
that
with
the
ldms
data
and
correlate
and
see
if
you
know,
if
we're
getting
the
the
same
answers
that
we
expect,
or
at
least
given
all
the
caveats
and
in
the
different
ways
of
collecting.
B
E
You
know
I
had
to,
I
had
to
walk
out
for
a
second,
and
I
think
maybe
you
covered
this,
but
you
know
I
you're,
probably
timing.
The
mpi
calls
from
when
you
know
when
they
are
are
called
and
when
they
return,
but
you
could
have
situations
where
you're
overlapping
communication
and
other
things
and
I'm
not
sure
how
that
might
be
handled.
B
D
B
Yeah,
this
limitations
are
like
only
four.
We
are
getting
the
data
from
just
the
four
ranks
and
for
the
analysis
also,
we
were
using
the
data
from
the
average
rank.
Not
not.
We
are
not
kind
of
discarded
the
when
we
try
to
use
okay.
Can
we
get
some
idea
of
the
load
imbalance
by
comparing
the
data
from
the
rank
with
maximum
mp
time,
the
minimum
mpa
time
on
the
average
every
time,
and
that
effort
was
not
so
fruitful
yeah,
I
think
for
the
loading
balance.
E
B
Yeah,
so
I'm
in
the
next
slide,
I'm
I'm
talking
like
this
is
as
this
is
one
of
the
limitations
for
the
auto
of
2.0.
We
do
capture
the
data
from
all
the
ranks
and
the
data,
the
overhead,
at
least
from
our
experiments
on
theta,
which
this
has
been
out
of
2.0,
has
been
in
production
running
on
theta,
for
I
think
close
to
three
months,
or
so
I
mean
we
did
not
have
any
complaints,
at
least
on
the
data
overhead.
B
B
Slides
and
yeah,
so
these
are
some
yeah
one.
Other
issue
is
okay.
For
for
an
mpa
operation,
only
average
time
is
recorded,
so
there
is
no
notion
of
time.
Distribution
is
not
captured,
so
then
also
the
mpa
opera.
The
message
is
also
just
if
you
have
send
with
different
sizes
what
we,
what
we
do
is,
but
all
that,
let's
say
if
they
send
with
10
calls
nine
calls
with
one
byte
message
and
one
call
with
the
say
10k.
The
average
will
be
computed
as
10k
divided
by
10
like
and.
B
B
B
And
the
other
issue
is
that
the
coverage
I
was
telling
you
that
it's
100k
logs
and
mera
et
cetera,
but
actually,
if
you
look
at
the
coverage
wise,
it's
25
of
the
jobs
only
in
that
two
year
or
three
year,
time
frame
data
that
we
looked
at
and
we
looked
into.
Why
why
we
were
not
able
to
collect
data
for
like
100
percent
of
all
the
all
the
jobs
that
are
running?
B
There
are
some
issues
we
detailed
those
issues
in
the
paper,
but
some
of
these
are
like
dependence
on
the
pmpa
profiling
interface
and
only
one
tool
can
use
this
pmp
profiling
interface.
So
if
an
application
is
linked
already
with
pmp
wrappers,
then
auto
puff
will
not
be
able
to
provide
the
log,
and
there
are
some
applications
like,
for
example,
like
5000.
That
run
quotes
not
reaching
the
finalize.
E
B
B
B
Yeah,
so
we
all
so,
if,
if
you
go
with
the
qmp
role,
at
least
the
coverage
at
least
that
issue
of
multiple
tools
linking
that
could
be
avoided,
we
are
now
coming
to
the
artup
of
2.0.
So,
as
I
was
mentioning
in
order
to
use
the
logging
and
reporting
framework
and
with
the
binary
log
we
have
made
out
of
as
a
part
of
darshan
and
the
darshan
3.3.
B
from
the
darshan
3.3.0
release,
auto
puff
sub
module
is
available,
and
so
what
we
did
was
that
as
kevin
was
mentioning
with
the
auto
puff
1.0
on
mira,
only
mp
2.0
standard
operations
were
collected,
but
now
we
kind
of
increase
the
number
of
operations
and
also
we
discarded
some
of
those
operations
which
we
think
are
not
heavily
used.
So
out
of
the
350
total
operations
in
the
mba
3.1
standard
74,
which
we
think
are
the
prominently
used,
are
intercepted
in
our
performance
zero.
B
C
C
B
In
a
report
format.
There
are
some
application
users
want
to
kind
of
okay.
They
want
to
use
this
auto
buff
as
a
profiling
tool
and
want
to
get
some
high
level
information
for
that
purpose.
We
have
this
some
single
log
analysis
tool
and
also
similar
to
our
sc
18.
Where
we
looked
at
the
multitude
log
analysis,
we
are
also
working
on
that
post
processing.
Once
we
have
kind
of
say,
hundreds
of
logs
how
to
kind
of
analyze
those
and
get
some
insights.
B
Yeah
and
so
theta,
which
uses
static
linking
and
on
the
theta
gpu,
which
uses
the
dynamic
linking
it's
not
a
straightforward
and
we
had
to
use
the
ld
preload
kevin.
Do
you
want
to
kind
of
come
in
here
like
to
explain?
Maybe
that
is
relevant
for
you
with
the
pearl
motor.
I
think
the
dynamic
linking
will
be
the
default
right.
B
D
B
G
Yeah,
so
at
alcf,
theta
is
still
running
with
static,
linking
we
never
converted
over
to
dynamic.
Linking
nurse
did
convert
over
to
dynamic
linking
on
corey.
So
I
think
you
guys
ended
up
resolving
how
you
would
put
darshan
in
a
dynamically
linked
executable
by
just
using
l
darshan
somewhere
in
the
link
line.
So
on
theta
gpu
we've
been
trying
to
to
decide
like
what
the
best
approach
is.
It's
nvidia
dgx
machine,
and
so
it
doesn't
have
like
a
a
programming
environment
similar
to
that
of
cray
or
hpes.
G
D
G
G
I
wasn't
sure
if
on
perlmutter,
you
guys
have
investigated
how
you
will
be
intercepting
if
you,
if
you're
still
planning
to
use
darshan
how
you
will
be
intercepting,
will
you
be
relying
on
craze
or
hp's
programming,
environment,
wrappers?
Or
will
you
be
doing
something
different?
If
you
assume
users
will
instead
be
using
nbcc
directly?
E
G
F
I'll,
just
chime
in
with
like
a
little
bit
of
a
comment,
I
mean,
I
think
that
this
this
stuff
is
great
and
I
hope
we
can
enable
the
auto
perf
2.0
type
collection
on
on
our
systems
as
well.
I
just
we
just
have
to
make
sure.
I
guess
that
everything
is
is
like
nicely
modular,
because
lots
of
these
performance
tools
can
conflict
with
each
other.
F
So,
for
example,
we're
using
like
the
dc
gmi
dcgmi
interface
for
the
whatever
the
data
center
gpu
thing
is
to
collect
some
usage
data
with
ldms,
and
so
you
know,
I
know
that
in
many
cases
you
can't
have
more
than
one
kind
of
profiler
or
sampler
trying
to
to
touch
that
the
same
data
and
the
same
thing
for
like
cpu
performance
counters
right,
like
ldms,
really
wants
to
control,
which
ones
you're
looking
at,
and
we
have
to
turn
that
off.
If,
like
a
user,
turns
on
a
profiling
tool,
for
example,.
C
But
there's
a
fundamental
difference
here:
right,
ldms
works
in
on
the
system
side
of
the
house,
it's
running
as
root,
basically
auto
purpose,
running
in
user
space,
so
that
gives
you
different
set
of
pros
and
cons,
but
as
you're
right,
brandon
meshing
them
together.
It's
actually
really
quite
difficult.
F
E
G
How
does
that
work?
I
can
just
provide
one
comment
we
can
get
into
this
more
later,
but
one
of
the
kind
of
aspects
that
we
have
with
the
combination
of
auto
perf
and
darshan
is
to
try
to
maintain
a
modular
approach
and
so
yeah.
If
you
were
decided
like
you,
you
provide
a
module
for
collecting
some
data,
but
we
don't
want
to
use
that.
You
know
you
can
just
build
your
version
without
that
module.
G
Like,
for
example,
auto
perf
contains
kind
of
two
modules
right
now
from
the
darshan
perspective,
they're
independent
modules,
but
we
just
pin
them
under
this
concept
of
auto
perf
one
is
network
counters
one
is
mpi
counters.
If
you
only
wanted
mpi
and
not
network
right,
you
can
just
enable
one
or
the
other,
or
vice
versa.
B
B
G
B
G
G
G
G
G
The
idea
is:
each
module
is
sort
of
standalone
in
the
sense
that
the
core
parts
of
darshan
don't
know
anything
about
the
counters
in
here,
but
the
module
itself
understands
all
the
counters,
so
you
can
just
make
kind
of
arbitrary
new
modules.
Have
them
add
counters
and
the
run.
You
know
the
darshan
runtime
will
manage
like
here's,
how
much
memory
you
have
here's
what
you
can
do.
It
will
help
you
with
the
reduction
step.
If
you
want
and
things
like
that,
and
so
we
ended
up
deciding.
G
Let's
just
use
all
of
this
existing
tooling
that's
got
a
lot
more
seasoning
than
the
stuff
we
wrote
for
auto
perf,
one
which
was
specific
to
mira
and
then
just
build,
auto
purfa's,
separate
modules,
and
so
when
we
created
auto
perf
2
slight
difference
is
that
this
is
the
first
module,
that's
external
to
the
darshan
repo.
So
all
of
these
other
modules,
you
would
just
find
in
the
darshan
repo,
but
ours
is
actually
lives
in
its
own
repo
and
there
is
still
some
source
that
requires
you.
You
have
to
modify.
G
It
would
be
nice
if
we
could
somehow
extract
this
part,
but
I'm
not
a
configure
expert
to
know
how
to
move
parts
external
and
have
them
included
in
the
configure
ac,
but
so
there's
still
some
remnants
that
have
to
go
into
the
main
darshan
source,
but
we
try
to
put
most
code
into
the
autoproof
repo,
that's
specific,
to
auto
perf
and
currently,
at
this
time
to
build
auto
perf2.
It
can
only
be
built
in
the
context
of
darshan,
so
it
doesn't
stand
alone
at
this
time
in
the
future.
G
That's
one
thing
we've
kind
of
tried
to
consider
is:
maybe
we
can
bring
back
like
the
most
basic
build
system
and
like
put
an
ascii
logging
that
uses
the
existing.
You
know
basically
darshan
callbacks,
but
just
can
write
out,
maybe
an
ascii
log
or
something
so
we
could
build
this
and
run
it
separately
for
russian
for
testing,
but
so
far,
that's
like
something
far
down
the
road
and
we
just
mainly
focusing
on
auto
perf
as
a
component
of
darshan.
G
I
think
we've
had
the
python
framework
for
like
two
years,
but
we
only
finally
were
able
to
like
productize
it
and
put
it
in
a
release
recently,
and
so
this
kind
of
lists
a
python
wrapper
around
the
existing
c
library
for
managing
log
files,
and
so
it's
called
pydarshan.
You
can
get
it
from
this
url
the
core
source
for
all
the
python
part
is
in
the
darshan
repo.
G
But
the
idea
is
this
is
going
to
be
the
future
of
analysis,
tools
and
building
them
on
this,
because
it's
a
little
bit
easier
to
build
tools
on
this
python
framework
than
it
is
directly
on
the
c
framework
and
so
for
the
auto
perf
2
we're
we're
focusing
on
building
all
our
tools
in
this
pie.
Darshan
framework
going
forward,
and
so
integrating
these
with
the
existing
summary.
So
on
the
right
side
is
like
graphs
from
our
old
summary,
then
on
the
left.
G
G
Yeah,
so
one
of
the
good
and
bad
things
is
the
the
core
report
object
can
give
you
data
in
three
different
formats,
and
I
think
we
have
three
different
formats,
because
we
didn't
know
which
one
we
should
have
so
we'll
give
you
back
data
as
dictionaries
or
you
can
get
back
data
as
numpy
arrays
or
you
can
get
data
back
as
a
pandas
framework
or
a
pandas
data
frame.
Sorry,
so
I
think
most
of
the
work
we've
done
is
all
using
the
pandas
data
frames.
C
G
E
What
useful
information
yeah
this
is
this
is
this
is
really
cool.
I
found
one
thing
that
was
all
was
useful.
I
thought
was
kind
of
a
two-dimensional
array,
kind
of
a
heat
map
of
various
things
you
want
to
look
at
because
you
can.
You
can
really
pick
out
load
imbalance.
You
know
at
a
glance
and
that
kind
of
thing.
G
G
E
One
issue
that
we
would
get-
and
I
guess
this
would
apply
to
darshan
too-
is
that
the
overhead,
mostly
called
by
io,
would
be
just
enough
to
put
push
the
job
time
limit
of
beyond
what
the
user
has
set
for
their
limit
and
so
basically
we'd
hit
a
wall
clock
limit.
I
don't
know
if,
if
you
see
that
often-
and
if
so,
how
you
might
deal
with
that.
E
G
G
E
G
Yeah,
yes,
so
so
far,
we've
it's
we've.
You
know
phil
and
shane
have
spent
some
time
working
on
that
and
the
log
sizes
are
still
relatively
small.
So
we've
set
some
general
options.
We
use
mpio
for
writing
this
data
out
so
that
it
only
uses
like
four
aggregators
and
writes
it.
It's
it's
usually
pretty
efficient,
like
the
the
log
file
names
actually
encode.
The
time
spent
writing
the
log
and
they're,
usually
not
more
than
two
seconds.
F
G
G
F
G
Yeah,
so
in
the
I
o
side
you
can
have
much
bigger
logs,
because
this
data
is
like
whatever
the
number
of
ranks
is
you're
having
a
fixed
number
of
counters.
But
if
you
open,
like
seven
million
files
on
the
I
o
side,
the
counters
are
per
file.
So
now
you
have
seven
million
records
yeah,
so
those
have
far
much
more
variability
than
the
mpi
side.
E
E
B
F
F
And
in
order
for
that
to
be
like,
you
know,
there's
a
little
progress
bar.
So
as
long
as
I
can
load
something
and
turn
it
into
an
image
in
less
than
say,
30
seconds,
or
so
that's
fast
enough
that
that
you
can
that
we
could
think
about
integrating
this
into
that
framework
versus
like
having
to
do
like
a
you
know,
a
button
to
request
an
analysis.
Try
again
in
an
hour
when
it's
done
sifting
through
logs
yeah.
G
G
C
D
G
E
B
So
we
have
some
so,
as
kevin
mentioned,
we
have
this
network
counter
data.
We
call
that
apxc
for
the
xc
system
and
we
would
like
to
do
that
for
the
slingshot
as
well
and
for
the
gpu
we
have
this
gpu
module
using
the
tau
infrastructure,
one
student
from
the
oregon.
He
worked
on
it,
so
he
he
thought
that
using
the
tau
profiling
infrastructure
would
be
the
ideal
thing
and
so
that
he
had
he
had
a
work.
He
has
a
working
prototype
of
that
on
the
nvidia,
gpus
and
yeah.