►
From YouTube: 2021-03-09 - Shane Snyder - Darshan: Enabling IO Understanding in an Evolving HPC Landscape
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title: Darshan: Enabling Application IO Understanding in an Evolving HPC Landscape
Speaker:
Shane Snyder (Argonne National Laboratory)
Abstract:
Darshan is a lightweight, application I/O characterization tool that captures detailed statistics describing an application's I/O workload. Installed and enabled by default at many production HPC facilities (including at NERSC), Darshan has become an invaluable tool for users, system admins, and I/O researchers to investigate and tune the I/O behavior of applications. While the initial focus of Darshan was on instrumenting file-based APIs (e.g., POSIX, MPI-IO) for MPI applications, much recent work has focused on extending Darshan to new contexts that are increasingly relevant in the HPC community, including object-based storage APIs (e.g., DAOS) and non-MPI computational frameworks (e.g., Spark, TensorFlow). In this seminar, we describe how users can leverage Darshan to better understand the I/O behavior of their applications. We provide details on how users can produce Darshan instrumentation data for their applications and how to further analyze this data, focusing specifically on the Cori system at NERSC. New and upcoming features are covered that aim to extend Darshan to exciting I/O instrumentation contexts for HPC, including instrumentation modules for HDF5 and DAOS libraries, as well as support for instrumenting non-MPI applications and frameworks. We further walk through a couple of Darshan log analysis examples to help illustrate the types of I/O insights that can be attained using Darshan log data and analysis tools.
Bio:
Shane Snyder is a software engineer in the Mathematics and Computer Science Division of Argonne National Laboratory. He received his master's degree in computer engineering from Clemson University in 2013. His research interests primarily include the design of high-performance distributed storage systems and the characterization and analysis of I/O workloads on production HPC systems.
B
All
right,
hopefully,
you
guys
can
hear
me
and
see
my
screen
and
everything
is
all
good,
but
I'm
going
to
be
talking
today
about
a
tool
called
darshan,
I'm
one
of
the
lead
developers
on
darshan,
it's
a
I
o
characterization
tool
developed
at
argonne,
and
today
I'm
going
to
be
talking
about
how
you
can
use
darshan
to
enable
application.
I
o
understanding
and
doing
so
in
a
hvc
landscape.
B
In
our
case
we're
looking
at
you
know
I
o
and
storage
systems,
and
in
this
case
you
know,
there's
a
deep
layers
of
I
o
libraries
that
have
to
coordinate
with
each
other.
So
you
know
traditional
mpio
stack
going
down
to
the
storage
layer,
but
that's
also,
you
know,
exposed
to
hdf5
and
other
higher
level.
Io
libraries,
that
really
kind
of
are
super
convenient
for
users,
but
they
obscure
a
lot
in
the
ways
of
the
performance
implications
of
doing
specific
things.
B
You
know
there's
also
entirely
new
to
hpc
storage
software,
so
things
like
object,
storage,
intel
deos
comes
to
mind
as
things
that
are
going
to
have
to
be
I'm
content
with
going
forward.
So
there's
already
kind
of
a
you
know:
a
rich
landscape
of
different
io
software,
libraries
and
middleware
and
stuff
that
has
to
be
contended
with.
B
B
Io
analysis
tools
are
really
helpful
for
helping
to
navigate
this
complexity
and,
to
you
know,
offer
users
tools
to
better
understand.
I
o
behavior.
So
this
this
could
be
individual
users,
characterizing
io,
behavior
of
their
jobs,
to
help
inform
tuning
decisions,
but
it
could
also
be
broader.
You
know
system-wide,
I
o
stack
characterization
using
you,
know
all
kinds
of
jobs
that
might
run
on
a
system
to
get
a
better
feel
for
you
know
how
storage
resources
are
used
more
generally
and
ideally
using
that
information
to
optimize
deployments.
B
So
I
mentioned
we're
going
to
be
talking
about
a
tool
called
darshan.
Today,
again,
it's
an
I
o
characterization
tool
for
applica
hpc
applications,
so
just
we'll
be
covering
a
lot
about
darshan
today,
but
I
want
to
start
with
some
some
basics,
so
hopefully
there's
a
little
bit
of
information
for
not
only
new
users,
but
also
existing
users,
as
we
kind
of
move
through
the
the
presentation,
but
we'll
start
with
some
some
background.
Obviously
so
darshan
is
just
a
lightweight.
I
o
characterization
tool
that
captures
concise
views
of
application.
B
I
o
behavior
it
basically
for
every
job
that
instruments
it
produces
a
summary
of
I
o
activity
for
for
the
for
the
job,
so
this
could
be
counters,
histograms,
timers
and
other
sorts
of
statistics.
Basically,
just
you
know
what
what
kind
of
information
we
can
capture
in
a
fixed
amount
of
space
for
for
each
job,
that's
intermittent
instrumented.
B
B
Some
characteristics
about
darshan
that
make
it
attractive
to
use.
First,
it's
it's
widely
available,
so
it's
deployed
and
usually
enabled
by
default
at
a
number
of
hpc
facilities
around
the
world,
including
at
nursk.
It's
easy
to
use.
So
there's
no
code
changes
required
on
users
behalf
they
can.
Just
basically,
you
know
the
most
they
would
have
to
do
is
recompile
their
application
to
get
darshan
instrumentation
and
there's
there's
negligible
performance
impact.
B
So
there's
on
this,
this
notion
that
you
basically
just
turn
it
on
and
leave
it
on
it,
just
kind
of
not
really
obvious
to
users
that
there's
something
happening
in
the
background
and
it's
also
modular.
So
this
is
a
this
one
gets
us
a
lot
of
bang
for
our
buck
and
that
we
can
continuously
enhance
or
extend
darshan
to
instrument
new.
I
o
interfaces
or
storage
components.
So
that's
also
another
another
really
nice
feature.
C
B
B
And
that's
that's
basically
what's
happening
at
nurse
now
is
there's
a
default
darshan
module
available,
that's
capturing,
for
you
know,
applications
that
have
been
compiled
there.
You
know
instrumentation
and
logs
that
users
can
access.
I
can
I'll
get
more
into
that
in
a
few
slides.
Hopefully,
that'll
that'll
answer
some
questions
and
if
please
please
stop
me
if
this
is
a
long
presentation,
so
it
might
be
nice
to
have
a
little
bit
of
a
conversation.
B
B
A
lot
of
times
is
for
static
and
dynamic
executables
just
offer
darshan
compiler,
wrappers
or
other
ways
to
integrate
into
the
compiler,
so
that
darshan's
linkedin,
and
that
you
get
your
instrumentation
directly,
there's
also
an
ability
to
use
led
preload
for
dynamic
executables,
in
which
case
at
run
time.
You
just
point
at
the
darshan
library
an
instrumentation
happens
that
way,
starting
in
version
320.
We
actually
have
the
ability
to
instrument
any
dynamically
linked
application.
B
This
is
important
because
traditionally
darshan's
been
paid
to
mpi,
but
we've
we've
broken
that
dependency,
at
least
for
dynamic
executables,
which
is
I'm
going
to
be
we'll
talk
about
this
more
in
some
future
slides,
but
something
that
we're
excited
about
more
about
how
darshan
works
so
essentially,
there's
this
notion
of
instrumentation
modules
that
you
can
see
on
the
right
side.
This
figure
here
so
there's
instrumentation
modules
for
different
layers
of
the
stacks
like
hdfi
and
mpio,
and
those
are
responsible
for
implementing
function.
Wrappers.
B
For
you
know,
I
o
calls
that
we
want
to
instrument
on
capturing
statistics
we
want
to
gather,
and
so
that
basically
happens
as
the
application
executes
and
then
what
happens
at
shutdown
time
is
that
darshan,
the
darshan
core
library
will
collect
all
this
information
aggregate.
It
compress
it
and
write
out
the
log
file
and
then,
after
this
you'll
have
a
log
file
that
you
can
use.
B
So
next,
we'll
cover
what
might
be
of
interest
to
a
lot
of
you
folks
is
how
to
use
darshan
on
corey
so
for
traditional
mpi
applications,
which
is,
as
I
mentioned,
what
darshan
has
traditionally
been
most
concerned
with
or
tied
to.
I
guess
is
we
have
you
know
on
and
or
installed
in
on
by
default
software
module
available
on
nurse
cory.
So
if
you
log
into
cory
and
use
module
list,
you
should
see
a
darshan
321
module
there.
B
That's
the
currently
default
version,
if
not,
for
whatever
reason,
if
you
mess
with
your
environment
or
whatever,
you
can
obviously
always
explicitly
module
load
it,
but
essentially
what
this
does
is.
It
gets
a
darshan
software
module
and
the
environment
that
knows
how
to
interact
with
cray
compilers,
to
insert
darshan
linker
flags
to
integrate
the
instrumentation.
So
it's
pretty
transparent.
B
For
hdf5
applications,
so
this
is
also
for
mp
applications,
but
those
that
are
using
hdf5.
We
have
a
separate
module
specifically
for
these,
so
if
you
have
an
hdf5
application
and
you
want
to
know
more
details
about
hdf5
api
usage,
we
have
a
darshan
instrumentation
module
that
can
provide
you
that
sort
of
information,
but
it
is
not
part
of
our
default
darshan
module
on
quarry.
B
The
reason
for
this
is
mostly
just
because
darshan
with
hdf5
support
obviously
has
a
library
dependency
on
hdf5,
and
we
don't
want
to
you
know,
carry
that
around
to
all
applications
running
or
compiled
on
on
quarry.
So
basically
we
don't
want
to
introduce
a
hdf5
dependency
for
non-hdf5
applications,
which
is
understandable
and
obviously
you
can
use
use
module
switch
to
switch
from
the
default
321
darshan
version
to
the
hdf5
enabled
one
as
I
show
here.
B
B
Okay,
so
we've
covered
how
to
basically
get
a
darshan
module
on
query
into
your
environment
so
that
the
question
is
now
what
so.
Basically
at
this
point,
you
can
just
compile
and
run
your
application.
Darshan
will
integrate
directly
with
the
cray
compiler
wrapper
and
insert
the
necessary
linker
flags
to
get
darshan
into
the
executable,
regardless
of
whether
you're
using
static
or
dynamic
blinking.
B
As
I
mentioned,
ld
preload's
another
option
for
dynamically
linked
executables
we've
noticed
that
this
is
definitely
a
requirement
for
python
frameworks,
so
things
like
mpf
or
pi,
and
h5
pi,
those
obviously
python's
dynamically
linked
itself.
So
you
can't
recompile
python
with
direction
support.
So
obviously
your
only
other
option
is
to
use
ld
preload
to
insert
the
instrumentation
at
runtime,
but
that
that
tends
to
work.
It's
not
necessarily
a
big
deal,
it's
just
something
worth
pointing
out
for
for
python
users.
B
This
ld
preload
mechanism
is
also
useful,
just
in
general
for
applications
that
can't
you
can
recompile
it
or
just
don't
want
to
recompile.
It
gives
you
an
option
to
insert
darshan
instrumentation
without
going
through
that,
if,
if
that's
useful
for
you
so
here
I
have
some-
you
know
quick
snippets
to
show
how
to
set
led
preload.
B
Another
thing
I
slightly
hinted
at
earlier
is
our
support
for
non-mpi
applications.
So
this
is
something
that's
relatively
recent,
so
starting
in
darshan
version
320,
we
support
instrumentation
of
non-mpi
applications
using
ld
preload.
So
again,
this
is
only
for
dynamically
linked
executables
users
must
additionally
set
a
different
environment
variable
here
to
to
signal
to
darshan
that
you
definitely
want
to
capture
non-mpi
information
and
to
write
logs.
B
The
reason
for
this
is
mostly
just
because,
if
users
set
ld,
preload
and
kind
of
forget
about
it,
that
darshan
will
just
generate
log
files
for
literally
every
process,
that's
being
run
in
the
environment,
so
you'll
get
log
files
for
ls
and
ps
and
all
kinds
of
different
things.
You're,
probably
not
really
interested
in
so
it's
important
to
maybe
kind
of
limit
the
scope
of
what's
being
instrumented
by
setting
this
for
regions
of
interest.
B
One
thing
worth
noting:
if
users
are
interested
in
trying
this
is
that
the
darshan
321
module
on
quarry
that's
available
by
default
has
a
bug
that
prevents
non-mpi
mode
from
working
properly.
So
you
can't
just
you
know,
load
that
module,
set
ld
preload
and
set
this
environment
variable
and
run
because
there's
a
bug,
there's
no
theoretical
reason.
This
shouldn't
work.
So
this
is
something
we'll
have
cleaned
up
in
future
releases
so
that
this
default
nurse
module
could
support
both
use
cases.
But
in
the
meantime
it
is.
It
is
broken.
B
So
what
what
users
can
do
to
work
around
this
if
they're
interested
in
the
mean
time
of
getting
non-mpi
instrumentation,
is
to
just
do
like
a
spac
install
or
an
otherwise
manual
install
of
darshan
themselves.
So
I
have
a
simple
slurm
recipe
here
for
installing
darshan
runtime
library,
without
npi
support
and
with
form
support,
so
that
could
you
know
something
that
you
could
do
yourself
manually
if
you're
interested
in
testing
this
out?
But
we
will,
you
know,
get
this
squared
away
going
forward.
B
So
after
your
application
terminates,
you
know,
regardless
of
whether
you
used
ld
preload
or
the
the
the
linker
options
for
darshan.
You
should
have
a
darshan
log
file
that
you
can
start
analyzing.
So
the
first
step
is
to
look
for
it
for
for
system-wide
installs
of
darshan,
like
the
one
available
to
by
default
on
corey.
B
You
can
query
darshan
using
this
darshan
config
utility
to
find
out
exactly
what
the
log
path
is,
and
you
can
see
on
query
it's
located
in
the
the
global
c
scratch
file
system.
So
that's
where
you
need
to
start
your
search
for
your
particular
log
file.
So
if
you
cd
there
you'll
notice
that,
within
that
particular
sitewide
log
directory
that
there's
further,
you
know
organization
of
the
directory
hierarchy
into
a
date,
so
there's
a
year
directory
a
month
directory
and
then
a
day
directory.
B
So
you
basically
have
the
cd
further
down
into
the
data.
You
ran
your
job
of
interest
and
then
you
can
find
the
log
file
there
and
all
the
log
files
have
the
same.
You
know
pattern
for
their
their
naming,
so
it
all
starts
off
with
the
username.
So
in
my
case
starts
off
with
s
schneider,
followed
by
an
executable
name.
So
this
one
was
mpio
test
and
then
you
can
see
it's
followed
by
a
job
id
and
some
other
kind
of
metadata.
B
One
thing
worth
mentioning
is
that
the
spec
installed
version
of
darshan
simply
puts
log
files
in
your
home
directory
by
default,
just
to
have
somewhere
to
put
things
you
can
override
that
with
an
environment
variable
to
give
it
another
directory.
You
just
kind
of
store
things
in
this
isn't
formatted
the
same
way
as
a
systemwide
directory.
It
doesn't
have
like
dates,
and
things
like
that,
so
it's
just
literally
a
huge
directory
to
store.
You
know
individual
log
files
in
so
it's
a
little
bit
different
but
should
be
manageable.
B
B
So
there's
some
darshan
log
utilities
that
can
help
here
so
generally.
What
I
do
as
a
starting
point
is
just
copy
the
log
file
somewhere
else.
You
know
either
to
my
laptop
or
to
another
directory,
just
to
kind
of
hold
it
and
make
sure
I
don't
lose
track
of
it,
and
then
you
can
just
operate
on
it
directly
using
the
darshan
utilities.
So
the
first
one
I'll
cover
is
just
this
darshan
parser
utility,
and
these
will
all
be
available
in
darshan
path,
if
or
sorry
on
the
path
environment
variable.
B
If
you
have
the
darshan
module
loaded.
So
keep
that
in
mind.
This
should
all
be
available
for
you,
but
essentially
you
run
darshan
parser
on
the
log
file
and
you
can
get
detailed
counters
describing
accesses
to
it.
So
I
have
some
example:
information
here
for
mpio
module
and
darshan's
posix
module.
That
shows
the
type
of
information
we
capture
and
the
kind
of
the
format
for
it.
All
the
modules
follow
this
this
general
format,
but
it
shows
the
module
that
captured
the
particular
counter
information,
so
posix
or
mpio,
then
the
rank.
B
So
in
this
case
negative
one
just
indicates
that
this
was
a
shared
record,
so
every
single
process
and
the
application
accessed
it.
Then
there
is
a
record
id
which
is
just
a
unique
identifier
that
darshan
has
for
records
and
then
there's
the
counter
information,
so
the
counter
value
and
also
the
counter
type.
So
we
have
things
like
opens
or
read
and
write
counts
and
things
like
that.
But
that's
only
just
like
a
you
know,
a
small
snippet
of
what
we
capture
just
to
give
an
idea
of
what
starstream
parser
can
can
provide
it.
B
B
So
if
you
know,
as
I'm
kind
of
alluding
to
this,
isn't
so
accessible
for
a
lot
of
users,
we
do
offer
another
tool
called
darshan
job
summary,
which
seems
to
be
pretty
popular.
It
generates
a
pdf
summary
of
application.
I
o
behavior.
So
it's
you
know
some
nice
figures
and
graphs,
and
things
like
that.
The
users
can
look
at
and
not
be
overwhelmed
by
the
the
parser
data,
this
particular
utility.
We
do
recommend
or
yeah.
B
I
typically
recommend
to
to
copy
it
somewhere
off
the
production
systems,
mostly
just
due
to
latex
and
pearl
dependencies,
that
kind
of
make
it
difficult
to
get
the
tool
to
work
properly.
If
you
do
it
on
your
laptop,
you
have
a
lot.
You
know
more
success
and
installing
whatever
packages
you
need
with
your
package
manager
or
whatnot,
so
just
a
a
general
tip
or
trick
for
that
in
case
things,
don't
work
directly
for
you.
B
In
any
case,
once
you
invoke
darshan
job
summary,
you
just
pass
it
a
log
file
like
the
ones
we
copied
and
then
you'll
see
down
at
the
bottom
here
that
there
is
an
output
pdf
file,
that's
generated
based
on
the
darsen
log
file,
name,
see
yeah,
so
just
generally
go
walking
through
the
kind
of
information
this
provides
just
starting
at
a
high
level.
It's
just
just
generates
a
multi-page
pdf
containing
a
bunch
of
tables
and
graphs
performance
estimates
and
things
like
that.
The
users
can
use
to
kind
of
orient
themselves
to
general.
B
So,
starting
with
the
header,
so
at
the
very
top
you'll
get
some
some
high-level
metadata
for
the
job,
so
that
includes
you,
know
job
id
and
user
id,
but
also
job
size.
In
terms
of
you
know,
number
of
processes
how
long
it
took
to
run
generally
useful
things
to
have,
after
the
effect
in
the
darshan
log
file.
B
There's
also,
I
performance
estimates
that
we
can
provide
at
different
interface
layers.
So
here
you
can
see
that
we're
able
to
provide
a
general
mpio
bandwidth
value
and
similarly
something
for
standard
io,
which
is
you
know,
the
file
stream
api.
So
we
have
a
few
different
estimates.
We're
able
to
provide
just
straight
off
the
bat.
B
Looking
at
some
of
the
graphs
one
of
the
the
first
ones
is
average,
I
o
cost
per
process
which
I
have
here
in
the
upper
right
corner.
You
can
see
that
this
captures
across
a
few
different
apis,
so
across
posix,
mpio
and
standard
io
read,
write
and
metadata
average
timings
for
each
across
all
processes.
So
I
can
show
you
like,
on
average,
how
much
time
is
being
spent
by
all
processes,
doing
metadata
operations
versus
read
or
write
and
and
so
on.
B
One
thing
that
you
can
look
for
in
this
particular
graph
is,
if
there's
a
lot
and
what
this
in
this
picture,
the
pink
this
other
or,
basically
anything.
That's
not.
I
o
so
compute.
If
you
see
the
computes
dominating
this
particular
graph
and
there's
limited
opportunities
for
I
o
tuning.
So
that's
one
thing,
one
obvious
takeaway
you
could
make
from
this
particular
one
in
the
bottom
right.
B
There's
some
more
io
operation
count
histograms
kind
of
showing
you
the
for
different
api
levels
or
api
layers,
what
what
operations
were
used
and
the
relative
intensities
of
them.
So
one
thing
you
could
look
for
here
is
you
know
like
a
a
lot
of
metadata
operations
or
things
like
that.
We've
noticed.
Obviously
those
can
have
a
adverse
performance
impact
for
applications.
So
it's
something
that
you
could
look
for
in
this
particular
operation.
Count
graph.
B
B
So
if
you
tend
to
see
a
lot
of
really
small
like
on
the
order
of
kilobytes
or
so
access
sizes,
that's
a
an
obvious
place
where
it
would
be
useful
to
to
try
and
do
some
optimizations
to
aggregate
those
requests
into
larger
access
sizes
that
perform
better
and
there's
also
some
some
tables.
So
as
an
example,
there's
a
file
count
summary
table
that
shows
you
know
for
different
file
types.
B
B
One
of
the
last
things
I'll
cover
is
a
little
bit
of
timing
information
and
these
pdf
summaries,
so
we're
able
to
provide
not
only
for
independent
files,
which
I
have
here.
So
you
know
files
that
are
accessed
by
a
single
processor.
Something
like
that
or
you
know
multiple
processes
independently.
In
this
case,
you
can
see
the
the
balance
for
when
read
and
write
activity
was
actually
happening.
B
B
D
B
Yeah,
that's
that's
absolutely
right!
So
this
this
captures
it's
not
a
trace
or
anything
so
it
doesn't
have
like
every
operation,
but
it
will.
It
will
keep
for
you
know
each
rank
the
time
that
it
first
read
something
and
the
time
that
it
last
read
something
and
you
have
to
kind
of
you,
don't
necessarily
know
what
the
intensities
of
the
operations
were
within
those
bounds,
but
it
does
at
least
give
you
a
starting
point
to
know.
E
D
B
B
B
If,
instead,
you
want
to,
you
know,
focus
it
in
on
a
particular
file
of
interest,
then
you
can
use
the
starting
convert
utility
to
transform
a
existing
darshan
log
into
a
darshan
log
that
contains
only
a
specific
file
record
of
interest
or
something
like
that.
And
then
you
can
pass
that
back
through
some
of
these
other
utilities
to
get
a
more
focused
analysis
and
and
kind
of
along
those
lines.
There's
a
darshan
summary
for
file
tool
that
will
just
do
that
version.
Job
summary
for
every
single
file.
B
So
that's
helpful
in
some
specific
use
cases
we
don't!
You
know
recommend
that
in
general,
because
there
could
be
a
lot
of
logs
or
a
lot
of
files
instrumented,
so
it
could
take
a
while
to
generate
all
the
summaries
for
them
so,
but
it
is
possibly
useful
first
for
applications
that
don't
touch
a
lot
of
files.
B
Going
back
to
this
timing,
stuff
that
I
shared
on
a
few
slides
back
but
for
shared
files
in
particular,
it
kind
of
seems
that
darshan
isn't
able
to
provide
as
finer
grained
information
as
you
might
want.
So,
as
you
can
see
here,
we're
looking
at
shared
file
access,
so
the
top
two
panes
are
related
to
independent
files.
So
there's
nothing
there
and
the
shared
files.
You
can
see
that
the
the
timing,
durations
kind
of
spannel
or
particularly
for
right,
span
the
entire
application
execution
time.
B
What
what's
happening
here
is
that
darshan
collapses
records
that
are
shared
across
all
processes
into
a
single
record
that
collectively
describe
the
access.
So
it
throws
out
a
lot
of
the
timing,
information
for
individual
processes
and
just
provides
you
like
a
global
view
of
it,
which
is
fine,
for
you
know
a
lot
of
depending
on
what
sort
of
analysis
you
want
to
perform.
But
if
you're
interested
in
some
of
the
finer
grain
details
of
collective
io,
then
you
probably
want
to
disable
that
shared
file
reduction.
B
So
there's
a
environment
variable
you
can
set
it
runtime
to
for
darshan
to
say,
don't
reduce
any
shared
records.
Just
keep
all
the
process
information
and
then
that
will
produce
these
sorts
of
graphs.
It'll
convert
the
shared
file
graph
from
the
previous
page
into
something
finer
grained
per
process
like
we
have
here,
and
it
will
give
you
obviously
a
little
bit
more
information
that
would
be
otherwise
obscured
by
the
darshan's
shared
file
reduction.
D
B
And
sorry,
lastly,
obtaining
finer
grain
traces
with
dxt,
so
dxt
is
this:
darshan
extended
tracing
module
that
comes
with
darshan.
This
is
something
that
you
can
request
at
runtime,
so
you
need
to
set
a
environment
variable
to
request
trace
data.
This
dxd
enable
I
o
trace
variable
and,
if
that's
set,
then
darshan
will
start
up
a
couple
of
additional
modules
to
instrument
mpio
and
posix
to
trace
those
back.
Basically,
so
it'll
use
a
fix
as
buffer
and
try
and
capture
as
much
tracing
information.
B
It
can
and
store
that
with
the
darshan
log
file,
and
similarly,
we
have
a
darshan
dxd
parser
that
can
be
used
to
dump
the
text.
Format
trace
data,
as
you
can
see
here,
it's
kind
of
like
a
similar
format
to
our
other
parser
tool,
but
it's
not
counters.
It's
actual
individual
read
and
write
operations.
B
We
also
we
don't
have
a
whole
lot
in
the
way
of
dxt
analysis
tools.
Yet
that's
something
we're
interested
in.
We
do
have
one
dxd
analyzer
tool
that
was
contributed
that
can
read
the
text
output
from
that
vxt
parser
tool
and
convert
it
into
a
graph
like
this.
That
shows,
you
know
for
ranks
on
the
y-axis
and
time
on
the
x-axis.
B
What
the
where
read
write
activity
was
happening
in
an
application,
so
this
is
again
useful
for
you
know
if
you,
if
you
don't
think,
darshan
was
able
to
provide
you
the
information
that
you
that
you
wanted
with
its
default
instrumentation
mode.
This
will
provide
you
a
little
bit
more
finer
grained
details
that
you
can
dig
through
yourself.
B
Okay,
so
that's
hopefully
that's
useful
for
folks
in
the
audience
who
don't
know
much
about
darshan
or
haven't
had
a
chance
to
use
it
a
lot.
What
I'll
cover
next
is
still
useful,
for
you
know
new
users,
but
also
really
probably
helpful
for
existing
darshan
users
that
don't
know
about
what
we've
been
up
to
so
one
thing
that
I've
already
teased,
that
was
this
hdf5
instrumentation
available
on
darshan,
so
just
to
motivate
this
something
we
did
in
our
ecp
project.
B
B
How
are
data
sets
organized
within
files
or
are
files
containing
tons
of
different
data
sets?
Is
it
generally
just
a
few
of
them?
Is
there
any
sort
of
performance
implication
from
having
a
lot
of
data
sets
in
a
file
and
so
on?
And,
lastly,
how
do
you
know
high-level
hdf5
library,
accesses
decompose
into
the
lower
level,
mpio
and
posix
accesses
where
performance
is
really
controlled
at
and
if
not,
do
any
optimizations
make
sense?
So,
just
kind
of
simple
questions
like
that
that
can
better
inform
on
hdf5
usage.
B
Just
to
kind
of
quickly
walk
through
the
types
of
information
we're
capturing
in
each
and
each
h
in
each
api.
Sorry
for
hdf5
file.
Api
there's,
really
not
a
lot
of
interest
that
we're
capturing,
there's
just
not
really
a
whole
lot
of
insightful
stuff
that
happens
at
this
layer,
mostly
it's
just
file.
B
Basically,
this
operation
counts
for
the
file
open,
so
anytime
the
user
opens
or
creates
a
file.
We
instrument
that
and
also
any
flush
operations
initiated
to
a
file
beyond
that.
We
capture
whether
the
user
requested
to
use
the
mpio
driver,
and
then
we
also
capture
some
metadata
timing.
So
the
amount
of
time
spent
in
like
open
calls
and
things
like
that.
B
So
not
really
a
whole
lot
super
interesting
here,
but
it
is,
you
know,
helpful
for
for
mapping
hdf5
file
apis
down
to
on
what's
happening
and
data
sets,
which
is
what
we'll
cover
next
so
for
hdf5
data.
Instrumentation.
There's
a
lot
more
of
interesting
stuff
happening
here,
because
this
is
where
the
actual
read
and
write
calls,
and
things
like
that
are
happening
so
again,.
C
B
B
We
also
capture
information
on
selection
types,
so
whether
users
are
using
just
like
regular,
hyper
slab
selections
to
select
big.
You
know
chunks
of
a
data
set
whether
they're
using
regular,
hyper
slabs
to
select
a
bunch
of
you
know
not
not
as
well
defined
portions
of
the
data
set
and
then
also
point
selections
for
use
cases
where
people
are
selecting
just
particular
points
in
the
data
set
space,
we're
capturing
information
on
data
space
dimensions
and
points
just
to
give
an
idea
of
you
know
the
dimensionality
and
how
large
these
things
are.
B
We're
capturing
chunking
parameters
to
see
if
users
are
setting
those
themselves
and
what
they're
selecting
for
them
we're
also
collecting
information
on
whether
mpio
collectives
are
used
in
hdf5.
So
that's
a
that's
an
option
that
users
can
set
in
hdf5
to
try
and
use
collectives
and
recapture
whether
they
did
so.
B
We
also
capture
whether
any
deprecated
hdf5
functions
are
being
used
still,
and
we
also
you
know,
capture
a
bunch
of
read,
write
metadata
timing,
similar
to
what
we
do
in
other
modules,
just
to
kind
of
quickly
walk
through
a
hdfive
example
that
kind
of
demonstrates
sort
of
insights
you
can
get
from
using
the
hdf5
instrumentation
module
in
darshan.
So
I
just
did
some
some
simple
test
runs
of
the
maxio
io
proxy,
so
it
has
a
hdf5
plugin
that
can
generate
some
interesting
io
workloads.
B
For
this
particular
example,
I
just
did
like
a
6d
process,
a
single
shared
file,
3d
mesh,
so
writing
about
a
gigabyte
of
data.
Using
this
benchmark
configuration
to
see
what
we
could
learn
from
the
hdf5
data.
One
thing
I,
or
the
only
thing
I
really
changed
in
between
runs-
is
whether
the
hdf5
module
was
using
collective
or
independent.
B
I
o
configurations-
and
you
can
see
down
here
on
the
bottom-
I've
plotted.
You
know
the
average
per
process
time
spent
in
io
in
each
case
for
different.
I
o
layers,
so
you
can
see
there's
a
huge
difference
in
what's
happening
here
and
that's
not
not
necessarily
surprising,
because
I'm
sure
a
lot
of
us
are
well
aware
that
collective,
I
o
can
be
a
huge
net
win
for
particular
I
o
workloads-
and
this
is
obviously
one
of
those.
B
So
on
the
left
side
you
can
see
for
the
independent
configuration
how
much
time
was
spent
in
different
api
levels.
You
know,
one
thing
that
stands
out
is
that
posix,
I
o
is
completely
dominating
the
time
here.
You
know
well,
over
two
thirds
of
the
application
runtime
is
in
posix.
Io
hdf5
has
a
pretty
high
overhead
itself
here,
and
mpio
is
basically
negligible.
B
You
know,
posix
accesses
and
at
the
posix
level,
these
accesses
themselves
are
generally
really
small
and
inefficient,
so
they
take
a
long
time
to
complete,
which
explains
the
the
high
posix
cost
here
and
also,
as
I
mentioned,
the
high
htf5
cost
mpio
is
doing
nothing
more
than
passing
the
independent
calls
down
to
posix.
So
it's
really
not
doing
anything
at
all
which
matches
with
the
the
timing.
B
B
As
an
example
of
some
other
information
that
we
can
get
from
instrumenting
hdf5,
we
were
looking
at
the
fluid
darshan.
Also
captures
you
know
the
most
common
accesses
issued
by
an
hdf5
application
and
it
captures
information
on.
You
know
like
what
dimensions
you
know
what
size
along
which
dimensions
were
being
accessed
in
the
the
data
set,
read
or
write,
and
we
plotted
those
here
with
some
radar
plots
and
there's
a
couple
of
different
configurations
here.
B
B
This
information
isn't
super
useful
in
itself,
but
what
we
think
could
be
useful
is
comparing
this:
these
sorts
of
access
patterns
back
with
hdf5
chunking
parameters
that
users
are
either
setting
themselves
or
maybe
not
setting
at
all
and
seeing
whether
the
ways
a
data
set
is
accessed
matches
with
the
way
that
it
was
configured
from
a
chunking
perspective,
because
that
obviously
can
have
big
performance
impacts.
So
that's
something
that
we're
kind
of
excited
to
try
here
is:
we
have
more
time
to
play
with
the
hdf5
data.
B
Let's
see
moving
along
one
thing
I
haven't
teased
yet
is
deus
instrumentation,
so
this
is
something
that
we're
actively
working
on
now.
So
it's
a
work
in
progress,
but
basically
we're
looking
at
how
to
integrate.
You
know:
support
for
deo's
apis
and
to
darshan,
as
that's
going
to
be,
you
know
useful
in
our
next
generation
systems.
So
for
anyone
who's
not
aware
deos
is
basically
this
object.
Storage
system
that
intel's
offering
that's
going
to
be.
You
know
a
new
interesting
use
case
for
hpc
applications.
It's
going
to
provide.
B
You
know:
object-based
storage,
directly
over
some
hardware,
like
storage
class
memory
and
ssd
devices.
That's
going
to
have
really
nice
performance
characteristics
that
are
good
for
applications,
but
we
also
you
know
this
is
kind
of
kind
of
new.
So
it's
something
that
we
don't
fully
understand
that
well,
so
it'd
be
great
to
ask
darshan
instrumentation,
better
characterizing
this!
B
So
there's
a
couple
of
different
apis,
we're
interested
in
for
deos.
The
first
is
dfs
libdfs.
This
is
deos's
posix
file
system,
emulation,
api
and
there's
also
libdeos,
which
is
their
native
object,
key
val
api,
and
you
know
we
want
to
instrument
both
of
these.
As
we
think
it'll,
you
know
be
helpful
for
gaining
insights
into
the
different
types
of
applications
that
are
using
theos.
You
know
whether
this
is
legacy
posix
applications
that
are
using
deo's
via
fuse,
which
you
can
see
over
here
on
the
graph
to
the
right
with
the
far
right.
B
There's
a
fused
daemon
that
posix
applications
can
use
to
indirectly
use
deos
and
there's
also,
you
know,
application
and
io
middleware
direct
usage
of
deos
apis.
So
this
would
be
happening
via
hdf5
fall
or
mpio
and
so
on,
but
I'm
basically
instrumenting
direct
access
to
deos
apis
as
well
to
provide
more
information.
B
B
There's
you
know
additional
things
like
punch
and
and
stat
that
dfs
implements
that
we're
also
capturing
we're
capturing
again,
bytes
read
and
written
access
size,
histograms,
we're
capturing
davis
parameters
like
file
chunk
size,
which
controls
the
distribution
of
devs
data
inside
files,
we're
capturing
information
on
dxt
usage
dxt.
Is
this
strict
consistency
mode
for
dfs
that
forces
it
to
behave
like
you
know:
posix,
compliant
consistency
as
opposed
to
a
weaker
consistency
mode.
They
also
offer
and
we're
also
capturing
the
corresponding
object
record
id
for
the
underlying.
B
Libdeos
and
that's
so
that
we
can
map
instrumentation
records
from
libdeos
back
to
libdfs
to
kind
of
compare
how
how
they,
how
they
work
with
each
other.
I
guess-
and
we
also
again
capture,
read,
write
metadata
timing
as
far
as
as
where
we're
at
in
this
process
with
integrating
data
support,
lib
dfs
instrumentation
is
basically
completely
done.
There's
a
few
tweaks.
We
still
want
to
do,
but
this
is
available
basically
to
use.
If
folks
were
interested
in.
D
B
So
if
you
are
feel
free
to
reach
out
to
me,
but
it's
not
something-
that's
been
released
yet
something
we
still
need
to
kind
of
fine-tune
for
libdeos
itself
we're
capturing
again.
This
is
the
native
object
api
that
the
deoc
exports
we're
capturing
more
operation
counts.
So
things
like
open
fetch,
which
is
the
lib
deo's,
read
equivalent
and
update,
which
is
the
right
equivalent
we're
able
to
capture
any
enumerations
that
are
done
on
object,
keys
and
things
like
that
again,
we're
capturing
total
bytes
read
and
written
exercise
histograms.
A
B
So
this
hasn't
been
much
of
a
challenge
in
the
past,
but
with
deo's.
You
know,
a
single
object
is
an
entire
key
value
source.
There's
a
lot
of
information
that
can
be
associated
with
a
single
object,
so
we're
trying
to
think
more
about
the
best
ways
to
capture
this
sort
of
information
to
characterize
how
how
applications
are
accessing
different
keys
and
so
on
and
so
forth.
B
B
B
So
for
most
starch
analysis
tools,
we're
relying
on
some
ad-hoc
mechanisms
for
getting
darshan's
data
on
one
of
the
popular
ways
of
doing
that
is
just
take
the
full
text:
output
of
the
darshan
parser
tool
and
extract
the
data
you
want
from
the
text
there,
which
is
really
really
cumbersome
and
not
the
way
we
would.
You
know,
want
to
do
things
if
we
could,
if
we
could
say
so
so
one
thing,
we've
thing
that
we've
added
to
darshan-
and
this
is
courtesy
of
a
of
jacob
lukau
at
dkrz.
B
He
did
most
of
the
development
work
here.
I'm
so
big
thanks
to
him,
but
we've
added
some
python,
tooling
and
modules
to
help
with
the
analysis
of
darshan
log
data,
so
pi
darshan's
based
on
python
cfi,
and
it
provides
some
python
bindings
to
our
lower
level
c
api.
But
it
also
defines
some
python
bits
to
help
organize
this
data
better
and
expose
it
to
analysis
scripts
that
are
interested
in
darshan
data.
So
it
exposes,
you
know,
information
in
pandas
or
numpy
or
in
dictionaries,
as
you
might
expect.
B
So
yeah,
it's
it's!
A
beta
version
of
pi
darshan
is
currently
available
in
pi
pi.
If
you're
interested
in
it.
So
you
could,
you
can
literally
go
out
and
pip
install
darshan
now
to
get
it
and
you
can
analyze
darshan
long
files
that
way,
but
you
can
also
install
pytarson
manually
alongside
darshan
util,
if
you
build
it
manually,
so
there's
a
couple
of
different
options
there.
B
B
Quickly
cover
a
few
other
upcoming
features
that
might
be
of
interest
and
darshan
3.3.0,
which
is
our
next
release,
which
we're
hopeful
to
get
out
next
week,
we'll
have
an
auto
perf
instrumentation
module.
This
is
contributed
by
sudir
chinduri
at
alcs,
but
basically
auto
perf
provides
two
different
modules.
B
One
provides
mpi
communication
counters
for
an
application,
so
it'll
be
nice
to
have
not
only
darshan.
I
o
analysis
data,
but
if
it's
desired,
you
can
also
get
some
information
on
mpi
communication
counters
and
there's
also
a
cray
xc
module
that
has
some
compute
and
network
counters
for
cray
xc
systems.
That'll
also
be
useful
to
have
alongside
darshan
log
data.
B
B
I've
also
worked
a
bit
on
instrumenting
some
ability
in
our
non-mpi
mode
to
gracefully
handle
applications
that
call
fork
we've.
You
know,
since
non-mpi
mode
started,
started
being
used
by
users.
This
is
something
that
we've
observed
that
wasn't
working
properly
with
darshan,
so
we've
integrated
some
code
that
corrects
that
and
gives
you
same
information
for
applications
to
call
fork
and,
as
I
was
alluding
to
on
my
previous
slide,
we're
very
much
interested
in
enhancing
our
analysis
tools
and
report
generation
based
on
pi
darshan.
B
B
So
it's
using
this
hdb
scan
hierarchical
clustering
algorithm
to
organize
logs
into
different
clusters
that
exhibit
similar.
I
o
characteristics.
So
it
starts
with
a
huge
job
population
at
the
top
and
then
kind
of
creates
smaller
and
smaller
clusters
that
represent
more
unique
kind
of
io
characteristics
as
you
move
further
and
further
down.
But
what
you
can
do
is
kind
of
analyze,
different
clusters
to
see
what
performance
characteristics
they
have
and
to
see
what
other
kind
of
applications
are
similar
and
get
similar
performance.
B
There's
a
lot
of
other
kind
of
machine
learning
aspects
of
this
study
that
I'm
not
covering
mostly
because
they're
out
of
my
technical
depth,
but
also
for
time
but
users
or
anyone.
That's
interested-
should
definitely
check
this
stuff
out.
But
if
you
select
a
particular
cluster,
then
you
can
see
that
gauge
is
able
to
provide
a
lot
of
performance
characteristics
for
applications
in
that
cluster.
So
it
can
provide
information
on
the
user
and
application
details.
B
They
can
provide
parallel,
coordinate
plots
that
show
the
general
performance
of
different
configurations
and
general
performance
general
access
characteristics
at
the
very
bottom.
So
for
these
jobs
like
what
kind
of
access
sizes
are
being
used
and
things
like
that,
but
it's
really
a
good
example
of
a
tool.
That's
using
a
bunch
of
darshan
data
representing
jobs,
running
our
system
to
offer
broader
insights
into
the
system.
Io
usage.
B
Yeah.
I
think
that's
that's
it.
I
know
we've
covered
a
lot
today.
Hopefully
it's
useful
for
folks
that
aren't
familiar
with
darshan
and
also
with
those
of
you
who
have
had
a
chance
to
use
it.
Hopefully
some
some
bits
stuck
out
to
you
as
being
useful
or
something
that
you
could
take
back
to
your
particular
research
topics,
and
things
like
that.
So,
hopefully
folks
got
something
out
of
it,
but
mostly
what
we.
B
I
o
interfaces
and
things
like
that,
like
mpi
or
sorry,
hdf5
and
deos,
and
also
kind
of
I
o
insights,
enabled
by
different
darshan
tooling,
whether
it
be
just
our
simple
tools
that
come
with
darshan
or
tools
like
gage
that
are
third
party
and
provided
additional
insights
on
darshan
data
and
again,
obviously
really
interested
in
seeing
how
how
di
high
direction
works
out
and
see
if
it
can
can
lead
to
a
lot
more
interesting.
Darshan
analysis
use
cases.
E
A
great
talk,
35
of
our
users
submitted
jobs
last
year
that
used
python
and
something
like
80
of
our
users
use
content
environments.
So
I
just
want
to
make
make
sure
I
understand.
So
if
a
user
wanted
to
profile,
I
o
with
darshan,
if
one
of
those
users
wanted
to
do
that,
I
guess
the
way
we
would
tell
them
to
do.
That
is
that
they
just
set
up
the
ld
preload
setting,
and
then
they
just
run
their
python
application
from
within
their
condo
environment
as
usual.
A
Yeah
yeah
you're
you're
right.
We
we
noticed
that
we
had
some
some
workloads
with,
for
example,
steve
ml
perth
workload
that
we
weren't
able
to
instrument
the
workload
with
darshan
because
it
was
coming
with
a
different
hd5
library.
So
we
had
to
rebuild
darshan
specifically
with
the
hdfi
library
coming
out
with
with
that
conda.
A
B
That's
a
good
question:
we're
something
we're
still
trying
to
to
figure
out.
I
don't
see
any
reason
why
it
shouldn't
work,
but
there's
obviously
it's.
It
complicates
the
the
process
of
you
know,
building
and
installing
and
setting
environment
variables
in
the
right
spot.
I'm
working
with
some
hep
workflow
systems
that
obviously
use
containers
very
liberally,
and
it's
it's
tough
to
to
get
things
completely
right.
So
part
of
the
challenge
is
me
trying
to
communicate
with
other
teams
and
not
necessarily
having
access
to
their
containers.
B
So
some
of
this
might
just
be
like
miscommunication
stuff,
but
from
a
technical
perspective,
I
don't
I
don't
see
why
it
should
work
and
we
have
some
success.
I
think
it's
just
more
of
a
getting
things
done
properly
if
that
makes
sense,
because
you've
got
to
find
the
right
place
to
install
darshan,
install
and
store
darshan
log
files
and
so
on,
and
it's
it's
a
little
more
complicated
in
a
virtualized
environment
like
that,
but
I
think
we
can
spend
some
more
time
to
probably
find
some
documentation
or
something
like
that.