►
From YouTube: 2021-11-02 - Hariharan Devarajan - Characterizing I/O of large scale scientific Deep Learning apps
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title:
Characterizing I/O behavior of large-scale scientific deep learning applications
Speaker:
Hariharan Devarajan, Lawrence Livermore National Laboratory
Abstract:
Deep learning has been shown as a successful method for various tasks, and its popularity results in numerous open-source deep learning software tools. Deep learning has been applied to a broad spectrum of scientific domains such as cosmology, particle physics, computer vision, fusion, and astrophysics. Scientists have performed a great deal of work to optimize the computational performance of deep learning frameworks. However, the same cannot be said for I/O performance. As deep learning algorithms rely on big-data volume and variety to effectively train neural networks accurately, I/O is a significant bottleneck on large-scale distributed deep learning training. In this talk, I will share our experiences of running large-scale DL applications on Theta supercomputer with a detailed investigation of the I/O behavior of various scientific deep learning workloads. Additionally, I will showcase our DLIO Benchmark, which accurately represents the class of applications previously characterized to foster I/O research in these classes of applications. I will share some key results and insights we discovered in modern scientific DL applications including, access patterns, integration with scientific data formats, and their I/O scalability in production supercomputers. Finally, I would highlight key pain points in doing I/O characterization of DL applications and discuss some research directions to improve these aspects.
Bio:
Hariharan Devarajan is a Postdoctoral researcher at Lawrence Livermore National Laboratory. He received his Ph.D. in Computer Science at Illinois Institute of Technology, advised by Dr. Xian-He Sun. His research is focused on accurate I/O characterization of distributed applications and building highly configurable storage systems on large-scale distributed systems. He has worked on I/O optimizations in several domains such as scientific simulations, AI, and Big Data Analytics and specializes in designing solutions for hierarchical storage environments. He is the recipient of the best paper awards at HPDC and CCGrid.
Host of Seminar:
Suren Byna
Computational Research Division
Lawrence Berkeley National Laboratory
A
Okay,
it's
my
pleasure
to
welcome
hari
haran
devrajan,
who
is
currently
a
postdoc
at
lawrence,
livermore
national
lab
he
he
graduated
from
illinois
technology
with
the
same
advisor.
I
was
working
with
so
he's
an
academic
brother
and-
and
he
recently
worked
on
characterizing
the
I
o
behavior
of
large-scale
scientific,
deep
learning
applications
and
he
presented
this
at
grid
and
won
the
best
paper
award
there.
So
welcome
hurry
and
yeah
just
for
the
audience.
A
B
B
We
have
been
trying
to
do
characterization
of
io
behavior
for
our
simulations
and
different
kinds
of
applications
throughout
time
and
now,
with
all
these
scientific,
deep
learning
applications
there,
it
was
kind
of
a
needed
step
in
a
way
to
kind
of
figure
out.
How
do
these
work
at
large
scale?
What
are
their
I
o
needs,
and
so
on.
B
B
Okay,
so
the
basic
idea
is
that
science
scientists
utilize
deep
learning
methods
to
reduce
the
computational
complexity
of
their
applications.
It
is
used
in
a
variety
of
domains
such
as
material
science,
biology
physics
and
many
more
are
shown
on
the
right.
Efficient
ingestion
and
processing
of
data
is
essentially
very
crucial
for
effectiveness
of
these
deep
learning
methods
and
it's
it's
been
kind
of
proven
in
the
cloud
environments.
B
However,
the
data
characteristics
of
deep
learning
methods
in
scientific
domains
is
not
well
understood,
just
to
give
a
brief
background
on
how
deep
learning
frameworks
and
methods
work
so
generally
popular
dl
frameworks,
you
such
as
tensorflow
or
python
caffe,
provide
an
infrastructure
to
optimize
data
management
required
by
these
deep
learning
models.
In
general,
you
could
look
at
a
deep
data
management
within
any
deep
learning
technique
having
three
phases.
The
first
is
the
data
ingestion.
That
is
how
the
data
is
read
by
several
sources
into
memory
for
input
processing.
B
Then
there
is
data
processing.
That
is
what
data
operators
are
applied
to
the
in-memory
data
before
feeding
it
to
the
training
and
finally,
it
has
data
communication.
That
is
how
data
exchange
occurs
between
various
workers
of
the
deep
learning
training
framework.
These
phases
have
their
own
software
and
hardware
optimizations
that
are
supported
by
these
deep
learning
frameworks.
B
The
code
is
publicly
available
at
the
top
ripple
mentioned
there,
and
I'll
also
have
the
slide
at
the
end,
which
shows
the
same
report
so
to
achieve
this
representative
benchmark,
we
need
to
first
understand
the
eye
of
behavior,
obviously
of
the
scientific,
deep
learning
applications
and
for
this
we
use
the
applications
present
at
argon
leadership
computing
facility,
because
they
they
had
a
lot
of
applications
from
new
projects
which
were
upcoming.
So
we
use
that
as
a
use
case
for
the
study.
B
So
in
this
study,
basically,
we
we
utilize
the
production
supercomputer
at
the
lcf,
which
is
theta
and
all
the
applications
were
part
of
ongoing
products,
projects
in
argonne
targeted
towards
the
next
excess
scale
era.
As
listed
on
the
right,
we
run
these
applications
with
128
kn
nodes
over
last
file
system,
which
we
saw
having
a
peak
bandwidth
of
240
gigabytes
per.
Second,
all
these
applications
utilize
tensorflow
framework.
Therefore,
we
utilize
its
internal
profiler,
along
with
darshan
for
low-level
io
profiling.
B
Also,
we
built
an
analysis
tool
which
will
basically
combine
the
higher
level
details
from
tensorflow
profiler
with
the
lower
level
io
profiling
from
darshan
and
provide
a
more
holistic
view
of
the
I
o
behavior.
All
these
all
the
results
I
will
be
kind
of
showing
during
this
categorizations
are
from
this
tool
so
I'll
first
kind
of
go
to
towards
the
eight
applications
we
looked
at
from
various
projects
in
this
work
and
we'll
see
how
each
of
these
applications,
what
are
their
high
level
goals
and
what
are
their
basic
data
set.
B
B
B
Then
we
looked
at
a
couple
of
tensorflow
based
applications.
Well,
one
of
them
was
cosmo
flow
benchmark
which
is
used
in
studying
the
features
or
in
the
distributed
dark
matter
in
the
universe.
So
this
application
is
basically
written
in
tensorflow,
with
a
two
terabyte
dataset
present
in
tf
record
format.
The
data
set
consists
of
256k
samples
spread
across
thousands
and
tf
record
files,
and
the
training
runs
over
256k
steps
with
one
large
one
image
in
each
training
step.
B
Then
we
have
the
popular
candle
application,
which
is
basically
cancer,
distributed
learning
environment
project.
This
application
basically
classifies
rna
sequence,
gene
to
detect
normal
or
tumor
issue
in
the
image
it
has
700
megabyte
data
set,
which
consists
of
100
1020
120
records
with
each
record
size
of
256
kilobyte.
B
Next,
we
also
looked
at
a
few
other
applications,
such
as
fusion
recurrent
neural
nets,
which
basically
was
used
in
prediction
disruption
in
fusion
plasmas.
The
application
in
this
case
used
a
six
gigabyte
data
set,
which
was
presented
in
numpy
array
files,
distributed
across
two
megabyte
files.
Each
containing
1024
samples,
the
training
model
used
hundred
estimates
with
a
batch
size
of
500
pro
samples.
B
B
B
Now,
let's
look
at
each
of
these
applications
and
what
they
actually
did
based
on
our
tool,
which
characterized
what
is
going
on.
So
in
all
of
these
tools,
you
would
see
at
the
top
like.
I
would
have
a
timeline
where
I
have
I
o
and
computation
below
that
and
then
and
then
there
could
be
some
distributions
of
request
sizes
so
I'll
be
talking
through
some
of
these
figures
and
what
were
the
main
conclusions
in
all
of
these
applications.
B
So
in
case
of
cosmic
tiger,
we
observed
that
22
percent
of
the
overall
time
was
spent
in
performing.
I
o.
The
data
set
essentially
is
read
sequentially,
but
it
achieves
a
very
low
bandwidth
of
11
megabytes
per
second
of
aggregate
across
all
processes.
This
low
bandwidth
was
basically
due
to
small
transfer
size
of
request.
B
If
you
remember,
the
each
sample
size
within
cosmic
tagger
is
40
kilobytes
around
a
median
and
they
do
32
batch
size,
so
it
should
not
have
been
smaller.
But
when
we
looked
at
the
actual
hdfi
data
set,
it
was
chunked
with
a
very
small
data
site,
matching
the
page
size
of
the
application
of
the
system,
which
is
like
four
kilobytes.
So
they
used
a
very
small
chunk
size
because
of
which
the
accesses
became
really
small
and
the
I
o
was
inefficient,
as
shown
in
the
timeline
on
tap
overall.
B
Only
like
10
percent
of
the
overall
io
was
overlapped
behind
model
computation,
which
is
kind
of
important
for
any
any
of
these
ai
applications,
because
computations
run
on
separate
resources.
So
you
generally
want
good
overlap
between
your
I,
o
and
and
computation
for
distributed
flat-filling
networks.
We
see
that
18
percent
of
the
overall
time
was
performed
in
io.
B
The
data
set
itself
consumed
was
consumed
by
multiple
processes
and
achieved
an
overall
bandwidth
of
11
gigabytes
per
second,
the
distribution
of
transfer
sizes
were
centered
around
64,
kilobytes
and
100
bytes,
and
essentially
again
we
saw
that
this.
This
was
due
to
the
chunking
how
the
two
files,
the
metadata
and
the
data
files,
were
chunked.
So
the
hundred
bytes
is
for
the
metadata
file
and
64
kilobytes.
You
see
it
is
for
the
data
file
how
it
was
jumped.
B
For
the
cosmo
flow
benchmark,
which
was
basically
written
in
utila
tensorflow
data
pipeline,
we
see
only
12
percent
of
the
overall
time
was
spent
in
io.
The
application
runs
over
four
epochs
with
an
aggregate
bandwidth,
36
gigabyte
per
second,
that's.
What's
the
maximum
bandwidth,
we
saw
in
any
of
these
applications
like
whenever
we
were
using
tensorflow
data
pipeline.
It
actually
gave
very
good
bandwidth
from
the
telephone
system.
B
The
data
was
evenly
distributed
among
all
the
processes
of
training.
We
saw
that
the
transfer
size
of
each
request
within
the
application
within
the
application
was
exactly
256
kilobyte,
even
if,
like
the
samples
were
smaller
or
bigger,
this
was
basically
due
to
the
way
tensorflow
internally
does
io.
It
uses
a
fixed
transfer
size
that
you
can
configure,
but
the
default
size
was
256
kilobytes.
So
that's
what
it
does
and
the
application
itself
completely.
B
Overlaps
io
with
compute,
and
one
of
the
other
interesting
things
we
saw
here,
was
that
out
of
the
23
person
that
was
spent
in
the
input
pipeline
overall,
eleven
percent
around
was
on
the
data
processing
pre-processing
in
memory,
the
imagenet
benchmark
again
similar
to
the
cosmo
flow
benchmark.
It's
also
written
in
tensorflow
using
tf
records,
and
this
this
application
spent
about
15
percent
of
its
overall
time
in
igo,
with
an
aggregate
bandwidth
again
of
32
gigabytes
per
second.
B
It
reads
the
data
set
on
demand,
unlike
the
cosmo
flow
benchmark,
which
reads
all
the
data
set
evenly,
it
kind
of
goes
based
on
how
many
times
you're
reading
it
so
based
on
the
time
step,
it
will
read
the
data
set
on
command
and
then
there
is
because
of
this
there
is
a
non-uniform
distribution.
If
you
look
at
the
lower
right
corner
here,
you
will
see
that
some
files
are
read
more
than
others.
B
That's
basically,
because
of
that
and
the
transfer
size
when
we
looked
at
it
again
was
the
default
256
kilobytes
and
all
the
I
o
that
we
saw
was
completely
overlapped
with
the
compute,
which
was
going
on
behind
the
candle
application,
which
is
the
cancer
application.
Basically
reads:
data
from
the
csv
data
sets
and
spends
36
percent
of
its
time
in
performing
io.
B
Basically,
in
this
case,
because
it's
a
csv
data
format
and
the
api
they
were
using
each
process
reached
the
whole
data
set
row
by
row
with
an
aggregate
bandwidth
of
8
gigabytes
per
second.
They
got
the
8
gigabyte
per
second
aggregate
bandwidth,
because
each
row
luckily
was
big
enough,
like
256
kilobytes
in
size,
so
they
got
good
transfer
size
on
each
process.
B
B
Also,
in
that,
as
we
observe
in
the
timeline,
like
only
one
thread,
were
basically
used
to
perform
this,
I
o
and
there
was
no
overlap
or
no
asynchronicity
between
io
and
computation.
Once
all
the
io
was
done,
you
do
the
training
and
then,
once
all
the
trust
data
was
read.
You
do
the
testing,
so
that's
kind
of
how
this
application
worked.
B
The
frnn
application,
which
used
numpy
arrays
as
its
data
set,
spends
about
23
percent
of
the
overall
time
in
io.
It
reads:
a
six
gigabyte
data
set
using
numpy
apis
with
an
aggregate
bandwidth.
28
gigabytes,
there
were
two
types
of
requests
on
this
in
this
particular
application.
One.
There
was
always
one
request
on
the
metadata
file,
which
is
a
smaller
file
and
then
the
whole
two
megabyte
chunks
of
signal
files.
B
They
call
it
were
read
in
one
go,
so
that's
kind
of
that's
why
the
frequency
of
metadata
calls
dominate
within
the
application.
If
you
again
look
at
the
bottom
right
corner,
the
smaller
io
is
all
from
the
metadata
and
the
bigger
ios
are
from
the
complete
chunk
of
reading
those
numpy
arrays
in
one
go
again
in
this
application.
We
saw
that
there
was
essentially
no
overlap
between
compute
and
io
for
the
last
two
applications.
As
I
said
before,
we
they
do
not
perform
any
disk
reading.
B
As
so,
however,
when
we
run
these
applications,
we
saw
that
they
do
some
periodic
checkpoints.
So
we
were
profiling
that
and
we
saw
that
even
for
the
they
they
actually
perform
like
models
and
billions
of
parameters
and
so
on.
So,
if
even
for
those
kinds
of
large
models,
the
amount
of
checkpointing
data
was
in
tens
of
megabytes
and
using
sjdio
interface
of
within
tensorflow,
and
we
saw
that
the
right
time
was
less
than
one
person.
B
B
B
B
They
use
a
single
large
file
with
pass
xpft
in
hdfi
file.
Now
there
is
no
framework
level
integration
within
tensorflow.
What
I
mean
by
that
is
generally,
they
are
executed
in
the
python
main
thread
and
then
everything
is
passed
down.
The
data
is
passed
down
to
tensorflow
because
of
which
they
they
run
on
a
different
runtime
than
the.
B
So
they
are
not
really
parallelized,
whatever
parallelization
happens
has
to
happen,
but
has
to
be
done
by
the
application
developers
themselves,
and
this
basically
forces
application
developers
to
manually
perform
sample,
shuffling
input,
pipeline
design
and
whatever
features
they
want
like
that,
that
even
tensorflow
kind
of
shows
that
they
have
those
features
and
they
could
be
useful
to.
Basically
developers
are
building
them
on
their
own,
and
because
of
that,
we
saw
that,
because
it's
running
on
different
time,
different
runtimes
and
not
utilizing
all
the
cores.
B
There
is
a
lot
of
serialization
between
I
o
and
compute
phases
within
the
application,
without
any
possibility
of
overlap.
So
that's
kind
of
one
of
the
things
we
saw
common
in
the
hdf5
set
of
applications
here
for
the
tensorflow's
tf
record
data
format.
Generally,
the
data
set
is
represented
into
multiple
files,
and
these
files
are
chosen
by
the
framework
at
random.
So
you
generally
give
a
glob
of
file
or
path,
and
then
they
they
choose
these
to
balance,
load
and
so
on
and
so
forth,
and
then
they
utilize
independent.
B
Finally,
we
observe
that
these
files
are
read
sequentially
on
demand,
so
the
way
tensorflow
works.
It's
kind
of
it
basically
runs
a
bigger
reach,
a
bigger
chunk
of
I
o
in
memory
and
then
shuffles
it
if
the.
If
the
application
wanted
it
to
shuffle,
so
it
does
not
do
shuffled,
I
o
within
the
file,
because
of
which
it
always
reached
large
big
chunks
of
I
o
and
then
deserialize
them
and
then
do
whatever
shuffling
they
want
to
do
in
memory
which,
which
turns
out
to
be
very
good
for
io
right.
B
So
they
work
inherently
well
with
the
pal
file
system
because
of
that
and
for
other
data
formats.
Basically,
it
kind
of
boils
down
to
what
interface
they
use,
for
instance,
for
csv.
They
use
the
normal
f
open
interface
because
of
like
the
sorry,
the
csv
interface
within
python,
which
does
not
let
you
do
anything
but
roberto,
but
if
they
have
used
something
smarter
like
let's
say,
pandas
or
desk,
they
might
be
able
to
do
something
more
smarter.
B
So
it's
kind
of
boiled
down
to
what
interface
you
use
and
how
you
read
the
data
and
there
is
no
inherent
like
optimization
logics,
at
least
in
those
two
applications
that
you
could
do
because
of
the
interfaces
they're
using.
So
so
that's
why
I
put
them
into
others,
because
they're
very
specific
and
rare
cases,
I
would
say
like
when
I
was
talking
to
the
developers
they
basically
didn't,
want
to
change
it
to
any
other
format
because
of
ease
of
use,
or
they
were
more
familiar
with
these
formats.
B
So
that's
kind
of
how
it
was
so
after
after
looking
at
this
classification,
what
we
thought
was
it
would
be
useful
to
build
a
benchmark
suit
which
can
represent
these.
These
features
of
these
eight
applications
accurately
and
any
other
application
which
might
have
a
symbol
because
most
of
ai
applications
work
on
a
kind
of
set
of
similar
features.
B
So
the
idea
was
to
use
these
eight
to
build
the
representative
benchmark
and
then
even
you
know,
change
start
building
other
benchmarks,
as
as
you
want
or
you
could
you
could
change
the
parameters
and
start
seeing
different
patterns
and
so
on
and
so
forth.
So
we
show
the
design
of
the
benchmark
itself
on
the
right.
It's
a
very
straightforward
benchmark.
The
benchmark
is
configured
using
arguments
which
configures
the
internal
parameters
of
the
benchmark
of
the
execution
of
the
benchmark.
B
B
Finally,
we
also
support
data
generation
so
that
we
can
emulate
data
scaling
tests
for
our
applications
because,
as
I
said,
most
of
these
applications
are
using
very
small
data
sets,
so
it
might
be
useful
to
generate
your
own
data
based
on
a
particular
pattern
and
then
and
then
run
your
application
on
top
of
it.
The
dlio
benchmark
in
its
current
state
on
the
repo
supports
the
following
configuration
such
as
tf
record,
hdf5,
csv,
numpy
erase
the
file
access
pattern
could
be
shared
or
multiple
files
with
independent
or
collective
io.
B
This
depends
upon
the
type
of
interface
you
selected.
Some
of
them
support
independent
collective
eyes,
some
of
them
don't
we
can
emulate
different
data
set
reading
patterns
such
as
sequential
or
random,
because
some
in
some
cases
you
want
to
see
that
again.
This
is
based
on
the
support
from
the
interfaces.
B
Also,
the
two
types
of
I
o.
We
look
at
here
is:
first,
is
the
major
one,
which
is
the
data
set
reading.
Also,
we
also
added
a
part
for
model
check
pointing
such
that
the
application
would
checkpoint
every
end
step,
so
it
basically
emulates
what
the
application
a
tensorflow
application
was
doing
as
I
described
before,
and
then
finally,
you
could
tune
the
request,
size
distribution
within
our
benchmark
for
these
interfaces,
so
that
you
could
kind
of
look
at
different,
I
o
patterns
and
so
on
and
so
forth.
B
So
to
kind
of
evaluate
this
benchmark,
we
use
a
similar
setup
as
before
on
the
theta
supercomputer,
the
workloads
we
use.
So
essentially,
what
we
do
is
we
run
the
emulated
versions
of
the
benchmark
with
different
regression
tests
like
okay,
let's
change
the
transfer
size,
let's
change
parallelism
so
on
and
so
forth,
and
these
run
fairly
quickly
because
they
are
not
actually
running
the
actual
computation.
B
We
emulate
the
computations
here,
so
you
could
kind
of
tune
them
to
not
do
any
computations
and
just
look
at
io
itself
and
based
on
that,
you
could
figure
out
what
are
the
best
parameters
for
optimizing.
Your
I
o
within
the
application,
and
once
you
have
that
you
we
apply
them
to
the
actual
scientific,
ai
applications
and
then
measure
their
in
the
impact
of
our
optimizations
and
how
well
it
works
before
we
do
the
actual
analysis.
What
we
wanted
to
make
sure
was
at
least
like
because
we
modeled
our
benchmark,
28
applications.
B
We
wanted
to
make
sure
the
characteristics
of
the
application
and
the
benchmark
are
similar
on
these
parameters
that
we
are
showing
in
the
table
and
in
general,
there
is
about
six
percent
variance
and
between
the
application
and
the
benchmark,
and
it's
this
is
primarily
because
of
in
the
application.
You
might
have
a
complex
distribution
of
request
sizes,
but
to
simplify
the
benchmark.
What
we
did
is
we
just
take
the
median
of
requests
which
we
saw
does
pretty
well
in
most
cases.
So
that's
what
we
use,
because
of
that.
B
Then
we
we
have,
I
have
more
optimizations,
we
have
shown
on
hdfi
as
well
of
the
paper,
but
I'm
I'm
showing
cosmo
flow
here,
because
I
think
that's
also
interesting
here,
like
for
the
cosmo
flow
application.
As
I
said,
it
gets
a
lot
of
bandwidth
already
right,
but
if
you,
if
you,
if
you,
if
you
think
about
it,
the
default
stripe
size
on
laster
is
about
a
megabyte
and
they
use
256
kilobytes.
So
the
idea
here
was
that,
can
we
apply
the
dlio
version
of
cosmoflow
application?
B
We
try
different
optimizations
such
as
different
transfer
sizes,
so
that
it
matches
your
matches.
Your
lustre
file
system,
better,
read:
parallelism
pre-processing
parallelism.
All
these
three
factors
we
kind
of
played
with
different
numbers
based
upon
the
machine,
and
then
we
measure
what
which
gave
the
best
performance.
In
terms
of
I
o
and
then
we
applied
those
parameters
back
to
the
actual
application,
the
ideal
ones,
and
essentially
what
the
paper
shows
is
that
you
could.
You
could
kind
of
get
very
consistent,
optimization
benefits.
B
It
might
not
be
the
same,
because
the
benchmark
only
does
sometimes
I
o
and
things
like
that.
But
you'll
see
the
benefit.
Still
there
like
a
relative
benefit
to
the
application,
is
still
there,
so
that
kind
of
is
nice
to
kind
of
quickly.
Do
the
all
your
regression
tests
on
the
benchmark
and
then
come
back
and
just
apply
the
optimization,
because
it's
basically
a
combinatorial
problem
right.
B
So
it
helps
with
all
these
benchmarks
to
do
that,
for
you
also,
as
you
have
seen,
the
eight
applications
don't
really
use
large
data,
because
these
data
sets
were
generated
by
simulations
and
they're
working
with
very
specific
problems.
So
what
we
did
here
is
the
data
generation
model
can
basically
generate
for
the
same
pattern.
We
could
generate
bigger
scale
so
that
we
could
start
doing
some
strong
scaling
or
weak
scaling
tests
on
the
application.
B
Whatever
we
want,
so
the
complete
set
of
results
are
there
in
the
paper,
but
essentially
they
all
look
as
follows.
Like,
for
instance,
on
the
left,
we
have
the
imagenet
benchmark,
which
uses
multiple
files
per
application
and,
as
we
as
we
increase
more
processes
to
do
the
same
work,
the
total
number
of
files
that
we
see
maintain
a
very
good
performance
as
the
scale
increases,
whereas
on
the
right
which
uses
a
shared
file
within
this
is
the
emulation
of
the
ddn
dffn.
B
B
B
So
in
that
case
you
it
makes
sense
to
like
look
at
isolated
and
you
can
do
it,
whereas
in
the
second
figure,
as
you
see
that,
if
some
push
all
the
I
o
is
always
smaller
than
the
computation
pushes
below,
so
you
don't
really
need
to
optimize.
I
o,
because
you
need
to
optimize
compute
in
that
case,
to
reduce
your
overall
time.
Otherwise
you
won't
see
any
difference.
So
that's.
B
Why,
like
that's
kind
of
important
to
look
at
io,
not
only
isolated
but
kind
of
together
to
make
sure
your
optimizing
optimizations
would
be
worth
it
and
that's
a
challenge
in
current
scenario
and
I'll
talk
about.
Why
that's!
My
next
point
is:
if
we
wanted
to
look
at
different
components
right
such
as
now,
you
have
cpus,
you
have
an
I
o
subsystem,
such
as
interfaces
you're,
looking
at
and
even
gpu.
In
that
case,
then
you
need
to
collect
these
traces
from
different.
A
B
Gpu,
you
might
be
collecting
it
from
some
nvidia
utility
cpu.
You
are
using
some
form
of
cpu
based
profiler,
I
o
you're
using
darshan,
and
then
you
have
to
now
combine
all
of
them
together
into
a
single
application
timeline,
and
we
saw
that
that's
not
that
trivial
to
do,
because
all
of
these
have
different
ways.
They
have
represented
log.
You
need
to,
even
if
you
parse
them
their
time
scales
where
that
exact
pointer
happens,
because
there
is
a
complex
software
stack
involved,
it's
kind
of
difficult
to
say,
oh,
which
part
is
executed.
B
When
and
if
you,
if
you,
if
you
are
doing
timeline
analysis
of
overlap,
you
need
to
be
as
accurate
as
possible.
So
for
this
work,
what
we
did
was
we
added
markers
on
darshan,
like
I
o
markers,
like
just
output,
a
simple
byte
out
at
the
start
of
the
application
and
end
so
that
we
could
kind
of
look
at
the
tensorflow
profiler
and
match
it.
B
But
you
can't
always
do
that
with
your
application
that
that's
kind
of
challenging
at
the
end,
to
figure
out
how
you
want
to
do
it
in
terms
of
I
o
requests
themselves
like
application.
In
a
lot
of
our
cases,
the
application
was
doing
bigger
io,
but
because
of
the
way
hdfi
was
chunked
or
the
data
set
was
chunked.
B
Honestly,
the
scientists
probably
didn't
do
it
now
they
just
whenever
the
simulation
generated,
those
datasets
they
just
been
using
it,
but
the
chunking
of
the
dataset
basically
determined
what
was
the
actual
transfer
size
and
because
of
which
we
saw
a
lot
of
inefficient
io
and
through
the
benchmark.
Some
of
the
results
actually
show
like
when
we
use
dlio
to
generate
larger
same
pattern
but
larger
chunk
sizes
within
the
data
set.
Everything
is
the
same
just
I
increase
the
chunk
size.
B
The
cosmo
flow
application
originally
had
its
data
set
in
hdr5
and
there
was
one
work
which
converted
this
hdf5
data
sets
into
tf
records
to
make
I
o
better
so
and
and
after
that
conversion
there
is
a
conversion
cost.
Obviously,
but
after
the
conversion,
the
I
o
within
the
application
is
way
better.
So
it
kind
of
helps,
maybe
like
you,
if
you
use
the
native
format,
because
the.
B
It
is
true,
a
lot
of
smaller
benchmarks
do
that,
but
in
in
this
case
I
want
to
kind
of
highlight
at
least
what
we
found
is
people
no
longer
do
that
they
even
in
the
cloud
world
they
either
come,
combine
them
into
tf
records
or
numpy
areas
or
some
format.
All
these
are
images
in
these.
In
these
examples
most
of
these,
except
for
candle,
which
was
numerical
numbers
all
of
the
else,
were
actually
images
and
they
all
combined
them
into
different
formats
to
to
have
a
bigger
group.
B
Instead
of
just
looking
at
individual
images,
so
they
do
that
now
and
so
it's
not
necessarily
always
that
it
will
do
random.
I
o
because,
for
instance,
the
way
cosmo
cosmic
tiger
and
dffn
were
actually
doing.
It
was
that
they
were
reading
32
images
at
a
time
from
the
hd5
file
and
then
to
avoid
doing
real
random.
B
I
o
they
would
go
and
monotonically
increase
their
offsets
so
that
it's
still
in
a
way
for
the
pal
file
system
sequential
like
and
then
they'll
come
back
and
do
other
chunks,
so
they
they
did
a
lot
of
smartness
point
being
to
optimize
their
io
within
from
the
hd5
file
itself,
so
that
they
are
working
it
well.
So
it's
not
like
it's
a
given
assumption.
Yeah
it
could
depend
on
the
applications,
but
the
applications.
We
see
generally
don't
directly
work
on
raw
images.
B
So
in
summary,
in
this
talk
we
kind
of
talked
about
the
behavior
of
modern
scientific,
deep
learning
applications
that
we
saw
at
alcf.
We
developed
a
benchmark
that
accurately
represents
these
applications
and
can
be
utilized
for
eye
optimization
and
finally,
through
benchmark
in
the
paper.
We
show
that,
like
you
could
do
scaling
tests
and
in
general
we
saw
that
you
need
better
support
for
scientific
data
formats
in
deep
learning
frameworks,
but
better
better
data
exchange
between
scientific
simulations
and
ai
to
work
better
in
these
systems.