►
Description
NERSC Data Seminars: https://github.com/NERSC/data-seminars
Title: ECP HDF5 - New features and applications
Abstract: HDF5 is a data model, file format, and I/O library that has become a de facto standard for HPC applications to achieve scalable I/O and for storing and managing big data from computer modeling, large physics experiments and observations. Several Exascale Computing Project (ECP) applications are currently using or planning to use HDF5 for I/O. The ExaHDF5 project team of the ECP is working on developing and productizing various features to improve the efficiency of parallel I/O to take advantage of exascale architectures. In this presentation, we will talk about these features, including Virtual Object Layer (VOL), asynchronous I/O, subfiling, Data Elevator, independent metadata updates, querying, etc. The presentation also include integration of HDF5 into ECP applications and co-design efforts, such as EQSIM and AMReX.
A
So
good
afternoon
everybody
I'm,
Quincey,
Koziol
and
Surin
I
know
we'll
be
presenting
with
me
about
the
ECP
work,
we're
doing
on
hdf5
and
a
tiny
bit
about
other
aspects
of
hdf5
and
other
projects.
We're
collaborating
on
this
we'll
start
off
with
a
brief,
hopefully
10-minute,
ish
intro,
to
hdf5
just
to
kind
of
refresh
everybody's
notions
of
things
and
give
you
a
little
bit
of
bootstrap.
In
case
you
haven't
seen
it
before
I
think
most
people
have
and
then
talk
mainly
about
the
actual
ADCP
work
that
we're
working
on.
A
Then
the
applications
are
engaged
with
so
hdf5
Weiss
hdf5
stands
for
hierarchical
data
format,
version
5,
don't
let
that
fool
you!
The
version
5
is
not
compatible
with
versions
1
through
4
just
for
fun,
but
hdf5
is
basically
composed
of
three
important
aspects.
It's
an
extensible
data
model
that
allows
you
to
organize
your
data
in
a
reasonably
scientifically
organized
perspective.
A
There's
a
software
package
that
comes
with
it
all:
io
libraries,
lots
of
command
line
tools,
there's
a
very
large
ecosystem
built
around
that
and
basically
does
all
the
I/o
according
to
the
data
model
through
the
API,
is
that
we
give
people,
there's
lots
and
lots
and
lots
of
different
ways.
You
can
operate
on
hdf5
data.
A
Traditionally
it's
been
through
POSIX,
but
you
can
reach
out
into
the
Amazon
Cloud
or
object
stores
like
this
or
other
ones
native
memory,
and
if
you
want
to
operate
a
dram
and
pretty
much,
anything
else
is
a
lot
of
extension
points
for
reaching
out
there.
The
third
aspect
that
we
won't
even
touch
on
today
is
the
the
file
format
that
is
corresponding
to
the
data
model,
so
data
models,
kind
of
the
abstract
programming.
What
you
think
about
and
whiteboard
about
software
is
such
a
program
and
the
files
are
what
you
get
read
about.
A
A
We
have
ported
the
library
and
had
bug
reports
and
interesting
discussions
with
people
from
every
different
level
of
the
hierarchy
from
embedded
systems
and
feature
phones,
requests
by
Nokia
in
the
early
2000s
all
the
way
on
up,
of
course,
to
supercomputers
that
we
have
today
and
in
the
future,
sorry
flexible.
As
I
said
earlier,
you
can
do
storage
and
lots
of
different
containers
or
formats
on
the
on
the
back
end
and
importable,
also
out
through
thoughts
and
lots
of
different
languages
and
application
wrappers
for
different
levels
of
abstraction
to
your
communities,
science
format.
A
We
really
try
to
have
a
flexible
model
that
allows
the
apps
to
evolve
and
that
add
new
things
and
rearrange
things
and
the
data
it
can
still
be
accessed
for
a
very
long
time.
That's
kind
of
the
final
thing
here
we
have
petabytes,
maybe
even
exabytes
remote
sensing
data,
that's
been
accumulated
by
NASA
and
lots
of
other
people
I
think
expect
to
get
that
data
back
in
50
years.
You
know
climate
simulations
need
accurate,
long-term,
real
measurements
and
that's
part
of
NASA's
goal
and
one
of
the
reasons
why
they
funded
development
at
hdf5.
A
It
other
pieces
like
the
DOA
and
then
si
Labs
really
place
a
very
high
value
on
retention
and
archiving
and
other
aspects
of
data
preservation.
Also
at
a
high
level.
Hdf5
files
are
containers
that
hold
a
bunch
of
objects
that
you
create.
The
different
types
of
science
can
be
described
and
stored.
Well
in
there
the
containers
look
a
lot
like
an
object,
oriented
situation
as
file
and
then
there's
datasets
groups
and
attributes
kind
of
the
main
high
level
concepts
and
links
and
data
types
and
data
space
are
kind
of
supporting
aspects
of
the
file.
A
A
This
is
all
really
useful,
but
it
would
be
awfully
better
if
we
could
organize
it
in
a
nicely
structured
way
that
says
hey
in
this
file,
I've
got
some
visualization
output
and
some
simulation
data.
That's
maybe
checkpoints
or
something
else
and
nicely
organize
it,
and
typically
a
hierarchical
manner.
Hdf5
allow
you
to
create
graphs,
but
most
people
choose
trees.
This
is
kind
of
a
trivial
graph.
In
a
sense,
the
the
biz
group
in
this
file
has
a
link
to
this
table
and
a
to
knowledge
in
temperature,
and
so
this
is
simulation.
A
Also,
this
off
to
the
side
here,
we're
saying:
hey
this
group
and
hdf5
can
point
to
data
in
another
container,
so
there's
a
mechanism
for
having
external
links
out
into
other
hdf5
files,
another
what
we
would
consider
raw
files
of
just
arrays
index,
so
very
high
level.
How
would
we
organize
things
in
hdf5
everything
starts
at
the
root
group
and
really
need
to
take
that
little
link
out
of
their
architecture,
aliso,
diving
down
into
software.
Most
of
this
talk
is
about
software.
A
The
API
that
hdf5,
the
very
top,
is
what
most
of
the
applications
program
sometimes
there's
another
layer
of
more
domain-specific
science,
libraries
that
people
are
programming
but
hdf5
proper,
is
a
public
API
and
some
language
bindings
and
then
immediately
below
that
I'll
describe
it
a
little
bit
more
later.
On
the
talk,
we
have
this
new
virtual
object
layer
that
abstracts
out
components
that
deal
with
storing
data,
and
that's
mostly
what
we're
interested
in
here
and
allows
data
to
be
passed
through
several
different
pluggable
layers.
A
Each
one
of
these
boxes
in
here
the
vol
connectors,
are
different
pluggable
mechanisms
for
doing
storage
to
a
particular
destination.
So
you
can
store
Amazon
s3
with
the
rest
driver
to
the
day,
object
store.
There
is
other
file
formats
or
even
possible
errors.
Being
Estados
is
one
of
the
connectors
that
we
have
and
over
here
on,
the
Left.
A
We
provide
the
native
connector
deposits,
and
that
also
has
extensions
to
do
the
efficient,
typically
collective
is
how
you
get
efficient
for
MPI
but
independent
as
well,
and
then
single
writer,
multiple
reader
for
kind
of
experimental
observational
data
in
lots
of
different
ways
that
go
out
through
the
POSIX
file
format
over
there.
But
these
other
mechanisms
you
can
store
data
any
way
you'd
like
through
the
virtual
object.
There
talk
like
I,
say
a
little
bit
more
about
this
general
later
programming.
A
C
Fortran
Java
and
C++
ship
with
the
main
hdf5
package,
the
name
spacing
that
we
were
able
to
create
and
push
on
a
little
bit
in
C
is
everything
starts
with
H
5,
so
then
a
character
that
determines
what
kind
of
object
you're
operating
out.
So
when
you
see
hdf5
programming,
api
is
everything
that
starts
with
H,
5,
isn't
GTA
5
and
then
DS
for
the
data
set,
calls
and
EPS
for
files.
A
Gotta
fix
this
here,
they're
more
than
380
I,
as
one
can
start
with
just
a
few
though,
and
some
of
those
guys
allow
you
to
do
optional
tweaking
on
the
files
or
the
objects
or
the
access
that
you
do
so
we'll
talk
occasionally
about
seeing
properties
and
an
object
or
an
access
mode.
I
know
that
you
changed
Chungking
storage
or
choose
a
different
I/o
driver
tax
as
the
file,
but
then
generally
you,
you
open
them,
create
an
object.
You
access
it
and
you
close
it's
kind
of
a
very
C
oriented
programming
model
very
standard.
A
This
is
what
a
simple
skeleton
would
look
like:
create
the
file
create
a
data
space.
That's
the
dimensionality
in
the
data
set
then
go
create
the
data
set
access.
It
close
things
down
pretty
straight
mark
in
the
parallel
world.
Knife
is
a
little
bit
more
complex,
though
the
number
of
factors
that
you're
dealing
with
goes
up
quite
a
bit.
It's
not
just
hdf5
writing
deposits,
but
it's
more
hdf5
talking
to
MPI
interfacing
to
the
parallel
file
system
and.
B
A
Complicated
set
of
things
at
the
bottom
and
for
the
storage
hardware,
it's
not
just
the
hard
drive.
This
sort
of
system
here
is
kind
of
very
familiar
to
HPC
people,
but
typically
your
compute
nodes
talk
over
some
interconnect
down
into
IO
servers
and
metadata
servers
that
handle
the
file
system
aspects
and
the
bulk
data
transfers.
Then
off
into
the
hardware,
through
some
storage
area
network
and
lots
of
grading
and
disks
and
possibility
and
B
Ram
these
days,
I
would
say
so
typical.
A
Apps
can
either
do
this
bottom
set
of
collective
buffering
on
their
own
or
hand
it
off
to
MPI
oh
and
trade-offs,
either
way.
I
do
tend
to
encourage
people
to
go
with
MPI
collectives,
at
least
there's
a
knowledgeable
bunch
of
people
who
understand
it
and
do
that
all
day
long
every
day
and
can
optimize
or
you
so
from
hdf5
perspective,
still
get
into
sort
of
a
very
large
number
of
moving
pieces.
A
So
when
you
do
use
again,
this
is
wrapping
up
the
sort
of
tutorial
part
of
this
when
you're
using
hdf5
in
parallel
mode.
That
parallel
hdf5
allows
multiple
MPI
processes
in
the
MPI
application
to
perform
I/o
to
a
single
hc5.
You
could
choose
multiple
files,
but
single
one
is
the
norm.
Typically,
now
we
layer
on
top
of
MPI,
you
can
move
your
app
across
platforms
very
nicely.
The
files
that
you
create
with
parallel
hdf5
are
exactly
hdf5
files,
they're
exactly
the
same
as
you
would
get.
A
If
you
wrote
that
data
is
sequentially
on
your
laptop
or
desktop
or
whatever
so
there's
they're,
totally
portable
between
the
platforms
that
you
create
the
data
and
move
things
around
and
there's
a
few
little
bits
of
extra
API
calls,
and
you
know
bells
and
whistles
to
go
apply
to
that.
But
we
really
are
gonna.
Try
to
stay
focused
on
the
ECD
feature
side
of
things
for
today.
So
there's
many
many
tutorials
on
parallel
hdf5.
How
to
do
that
well
and
I'll.
Leave
that
to
that
aspect
of
things.
C
So
I'm,
surrounded
by
Anna
from
CRD,
so
I
will
be
talking
about
the
ECP
project
that
is
called
xio
hdf5.
We
were
previously
known
as
xh
d
fi,
but
recently
we
have
been
merged
into
two.
Major
projects
have
been
merged
and
both
of
them
are
supporting
io.
So
we
are
calling
it
X,
IO
team,
so
several
applications
in
the
ECP,
a
majority
of
them-
are
using
hdf5
file
format.
C
C
The
mission
of
the
xh
t,
fi
project
has
been
working
with
ECP
applications
and
making
sure
those
applications
perform
well
and
also
work
with
the
facilities
to
meet
the
hdf5
io
requirements.
I
will
be
talking
a
bit
about
some
of
these
applications
today
and
we
Quincy
would
switch
to
productizing
the
HTF
5
features
and
also
their
R&D
toward
future
architectures
that
we
are
ok
next
one.
C
So
we
are
interacting
with
a
large
number
of
things.
This
list,
some
of
the
applications
that
we
are
working
with
and
what
type
of
engagement
we
are
doing,
and
the
PRC's
protease,
for
example,
some
of
the
applications
are
at
LPL
subsurface
simulation
at
LBL.
We
have
been
working
with
them
when
chambo
started,
so
it's
been
a
while
and
we
optimize
the
performance
for
that
team
and
EQ
sim
is
another
application
that
we
have
been
working
with.
Recently
we
are
integrating
HD
FII
over
there
main
code
calls
este
before
I'll
talk
a
bit
more
about
that.
C
Warp
X
is
an
Berkeley
code
that
he's
using
hdf5
through
h5
pi.
The
XFL
we
have
been
is
at
SLAC.
We
have
been
working
with
them
in
optimizing
performance
using
similar
enhancements.
Xs
kai
is
another
code
from
Oregon
and
we
have
been
tuning
their
performance
and
mixes
another
code
at
LPL
we
integrated
hdf5
into
the
aircraft
into
their
code
and
optimized
performance
of
going
to
the
next
slide.
We
are
also
working
with
various
software
technology
teams.
C
So
we
are
also
working
with
our
team
to
provide
some
in
situ
capabilities
for
hdf5
Jun
min-gu,
who
is
on
the
call
has
been
working
on
so
it's
aspects
of
it
with
the
data
Lib
team
from
Oregon.
We
are
working
on
a
couple
of
things.
One
is
the
virtual
object
layer,
connector
for
P
netcdf
data
and
also
one
of
the
most
useful
team.
C
U
useful
topics,
maybe
darshan
profiling
for
hdf5,
which
is
being
added
right
now,
and
a
developer
version
is
available
right
now,
so
that
captures
the
profiling
for
hdf5
calls
such
as
hdf5
open
flow.
The
file
open
file,
close
dataset,
create
dataset
to
read,
write
and
flush
operations,
so
that
gives
us
how
to
identify
the
bottlenecks
in
hdf5
performance
bottlenecks
in
hdf5,
so
that-
and
that
would
be
one
of
the
very
useful
tools
we're
also
working
with
unify
FS,
which
is
a.
C
Uniform
unified
node,
local
storage,
single
and
namespace
name
space
book,
so
we
they
are
part
of
our
work,
our
project
and
we
are
closely
working
with
them
in
developing
the
hdf5
api
in
their
in
their
system.
Ok,
going
to
the
next
slide
now
I'll
go
through
some
of
these
applications
and
talk
a
bit
more
about
it.
Next
slide
in
the
EQ
sim.
The
India
sim
is
a
high-performance
original
scale.
Earthquake
has
a
trip
hazard
and
risk
assessment
code,
the
main
code
there
is
called
sw4.
C
So
the
hdf5
instances
where
we
are
converting
are
listed
in
red
here,
so
they
previously,
they
had
an
ASCII
base
to
our
file
and
now
and
we
converted
into
a
binary
as
file
and
then
those
reading,
those
hdf5
files
and
then
produce
it.
Data
through
simulation
would
be
written
into
hdf5
files
as
well,
and
the
visualization
tools
are
also
tell
by
some
of
their
our
team
members
to
them
next
slide.
C
Okay,
going
to
the
next
slide,
we
are
working
with
a
Amaury
X
team
as
well.
This
is
a
core
design
center
building,
massively
parallel
block
structure.
The
adaptive
mesh
refinement
applications,
several
applications
in
combustion,
accelerator
physics,
carbon
capture,
cosmology
applications
are
using
a
mouriak
software
framework.
C
C
Compared
to
that
using
hdf5
gives
much
better
performance
at
larger
scale.
We
could
hum
their
native
format
a
bit
more
to
get
more
performance,
but
as
the
systems
change,
the
native
performance
and
native
format
performance
may
have
to
be
brought
up
as
well,
but
it
HD
a
file.
You
have
a
standard
format
and
I
mean
you
can
get
most
of
the
performance
with
all
the
optimizations.
We
are
implementing
okay
going
to
the
next
slide.
Okay,
so
this
is
working
with
war,
packs
and
Kimsey
pack
applications
and
warp
X.
C
We
that
they
were
previously
using
h5
PI
and
we
notice
it's
some
performance
bottlenecks
there.
One
of
the
performance
bottlenecks
was
writing
to
writing
to
one
lustre
OST
on
quarry
and
that
was
giving
very
low
performance.
Then
we
use
it
multiple
luster
luster
OS
trees
and
that
got
some
somewhat
better
performance.
Then
we
also
identified
there
is
a
and
there
was
a
Sun
with
a
hitch,
five
PI
and
fixing
that
but
improved
the
performance
much
too
much
larger
and
and
I'm
not
going
to
the
details
of
the
buckets
or.
C
But
if
you
are
interested,
let
me
know-
and
we
can
discuss
those
things
and
in
the
qmc
pack,
with
an
older
version
of
hdf5
I.
Think
one
point:
10.1
the
read
and
write
performance
was
much
lower.
The
time
was
my.
It
was
taking
much
longer
time.
We
notice
it
that
five
flows
during
the
file
close
the
the
it
was
taking
much
longer
time
because
of
the
truncate
tear
call
and
we
fixed
that
in
110
and
now
the
performance
is
much
better
for
the
KMC
pack
application.
C
Okay
next
slide
and
we
are
working
with
a
CFD
code
that
is
used
heavily
at
an
NSA
applications
at
Sandia
and
Lana.
We
notice
that
they
have
a
pattern
where
all
the
processes
write
or
read
from
their
metadata
or
the
data,
which
is
very
small,
so
that
was
causing
a
read
and
reader
rights
Tom
that
in
that
makes
the
performance
much
worse,
because
all
the
processes
are
going
to
the
same
location
in
the
file
and
that
sequential
Isis
things
away,
and
it
basically
slows
down
the
performance
significantly
going
to
the
next
slide.
C
So
then,
instead
of
writing,
instead
of
making
all
the
processes
write
that
there's
a
1:1
process
rights
that
into
a
compact
storage
and
that
improves
the
performance
by
thousandfold
or
something
so
next
next
slide
shows
that
so
the
original
performance
was
much
worse
and
when
we
change
it
to
compact
storage,
we
get
pretty
close
to
file.
/
crosses
performance
with
that,
and
this
was
a
much
larger,
win
and
I
believe
this
was
part
of
the
project
outside
ECP,
but
we
we
are
still
taking
advantage
of
all
those
features
and
adding
this
into
the
hdf5
library.
C
Okay,
going
to
the
next
slide,
so
we
are
also
benchmarking
hdf5
on
the
major
systems
that
are
nurse
and
other
leadership
computing
facilities,
for
example.
In
this
case,
I
have
been
benchmarking.
One
VP
Keio
benchmark,
which
have
writes
the
data
from
large
number
of
Trustees
so
with
here,
I
show
that
performance
for
me
big
right
thing
to
luster
on
quarry
and
data
war,
so
I
do
this
every
quarter,
so
this
was
something
that
I
observed
about.
C
September
of
2019
I
do
see
quite
better
performance
around
two
thirty
two
to
two
forty,
eight
gigabytes
per.
Second,
on
both
cluster
and
data
board,
for
some
reason,
when
we
switch
it
to
32,
32,
megabytes
and
32,
make
oh
sorry,
32k
processes,
the
date
of
our
performance
was
going
down
significantly
and
when
I
did
the
latest
one
which
I
am
still
running
some
of
them,
some
of
the
jobs
are
still
in
the
queue.
I
see
that
performance
profile
is
much
different,
so
we
have
to
investigate
why
that
is
happening.
C
C
C
The
Athena
is
astrophysics
code
that
was
spending
40
percent
of
the
execution
time
in
IO,
Quincy
and
Jo
and
Lou
was
asked
I
looked
into
that
and
they
notice,
if
that
the
collective
IO
was
actually
not
happening.
So
they
looked
at
some
of
the
introspection
interface
of
hdf5
and
identified
the
bottlenecks
and
solving
that
making
sure
that
IO
is
jelly
in
done
collectively.
Yeah
reduces
that
portion
to
1%
of
the
execution
time.
C
Similarly,
another
unnecessarily
and
unnecessarily
doing
offer
copy
was
taking
significant
amount
of
time
and
the
neurological
disorder
I
a
pipeline
there
and
observing
that
using
the
same
introspection
interfaces
there,
a
deucedly
perform
the
drug
use
at
the
execution
time
of
IO
time
from
40
minutes
to
2
minutes.
So
that
is
a
major
win
as
well.
I'm
going
to
the
next
slide
think
those
are
all
the
application
related,
slides
Quincy
will
take
over
to
talk
about
the
actual
features
that
we
have
been
developing
in
the
ECT
project.
Oh
that's.
A
Great
thanks
very
so,
diving
into
the
more
technical
details
of
things.
I
talked
a
little
bit
earlier
and
show
the
slide.
That's
about
the
hdf5
architecture
that
showed
this
virtual
object.
Layer
inside
the
library
and
I.
Can
it's
a
new
capability
that
we've
implemented
using
HTTP
funding
and
talk
a
little
bit
more
about
it's
a
really
important
concept
and
future
expandability
option
for
hdf5.
A
So
this
new
layer
that
we
put
inside
the
library
it
drives
a
distraction
or
abstraction
layer
all
the
way
across
the
API
routines
that
deal
with
file
and
data
set
and
group
object
IO
into
containers
that
allows
every
IO
operation
to
be
redirected
even
at
runtime,
and
not
have
to
be
always
using
the
hdf5
native
file
format.
So
these
connectors
they
actually
implement
the
storage
or
pass-through,
but
I'll
talk
about
that
for
the
objects
and
then
they're
invoked
with
callbacks
that
are
effectively
methods
on
those
objects.
A
So
they'll,
look
more
object-oriented
level
of
things
like
create
the
data
set
or
read
and
write
data
set
elements.
Query
things
close
an
object,
they're
super
beneficial
and
flexible
for
applications,
because
they
can
be
transparently
invoked
with
a
dynamic
library
with
an
environment.
Variable
application
doesn't
even
have
to
be
recompiled.
All
it
has
to
be
is
you
know
originally
built
with
a
version
of
hdf5
includes
this
fall
capability
in
the
1.12
2-0
release,
which
just
came
out
this
month.
Has
that
and
then
the
app
can
take
advantage
of
whatever
the
thing
is.
A
You
can
retarget
storage
containers
or
add
capabilities?
I'll
show
a
few
here
with
this
new
layer
and
it's
tremendously
flexible,
the
fall
connectors
themselves.
They
can
be
stacked
with
as
pass-through
connectors
and
one
or
more
zero
are
passed
through.
Connectors
can
be
layered
on
top
of
the
terminal
connector
at
the
very
bottom
and
right
now,
we've
got
several,
perhaps
through
connectors
and
several
terminal
ones.
A
I'll
talk
a
little
bit
more
about
this
asynchronous
a
connector
and
a
connector
that
allows
us
to
do
independent
metadata
operations
on
hdf5
leverage,
the
burst
buffer
and
Cori,
another
nvram
storage
and
HPC
systems
with
data
elevator.
You
can
interface
in
python
plug-ins
connectors,
which
every
luck
and
keep
going,
and
these
can
all
be
stacked
on
top
of
one
another
nicely.
At
the
bottom
level,
we
can
target
different
storage
formats
and,
of
course,
ship.
The
one
that
deals
with
hdf
is
a
native
API
and
all
the
library
that
was
currently
in
development
or
20
years.
A
A
So
the
architecture
changed
quite
a
bit,
but
fortunately
it's
just
moving
this
all
the
native
stuff
down
into
the
native
terminal
driver
and
then
putting
this
layer
in
here
and
all
the
other
hdf5
routines.
They
just
go
into
the
normal
and
she,
if
I,
have
infrastructure,
be
really
only
put
this
and
things
that
operate
in
a
container.
So
it's
an
example
of
that
part
of
the
ECP
funding
was
to
develop
asynchronous
I/o
operations
for
hdf5.
A
This,
as
this
diagram
here
on
the
right-hand
side,
shows
allows
the
I/o
to
be
overlapped
with
the
applications
compute
with
the
goal
of
taking
iOS
overhead
completely
out
of
the
pipeline
for
an
application
there.
Hopefully,
in
the
middle,
we
can
overlap
the
I/o
completely
and
there's
always
going
to
be
some
final,
get
the
last
time
step
out
piece,
but
in
general,
quite
a
bit
of
time
savings
for
an
application
if
they
could
manage
to
do
the
IO
asynchronously
and
overlap
it
entirely
with
the
compute
cycles.
So
we
employed
in
this
two
different
ways.
A
If
you
don't
modify
your
app
at
all-
and
you
do
you
know-
Lincoln
the
asynchronous
small
connector,
the
asynchronous
I/o-
it's
transparently
invoked
what
we
call
implicit
asynchronous
I/o
and
it's
very
conservative.
It
does
all
the
right
things,
but
there's
certain
places
where
it
waits
or
it
has
to
like
make
certain
the
buffer
is
ready
in
order
to
give
back
to
the
application,
because
the
application
doesn't
know
what's
happening.
A
Asynchronously
still
get
some
nice
speed
ups,
but
it's
not
as
good
as
the
explicit
mode,
and
if
the
apps
want
more
control
over
the
asynchronous
operations,
they
can
modify
their
code.
Some
so
they'll
have
to
rebuild
and
then
use
the
asynchronous
operations
explicitly
inside
hdf5.
That
return,
requests
tokens
and
then
manage
those
and
see
when
they're
completed
and
things
like
that
when
the
buffers
available.
So
this
looks
similar
to
this
diagram
here,
the
app
on
the
Left
starts
up
it
initializes
the
asynchronous
small
connector
and
then
performs
all
its
operations
internally.
A
A
All
those
file
open
and
create
an
object
write
to
the
object,
all
that
gets
put
into
a
task
queue
internally
to
the
ball
connector,
and
then
we
monitor
and
see
when
the
app
goes
off
in
this
busy
computing,
and
whenever
that
happens,
then
we
started
executing
IO
things
in
the
background
for
those
yep.
Let's
go.
Get
these
things
out
to
disk.
Well,
you're,
off
busy
running
your
actual
computer
simulation
code.
All
happens
in
the
background.
A
We
do
steal
a
thread,
but,
generally
speaking,
most
apps
are
beginning
to
run
in
environments
where
there's
more
cars
than
compute
to
fill
them.
They're
memory
constrained
or
something
else,
and
they
can
get
forward
to
give
us
a
thread
to
top
rate
with,
of
course,
there's.
You
know
how
much
memory
is
the
app
dealing
with
and
all
these
other
aspects
to
it.
A
If
you
do
it
explicitly
better
modified
the
V
pic
IO
benchmark,
so
that
it
would
overlap
the
IO
with
the
simulated
compute,
it's
a
bio
benchmark,
so
it
doesn't
actually
have
any
compute,
but
basically
sleeping
as
the
main
thread,
and
you
can
see
that
if
you
don't
have
any
overlapping
I'm
sorry
we
get
this
blue
curve.
That's
basically
flats
like
yep,
you
get
the
same
I/o
performance
and
it's
you
know
pretty
constant,
as
you
scale
up
the
app.
But
when
you
do
that,
then
you
do
the
overlapping.
A
You
can
push
the
perceived
gigabyte
per
second
ratio
for
your
IO.
You
know
up
to
infinity
right.
You
know
down
to
zero
time,
reduce
the
cost
quite
a
bit,
so
you
can
see
the
same
thing
happening
over
here
in
the
right
hand,
side
with
the
cation
benchmark.
So
this
is
some
sizable
benefits
and
if
apps
can
modify
their
code
to
overlap
the
compute
in
the
IO
and
then
have
enough
I'm,
just
a
spare
cord,
so
they
can
spare
thread.
They
can
get
some
very
sizable
benefits.
B
A
A
A
A
So
another
beverage
for
the
fall
connectors
here
is
this
capability
called
data
elevator
right-
and
this
is
basically
write.
Caching
using
burst
buffers
originally
on
Cori,
but
is
we're
you
know
pushing
it
out
into
all
the
other
recent
systems
that
have
mb
ram
and
some
mechanism
for
doing
some,
no
local
or
global
first
offering
for
the
system,
it's
transparent
to
the
application
and
it
moves
data
in
and
out
of
the
NVRAM,
the
first
buffer
on
its
own.
A
The
app
says:
please
do
that
file
and
then
the
data
elevator
component
is
in
charge
of
moving
pieces
of
that
file
into
the
first
buffer
as
it
needs
them
so
tested
with
a
particle
and
cell
and
the
trumbo
I/o
benchmarks.
A
couple
more
ECP
applications
are
evaluating
the
data
elevator,
so
this
is
relevant
to
the
ECD
apps,
we're
seeing
all
along
the
way,
and
it's
available
now
on
khari.
If
you
really
want
to
experiment
with
it,
and
you
think
your
app
it's
gonna
get
some
good
benefits.
You
can
just
modulo
the
data
elevator
and.
C
A
You
know
getting
some
benefits
or
experimenting
with
it.
You
can
see
here
that
Buster
in
the
data
war,
BPI
and
the
data
work
command
itself
and
then
compared
to
the
data
elevator
slightly
better
than
the
data
work
directly
and
definitely
better
than
muster
and
other
aspects
of
the
system.
Typically,
as
you
scale
up,
this
muster
is
coming
down,
but
the
elevator
is
still
performing
very
well
I
think
it's
scale
we're
limited
by
the
move
in
but
paging
from
more
page
ioads
in
this
case
to
to
the
olestra
file
system
on
the
back
end.
A
So
this
also
applies
to
the
read
side
of
things.
So,
if
you'd
like
to
bring
the
data
in
and
do
analysis,
there's
a
capabilities
we've
worked
on
here
to
to
predict
the
data
accesses
based
on
the
history
for
the
application,
prefetches
them
into
the
faster
storage
and
then
tries
to
you
know,
give
the
app
the
benefit
of
these
things.
You
can
see
over
here.
In
the
right
hand,
side
that
this
is
you
know
a
big
win
for
against
luster
and
data
warp
in
general
and
quite
a
bit
with
the
burst
buffer
and
SSDs
involved.
A
A
In
our
final
fall,
connector
component
here
that
I
wanted
to
mention
is
this
independent
metadata
modification,
which
is
a
a
bit
obscure
in
a
sense
but
historically
currently
hdf5
metadata
modifications
that
change
the
namespace.
What
the
files
contents
look
like,
not
the
raw
data
elements
in
the
array,
but
the
actual.
What
are
the
groups
and
datasets
in
here?
Those
must
be
performed
collectively
by
all
the
ranks
in
your
application.
A
This
is
a
tremendous
restriction,
it
dates
back
to
ASCII,
red
and
the
late
1990s
and
the
lack
of
waste
set
aside
threads
and
nodes
for
the
applications.
Everybody
did
everything
collectively
and
it's
kind
of
been
baked
in
for
a
while,
but
with
a
ball
connector
we
can
implement
a
pass-through
ball
connector
and
then
that
does
all
the
stitching
together
and
the
voting
and
everything
else
that
gets
all
the
ranks
to
eventually
trickle
out
the
change
that
1-ranked
made
on
the
file.
A
A
It
does
a
lot
of
work
for
you,
it's
it's.
It's
broadcasting
that
hey
somebody
made
a
change
in
here
is
that
okay,
everybody
gets
a
chance
to
vote
it
records
changes
from
other
ranks
and
then
eventually
starts
sending
those
guys
out
to
disk,
and
we've
done
the
considerable
above
work
here
to
get
all
these
pieces
working
correctly,
so
that
the
app
doesn't
have
to
think
about
what's
going
on
and
we
stay
out
of
their
way
with
the
way
these
MPI
and
very
non
blocking
mechanism
considerably
useful
piece
of
this.
A
We
haven't
never
adjusted
yet,
but
the
voting
manager
here
another
component
to
this
our
pluggable
and
we
will
be
able
to
add
in
and
change
the
mechanism.
So
instead
of
MPI
for
doing
the
voting,
we
could
use
0,
mq
or
some
other
inter
process,
not
an
MPI
process
communication
mechanism
and
that
more
distributed,
apps
start
modifying
hdf5
files
because
there's
a
mechanism
for
voting
controlling
and
becoming
aware
of
changes
to
those
files.
This
really
moves
us
into
a
realm
where
hdf5
s
are.
The
files
are
like
it's
a
even
closer
to
the
file
system.
B
These
are
really
interesting
features,
though,
over
the
role
connectors
and
is
there
I'm
trying
to
think
of
what
the
leading
use
case
is
or
a
leading
use
case?
Are
there
any
user
communities
that
have
picked
this
up
in
a
distributed
sense?
You
know
I'm
more
familiar
with
HDFS
a
great
way
to
access
the
parallel
firepower
of
a
file
system,
but
I'm
curious,
like
in
kind
of
a
different
different
orthogonal
direction.
Are
there
people
who've
kind
of
jumped
on
the
distributed
aspect?
Well,.
A
There's,
coincidentally,
you
know
because
I
know
you
work
with
slack
or
not
the
kinds
of
environments
from
light
sources
where
there's
multiple
cameras,
each
warning
record
its
data
and
into
a
shared
structure.
Eventually
that
kind
of
multi-process,
but
not
an
infinitely
num
large
number
of
writers
makes
a
lot
of
sense
for
this
kind
of
architecture.
We
would
use
0
mq
instead
of
MPI
or
some
other
mechanism,
or
you
know,
voting
and
doing
the
ledger
management.
But
we
worked
pretty
well
there
I
would
say
yeah.
B
A
B
Under
thinking
it,
but
as
an
example
of
something
to
kind
of
show
the
distributed
aspect
of
some
of
these
resilience
type
features
and
data
management
features,
really
you.
You
could,
for
instance,
land
a
hdf
file
with
a
Vall
connector
to
distribute
super
facility
information.
You
know
so
that
there
was
a
resource
status
file
in
hdf
right
that
was
maintained
via
these
mechanisms.
You
know
something
like
that
to
do.
It
wouldn't
be
solving
a
scientific
workflow
per
se,
but
these
types
of
functionalities
may
be
advanced
enough.
That
a
toy
example.
B
A
A
All
right
so
wrapping
up
the
independent
metadata
modifications.
You
can
see
that
before
you
had
to
collectively
MA
you
know
everybody
had
to
participate
in
greeting
all.
The
datasets
in
the
file
was
bottom
overhead,
especially
is
the
number
of
ranks
increased,
but
afterwards
it's
like
boom.
You
know,
hey
I,
want
to
new
Dana,
said
I,
create
the
dataset
I
just
go
it's
very
much
simpler
they're,
not
this
crazy
unscalable
mechanism
for
modifying
files.
A
So
on
towards
you
know,
lower
levels
of
the
storage
stack
with
hdf5
if
one
of
the
features
that
the
team
at
Argonne
is
is
really
heavily
looking
into
is
exploring
and
expanding
on
their
topology
awareness
work
that
they've
done
before,
and
this
is
a
mechanism
for
putting
another
at
a
lower
level
in
hdf5
by
a
virtual
file
driver
which
is
down
inside
the
native
ball
connector.
But
it's
a
replacement
for
the
MPI
driver
in
there
and
that's
it's
a
version
of
that
that
is
uses
system
calls
to
become
aware
of
well
just
exactly.
A
Where
are
the
you
know
interconnects?
And
how
is
everybody
all
these
ranks
are
on
the
same
node
and
therefore
I
can
aggregate
more
quickly
from
I/o
or
how's.
The
storage
interface
going,
and
maybe
I
should
ship
my
data
over
to
that
rank
because
he's
closer
to
the
storage
system.
So
that's
the
exploration,
work
they're
doing
here
and
you
can
see
that
when
done
correctly,
you
can
get
some
very
nice
performance
boosts
and
benefits
from
the
the
increasingly
stretched
out.
Elongated
storage
stack
that
we're
moving
towards.
A
I'll
long
those
lines
and
a
little
bit
higher
up
inside
hdf5,
typically
htif
I'd,
writes
to
a
single
file
and
it's
been
a
historical
mode
for
things.
We
have
got
mechanisms
for
kind
of,
avoiding
that
in
other
ways
than
sub
filing,
but
that's
been
the
norm,
and
although
people
do
create
trial
for
process
do,
but
if
you'd
like
to
avoid
that,
maybe
we
can
do
everybody
thinks
they're
writing
to
the
same
shared
file,
but
we
avoid
the
locking
contention,
that's
the
overhead
for
that.
A
So,
instead
of
a
single
shared
file,
we
create
a
metadata
file
that
describes
things
so
single
shared
metadata
file
in
that
case,
but
then
multiple
smaller
sub
files
that
are
the
back-end
for
storing
the
actual
raw
data
underneath
to
the
user.
They
won't
see
it
so,
hopefully,
we
can
kind
of
combine
this
with
the
topology
aware
thing
and
we
create
sub
files
that
are
logically
organized
according
to
the
topology
of
the
system.
A
Implementing
you
know
very
nice
speed
ups,
possibly
even
to
no
local
storage
and
everything
else,
and
again
this
is
coming
in
with
the
ball
connector.
It's
it's
kind
of
a
partial
pass-through.
It
passes
through
all
the
metadata
to
a
single
shared
file,
but
anytime
it
seeds,
raw
data.
It
sends
it
out
to
these
other
sub
files
that
it's
creating
off
to
the
side
looks
fairly
promising
we're
just
in
the
final
design
stages
for
this,
but
it
looks
very
promising.
A
They
tried
queue
for
query
creation
and
then
combine
queries
and
then
apply
those
unto
a
file
to
get
a
result
set
back
and
trying
to
find
good
ways
to
encourage
more
applications
to
to
use
this
query
an
index
capability,
it's
a
little
tough
because
it's
not
necessarily
one
of
their
built-in
I
use
this
all
the
time
many
people
already
have.
You
know,
flows
that
they
may
use
for
their
workflow.
That's
not
reliant
on
this
and
they
have
different
mechanisms
for
it.
A
So
done
with
EGP
things,
we're
doing
work
with
experimental
and
observational
data
from
some
funding
from
Oscar
as
well,
and
this
does
kind
of
a
general
bullet
list.
We've
done
a
whole
talk
about
this
if
you'd
like,
but
this
is
the
one
of
the
places
where
we
expect
to
do
that.
Multiple
writer
consumer
of
the
data
delay
I,
was
mentioning
earlier
with
the
independent
metadata
plug
ability.
A
That's
been
funded
for
a
you
know,
prototype
project
this
year
and
as
part
of
that,
we're
on
developing
an
xtc2,
vol
connector,
again
all
connectors.
This
is
a
file
format,
adapter
vault
connector.
So
it's
read-only
for
now,
XTC
files
are
produced
by
slack's
systems
right
now,
but
there
are
custom
format
and
they
have
their
own.
You
know
subset
of
tools
that
understand
them
and
that's
harder
to
get
larger.
You
know
communities
to
build
off
the
xtc2,
because
it's
mostly
a
slack
thing.
A
So,
instead
of
doing
that,
how
can
we
open
up
the
xtc2
file
format
and
the
data
that's
producing
with
without
changing
the
actual
data
itself?
So
we're
gonna
we're
almost
done
now
with
a
ball
connector,
that's
a
reader
and
makes
all
the
XTC
files
look
like
hdf5,
so
you'll
be
able
to
open
up
that
xtc2
file
and
do
hd5
API
calls
data
sent
open
data
to
read
close.
A
A
Lots
of
the
data
from
E&E
sources
has
variability
to
it.
In
some
sense,
it's
sensor.
Values
are
negative
strings
in
there
and
hdf5
supports
this.
But
it's
really
not
in
a
high
performance
aspect.
It's
never
been
released,
stressed
and
especially
not
for
streaming
applications.
So
we
want
to
design
and
prototype
out
improve
support
for
that
and
with
the
goal
of
getting
to
the
same
level
of
performance
for
the
variable
length.
Elements
is
for
fixed
size,
arrays
of
floats,
and
it's
which
hdf5
has
been
optimizing
for
20
years
right
now,
the
redesign
looks
very,
very
good.
A
I
would
say
we
can
probably
achieve
you,
know
near
disk,
speed
or
variable
length
data,
basically
constant
time,
access
or
append
and
then
later
iteration.
Currently,
that
would
be
kind
of
poor
and
then
year
time
I
would
say
for
variable
length
and
actually
better
than
logarithmic
access
pattern.
We
have
now
for
fixed
length
data,
so
it
will
probably
twist
the
fixed
length
data
to
take
advantage
of
this
for
when
it's
streaming.
A
So
that's
the
last
hdf5
thing,
just
a
quick
blurb
about
unify
FS
who's,
our
collaborator
for
the
excess
iodine,
a
lot
of
work
to
make
no
local
births
buffers.
Look
like
a
single
shared
file
system
across
all
of
those
ranks.
That's
temporarily.
You
know
set
up
when
the
app
starts
running
and
then
IO
happens
and
then
flushed
out
the
disk.
A
So
it's
a
it's
a
runtime
generated
burst
buffer
in
a
sense,
a
lot
like
quarries
and
we're
working
with
them
to
make
certain
that
hdf5
has
an
adapter
that
works
directly
on
the
top
of
unify
FS
and
that
we
don't
use
POSIX
calls
that
unify
FS
gets.
You
know
it
doesn't
have
support
for.
So
it's
collaboratively
building
out
both
sides
of
that
equation
over
time.
I
think!
That's
it.
A
If
you
need
help,
it's
a
great
forum
for
HTF
group
on
HTF
group
website,
there's
an
actual
dive
helpdesk
and
mention
that
you're
with
ECP
project
and
they
will
expedite
getting.
You
did
the
right
people.
If
you
have
direct
questions.
Serena
and
I
are
always
available
and
happy
dance
or
email
or
talk
instead
of
telecoms.
Whenever
any
questions.