►
From YouTube: 2020-01-10 - Tonglin Li - Independent metadata updating for large scale parallel I/O systems
Description
NERSC Data Seminars: https://github.com/NERSC/data-seminars
Abstract: For bridging the ever-widening performance gap between computation and storage systems, new tiers are introduced to the already deep storage hierarchy. I/O middleware, such as HDF5, has been developed and used for decades to provide applications relatively simple APIs and hide all the low-level details of the underlying I/O and storage systems. As the systems scale out, some of the old designs such as the collective metadata updating mechanism that fits smaller scales start to show the performance penalty. For addressing this problem, we have built a new HDF5-based I/O middleware prototype that enables independent metadata updating. In this presentation, I'll talk about the challenges, the design and implementation of the solutions, and the lessons we learned from building the system.
A
A
So
in
this
talk,
I'm
gonna
cover
a
couple
of
topics,
so
in
the
background,
I'm
gonna
introduce
a
little
bit
the
evolving
process
of
the
scientific
application,
along
with
supercomputers
and
parallel
storage
and
aisle,
and
the
why
parallel
aisle
and
the
middleware
and
metadata
they
matter
to
the
scientific
applications
and
for
the
motivation.
I'm
gonna
show
what's
the
problem
of
the
current
method
of
operation
with
IO
middleware,
so
here,
I'm
gonna
use
the
f5
as
the
example
and
that's
also
the
context
for
the
whole
whole
talk
here
so
the
to
solve
that
problem.
A
We
have
a
couple
challenges
and
also
we
gave
some
solutions
to
solve
out
so
majorly.
This
is
also
a
system,
design
and
implementation,
especially
in
context
distributed
and
a
parallel
environment,
and
then,
after
that,
I'm
gonna
show
some
preliminary
results,
because
this
is
just
just
a
prototype,
although
it's
functioning,
oh,
the
performance
is
not
that
well
of
function-wise,
worse
pretty
good.
Now
and
then
the
conclusion.
A
So,
first
of
all,
in
the
beginning,
a
sin:
haste
will
have
a
problem
to
solve
like
a
weather
simulation
or
a
particle
simulation
simulation.
So
we'll
have
a
problem
and
you
have
a
computer
to
solve
that
and
you
will
either
machine
this
app.
Do
all
those
commendation,
so
the
resource
you
have
is
the
machine
hard
drives
and
you
can.
You
can
use
my
spreading
to
do
the
to
your
class,
your
multiple
cores
and
also
at
this
stage
pretty
much
most
of
the
application
have
their
own
data
format.
A
You,
you
find
people
stop
to
talk
you
stuff
to
call
you
a
scientist.
They
start
to
call
you
a
computer
scientist,
and
you
feel
this
is
quite
distracting
because
you
are
out
of
track
of
your
own
research,
and
this
is
the
why
you
want
you
to
leave
all
all
of
these
complexity
to
the
to
the
qu,
a
computer,
scientists
and
computer
engineers,
so
the
supercomputers
involved
in
this
way.
A
So,
as
more
course
and
computer
resource
introduced
to
the
whole
system,
you
also
get
a
deeper
story
hierarchy,
because
the
performance
gap
between
the
compute
and
a
star
b
resource
getting
one
getting
wider
and
generally
at
the
which
you
fix.
That
problem
is
to
introduce
more
layer
of
cache
buffers
and
also
many
more
so.
In
this
case,
we
can
all
can
see
in
the
figure
it
actually
shows
types
of
risk.
A
So
it's
the
almeida,
we're
here
to
hide
all
of
these
and
the
project
like
the
h,
305
and
neccessity
apps
play
an
important
role
here
and
not
just
to
provide
the
Irish
features
like
to
give
you
a
view
of
the
single
file
to
hold
all
your
data,
which
is
much
larger
than
any
of
the
name.
Your
hard
drives,
and
also
you
you'll,
transparently,
you've
lost
all
those
newly
introduced
layers.
Lakum
refers
new
caches.
A
A
To
avoid
the
conflicts
running
to
the
simplement
same
place,
they're
asked
to
only
write
with
their
own
their
own
chunks
or
blocks
here.
So
each
of
them
write
children
either
is
read
or
write.
There
are
not
hidden
the
same
place,
so
you
can
see
pretty
much
the
end.
They
owns
their
own
data
entries
here
and
for
metadata.
A
If
that's
the
rate
case,
it's
also
pretty
similar,
but
the
only
difference
here
is
where,
when
you're
doing
the
method
read
like
you
want
to
open
the
dataset
here
in
your
use
case,
each
of
the
rank
open
the
dataset
and
they
own
they
own
their
own
data
set.
So
they
all
they
will
load
corresponding
metadata
into
their
own
local
cache.
Although
these
pieces-
although
all
of
these.
A
Can
have
something
you
can't
have
some
overlap,
so
something
like
the
describe
the
file
structure
and
things
like
that,
and
also
we
have
some
something
of
their
own
like
metadata,
describing
their
own,
their
own
data
entry
and
that's
a
partner
not
overlapped,
but
because
there
there's
no
change
made
in
any
of
this
any
of
this
hash.
So
technically,
all
of
these
cash
are
still
consistent,
although
they,
although
they
may
not,
look
like
the
same
because
they
are
just
different
pieces
of
the
whole
picture
and
the
whole
picture
are
still
the
same.
A
So
when
you
try
to
write
something
like
you
want
to
create
it,
it
is
that
from
each
of
the
rank,
none
problem
happens
because
each
rank
hadn't
have
their
own
local
metadata
cache.
They
change
the
logo,
change
will
for
their
first
and
and
then
to
flush
to
the
disk
right
right
to
the
material
file,
but
the
problem.
The
problems
here
is
because
each
change
made
by
the
ranks
are
different.
So
to
merge
all
of
these.
A
So,
to
avoid
all
those
meta,
corruption
or
conflicts,
the
actual
way
used
used
as
your
father
and
also
similar
mechanism
in
the
NASA
TF,
is
the
collective
marketer
update.
So
in
this
way,
each
of
this
every
single
process
work
rats,
are
asked
to
to
do
exactly
the
same
thing
so
here
here
you
can
see
it's
not
like
process
one
create
that
it
was
at
a1
and
rank
you
credit.
It
is
at
you.
A
So
the
way
the
way
they
are
doing,
this
is
try
to
create
a
consistent
view.
So,
each
of
them,
each
of
this
small,
small
master
cash,
will
look,
look
like
the
same,
and
then
they
will
reach
the
clock,
hips
sinking,
sinking,
point
and
their
barrier
there,
and
also
adapt
at
that
place.
The
system
which
act
sanity
and
to
make
sure
all
the
cuz
all
the
metal
master
change
submitted
are,
are
consistent
and
then.
A
So
the
biggest
problem
is
the
program
program
ability,
because
if
you
want,
you
write
a
MPI
program
and
if
you
do
there
is
that
a
create
in
one
of
the
ranks?
Actually
you
mean
I
I
just
want
to
create
my
own
rank.
My
own
said
here,
but
you
actually
have
to
do
the
similar
thing
for
all
other
ranks,
so
the
programmer
bottom
will
look
like
this:
the
lower
box.
You
have
to
do
these
in
the
in
the
for
loop
or
a
similar
thing
for
all
the
ranks.
This
is
not
a
big
problem.
A
A
is
running
with
a
small
faster,
but
if
you
running
with
any
of
the
large-scale
supercomputers,
this
will
you
thinking
if
you
have
to
run
every
wrong
on
that,
every
single
rank
and
to
create
a
million.
That
is
this
that,
on
each
of
the
issue
of
the
ranks,
then
this
will
be
really
mass
and
also
it's
not
just
the
just.
The
code
look
more
complicated.
It
also
caused
a
big
bottleneck
over
there,
because,
because
we
see
you,
there
will
be
out
like
I'm
sinking
point
here,
so
to
also
to
make
sure
everything
is
right.
A
So
it's
not
just
the
pencil
great.
There
are
quite
quite
a
few
other
costs,
and
all
of
these
actually
are
collective
and
they
happened
very
often
in
during
an
application
running,
especially
when,
when
there's
a
change
frequently
so
the
impact
either
to
the
performance
or
to
the
to
the
code,
logic
will
be
quite
big
so
to
solve
this
problem
were
people
a
system
to
do
the
independent
Mathematica
changes
and
to
achieve
this
goal
we
have
three
major
challenges
to
address.
The
first
one
is
the
most
obvious
one.
A
Another
one
is
the
current
MPI
communications,
especially
those
one
to
all,
and
all
communications
like
be
cast
and
all
reduced.
They
all
need
a
route
to
function.
Well,
so
meaning
you
have
to
specify
a
route
to
start
the
a
big
house
and
at
the
same
time
you
have
to
set
up
a
bunch
of
receive
to
receive
this
because-
and
this
have
to
be
down
in
the
explicit
way,
but
in
the
real
real
world
you
don't
know
when
and
where
a
process
wants
to
really
do
the
big
cast.
A
So,
typically
the
system
we
use
the
rank
0
to
as
a
as
a
route,
and
everybody
is
since
that
it
has
to
this
one
and
we'll
use
this
one,
but
still
this
will
be
a
bottleneck
in
the
large
large
machine.
Oh
another,
one
is
modest
writing,
because
current
MPI
is
not
fully
supporting
the
modest
Freddy.
Although
there
are
some
research
projects
still
is
not
quite
mature.
So
on
the
single
node,
how
to
handle
the
concurrency
is
to
still
a
problem
so
to
solve
that
problem
solve
all
these.
A
For
the
first
one
to
make
the
consensus
we
we
design,
distributed
voting
mechanism
to
reach
the
consensus
and
for
the
second
challenge,
we
implement
a
set
of
new
server
list
operations
to
do
the
big
cast
and
all
radios.
So
to
call
this
function,
you
don't
have
to
you,
don't
have
to
specify
a
route,
and
you
just
start.
You
just
started
and
everybody
will
receive
the
results
for
the
motif
reading,
because
it's
not
supported
so
we're
using
where
actually
we
implement
our
own
progress
engine.
A
So
it's
the
state
machine
with
a
bunch
of
message,
queues
to
track
different
states
of
each
of
the
events
or
the
messages,
and
we
encapsulate
all
of
these
into
a
single
layer
of
the
issue
of
five
lakh
nectar,
and
this
is
the
overall
texture
of
the
H
of
5,
lakh
Nitro,
so
a
mall
which
is
virtually
a
layer.
This
is
this
is
just
a
layer
of
the
hosts,
a
capacity
of
5.
A
So
this
intercepts
all
the
requests
from
the
application
and
to
change
this
either
to
change
this
request
or
to
outer
this
or
to
reroute
this
and
then
to.
After
all,
all
of
the
these
operation.
You
will
flash
to
the
HT
a
5/5
and
words
to
note
that
the
ball
plug
the
ball
connectors
here,
its
stackable.
That
means,
if
you
have
a
bunch
of
different
functions
for
different
balls,
you
can
stack
them
all
together
and
have
the
feature
have
all
of
the
features
together.
A
So
when
the
process
aj5
receive
the
request,
you
will
look
at
that.
You
will
look
at
if
this
is
the
change
to
the
method
to
the
metadata
and
because
for
now,
only
the
only
the
collective
of
method
operations
need
to
be
modified
here.
So
we
only
cover
that
part
and
the
local
matter
that
change
will
be
revolved
to
this
medical
manager.
And
this
all
these
big
box
and
small
components
are
all
stay.
Inside
of
this
ball
layer.
A
A
So
for
implementing
this
on
the
first
one
is
the
ruthless
or
the
serverless
communication,
which
is
the
PCAST
and
I
all
reduce.
So
we
implement
this
over.
This
skip
ring
overly
in
network,
so
we
we
arrange
all
the
participant
ranks
into
this
skip
ring.
This
is
the
modified
structure
of
this
skip
list,
so
each
of
them
have
a
static
receiving
channels
and
sending
channels,
so
they
don't
have
to.
A
They
don't
have
to
change
the
of
those
parameters
during
the
program
program
running-
and
the
big
has
happened
like
this.
So
if
the
note2
wants
to
do
a
big
house,
so
here
you
just
send
some
message
to
to
all
the
available
standing
channels
and
then
the
receivers
the
kaabah
I
receive
already
posted
and
it
was
you
will
decide
if,
if
you
forward
this
to
the
to
the
next
stage
and
then
during
the
message,
transferring
the
massive
numbers
actually
doubling
almost
every
a
very
simple
step.
So
the
communication
time
is
logarithmic.
A
So
to
do
that,
you
sort
of
voting
that's
to
happen
on
them
on
this
skippering
network,
so
to
abstract
the
all
the
nodes
participated.
We
can.
We
can
look
at
that
on
the
right
side.
Actually,
this
this
is
the
the
commissioning
tree
extracted
from
from
the
left
side
the
ring.
So
when
the
proposal,
we
call
a
message
and
a
proposal
before
it's
really
really
being
in
made
to
the
file
if
a
proposal
is
is
initiated
from
from
rent
one
and
the
receivers
will
because
and
to
the
forward
to
the
next
stage.
A
If
this,
if
the
proposal
is
compatible
with
this
own,
so
in
this
case
run
to
and
rank
for,
they
all
accepts
the
that
change
and
think
that
that
change
doesn't
break
my
own
thing
and
also
other
our
other
ranks,
except
for
the
run.
Six
here
run
six
fund,
you
will
break
my
own
metadata,
so
I
say
no,
and
then
you
will
go
back
with
a
no
answer
and
within
this
tree
that
the
response
is
being
aggregated.
A
A
A
So
for
the
nos
radical
currency
handling,
we
use,
we
actually
build.
Three
are
two
different
progress
engine
on
different
levels.
One
is
for
the
communication.
Another
is
for
the
distributing,
and
this
this
involves
earlier.
The
more
technical
details
and
and
I
put
this
into
a
backup
slides
if
you're
interested
I
can
I
can
get
to
that
part
of
it
later
so.
A
In
terms
of
the
results
here,
the
baseline
is
the
is
the
default
which
is
collecting
metadata
and
the
yellow.
The
yellow
bar
here
is.
The
Foley
is
riddled
boating
and
independent
metadata.
So
you
can
see
here.
Actually
the
bring
the
performance
is
pretty
poor
to
be
on,
to
be
honest,
is
quite
slow,
so
the
lack
ring
one
so
that
direct
direct,
because
one
is
the
one
we
try
to
make
some
compromise
between
these
two
and
the
performance
it
between
these
two.
A
And
the
reason
we
got
this
performance
mostly,
is
because
we
didn't
get
enough
enough
time
and
resource
to
all
those
optimization
and
ht5
having
optimized
on
this
for
more
than
20
years.
So
a
so
is
mostly
it's
just
a
the
matter
of
the
time
and
money
resource
and
for
the
application
developers.
Person
for
some
perspective.
So
here
is
this
is
the
the
simplest
case
when
you
try
to
dude
o
matic
the
update
so
to
just
as
create
a
single
data
set
and.
A
If
you
use
this
framework,
then
the
application
can
be
simplified
quite
a
bit
and
especially
when
the
when
the
structure
within
your
su5
is
complicated.
Like
you
have
a
lot
of
data
stat
in
in
the
group
and
in
each
of
file,
you
also
have
multiple
layers
of
the
groups,
and
then
you,
when
you
create
and
change
the
all
of
these
structure,
you're
touching
the
metadata
and
the
be
a
lot
operation
and
the
code
will
be
much
more
complicated
than
than
this.
A
A
To
improve
the
performance
of
the
system,
the
first
one
is
to
invest
more
time
and
resource
I.
Just
like
I
said
we
have
only
a
year
and
a
half
to
work
on
this
and
hdf5
group
spent
twenty
years
to
optimize,
and
also
because
the
current
code
base
is
using
the
progress
engine
to
handle
the
concurrency.
So
the
parallelism
of
the
available
parallelism
provided
by
all
those
fancy,
harbors
and
multiple
cores
and
hyper-threading
actually
are
not
only
utilized.
A
So
if
we
can
use
a
fully
supported
multi-threading
and
you
use
the
teddicus
thread
to
play
the
role
of
each
of
those
progress,
engine
I
believe
that
the
system
will
will
be
running
way
more
efficient
because
in
the
current
in
the
current
implementation,
we're
using
a
sliding
window
technique
to
to
avoid
too
the
misalignment
of
all
those
operations,
because
the
the
order
of
them
had
the
operation.
Actually
matters
quite
quite
much
here
and
also
we're
trying
to
are
trying
to
contact
the
MPI
foreign
so
to
include
these
ruthless
operations
into
their
to
their
standard.
A
So
as
a
conclusion,
so
these
will
be
useful
for
application,
designers
and
the
users,
and
this
is
by
largely
simplify
the
metadata
operations
like
create
change,
resize
and
to
add
work
to
to
write
new
attributes,
and
all
of
these
are
all
of
these
can
be
simplified
and
the
the
code
will
be
more
intuitive.
So
it's
not
like
you
have
to.
Everyone,
have
to
do
everything
every
single
place
and
the
so
in
the
new
way.
A
Actually,
it's
more
natural
way,
you
only
do
your
own
thing
on
your
own
place,
so
the
code,
the
application,
will
also
see
the
actually
five
more
like
a
local
file
system,
and
actually
this
is
the
one
of
the
initial
goal
of
actually
five
four
system.
Software
designers,
these
these
techniques
and
components
actually
can
be
used
as
building
blocks.
So
the
MPI
ruthless
primitives
itself
is
generic
and
it
can
be
used
with
any
MPI
application
for
the
consensus
making
platform.
A
You
can
also
be
used
used
by
other
software
and
don't
have
to
change
a
lot
of
code
and
because
we
designed
them,
we
designed
the
consensus
making
using
a
plugging
mechanism.
So
actually
you
can.
You
can
choose
different
different
ways
to
do
the
voting
and
choose
to
use
different
policy
to
decide
when
to
approve,
and
once
you
deny
a
proposal
in
one
of
our
ongoing
project
where
you
were
using
POSIX
file
system
to
reach
consensus-
and
that's
also
can
be
plugging
in
into
this
independent
independent
platform
and,
of
course,
performance,
because
it's
still
poor.