►
From YouTube: 2021-08-10 - Brandon Cook - LDMS data at NERSC: doing something useful with 8 PB of CSV files
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Speaker:
Brandon Cook
Application Performance Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
Title:
LDMS data at NERSC: doing something useful with 8 petabytes of CSV files
Abstract:
Analysis of telemetry data from NERSC systems offers the potential for deeper quantitative understanding of the NERSC workload - providing insights for future system design, operation and optimization of the current platform, feedback to developers about workflow performance and diagnostics to uncover issues with workflows. NERSC uses the Lightweight Distributed Metric Service (LDMS) for lightweight collection of a large variety of metrics on NERSC systems; including memory usage, CPU HW counters, power consumption, network and I/O traffic. Across the compute nodes of Cori currently a total of ~400 MB/s worth of data is being collected and stored in CSV file format. With current retention policies there are approximately 8 petabytes of data in CSV format. The size and number of the CSV files along with the desire to integrate with other sources such as Slurm accounting poses several challenges for anyone who wants to work with this data. In this talk, I will walk through this data set: how it is collected, what is in it, where it is located. Then I will discuss a post processing pipeline that transforms, filters, joins this data with Slurm accounting information, and finally stores it 20 - 200 times more efficiently than CSV. Finally I will discuss how the results of the pipeline are used in Iris through the Superfacility API to provide plots directly to users for all non-shared jobs on Cori in O(seconds). Throughout the talk I will highlight how these resources can be accessed and extended.
Bio:
Brandon leads the simulations area of NERSC's application readiness program (NESAP) and works on understanding and analyzing performance on a system and application level, developing future benchmark suites, analyzing future architectures, developing tools to help NERSC users/staff be more productive, engaging users through consulting, acting as NERSC liaison for several NESAP teams, and exploring future programming models. Brandon received his Ph.D. in physics from Vanderbilt University in 2012, where he studied ab initio methods for quantum transport in nanomaterials. Before joining NERSC he was a postdoc at Oak Ridge National Laboratory where he developed and applied electronic structure methods to problems in material science.
Host of Seminar:
Wahid Bhimji
Acting Group Leader, Data & Analytics Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
Yeah,
so
thanks
for
for
hosting
me,
I'm
just
gonna
talk
a
little
bit
about
the
ldms
data
that
we
have
here
at
nurse,
and
this
talk
is
is
not
quite,
I
would
say,
like
your
typical
research
talk,
it's
kind
of
a
a
mix
of
like
a
tour
of.
What's
there,
how
you
can
get
a
hold
of
it,
some
of
the
things
that
you
could
potentially.
A
C
A
So
first,
I
just
want
to
acknowledge
that
I
didn't
do
everything
by
myself.
Lots
of
people
have
contributed
in
in
one
way
or
another,
I'm
sure
I'm
forgetting
some
people
and
that
that
ranges
from
from
really
configuring
and
maintaining
the
ldms
thing
to
providing
some
valuable
comments
and
insight
just
through
discussions.
A
So
some
many
of
you
are
on
here.
So
thank
you
so
first,
I
just
want
to
take
a
minute
to
motivate.
You
know
why
bother
dealing
with
eight
petabytes
of
csv
files
and
and
why
make
them
in
the
first
place
seems
like
a
good
place
to
start
so
one
you
want
to
ideally
be
able
to
give
some
feedback
in
an
automated
way
to
to
users.
So
these
are
like
diagnostics
and
a
lot
of
times.
A
This
you
know,
if
you,
for
example,
know
very
detailed
information
about
the
memory
usage
you
could
proactively
inform
them
hey.
You
know,
it
looks
like
your
code
has
a
linearly
increasing
with
time
memory
footprint.
So
it
looks
like
you
might
have
a
memory
leak.
You
should
check
that
out
or
you
know,
after
the
fact
we
can
go
in
and
say:
oh
well,
look,
you
know,
you
blew
up
the
memory
and
that's
why
your
job
crashed
other
things.
A
Can
you
get
a
roof
line
and
then
there's
kind
of
a
higher
level?
I
would
say
system
level,
questions
that
you
can
ask,
so
you
know,
can
you
make
scheduler
more
efficient
by
adding
some
some
data
that
it
can
draw
from?
You
know:
prediction
of
power
usage
or
a
prediction
on
the
usage
of
some
contended
resource
like
io
or
network,
and
then
just
in
general,
for
informing
the
design
and
operation
of
of
new
systems.
A
You
know
just
generically,
maybe
this
higher
resolution
data
can
can
give
us
more
insight.
You
know
questions
like
does
disaggregation
make
sense,
or
can
we
use
system
level
data
to
kind
of
draw
a
line
through
a
whole
workflow
that
might
consist
of
of
multiple
applications
and
multiple
jobs
and
that
that
can
potentially
help
answer
some
questions
when
you
know
building
nurse
10
or
n
plus
one
so
that's
kind
of
the
the
promise
of
of
this
source.
D
A
A
B
E
A
A
The
free
memory
memory
cons,
that's
in
buffers,
cached
and
then
there's
a
few.
You
know
the
whole
list
of
what's
in
there
and
in
addition,
so
this
is
just
one
example,
so
there
are
samplers
for
I,
o
traffic,
there's
samplers
for
the
aries
network,
there's
samplers
for
the
cpu
performance
counters.
So
if
you're
familiar
with
papi,
it
sort
of
builds
on
the
papi
interface,
so
things
that
are
like
instructions
per
cycle,
cache
misses
et
cetera.
A
A
A
This
amounts
to
about
400
megabytes
a
second
and
with
with
this
data
rate
and
our
current
retention
policy.
That
means
we
have
about
eight
petabytes
of
of
these
metrics
from
these
samplers
in
in
a
bunch
of
csv
files,
and
so
that's
where
the
eight
petabytes
comes
from
and
then
each
it's
not
just
one
big
file.
Each
file
is
somewhere
between
about
one
and
and
one
gigabyte
and
about
half
a
terabyte.
A
F
F
B
A
I
think
back
of
the
envelope
you
can
see
about
an
order
of
magnitude,
maybe
two
order
magnitude
and
a
half
reduction
in
in
storage
space.
There's
a
couple
trade-offs
right.
If
you,
if
you
go
to
a
since
I'm
I'm
processing
files
in
bulk,
I
can
there's
certain
compression
algorithms
that
are
available,
that
don't
really
work
in
a
streaming
sense.
A
B
E
G
F
G
H
F
C
D
Yeah,
partly
that
depends,
if
you
want
to
retrospect,
you
know
on
on
things
that
came
before,
and
my
guess
is
for
these
columns
that
they
might
have
different.
You
know
compressibilities,
based
on
on
on
what
type
of
column
it
is.
You
know
the
text.
One
would
probably
have
a
lot
more
repeated
strings
than
than
a
yeah.
A
That's
fine.
I
I
think
I
I'm
pretty
sure
I'm
gonna
answer
a
number
of
these
questions
as
I
get
through
so
so.
This
is
just
sort
of
a
a
starting
point
of
okay.
What
what's
in
the
these
files
and
we'll
we'll
get
through?
I
think
some
of
the
questions
about
compression
and
the
different
data
types
as
well
and
just
to
kind
of
follow
up
on
some
of
the
I
see
a
question
or
two
in
the
chat.
Now
the
the
collecting
an
actual
sample
is,
is
really
cheap
right.
A
B
A
Kind
of
overhead
being
and
then
the
other
thing
I'll
say,
is
just
running
gzip
isn't
so
simple.
First
of
all,
you'll
be
waiting
forever,
and
then
you
have
a
problem
of
information
retrieval
right.
So
if
you
just
gzip
this
stuff,
how
do
you
get
out
and
extract
the
the
specific
rows
and
and
columns
that
you
care
about,
and
then
on
top
of?
That
is
something
that
david
alluded
to.
A
A
I
A
C
I
A
A
So
I
I
guess
one
of
the
questions
that
sort
of
came
up
in
the
discussion
is,
you
know,
is
this
stuff
any
good
right?
Do
you
want
to
make
a
decision?
Are
you
actually
going
to
want
to
make
a
decision
based
on
on
it?
I
think,
honestly,
the
answer
is,
for
the
most
part
we
don't
know
and
so
to
kind
of
answer
this
particular
question
we've
started
an
effort
that
is
actually
it's
pretty
arduous.
If
you
want
to
kind
of,
do
it
properly
right.
C
A
D
A
A
So,
in
addition
to
just
the
raw
metric
values
which
are
ascii
floats
typically
or
integers,
there's
a
producer
name,
which
is
the
node
id
that
that
the
samplers
run
on
there's
a
timestamp
there
and
then
there's
a
job
id
column,
and
this
is
kind
of
the
first
caution
I
would
say
to
anybody.
Looking
at
this
stuff
is
that
the
the.
C
A
B
A
A
So
this
is
a
a
column-oriented
data
storage
format,
kind
of
similar
in
some
ways
to
hdf5,
except
that
this
one
is,
I
think,
a
little
bit
less
complicated,
and
this
is
really
the
I
guess,
the
native,
these
archival
type
format
for
the
apache
hadoop
ecosystem,
and
so
this
gives
you
a
couple
benefits
one.
It's
a
file
format
designed
specifically
for
this
like
column,
oriented
data,
which
means
that
you
can
very
efficiently
extract
like
a
single,
a
single
metric
out
of
a
giant
file.
A
A
Some
versions
of
it
support
something
called
bit
packing
and
then
yeah,
it's
got
kind
of
native
gzip
or
snappy
is
a
another
compression
algorithm.
That's
particularly
good
for
this
kind
of
thing
built
in
and
sort
of
a
byproduct
of
it
being
one
of
these
a
thing
that's
heavily
invested
in
by
industry.
Is
that
there's
kind
of
language
support
for
it
everywhere?
You
look,
you
can
there's
cuda
bindings,
there's
native
pandas
and
python
support.
A
Yeah
c
plus
plus
go
java
rust,
you
name
it
and
you
can.
You
can
have
a
native
supported
interface
to
interact
with
these
files
and
then
it's
also
self-describing,
so
it
does
have
some
capabilities,
for
it
describes
its
own
schema,
but
you
can
also
embed
some
metadata
into
the
files
and
yeah.
As
rollin
pointed
out,
this
there's
really
great
gpu
support
and
I
think
nvidia's
invested
a
lot
of
effort
making
sure
that
they're,
I
don't
know
what
they
call
gpu
direct
io
or
anyway
their
their.
B
A
You
know
why
are
you
giving
me
a
talk
about
this?
Can't
you
just
write
five
five
lines
of
python
code
and
called
a
day,
and
so
this
th
this
is
sort
of
like
the
maybe
naive,
step,
zero
and-
and
actually
it
is
a
viable
thing
to
do.
The
only
downside
for
this
or
well
there's
a
couple
you're
still
stuck
with
a
bad
job
id
column.
A
A
A
There
are
some
missing
pieces
every
once
in
a
while,
the
most
the
one
I've
seen
the
one.
That's
there
reliably
is
that
occasionally,
when
the
you,
you
rotate,
which
csv
is
being
written
to
right,
so
the
aggregator
is
continuously
writing
or
pending
to
a
csv
file
and
that
at
a
particular
time
stamp
it'll
switch
and
start
writing
to
another
one.
And
occasionally
what
can
happen?
A
Is
you
get
a
partial
final
line
in
the
csv
file,
which
really
throws
a
wrench
in
and
all
a
lot
of
this
stuff,
so
that
one
final
row,
if
you're
just
missing
a
couple
elements?
Well
many
of
these
things
will
just
say:
oh
that's,
not
a
number
or
like
they'll,
throw
it
in
as
a
just
a
blank
string
or
something
like
that,
and
then
that
throws
off
your
type
inference
and
can
can
kind
of
confuse
these
things
a
little
bit
so
you're
stuck
either.
A
A
B
A
A
C
A
A
Database
in
memory
then
does
the
join
and
then
backs
it
back
out.
So
that's
that's
not
ideal.
There's
I
think
I
tried
almost
every
other
one
of
these
kind
of
solutions
that
that
says.
Oh,
you
can
do
sql
on
a
pandas
data
frame
and
everyone
I
tried,
doesn't
support
this
type
of
query.
They.
They
really
want
to
do
simple,
really
basic
inner
joins
kind
of
thing.
A
H
C
A
And
then
maybe
I'll
just
comment
so
so
nestor
you're
right,
I
the
I
have
used
the
the
rust
csv
crate.
The
issue
with
doing
rust
is
that
the
I
sort
of
cheated
a
little
bit
when
I
said,
there's
good
language
support
for
parquet,
the
the
parquet
writer
in
rust
in
the
official
rust
crate
is,
is
less
than
ideal
for
a
couple
reasons,
and-
and
maybe
I
won't
go
into
all
of
them
now,
but
it's
it's
not
great
for
kind
of
easily
writing
these
things.
A
A
So
there's
sort
of
two
approaches:
one
involves
using
a
ton
of
memory
and
that's
what's
up
on
the
screen
now,
and
this
is
almost
an
exact
copy
and
paste
from
the
the
code
that
I've
I've
got
running
so
first,
you
get
your
your
data
frame
that
has
all
of
your
jobs
from
that's.
Basically
just
pulling
the
output
of
an
s
account
command
for
a
time
frame.
A
Then,
when
you
do
that
you
get
the
host
list
comes
in
a
compressed
form
right,
so
you
have
job,
and
then
you
have
the
short
short
string
that
has
all
the
draw
by
all
the
note
id's
that
the
job
is
on
so
first
you
need
to
unpack
unpack
that
string
and
then
there's
this
nice
explode
thing.
That's
in
relatively
recent
versions
of
pandas
and
what
that
does
is
it
takes
a
list?
If
you
have
a
column,
that's
full
of
list
entries.
A
So
this
really
addresses,
I
think,
richard's
comment
about
it
being
like
random
searches.
So
this
is
sort
of
what
you
would
do
to
to
not
do
a
random
search
with
a
multi-index
like
this,
you
a
multi-index,
that's
sorted
right,
the
the
producer
name,
you
can
have
a
hash
table
that
looks
up.
That
tells
you
what
indices
you
want
and
then
for
the
time
range.
If
you
know
it's
a
sorted
sorted
list,
you
can
do
a
binary
search
and
that's
what
what
pandas
does
in
the
this
dot?
A
A
A
Of
the
the
things
like
desk
or
moden,
or
any
of
the
other
ones
that
can
purport
to
make
kind
of
drop
in
parallelized
data
frames
for
for
python,
none
of
them
support
multi-index
objects.
They
only
support
regular
index
objects
and
if
you
can't
index
on
both
the
job
id
and
the
or
the
both
on
the
node
id
and
the
time
timestamp,
then
it's
just
super
inefficient
and
you
don't
want
to
be
doing
that.
A
That's
what
you
need
to
do
to
get
a
good
job
id
column,
but
then
what
about
derived
metrics?
So,
for
example,
the
per
cpu
counters
come
in.
As
you
know,
let's
say
the
total
cycles
counter
this
one
cut
you
get
one
of
those
for
every
hyper
thread
and
in
general
you
know
per
hyper
thread.
Isn't
the
most
interesting
use
case
here
this
this
date
is
all
pretty
coarse
grained
anyway.
So
what
you
probably
just
want
is
like
the
cycles
per
node
right
and
that's
a
pretty
basic
sum.
A
Go
look
up,
and-
and
maybe
just
as
an
aside
in
case-
you
don't
know
in
later
version
the
versions
of
the
arc
parse
library
in
python.
There's
support
for
subclassing
the
artparse.action
object,
and
this
lets
you
make
really
powerful
command
line
interfaces
for
things
like
this,
so
that
whole
pipeline
and
post
processing
tool
also
supports
all
the
same
features
from
the
config
file
via
command
line
options.
A
A
Metrics,
I
guess,
as
I
sort
of
mentioned,
that's
a
bit
slow
in
in
python
and
it's
you're
also
dealing
with
the
occasional
like
incomplete
row.
So
the
the
approach
I
took
was,
I
wrote
a
c
plus
code,
that's
multi-threaded
and
I
use
the
pi
bind
interface
to
to
go
through
and
do
the
the
column
selection,
applying
any
operations
and
and
so
on,
and
so
by
switching
to
c,
plus,
plus
and
and
kind
of
by
hand.
F
C
B
A
B
B
E
A
A
A
B
A
A
What
do
you
get
out
of
doing
all
this
stuff-
and
this
is
this-
is
about
a
20
to
200x
size
reduction
and,
let's
see
so
on
the
on
the
left
and
blue,
is
the
or
are
two
of
the
are
two
histograms
of
like
csv
file
size
and
then
on.
The
right
is
a
histogram
of
the
same
same
set
of
samplers,
but
in
parquet
format,
and
you
can
see
that,
for
example,
on
the
the
cis
papi.
So
this
is
like
the
cpu
counter
stuff.
H
A
In
a
regime
where
you
can
kind
of
load,
one
of
these
files
up
on
your
laptop
or
just
a
regular
non-high
memory
node
and
do
some
some
analysis
so
nodes
accessing
the
parquet
files
at
once.
Typically
one
there's
that
may
or
may
not
change
in
the
future.
If,
if
this
stuff
becomes
super
popular
and
everybody
wants
to
to
read
all
their
stuff
at
once,
but
but
usually
just
one
just
one
node.
A
H
A
B
A
C
A
You
don't
need
to
deal
with
the
csv
files
at
all
anymore.
If
you
don't
want
to
there's
a
there's,
a
gitlab
project
that
has
got
a
very
nice
readme
and
all
you
have
to
do
is
send
a
pull,
request
or
or
tell
me
which,
which
metrics
you
want
out
of
it
like
which
columns,
and
so
you
don't
have
to
do
anything
other
than
add
something
to
this
repo
and
then
wait
and
then
the
next
time
this
pipeline
runs
it'll
start
appearing
from
that
time
going
forward.
A
But
if
you
really
want
to
this
is
all
set
up
to
not
depend
on
any
particular
person's
environment.
So
you
can,
you
can
clone
the
project,
make
make
new
content
environment
and
then
install
the
install
the
package
into
that,
and
it
comes
with
a
set
of
tests
and
everything
to
make
sure
that
you've
done
it
properly.
A
Database
questions
in
general,
I
mean
the
so
it's
true
that
I've
done
like
quite
a
lot
of
reduction
here,
but
it's
not
quite
enough
unless
somebody
wants
to
cough
up
some
a
real
amount
of
money
to
run
some
dedicated
time
series
database
hardware,
and
I
think
my
ballpark
estimation.
If
you
wanted
to
do
something
like
a
cassandra
database,
you
you
might
need
on
order
20
or
30
or
40
nodes
in
a
distributed
database
cluster
dedicated
to
this,
and
so
that
that's
that's
one
option.
A
F
A
A
C
A
E
Think
we're
getting
close
to
the
time
yeah
yeah
yeah
we're
getting
close
to
the
top.
That
we've
obviously
sparked
a
lot
of
discussion,
so
we
should
probably
just
cover
some
more
of
your
material
and
then
answer
people's
questions
at
the
end.
In
case,
people
need
to
drop,
but
I
can
hang
out
for
a
bit
at
the
end
for
the
discussion.
B
B
A
B
A
You
can
read
the
same
thing,
but
now,
with
the
job
step
joined
in,
which
is
an
extra
step,
you
can
do
some
analysis.
So
this
is
like
memory,
a
histogram
of
memory
used
by
job
grouped
by
if
it
was
on
haswell
or
knl,
and
then
you
can
have
time
series
that
are
either
you
know
just
the
whole
node
time
series
or
you
can
have
it
split
out
by
job
step
and
all
of
that
is
sort
of
wired
into
iris
today.
A
A
A
So
if
you
want
to
add
a
new
plot,
that's
all
very
modular.
It's
you
really
just
write
one
python
function
that
defines
a
plot
in
and
spits
out.
As
long
as
you
can
output
png,
encoded
data,
you
can
have
a
new
plot
and
embed
it.
So
I've
got
a
project
where
I'm
trying
to
put
a
roof
line
into
into
this
you'll
get
a
roof
line.
You
could
have
like
a
spider
plot.
A
Some
things
that
are
ongoing
is
our
gpu
support,
kind
of
mapping
of
steps
to
gpus,
more
gpu,
metric
validation
and
analysis,
support
for
uncounters,
that's
like
memory
controller
data
and
then
there's
there's
lots
of
questions
like
you
know.
I
see
people
discussing
this
like
do
you?
Do
you
want
a
different
data
format?
A
A
Theoretically,
you
can
insert
directly
into
something
like
victoria
metrics,
although
to
be
fair,
I
don't
know
what
the
interface
for
getting
data
back
out
of
that
is
or
the
interface
for
doing
a
join
and
then
there's
some
questions
about.
Maybe
we
can
provide
this,
this
data
like
resampled
or
not
through
some
kind
of
api
as
well.