►
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title: Monitoring Scientific Python Usage at NERSC
Speaker:
Rollin Thomas
Data & Analytics Group
National Energy Research Scientific Computing Center (NERSC)
Abstract:
Last year about 30% of all NERSC users ran jobs involving Python in some way. The most popular Python package in use in Python jobs isn't NumPy, it's multiprocessing. And among Python-based MPI jobs, AstroPy appears to be the most significant package in use. How do we know this and how does this information help us? In this talk, I will discuss how NERSC monitors the use of Python on its systems, as part of its Monitoring of Data Services (MODS) project. The talk will cover how the data is collected, stored, analyzed, and published for consumption by various stakeholders (staff, management, developers, etc.) using a Jupyter notebook-centric workflow involving GPUs, Dask, Papermill, Voilà, and Spin.
Bio:
Rollin Thomas is a Data Architect in the Data and Analytics Services group at NERSC. From 2015 to 2020 he was in charge of Python support on Cori.
Host of Seminar:
Wahid Bhimji
Acting Group Leader, Data & Analytics Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
Anyway,
good
well,
I
I
I
vaguely
introduced
you,
but
I
don't
think
that's
fully
necessary.
As
everyone
knows,
ron
is
in
the
data
analytics
group
at
nursk
and
yeah
and
led
a
lot
of
the
python
development
at
nurse,
putting
it
on
the
quality
footing.
It
is
now
and
now
him
and
laurie
and
others
have
are
taking
it
to
the
next
level,
with
advanced
python,
monitoring
and
so
yeah.
This
is
going
to
be
a
talk
at
scipy
and
he's
kindly
giving
us
a
preview.
B
B
It's
a
25-minute
talk
and
there's
supposed
to
be
five
minutes
for
questions
at
the
end,
and
so
this
is
a
practice
and
I'm
trying
to
get
the
time
right.
So
if
there's
questions
unless
there's
like
some
emergency
question
or
something
like
that-
maybe
hold
it
till
the
end
and
since
we'll
have
lots
of
extra
time,
we
could
go
into
extra
background
material
if
people
are
interested
in
hearing
about
that.
So
with
your
indulgence,
I'm
going
to
go
ahead
and
give
this
just
like
I'm
going
to
give
it
tomorrow.
B
So
all
right,
so
I've
started
my
timer
here
all
right,
so
hi,
I'm
roland
thomas,
I'm
from
the
data
and
analytic
services
group
at
nursk,
and
I'm
going
to
tell
you
today
about
the.
Why
and
how
of
our
monitoring
of
scientific
python
usage
on
our
supercomputers,
and
I'm
going
to
tell
you
about
the
framework
that
we've
put
together
to
analyze
the
data
that
we
collect,
also
using
python
and
share
with
you
some
of
the
more
interesting
results
that
we've
gathered
over
the
past
year
of
of
monitoring
all
right.
B
I
mentioned
that
I'm
from
nurse
nursk
is
the
national
energy
research
scientific
computing
center.
It's
located
at
berkeley
lab
in
berkeley.
California
nursk
is
the
primary
scientific
computing
facility
for
the
us
department
of
energy's
office
of
science,
which
in
turn,
is
the
largest
federal
sponsor
of
basic
research
in
the
physical
sciences
in
the
united
states.
B
To
do
this,
we
operate
supercomputers,
some
of
the
largest
and
most
powerful
systems
fastest
systems
on
the
planet.
We
have
two
supercomputers
on
the
floor
right
now
at
nursk,
the
older
one
is
corey.
It's
been
running
in
operation
for
about
six
years
now,
and
promutter
is
the
new
system
which
is
getting
ready
for
its
first
users
and
is
number
five
on
the
top
500
list.
Since
corey
is
the
production
system,
that's
where
all
the
monitoring
of
python
that
I'm
going
to
be
talking
about
was
done.
B
So
that's
going
to
be
the
focus
of
the
talk
today,
all
right.
What
do
we?
Why
are
we
trying
to
gather
data
from
user
activity
in
the
first
place?
Well,
workload,
analysis
workload
data
informs
how
we
deploy
and
support
supercomputers
at
nursk.
What
we
want
to
do
is
understand
what
the
users
are
actually
doing,
not
from
what
they
say
they
do,
but
from
the
data
that
their
interaction
with
the
system
actually
generates.
B
Surveys
of
what
users
are
doing,
what
their
requirements
are
gathering
requirements.
All
of
that
goes
alongside
the
data
collection,
but
what
we
find
is
that
the
data
often
gives
us
a
nice
way
to
open
conversations
with
users
and
to
ground
the
conversations
with
data
this.
This
is
used
in
two
main
ways.
The
first
one
is
for
procurement
planning.
We
need
to
understand
where
users
are
now
in
terms
of
their
use
of
the
current
system.
What
kind
of
things
are
constraining
them?
How
we
can
improve
on
this?
B
B
So
what
about
scientific
python
usage
on
our
super
computer
python
is
the
newcomer
in
high
performance
computing
compared
to
fortran,
c
and
c
plus
plus,
but
what
we
and
others
have
observed
is
really
rapid
growth
in
the
use
and
need
for
support
requests
over
the
past
five
years
in
python,
to
the
point
that
vendors
and
policy
makers
are
actively
curious
about
it.
So
program
managers
at
doe
want
to
know
how
people
are
using
python
on
our
systems
and
we
want
to
know
how
python
should
impact
procurement
of
systems
and
how
we
support
our
users.
B
So
we
do
our
we.
The
last
workload
analysis
that
was
done
was
in
2018
and
looking
at
that,
we
we
seem
to
be
missing
the
full
extent
of
python
use
on
the
system.
The
focus
of
of
workload.
Analysis
is
necessarily,
of
course,
on
batch
kind
of
workload,
which
which
makes
a
lot
of
sense,
but
we
were
kind
of
curious
about
whether
or
not
that
might
be
sufficient
to
explain
the
full
breadth
and
depth
of
python
use
on
the
system,
in
particular
we're
especially
interested
in
what
libraries
or
frameworks
are
in
use
by
python
users.
B
On
the
system,
just
saying
that
30
of
users
are
using
python
is
a
bit
like
saying
they're
using
bash.
What
are
they
actually
doing
with
python?
Are
they
doing
things
with
numpy,
scipy,
scikittler
and
deep
learning?
That
kind
of
thing
so
trying
to
figure
out
where
users
are
in
terms
of
libraries
and
frameworks
in
python
is
very
important.
B
So,
overall
we
wanted
to
figure
out
how
we
could
enhance
the
workload
analysis,
effort
by
gathering
more
detailed
data
on
python
as
part
of
this
project
called
the
monitoring
of
data
services,
project
or
mods,
which
is
our
business
intelligence
project
in
in
the
das
group,
where
we're
trying
to
help
represent
the
data
stack
and
workload,
analysis
and
python
monitoring
is,
of
course,
a
big
part
of
that
right.
So
if
we
want
to
do
python
monitoring,
what
do
we
actually
need
to
do
want
to
do?
What
kind
of
information
do
we
need
to
gather?
B
And
how
are
we
going
to
do
this?
So
our
goal
was
to
try
to
monitor
any
python
3
that
was
installed
on
corey,
whether
or
not
that
was
the
python
that's
installed
by
the
vendor
or
by
ourselves.
Through
software
environment
module,
we
also
wanted
to
capture
installations
of
python
environments
done
by
users
using
condo
or
spac,
or
if
they
build
from
source
or
whatever
they're
doing.
We
also
wanted
to
make
sure
that
we
captured
python
use
in
containers.
B
B
We
also
want
to
capture
python
use,
not
just
in
batch
jobs
on
the
compute
nodes.
We
also
want
to
capture
its
use
on
login
nodes,
jupiter
nodes,
which
are
not
in
the
batch
in
the
in
the
batch
administered
by
the
the
batch
queue
system,
workflow
nodes
and
other
special
nodes
like
gpu
and
large
memory
clusters
that
are
attached
to
corey
as
well.
B
We
want
to
provide
users
with
a
way
to
opt
out
if
there's
a
reason
for
them
to
opt
out,
they
can
do
that
for
other
monitoring
technologies
that
we
have
installed
on
the
system,
and
so
this
should
be
no
different.
However,
their
participation
in
this
effort
is
really
important
to
us
and
it
helps
us
help
them.
So
we
really
hope
that
they'll
continue
to
opt
in,
and
that's
part
of,
why
we're
talking
about
this
in
this
venue.
B
And
finally,
one
of
the
things
we
don't
want
to
do
is
cause
hardships
for
ourselves
or
for
users.
So
we
need
to
make
sure
that
we
follow
our
standard,
operating
procedures
and
testing
of
the
software
and
make
sure
that
everything
just
kind
of
goes
right
in
with
all
the
other
software
that
we
install
on
the
system,
but
also
we
don't
want
to
mess
up
user
workflows.
B
B
So
the
really
the
first
group
that
we
found
doing
this
outside
of
those
other
outside
of
those
other
possible
solutions,
was
done
on
blue
waters
by
our
our
friend
and
colleague,
colin
mclean,
who's
actually
now
at
berkeley
lab,
and
he
was
managing
bw
pi,
which
is
the
blue
waters,
installation
of
python
on
blue
waters,
and
he
presented
his
metrics
at
the
cray
user
group
meeting
about
four
years
ago
and
was
the
inspiration
for
what
we
do
here.
So
the
basic
idea
is
to
instrument
the
python
installation.
B
Of
course
they
have
a
python
installation
that
they
provide
for
users
and
so
they're,
mostly
interested
in
instrumenting,
that
the
technique
is
to
use
something
called
sitecustomize.pi,
which
is
a
part
of
the
python
standard
library.
It
runs
at
normal,
startup
anytime.
You
invoke
the
python
the
python
binary
within
that
you
register
an
exit
handler
that
runs
on
normal
shutdown.
That
handler
reads:
sys
modules,
which
is
the
dictionary
that
keeps
track
of
everything.
That's
been
imported
up
to
that
point
and
at
blue
waters
they
serialize
this
to
disk.
B
B
B
We
want
to
make
sure
that
we
install
the
libraries
local
to
the
node
to
avoid
trying
to
kind
of
clog
up
the
network
with
attempts
to
access
these
these
libraries
over
the
network.
We
need
a
robust
solution
for
storing
the
data.
Putting
them
on
disk
didn't
seem
like
a
really
great
solution
in
our
context,
and
of
course
you
need
to
be
careful
so
testing
in
terms
of
unit
tests,
integration
tests,
making
sure
that
the
things
are
actually
working
properly
on
the
test
system.
B
B
You
just
install
something
that
you
want
to
import
and
see
it
show
up,
and
then
what
you
do
is
you
insert
kind
of
a
script
or
a
module
at
sitecustomize.pi
at
the
path
shown
here?
Just
basically
anywhere,
I
think
insist.
Paths
will
actually
work,
but
this
seems
to
be
the
best
place
and
then
inside
of
that
install
and
exit
handler,
which
is
just
a
function
that
you
write
and
it
can
take
arguments
and
it
can.
B
It
can
see
sys
modules
and
things
like
that
and
you
register
it
using
this
at
exit
register
function,
and
then
I
have
a
little
print
to
say
that
the
site
customize
is
actually
running
and
when
you
do
this
and
you
do
something
like
import
scipy.
The
first
thing
you
see
is
hello
from
site
customize
because
that's
running
in
the
site,
customize,
it's
actually
running
at
python,
startup
and
then
the
import
happens,
and
then
the
exit
handler
fires-
and
you
see
hello
from
the
exit
handler
and
you
see
all
the
stuff
that
got
imported.
B
While
we
were
importing
scipy
and
among
them
you
see
the
dependencies
of
scipy,
lots
of
other
stuff
and
and
sci
pi
itself
all
right.
So
that
can
be
a
lot
of
a
lot
of
data
for
invocation
of
python.
Obviously,
all
right!
So
that's
the
basic
idea.
It's
actually
a
little
bit
more
complicated
than
that
and
we
try
to
make
it
easy.
Easy
to
manage
and
more
organized,
and
so
basically
there
are
three
things
that
you
need
to
worry
about.
B
B
B
The
reporter
object
then
forwards
all
the
hits
and
the
context
data
to
storage
and
in
order
to
make
sure
that
we're
not
missing
any
kind
of
emerging
interesting
packages,
one
in
ten
thousand
python
implications
forwards,
the
entirety
of
cis
modules,
to
kind
of
to
kind
of
figure
that
out,
so
that
actually
happens
several
times
an
hour.
Basically,
if
you're
curious,
how
often
that
happens,
some
of
the
extra
context
metadata
is
shown
in
the
little
table
on
the
side
there.
B
B
This
is
this
is
a
table
that
summarizes
kind
of
differences
in
similarities
between
the
blue
waters
python
and
quarry
python
monitoring.
The
main
ones
to
mention
here
are
that,
instead
of
doing
the
kind
of
append
only
logging
that
was
being
done
on
blue
waters,
we
forward
everything
that
we
gather
to
omni
the
operations,
monitoring
and
notification
infrastructure,
basically
just
using
syslog
and
forwarding
that
through
the
elasticsearch
database,
there
it
supports
containers
through
local
volume,
mount
on
the
compute
nodes
or
on
the
login
node,
where
it's
being
used.
B
B
So
that's
how
we
get
data
to
the
central
data
collect.
What
do
we
do
with
it
once
it's
there
part
of
omni.
Our
omni
includes
a
kibana
instance,
and
we
use
that
a
lot
for
visualization
for
creating
dashboards
and
it
works
fine.
Once
you
get
the
hang
of
the
the
query
syntax,
but
we
found
that
we
often
have
deeper
questions
and
that
we
kind
of
needed
a
bit
more
of
a
python-based
workflow
that
we
were
comfortable
with
for
analyzing
the
data
in
more
detail.
B
So
we
our
requirements
for
this,
are
based
on
the
kind
of
natural
workflow,
which
is
that
we
want
to
be
able
to
interactively
explore
as
much
of
the
data
as
possible
early
on.
We
want
to
firm
up
that
kind
of
prototype
exploration
and
turn
it
into
something
that
we
can
run
in
production,
and
we
want
to
be
able
to
communicate
the
findings
we
have
to
stakeholders,
and
so
I
mentioned
the
word
interactive
exploration,
and
so
people
are
probably
going
to
assume
that
jupiter
is
a
part
of
this
and
it
is.
B
B
Dashboards
are
much
better,
so
the
solution
that
we
decided
to
try
kind
of
as
an
experiment
was
to
see
if
we
could
kind
of
string
together
the
pieces
of
the
python
data
ecosystem
and
see
if
we
could
just
use
that.
So
we
used
jupiter
notebooks
for
interactive
exploration
of
the
data.
This
inside
of
those
notebooks.
B
We
use
das
and
pdf
coupon
the
rapids
ecosystem
on
the
gpu
cluster
that
we
have
to
be
able
to
chunk
through
as
much
data
as
we
can
there's
a
package
called
paper
mill
which
is
for
parametrizing
and
executing
those
notebooks
in
batch,
and
we
submit
those
as
batch
jobs
periodically
using
cron
and
there's
another
package
that
can
take
jupiter
notebooks
and
render
them
as
dashboards
called
voila.
And
we
run
that
in
spin
the
kubernetes
rancher
setup
that
we
have
for
auxiliary
services.
B
We
can
do
this
with
a
single
notebook
for
each
analysis,
with
a
little
bit
of
work
and
we'll
talk
about
how
that
worked,
and
we
use
a
single
docker
container
for
interactive
batch
and
dashboard
use
in
kubernetes.
All
these
tools
are
things
that
our
own
users
can
use,
so
we're
kind
of
sampling.
Our
own
champagne,
here
I
mentioned
that
we
try
to
use
a
single
jupiter
notebook
to
manage
all
of
this.
B
So
your
notebook
kind
of
turns
into
this
kind
of
multi-part
document
where
there's
an
analysis
exploration
analysis
part
at
the
top,
which
is
run
on
hpc,
using
gpu,
enabled
code
for
data,
cleaning
reduction,
trying
different
models
out
and
then,
ultimately,
writing
reduced
data
to
disk.
This
gets
executed
interactively
by
the
user.
It
can
be
run
in
batch
using
jupiter
and
paper
mill.
The
dashboard
part
which
runs
on
on
spin
spin,
doesn't
have
gpus,
so
we
rely
on
reduced
data
there
to
create
the
visualizations,
but
you
can
also
run
the
dashboard
testing
part
interactively.
B
So
the
kind
of
downsides
that
we
encounter
with
this
is
that
managing
tags
can
be
a
little
bit
cumbersome.
It's
easy
to
forget
to
apply
them
or
to
remove
them,
and
also
you
tend
to
generate
a
little
bit
of
extra
boilerplate
code
in
the
notebook.
So
you
might
end
up
importing
packages
in
two
contexts
which
isn't
harmful
but
might
look
weird.
B
I
mentioned
paper
mill
for
productionizing
notebook
runs
basically,
once
you
have
a
template
notebook
in
place,
you
just
can
kind
of
import
it
and
execute
it,
and
we
can
use
the
cell
metadata
tags
to
kind
of
pre-filter
out
the
cells
that
we
don't
want
to
run
in
batch,
and
then
we
submit
a
batch
job
all
right.
The
dashboard
since
demos
are
kind
of
dangerous
to
do
we
put
together
live
demos
are
a
little
bit
dangerous.
We
put
together
a
little
video
here
to
show
you.
B
This
is
a
public
dash
version
of
the
dashboard
at
mods.nurse.gov
and
there
are
a
few
different
dashboards
on
the
right
for
python
and
jupiter,
and
so
this
is
a
picture
of
the
quarterly
quarterly
use
python
dashboard.
So
it
shows
the
most
popular
python
libraries
correlations
between
python
libraries,
I'm
going
to
mention
later
how
much
python
usage
on
different
parts
of
the
system.
There
was
over
that
quarter
and
so
on.
So
it's
an
interactive
dashboard,
so
you
can
kind
of
move
around
and
inspect
different
years.
B
This
version
here
is
going
to
be
all
of
the
python
data
that
we've
been
collecting
over
the
past
year,
or
so
we've
actually
been
collecting
data
for
a
bit
longer
than
that,
but
about
a
year
ago
is
when
we
were
able
to
get
the
full
system
monitoring
into
place.
So
I
think
that's
not
the
whole
video,
but
I
think
that's
kind
of
enough
to
communicate
what
we're
doing
with
the
dashboard.
B
B
It
turns
out
that
over
30
percent
of
users
are
running
jobs
that
are
using
python.
This
is
python
in
any
form
or
fashion.
The
vast
majority
of
jobs
that
are
run
using
python
are
running
on
the
haswell
nodes.
It's
a
little
bit
easier
to
get
performance
in
python
on
the
on
the
haswell
nodes
than
on
k.
L,
neither
of
those
are
that
surprising,
but
on
the
right
hand,
side
are.
B
B
Here's
the
thing
that
we
were
mainly
interested
in
was
the
top
track
library.
So
these
are
the
top
20
libraries
that
we
were
tracking
and
our
expectations
were
that
scipy
and
the
pi
data
stack
are
going
to
be
very
highly
ranked
that
we'll
see
a
lot
of
visualization
libraries
things
for
parallelism,
jupiter,
astro
pi,
because
we
have
a
lot
of
cosmologists
and
deep
learning
will
probably
rank
pretty
low
because
we
don't
have
a
lot
of
hardware.
That's
really
optimal
for
that.
B
B
But
what
we
find
from
users
is
that
it's
a
combination
of
both
direct
use
of
multi-processing
and
indirect
use
and
for
people
who
aren't
familiar
with
multi-processing,
it's
a
real,
easy
way
to
get
kind
of
process
level
parallelism,
but
it
doesn't
scale
beyond
beyond
a
single
node
right.
So
that's
that's
an
interesting.
B
We
thought
that
was
an
interesting
data
point
in
terms
of
the
distribution
of
batch
job
sizes.
Most
jobs
are
small.
Some
are
large.
That
is
not
unique
to
python.
Most
of
the
workload
at
nurse
kind
of
looks
like
this.
At
the
largest
scales,
the
use
of
python
is
pretty
traditional,
numpy
and
mpi
for
pi,
which
is
kind
of
what
you
would
expect.
B
Pi
torch
seems
to
be
able
to
scale
bigger
than
tensorflow,
or
at
least
users
find
it
easier
to
scale
pine
torch,
or
maybe
they
don't
need
to
scale
tensorflow
as
big
as
my
torch,
I'm
not
sure
das,
which
is
another
library
for
parallelism
that
does
scale
across
multiple
nodes
process
level.
Parallelism
most
of
those
are
staying
below
a
thousand
nodes.
So
we
need
to
do
some
work.
B
I
think,
in
order
to
help
users
make
it
easier
for
users
to
scale
up
their
python
analyses,
I
think
that's
kind
of
on
us
in
terms
of
interesting
things
you
can
do
with
the
data
we
created
a
correlation
matrix
to
see
what
things
kind
of
got
imported
together.
Now,
there's
an
obvious
problem
with
this,
which
is
that
things
depend
on
other
things
and
it's
kind
of
hard
to
disentangle
dependencies
between
libraries
in
order
to
make
this
kind
of
plot.
B
So
we
focused
on
libraries
that
in
enable
parallelism
mpi
for
pi,
multiprocessing
and
desk,
and
one
of
the
things
we
saw
was
that
astro,
pi
and
npi
for
pi
happen
together
a
lot,
and
so
this
might
be
an
opportunity
for
the
astropyte
community
and
the
users
of
the
astropy
package
to
maybe
collaborate
and
develop
tooling.
Around
improved
use
of
mpi.
Also,
maybe
there's
opportunities
for
improving
performance
of
astropy
in
terms
of
I
o
multiprocessing
seems
to
be
kind
of
spread
over
its
use
is
correlated
with
a
whole
bunch
of
other
things.
B
B
Finally
desk
and
desk
goes
with
x-ray
and
net
cdf.
This
is
representing
the
climate
and
earth
sciences.
Community
some
use
of
desk
appears
to
be
incidental.
Some
users
are
not
even
when
we
reach
out
to
them
about
their
desk
use
they.
They
seem
a
little
bit
confused
about.
Why
we're
talking
to
them,
and
I
think
it's
because
they're
they're
not
using
it
for
parallelism
they're
using
some
of
the
some
of
the
auxiliary
things
that
it
provides
like
maybe
reading
csv
files
or
something
like
that.
B
But
what
we
need
to
do
here,
I
think,
is
again
make
desk
a
bit
easier
to
use
okay.
So
those
are
some
of
the
interesting
results,
and
this
is
that's
going
to
bring
us
to
the
conclusion
of
the
talk.
So
the
summary
is
what
we've
done.
We've
met
our
goal
to
enrich
our
insight
into
python
use
on
corey
we've
created
a
pretty
easy
framework
for
inspecting
and
reporting
imports,
and
we've
also
put
it
on
github
in
case
anybody
wants
to
take
a
look
at
it
and
maybe
try
to
extend
it.
B
Our
future
work
is
going
to
be
applying
is
installing
monitoring
for
python
on
the
perlmutter
sisters
system.
So
we
can
understand
what
users,
especially
of
the
cpu
nodes,
are
doing
and
see
whether
or
not
there
are
opportunities
to
help
them
migrate
to
better
use.
The
gpu
hardware
we're
going
to
also
study
how
users
respond
to
the
new
system
and
see
what
how
their
work
workloads
or
their
behavior
change,
try
to
put
together
the
data
with
identity
and
banking
information
that
we
have
identify
points
of
collaboration
with
users
around.
B
Oh.
I
already
said
that
I
should
fix
that
and
possibly
figure
out
ways
to
integrate
this
kind
of
monitoring
into
x-alt,
which
is
kind
of
the
the
standard
tool
used
for
monitoring
all
right.
So
there's
some
links
at
the
bottom,
the
dashboard
and
the
customs
module,
and,
if
you're
interested
in
following
up
with
us,
you
can
email
us
directly.