►
From YouTube: 12 - Jupyter at NERSC
Description
Part of the NERSC New User Training on September 28, 2022.
Please see https://www.nersc.gov/users/training/events/new-user-training-sept2022/ for the training day agenda and presentation slides.
A
Okay,
hi
everybody,
I'm
Roland,
Thomas
I'm
from
the
data
science
engagement
group
and
I'm,
going
to
be
giving
a
little
introduction
today
to
to
Jupiter
at
nurse
right.
A
Okay,
so
I'm
not
really
sure
whether
or
not
everybody
on
the
call
is
familiar
with
what
Jupiter
is,
but
I
thought
I'd
give
a
slide
to
kind
of
level
set.
Everybody
Jupiter
is
an
interactive
open
source
web
application
that
allows
you
to
create
and
share
documents
called
notebooks,
and
these
notebooks
can
contain
Live
code
equations,
visualizations,
narrative
text,
interactive
widgets.
A
You
can
use
Jupiter
notebooks
for
data
cleaning
data
transformation
tasks,
numerical
simulation,
statistical,
modeling,
dataviz,
machine
learning,
workflows
and
analytics.
Frameworks
Jupiter
is
extremely
popular.
It's
becoming
ever
more
popular,
especially
in
the
data
sciences,
and
we've
noticed
that,
obviously
at
nurse
amongst
our
user
base.
A
So
basically
this
is
a
slide.
That
is,
that
explains
kind
of
why
nurse
cares
about
Jupiter
and
what
our
engagement
is.
Obviously,
as
I
mentioned,
it's
become
an
integral
part
of
big
data.
Science,
and
especially,
it
seems
to
be
popular
among
users
of
who
are
doing
experimental,
observational
data
science,
so
experiments
like
lsst
desk
Desi
user
facilities
like
the
ALS
lcls
projects
like
materials
project
and
so
on,
have
workflows
that
are
based
around
Jupiter
notebooks.
A
We
find
that
new
users
who
are
coming
in
the
I
think
it
was
mentioned
earlier.
Something
like
almost
half
of
our
users
are
early
career
users
come
in
and
expect
to
be
able
to
use
jupyter
notebooks
at
our
facility.
So
we
need
to
be
able
to
to
meet
the
needs
there.
A
Over
the
last
four
or
five
years,
usage
of
Jupiter
at
nurse
has
increased
by
about
a
factor
of
five,
and
it
seems
to
always
be
going
up.
So
over
the
summer
we
hit
at
least
a
thousand
users
using
Jupiter
at
nurse
per
month
and
usage
Peaks
generally
over
the
summer,
when
we
have
interns
showing
up
and
and
doing
summer
research
projects.
A
This
number
just
in
case
you're
interested
for
comparison
about
three
thousand
users
per
month,
connect
to
our
super
computers
via
SSH.
So
the
number
of
users
that
use
Jupiter
is
is
comparable.
It's
the
same
kind
of
order
of
magnitude
as
all
of
the
users
who
SSH
into
into
one
of
our
systems.
A
This
slide
shows
that
we've
been
engaged
with
the
Jupiter
project
for
about
the
last
five
to
seven
years.
Really,
I
want
to
make
it
clear
to
everybody
on
on
the
call
that
when
I
say
engagement,
what
I
mean
is
we
actually
work
with
the
developers
and
we
engage
with
the
the
Jupiter
developer
community
in
order
to
make
sure
that
we're
able
to
meet
the
needs
of
of
users
of
Jupiter
on
our
systems.
A
So
this
timeline
kind
of
demonstrates
different
milestones
for
us
and
the
development
of
our
deployment
of
Jupiter
at
nurse
so
across
the
top.
There's
a
list
of
of
things
that
we've
done
with
Jupiter
over
the
years
to
kind
of
improve
our
ability
to
provide
a
stable
service,
but
also
to
increase
the
capacity
be
able
to
to
serve
more
and
more
users
and
to
expose
new
capabilities
to
them
as
new
systems
arrive
all
right.
A
So
how
do
you
actually
use
Jupiter
at
nurse
if
you've
used
another
Computing
facility,
maybe
one
of
the
NSF
facilities?
You
may
be
familiar
with
open
on
demand.
That's
one
way
that
you
can
use
Jupiter
notebooks
on
HPC
systems.
Another
way
is
Jupiter
Hub
and
we
actually
have
a
jupyter
hub
deployment
at
nurse.
A
What
that
does?
Is
it
redirects
you
to
authenticate
if
needed
it's
located
at
jupiter.nurse.gov?
So
you
point
your
web
browser
there.
If
you're
not
already
authenticated,
say
you
haven't
logged
into
Iris
today
or
whatever
then
you'll
be
redirected
to
authenticate,
then
you
can
spawn
a
notebook
server
somewhere
at
nurse
either
Quarry
or
Pro
Mudder
The
Hub
manages
communication
between
you
and
your
notebook,
and
it
also
manages
the
life
cycle
of
the
notebook.
A
So
once
you
start
the
notebook
server,
it
makes
sure
that
it's
running
every
five
or
ten
minutes
or
so
and
then,
if
you
leave
it
idle
for
several
hours,
then
it
will
shut
it
down
for
you,
it
keeps,
and
it
also
includes
some
helpful
additional
Services
if
you've
been
to
Jupiter
recently
jupiter.nurse.gov
recently,
you
may
have
noticed
that
there's
a
banner
across
the
top
mentioning
that
Pro
letter
is
having
problems
right
now.
I
think
we
mentioned
earlier
that
there's
some
work
going
on
to
address
some
some
performance
issues.
A
Super
computers
are
complicated,
but
anyway,
if
there's
any
kind
of
degradation
or
performance
issues
that
you
may
need
to
know
about
that,
that
may
be
posted
there
on
the
Hub.
Okay.
So
again
at
the
bottom
summaries
you
authenticate
to
the
hub,
then
you
choose
where
you
want
to
run
a
notebook
and
then
you're
redirected
to
your
notebook
once
it
starts
up.
A
A
As
I
mentioned,
you
can
start
a
notebook
on
either
promoter
or
query.
Most
people
start
a
notebook
on
a
login
node
on
one
of
those
two
systems.
An
important
thing
to
note
is
that
on
prometer,
if
you
click
this
button,
the
top
left
button.
That's
the
promoter
shared
CPU
node.
What
that
is,
is
that's
actually
a
notebook
server
running
on
a
login
node
and
since
promoter
has
about
40
login
nodes,
your
notebook
server
is
going
to
be
started
up
on
one
of
those
you'll
be
running.
A
Alongside
other
users
who
are
either
SSH
genre
running
jupyter
notebooks
themselves
from
there,
you
can
see
the
community
file
system,
your
home
directory.
You
can
see
the
promoter
scratch
directory.
It's
got
the
same
python
environment
as
you
would
get
if
you
did
modulo
load
python
from
SSH
and
you
can
also
submit
batch
jobs,
even
if
you
want
to
using
the
terminal
there
or
using
Magic
command
like
s
batch
on
Quarry.
A
All
of
the
notebook
servers
that
get
started
up
on
the
login
nodes
are
actually
just
restricted
to
four
login
nodes.
Okay,
so
is
probably
a
better
idea
these
days
for
you
to
start
up
a
Jupiter
notebook
server
on
a
promoter,
login
node,
and
not
on
a
query.
Login
node,
before
the
talk
here,
I
just
went
and
counted
them
up.
There's
about
300
notebook
servers
running
right
now
on
Corey
that
are
crammed
into
those
four
nodes
and
there's
about
a
hundred
running
on
Quarry
login
nodes
spread
out
all
all
over
those
40
login
notes.
A
So
it's
it's
probably
better
for
you
to
to
get
on
to
to
prometer
with
Jupiter
on
Corey,
like
I
mentioned
it's
you're
kind
of
crammed
into
those
four
nodes.
You
can
see
the
same
CFS
and
home
directories.
You
can
see
Quarry
scratch,
you
cannot
see
promoter
scratch
from
Corey
and
vice
versa.
A
Same
python,
environment,
as
as
SSH
login
on
on
Quarry
as
module
in
Python.
There
are
some
other
options
as
well.
If
you
want
to
be
able
to
run,
say
a
deep
learning,
a
deep
learning,
workflow
using
a
Jupiter
notebook
with
four
gpus,
or
something
like
that,
you
can
run
a
jupyter
notebook
in
a
batch
job.
A
Basically,
you
push
one
of
the
other
buttons.
If
you
want
to
be
able
to
run
a
notebook
on
a
compute
node
that
doesn't
have
any
gpus,
it's
just
a
regular
CPU,
you
can
press
the
exclusive
CPU
node
button,
the
configurable
jobs
on
the
far
right
of
the
interface.
Allow
you
to
configure
the
job
parameters
in
case
you
need
say
more
than
one
node
or
you
want
to
change
what
the
layout
is
for
for
scheduling
the
steps
for
individual
jobs.
A
There's
also.
You
might
also
see
buttons
for
query
GPU
nodes.
You
may
or
may
not
see
those
it
depends
on
whether
or
not
you've
been
you.
You
have
access
to
the
the
qos
for
that,
but
these
days
everybody
should
probably
be
getting
onto
prometer
and
using
the
GPU
nodes.
There.
Final
thing
to
remember
is
that
the
shared
notebook
server
options
mean
that
you're
running
alongside
other
users,
and
so
generally
we
do.
A
We
do
things
to
keep
users
from
colliding
with
each
other,
but
sometimes
users
can
do
a
lot
of
I
o
and
that
can
affect
that
can
affect
other
users
on
the
same
node.
So
we
we
do
ask
that
people
observe
the
guidelines
for
good
behavior
on
login
nodes.
A
A
Hpc
file
system
is
a
big
gigantic
hierarchical
place
where
it
can
be
easy
to
get
lost,
especially
if
you're
kind
of
clicking
around
in
in
the
Jupiter
lab
interface.
So
we
actually
created
this
favorites
plug-in
that
allows
you
to
bookmark
favorite
places
on
the
file
system
and
it's
pre-populated
with
your
home
directory
scratch
directories.
And
you
can,
you
can
add
the
current
directory
that
you're
into
the
list
of
favorites
by
clicking
on
the
star
icon
there.
A
Another
goodie
that
we've
got
is
a
list
of
recent
locations,
places
that
you've
been
recently
on
the
file
system.
That
can
be
handy
for
again
navigating
the
file
system,
which
can
be
kind
of
challenging
through
a
UI
like
this
and
also
we've,
we've
added
a
feature.
A
I
think
we
got
this
upstreamed
into
the
Jupiter
Lab
Core,
which
is
the
ability
to
open
any
file
using
the
open
from
path
element
of
the
file
menu,
all
right,
so
once
you're
running
jupyter
lab,
how
do
you
actually
compute
in
order
to
compute?
A
What
you
need
to
use
is
a
process
called
a
kernel
which
is
already
an
overloaded
term,
of
course,
but
basically
it's
just
a
python
interpreter
or
some
other
process
that
the
notebook
server
starts
up
on
your
behalf
that
manages
compute
for
you
and
keep
State,
and
things
like
that.
So
this
diagram
depicts
you
on
the
left,
with
a
smiley
face
operating
your
web
browser
on
your
laptop
and
that's
connected
to
a
notebook
server,
that's
running,
say
on
promoter
and
say
a
promoter,
login
node.
A
So
that's
the
the
green
box
that
says
notebook
server
and
attached
to
that
is
a
kernel
process
generally
that'll
probably
be
a
python
interpreter.
That's
that's
running
and
it's
communicating
The
Notebook
server
is
communicating
with
the
kernel.
The
Notebook
server
is
responsible
for
taking
care
of
your
actual
notebook
itself.
Okay,
so
the
question
is:
if
you
want
that
kernel
process
to
be
doing
something
kind
of
custom,
how
do
you?
How
do
you?
How
do
you
get
your
own
kernel
set
up?
A
Normally,
you
could
run
R
kernel,
which
is
just
a
python
interpreter
that
you
get
from
module
and
python.
But
what
if
you
have
your
own
set
of
packages
that
you
want
to
use
that
turns
out
to
be
kind
of
the
most
common
question
that
we
get,
which
is
how
do
I
take
a
Content
environment
that
Daniel
just
talked
about
in
the
previous
talk
and
use
that
from
Jupiter
there's
a
few
different
ways
to
do
this?
A
Here's
the
easy
one
which
is
to
activate
the
environment,
make
sure
that
you've
added
the
IPI
kernel
package
to
that
and
then
once
you
have
done
that
run
the
command
shown
at
the
at
the
bottom
of
the
listing
here
on
this
slide,
python-mi
Pi,
kernel,
install
and
so
on.
What
that
does
is
that
creates
a
kernel,
spec
file
and
then,
when
you
bring
up
Jupiter,
you
may
need
to
either
restart
the
notebook
server
or
just
reload
the
page.
A
Actually,
you
should
see
your
kernel
listed
there
on
the
on
the
Kernel
launcher
in
Jupiter
lab.
What's
actually
in
the
kernel
spec
file,
when
you
create
the
kernel
spec,
it
actually
nicely,
it
tells
you
exactly
where
I
put
it.
So
you
can
just
go.
Look
at
it.
It's
a
Json
file
that,
basically
just
is:
how
do
I
run
this
command,
so
the
The
Notebook
server
is
going
to
run
the
stuff.
That's
in
the
arc,
V
array
that's
listed
there
and
then
there's
a
little
bit
of
metadata
after
that.
A
You
really
don't
need
to
mess
with
this
generally
too
much
unless
you
want
to
do
further
custom
organization,
besides
just
making
sure
that
you
have
the
packet,
your
favorite
packages
available
in
your
in
your
kernel,
things
like
environment
variables
can
be
added
to
to
the
Json
here
and
you
would
do
that
by
hand,
but
that
can
be
kind
of
inconvenient
because
you
can't
use
semantics
like
you
can't
extend
a
path,
the
path
variable.
You
can't
concatenate
the
path
like
you
do
in
a
shell
like
in
a
shell
script.
A
A
Is
you
just
delegate
everything
that
you
want
to
happen
when
it's
time
to
run
your
kernel
to
the
to
the
shell
script
like
set
up
some
environment
variables,
load
some
module
and
then
start
the
python
process,
and
you
can
kind
of
mix
and
match
this
with
all
kinds
of
different
modules
and
variables.
A
You
can
also
create
a
kernel
based
off
of
a
shifter
container.
Lori
is
going
to
talk
about
shifter
next,
but
here,
basically
again,
you
just
tell
Jupiter
how
to
run
the
process
that
you
want
to
run
inside
the
kernel
spec
file.
This
is
also
documented
on
the
nurse
website
as
well.
A
I
have
a
couple
final
pointers
about
debugging
things
if
you're
having
trouble
setting
up
a
kernel,
spec
or
something
like
that,
how
do
you
figure
out?
What's
going
wrong
with
it?
The
interface
will
probably
just
show
you
that
your
that
your
kernel
doesn't
start
or
something
like
that.
Everybody
gets
a
log
file
that
captures
the
output
from
the
Jupiter
notebook
server
when
you
run
it
on
either
machine
and
the
name
of
this
log
file
is
it's
in
your
home
directory.
A
It's
dot,
Jupiter,
Dash
machine
name,
DOT
log,
so
on
core8.jupyter
dash,
query.log
and
on
promoter.
It's
Dodge,
Jupiter,
Dash
promoter.log
and
you
can
open
this
file
up
and
you
can
usually
see
error
messages
or
Warnings
or
information
that
might
tell
you
what's
going
wrong
with
your
setup,
and
you
know
if
you
have
trouble
and
you
you
know,
you
can't
figure
out
exactly.
What's
going
on
from
your
from
your
log
file,
you
can
you
can
let
us
know.
A
You
took
a
look
at
your
log
file
and
didn't
you
know,
didn't
see
information,
they
told
you
how
you
could
debug
it
yourself
and
we
can
take
a
look
at
it
for
you
and
try
to
help.
Get
you
back
on
your
way,
but
I'd
say
before
you
before
you
open
up
the
interface
to
start
filing
a
ticket.
You
might
want
to
just
take
a
look
at
this
log
file
and
see
if
there's
anything
there,
that
might
be
a
clue
that
helps
you
debug
your
Jupiter
environment,
okay,
so
this
is.
A
A
You
can
customize
your
Jupiter
experience
at
nurse
using
a
kernel
spec
with
a
conda
environment,
so
you
can
have
whatever
package
it
is
packages.
It
is
that
you
want
to
use
from
a
Jupiter
notebook.
You
can
customize
those
kernel,
spec
files
in
many
different
ways
you
can
use,
you
can
add
environment
variables.
You
can
add
modules.
A
You
can
even
use
a
shifter
container
if
you
like,
and
all
of
that's
documented
on
the
Jupiter
at
nurse
documentation,
page
and
I
just
wanted
to
also
mention
that
we're
always
looking
for
ways
to
help
our
users
use
Jupiter
for
in
for
in
new
ways.
A
So
please,
if
you
have
feedback
or
advice,
or
you
figured
out
how
to
solve
a
problem,
and
you
want
to
share
it
with
us.
Please
go
ahead
and,
and
just
let
us
know
by
ticket
or
you
can
you
can
even
email
me?
Okay,
so
I
think
that's
I
think
that's
everything
that
I've
got.