►
From YouTube: 13 Jupyter at NERSC
Description
Part of the NERSC New User Training on June 16, 2020.
Please see https://www.nersc.gov/users/training/events/new-user-training-june-16-2020/ for the training day agenda and presentation slides.
A
This
talk
is
about
how
to
use
jupiter
at
nursk,
I'm
rollin,
I'm
in
the
data
and
analytics
services
group,
and
I
take
care
of
jupiter
at
nurse
okay.
I
did
that
okay,
what
is
jupiter
jupiter,
of
course,
is
a
interactive
open
source
web
application
that
lets
you
create
and
share
these
documents
that
are
called
notebooks.
A
It's
been
around
a
while.
I
think
a
lot
of
people
use
it
for
all
kinds
of
stuff,
because
you
can
put
all
kinds
of
neat
things
in
notebooks
live
code,
equations,
visualizations,
narrative
text.
Even
these
cool,
interactive
widget
thingies
people
use
jupiter,
notebooks
for
all
kinds
of
things:
cleaning
up
their
data,
transforming
data
actually
running
simulations
modeling,
making
statistical
models.
A
A
Why
do
we
care
about
nurse?
Why
do
we
care
about
jupiter
and
nerves?
There
are
a
lot
of
users,
obviously
that
use
jupiter
nurse,
but
there
are
also
a
lot
of
experiments
that
have
made
jupiter
part
of
their
data
analysis
framework.
So
some
of
them
are
listed
here.
Some
of
you
may
even
be
members
of
some
of
those
collaborations
and
experiments.
A
I
don't
doubt
that
there
are
a
lot
of
people
who
are
new
to
nurse
but
are
really
not
new
to
jupiter
at
all.
Some
of
the
people
on
this
call
may
actually
have
taken
data
eight
at
uc
berkeley
and
maybe
in
this
picture
of
all
of
the
students
in
seller
rock
hall.
A
That
class
is
completely
taught
on
jupiter
notebook,
so
people
come
out
of
that
class
expecting
to
be
able
to
do
data
science
anywhere
using
a
jupiter
notebook,
and
so
those
users
come
to
nurse,
and
so
not
we
have
to
be
able
to
help
them
and
also
jupiter
notebooks
have
become
this
new
tool
for
aiding
and
reproducibility
and
science
outreach.
So
there's
an
example
here,
on
the
left
hand,
side
of
the
screen
of
jupiter
notebook
that
you
can
download
and
you
can
reproduce
this
binary
black
hole.
Merger
plot
we've
been
running.
A
Supporting
jupiter
at
nurse
for
about
five
six
years.
This
is
the
timeline.
I'm
not
going
to
go
into
a
lot
of
detail
here,
but
jupiter
has
been
running
on
quarry.
Basically,
from
the
first
year
that
corey
had
been
deployed,
the
resources
that
are
set
aside
on
corey
for
running
jupiter
right
now
there
are
three
nodes
that
can
run
jupiter
notebooks
that
are
not
batch
batch,
scheduled
badge
schedule.
Jupiter
is
possible,
running
jupiter
on
the
quarry
gpu
nodes.
A
If
you
have
access
to
that,
that's
possible
too,
we'll
be
adding
another
node,
so
there
will
be
four
nodes
after
the
next
maintenance
for
running
shared
node
for
running
notebooks.
I
mean
this
is
a
plot
of
how
many
jupiter
users
there
are
per
month.
So
the
first
thing
you'll
notice,
is
that
there's
quite
a
lot
of
them
and
the
new
record
for
the
number
of
users
per
month
is
beating
last
month
and
this
month
isn't
even
over
yet
so
really
it's
a
very
popular
service.
A
You
could
compare
this
to
the
number
of
users
who
log
in
across
all
of
the
query:
login
notes
all
12
of
them
in
a
month.
That's
about
2900
users,
so
there
are
a
lot
of
jupiter
users
if
you're
a
jupiter
user
you're
a
very
good
company
at
nurse
okay.
How
do
you
do
it
at
nursk?
Jupiter
is
provided
through
a
jupiter
hub
deployment.
The
hub
is
a
thing
which
authenticates
you
and
starts
up
your
notebook
for
you,
okay.
A
This
is
a
lot
better
than
starting
up
a
jupiter
notebook
in
an
ssh
tunnel
and
then
trying
to
connect
to
it.
We
saw
users
doing
that.
We
thought
that
was
really
painful,
so
we
set
up
a
hub
for
our
users.
Okay,
the
way
you
do
it
there's
three
steps.
It's
very
easy!
You
authenticate
at
jupiter.nurse.gov.
A
You
need
to
use
your
one-time
password,
it's
a
multi-factor
authentication
supporting
service.
Then
you
choose
where
you
want
your
notebook
to
start
up
and
there
are
usually
two
or
three
choices.
Most
users
are
just
going
to
see
the
two
buttons
here,
the
query
button
and
the
spin
button.
These
are
labeled
shared
cpu
nodes.
That
means
that
the
node
is
shared
between
you
and
maybe
50
other
users.
Okay.
A
So
it's
a
lot
like
a
regular
login
node
that
you
heard
about
this
morning
that
you
ssh
into
and
you
can
do
stuff
like
compiler
look
at
your
data
or
whatever
you
have
to
be
mindful
that
there
are
other
people
on
there.
So
you
can't
do
something
like
take
all
of
the
memory
and
not
let
anybody
else
do
anything
that
isn't
nice
and
we
don't
allow
that
so
be
mindful
of
the
fact
that
you
might
be
sharing
with
with
quite
a
few
other
people.
A
There
are
other
buttons
that
show
up
here,
depending
on
what
kind
of
permissions
you've
got.
If
you
have
access
to
the
query
gpu
cluster,
then
you
would
see
a
button
here
that
would
allow
you
to
start
a
notebook
on
the
gpu
cluster,
so
first
step
authenticate,
then
you
choose
and
then
you
just
go.
You
click
one
of
these
buttons
and
your
notebook
should
hopefully
spin
up.
A
Okay,
how
do
you
make
that
decision?
So,
as
I
mentioned
before,
the
shared,
the
shared
notes
are,
on
the
left
hand,
side
here.
There
are
these
three
nodes
set
up
for
corey.
What
can
you
do
with
that?
You
can
see
all
of
the
file
systems,
so
you
can
see
community
file
system,
your
home
directory.
You
can
see
corey
scratch,
so
you
can
interact
with
any
data.
That's
in
any
of
those
file
systems.
A
It's
the
same
exact
python
environment
as
if
you
logged
in
via
ssh,
which
is
a
big
plus.
So
there's
no
difference
between
how
you
interact
with
python
through
jupiter
and
how
you
interact
with
python
over
ssh
and
you
can
submit
jobs
and
you
can
interact
with
the
batch
cues.
We've
created
this
kind
of
slur
magic
that
some
people
like,
but
other
people
don't
like
it
and
they
just
use
shell
magics.
A
It's
fine,
whatever
the
spin
shared
cpu
note
is
really
kind
of
a
backup
option
if
corey
is
down
for
some
reason
like
we're
having
a
maintenance,
but
the
spin
node
is
up
it's
external
to
cory.
It
cannot
see
corey
scratch
and
you
cannot
run
jobs
from
it.
But
you
can
see
the
community
file
system,
you
can
see
your
home
directory
and
all
of
that,
so
you
have
data
there
and
you
need
to
make
a
flat
because
your
paper
deadline
is
tomorrow.
A
You
can
use
the
spin
shared
cpu
node
option
and,
as
I
mentioned,
the
gpu
node,
if
you
click
that
button,
if
you
can
see
it,
not
everybody
can
see
it.
It
launches
a
notebook
for
you,
it
runs
in
batch,
but
it
doesn't
get
charged,
but
you
can
only
run
for
up
to
four
hours.
It'll
turn
off.
A
Okay,
all
right
on
the
hub.
We
have
a
couple
of
additional
services
that
you
might
want
to
know
about
that
are
going
to
be
ruled
out
here
within
the
next
month.
The
hub
allows
us
to
run
these
kind
of
user-facing
services
or
management
services,
and
we
use
them
to
provide
a
another
route
for
providing
users
jupiter-specific
information
about
the
status
of
the
service.
So
if
there's
going
to
be
an
upcoming
maintenance
for
for
jupiter,
you
might
see
a
banner
on
all
the
hub
pages
saying
when
it's
going
to
be
so.
A
A
So
this
is
a
way
where
you
could
say
pick
up
a
canned
notebook
that
somebody
else
has
already
written
and
copy
it
over
along
with
its
kernel
and
then
start
it
up
and
and
customize
it
to
do
whatever
you
need
to
do
with
it.
So
if
you
log
into
jupiter.nurse.gov
today,
you
won't
see
this
little
drop
down
here
with
these
two
service
links.
A
But
after
we
do
the
upgrade
which
is
going
to
be
coming
in
the
next
month,
you'll
see
those
services
there
once
you're
in
the
default
way
of
running
a
jupiter.
Notebook
at
nurse
is
through
jupiter
lab
and
it
has
been
basically
since
it
was
in
beta.
I
think
it's
got
a
file
browser.
It's
got
your
notebooks
over
here
and
tabs.
You
can
start
terminals
and
all
other
kinds
of
widgets
and
cool
stuff.
A
One
of
them
is
called
jupiter
lab
favorites,
one
of
the
things
that
happened
when
we
first
deployed
jupiter
lab
was
we
observed
that
it
was
really
easy
for
users
to
get
lost
on
the
file
system
if
they're
just
clicking
over
here
on
the
on
the
file
browser,
it's
really
easy
to
get
lost
and
not
be
able
to
find
your
way
back
to
your
home
directory,
plus
it's
kind
of
tedious
to
click
around
a
massive
parallel
file
system.
Trying
to
find
you
know
your
scratch
directory
or
whatever.
A
So
we
decided
to
make
that
easier
by
making
kind
of
bookmarks,
and
so
this
is
an
extension
to
jupyter
lab.
It's
not
part
of
jupiter
lab.
If
you
install
your
own
jupiter
lab
on
your
laptop
you're,
not
gonna
see
it,
but
we
we
pre-seed
it
with
these
two
directories,
because
most
users
need
access
to
those
so
play
with
that.
It's
pretty
neat
it'll
save
a
lot
of
time
and
clicking
a
couple.
Other
things
is
this
open
from
path
thing?
A
We
we
made
it
so
that
you
could
hop
to
any
place
on
the
file
system.
That's
handy
in
tandem
with
the
favorites
tool
and
also
recents
a
list
of
recently
accessed
files
and
pads
is
located
in
the
file
file
menu
as
well
all
right.
So
if
you
use
jupyter
you're,
you
might
be
aware
that
the
way
that
you
actually
run
code
is
through
a
thing
called
a
kernel.
So
here's
you
you're,
happy,
here's
your
browser
and
then
over
here
is
the
notebook
server.
A
A
So
how
you
configure
this
kernel,
basically
tells
you
what
you
can
do
in
a
jupiter
notebook,
it's
basically
just
a
python
process
or
a
julia
process
from
our
process
or
something
that
you
send
commands
to,
and
it
sends
the
output
back
all
right.
The
number
one
question
we
get
about
jupiter
at
nurse
for
the
most
part,
is:
how
do
I
take
a
condo
environment?
So
what
one
of
the
things
that
lori
just
talked
about?
The
previous
talk
was
kind
of
environments.
A
How
do
I
take
my
condo
environment
that
I've
made
and
just
use
that
from
jupiter?
So
there's
a
few
different
ways
to
accomplish
that?
The
one
that
I
like
the
best
is
shown
here.
So
you
create
your
content
environment,
you
put
it
whatever
it
is.
You
need
it
and
then
you
add
this
additional
package
called
ipi
kernel.
A
Okay,
once
that's
all
installed,
then
you
run
this
command
here
and
that
will
create
a
kernel
spec,
which
is
a
special
json
file
that
gets
dropped
in
a
kind
of
secret
place
in
your
home
directory,
and
your
content
environment
will
show
up
basically
in
the
list
of
kernels
that
are
available
on
jupiter.nurse.gov
you'll
probably
need
to
restart
your
notebook
server.
If
you
do
this,
while
your
notebook
server
is
running,
this
is.
B
A
A
A
A
template
for
how
you
do
that
now.
Why
do
you
need
to
know
this?
Because
you
probably
want
to
customize
it
further?
If
you
have
other
executables
that
you
want
to
be
able
to
run
while
you're
running
your
kernel,
you
need
to
put
those
in
a
path
or,
if
there's
some
dynamically,
linked
application
that
you
need
to
include.
You
need
to
extend
the
ld
library
path.
You
can
do
that
through
this
end,
part
of
the
kernel
spec,
but
in
in
actuality.
A
I
don't
really
like
this
way
of
customizing
a
jupiter
kernel
at
nursk,
because
this
kind
of
obliterates,
whatever
path
you
had
before,
and
it
completely
overwrites
the
ld
library
path
and
that's
kind
of
inconvenient.
A
A
better
option
is
for
you
to
just
say:
hey,
I'm
going
to
instead
of
running
python,
which
is
what
this
first
rv
argument
is,
I'm
going
to
run
some
other
bash
script
and
you
can
put
whatever
you
want
in
there,
but
the
very
last
thing
needs
to
be
exact:
python
dash
and
ipi
kernel.
So
that's!
Basically,
whatever
you
were
running
before
right,
so
python,
dash
m,
ipi
kernel,
f
connection
file,
whatever
it's.
Basically,
the
same
stuff
gets
passed
in
here
and.
A
Can
load
whatever
other
module?
You
want
to
load
a
lot
of
people.
This
is
a
common
one,
because
people
like
to
make
latex
labels
on
matplotlib
plots,
I
don't
know,
go
figure
or
you
could
set
some
environment
variable
or
whatever.
I
think
this
is
really
the
most
flexible
approach
because
of
the
module
thing
and
because
of
environment
variables,
but
really.
A
So
if
you
have
a
docker
container,
where
you
have
all
your
stuff
already
set
up-
and
you
don't
want
to
mess
around
with
a
con
environment
for
whatever
reason
at
nurse
then
cool,
this
is
really
handy
if
you've
got
like
a
collaboration
and
everybody
agrees,
they're
going
to
use
this
docker
container
for
all
their
python
stuff
say,
but
it's
really
basically
just
whatever
command
it
is.
You
would
run
on
the
command
line.
You
stick
it
in
there
right.
A
What
do
you
do
when
things
go
bad
with
jupiter?
Sometimes
problems
happen.
I
would
say
the
number
one
problem
we
get
with
with
jupiter
users.
Is
they
fill
their
home
directory
up
and
then
they
can't
run
jupiter,
and
what
will
happen
is
you'll
see
that
your
notebook
fails
to
start
up
and
you'll
get
like
a
little
message,
507,
which
in
http
means
you're
out
of
space,
and
you
just
need
to
go
back
and
move
some
data
out
of
your
home
directory,
and
then
you
can
run
jupyter
again.
A
We
might
make
a
couple
of
changes
in
the
future
that
will
allow
your
jupyter
notebook
to
keep
running.
But
if
it
were
me,
I
would
want
to
actually
know
that
my
home
directory
was
full,
but
that's
probably
the
thing
that
people
are
most
likely
to
run
into
next,
one
will
be
their
environment
is
kind
of
messed
up
like
lori
was
talking
about
their
rc.
A
Maybe
has
lots
of
python
path
stuff
that
kind
of
confused
jupiter
lab
about
what
what
function
calls
it's
making
basically
in
its
search
path,
it's
library
search
path,
that's
a
kind
of
other
one
that
happened,
so
those
are
kind
and
disentangling
those
can
be
kind
of
messy,
but
the
way
to
figure
out
what's
going
on,
especially
with
the
second
one.
Is
that
you
look
at
this
awesome
log
file
called
dot
jupiter.log
everybody
that
runs
jupiter
nurse
gets
one
of
these
in
their
home
directory,
and
you
know
nobody
else
can
look
at
it.
A
Unless
you
like
change
the
permissions,
we
force
the
permissions
to
be
kind
of
so
that
only
you
can
read
it
really.
But
basically
this
tells
you
what's
going
on
with
your
jupiter
notebook
while
it's
running
so
there's
all
kinds
of
stuff
that
comes
out
and
if
there's
a
problem,
usually
you'll
see
some
kind
of
traceback.
That
kind
of
tells
you
where
the
answer
is
like.
If
you
mess
up
your
kernel
spec
and
you
mess.
A
You'll
see
a
json
parsing
error,
or
something
like
that
right.
So
before
you
fire
off
a
ticket
to
me
saying,
hey
jupiter
is
broken
fix
it.
You
should
go,
take
a
look
at
the.jupiter.log
and
see.
If
there's
anything
you
can
do
about
it.
It's
really
your
friend
all
right.
What
are
we
going
to
do
in
the
near
future
with
jupiter
at
nurse?
A
I
mentioned
that
we're
going
to
add
another
node,
so
we'll
be
able
to
support
more
users
and
we'll
be
able
to
do
so
in
a
more
stable
fashion
than
we
have
recently
we're
working
on
what
the
strategy
for
jupiter
with
pro
mutter
is
going
to
look
like
there
will
be
jupiter
running
on
pro
mutter
in
some
form
or
fashion,
so
that'll
be
cool
because
there's
all
those
gpus
there
and
there's
all
kinds
of
deep
learning
frameworks
that
come
in,
that
you
can
access
from
jupiter
notebooks,
so
that
ought
to
be
a
lot
of
fun.
A
One
of
the
things
that
I'm
focusing
on,
especially
in
the
next
year,
is
distributed
analytics
frameworks
like
desk
or
ipi,
parallel
or
array
or
whatever,
and
trying
to
streamline
those
on
our
system.
They
work
great
in
the
cloud,
but
you've
got
to
do
a
little
bit
of
adaptation
to
make
them
actually
work
as
nicely
on
hpc
systems.
So
I'm
going
to
try
to
make
that
easier,
enabling
users
to
control
what
jupiter
lab
notebook
is
running
is
another
thing
I'm
working
on.
A
That
means
that
you'll
be
able
to
install
your
own
custom
extensions
for
some
reason.
You
don't
like
jupiter
lab
reasons
you
could
make
it
go
away,
say
and
then
providing
more
access
to
running
kind
of
bigger
jupiter
jobs.
Things
like
jupiter
on
compute
nodes,
extending
jupiter
lab
to
say,
have
a
nice
interface
to
slurm,
or
something
like
that.
Those
are
other
research
projects
we're
doing
okay.
So
that's
the
jupiter
talk.
You
know
if
you
want
to
use
jupiter,
go
to
jupiter.nurse.gov
and
figure
it
out.
Kernel
spec
refer
to
these
slides.
A
If
you
get
lost
on
how
to
actually
convert
a
con
environment
into
a
kernel
and
use
it
from
jupiter,
there's
also
documentation
basically
on
the
website,
it's
the
same
thing
and
customize
away,
and
if
you
get,
if
you
get
in
trouble,
just
file
a
ticket
email
me,
let
me
know
how
it's
going
and
I'll
help
you
out.
B
Hey
thanks
roland.
We
have
one
question:
we
have
someone
who'd
like
to
use
julia
and
jupiter.
Can
you
recommend
the
best
way
to
do
that.
A
Yeah,
you
can
do
that,
so
what
I've
done
in
the
past,
with
users
who've
tried
to
use
ijulia
is
just
let
them
install
their
own.
A
Julia
stack,
which
is
actually
pretty
straightforward,
and
a
lot
of
julia
users
are
used
to
doing
that
anyway,
there's
a
pretty
decent
installer,
it
gets
you
a
pretty
performant
julia
julia
already
out
of
the
box,
and
then
you
just
follow
the
instructions
for
installing
the
ijulia
kernel,
and
I
think
I
think
that
that
just
kind
of
works,
if
there
are
problems,
send
me
a
ticket,
because
I
have
gotten
enough
at
least
a
couple
other
users
using
I
julia
at
nurse,
and
so
it
can
be
done.
People
have
done
it.
B
Okay,
all
right
well,
thank
you
very.