►
Description
June 13, 2019 Jupyter Community Workshop talk by Thomas Stitt, Lawrence Berkeley National Laboratory
A
Hi
everyone
I'm
Tom,
stit
I
work
at
lol.
I
am
a
software
developer
on
a
project
card,
marble
which
is
one
of
our
production
physics
simulation
applications.
So
I'm
going
to
talk
today
about
a
little
bit
different
than
what
we've
maybe
seen
before,
but
we
are
trying
to
extend
our
command-line
interface
to
use
Jupiter
notebooks
as
an
interface
option.
A
So
first,
some
acknowledgments
big
thanks
to
Thomas
Mendoza,
who
I
don't
know
if
you
see
right
now,
but
he
presented
yesterday
without
him,
we
wouldn't
really
be
able
to
do.
Jupiter
are
on
our
computing
network
on
the
pink
of
marble
team
because
without
demo
and
not
a
code
to
play
with
and
all
of
you
guys
most
everyone
else
who
isn't
here
from
the
Dupree
community
they're
making
such
a
great
product.
A
So
marble
is
a
higher
energy
density
physics,
application
simulation
application,
it's
built
off
to
other
applications,
I'm
not
really
going
to
go
into
detail,
but
it
uses
super
little
Fortran,
it's
all
MPI
based
and
it
has
the
shared
layer
of
C++
and
Lua
at
the
top.
So
our
users
write
code
in
Lua
to
define
their
input
problems
and
then
also
if
they
want
to
use
a
shell
with
our
application,
they
have
a
little
shell.
A
So
a
lot
of
you
probably
already
know
how
applications
are
run
with
these
HPC
facilities,
but
I'll
get
a
little
bit
over
you
again.
So
one
of
the
mode
is
patched.
Batched
is
basically
you
write
this
script
that
defines
your
resource
needs,
so
maybe
how
many
nodes
you
need
how
long
you
want
to
run
if
you
need
GPUs
or
not,
and
you
submit
it
to
this
batch
system.
The
batch
system,
whenever
it
has
resources
available,
put
your
code
onto
the
resources
its
allocated
for
you.
It
runs
for
a
while.
A
A
Another
option:
if
it's
available
is
an
interactive
interactive,
is
more
of
like
you're
dropped
into
a
shell.
You
can
say,
run
400
time,
steps
maybe
for
some
values,
maybe
crates
and
visualizations.
If
the
code
supports
it,
so
our
production
applications
widely
vary.
Some
are
only
badge:
some
have
interactive
custom
language.
Some
are
interactive,
Python
or
marble.
Like
I
said
this
is
Lua.
We
have
type
of
history
and
we
use
an
application
called
geo
vez
to
allow
in
line
visualization.
So
you
can
visualize
the
density
field
interact
with
it.
A
But
to
use
this,
you
have
to
start
a
visualization
server,
configure
marble
to
look
where
the
Vigilant
earthen
server
is
and
then
in
the
interactive
shell,
say:
ok
or
to
visualize
the
energy
field
and
and
x
windows
will
pop
up,
and
you
think
they're
app
with
that,
but
it's
a
little
clunky,
sometimes
and
very
slow.
If
you
are
BPM.
A
So
we
were
thinking
when
I
started.
Some
other
people
have
been
thinking
about.
What
do
we
want
after
the
CLI,
how
we
give
users
a
richer
experience?
There
are
some
initial
mock-ups
which
is
using
some
open
source
web
terminal
just
to
connect
from
a
browser
to
the
app
but
I
thought
Jupiter
might
be
a
good
fit,
especially
the
true
Patil,
a
very
interested
in
having
that
be
the
entire
space
for
the
user
to
do
that
workflow.
So
they
want
to
make
their
mesh
and
maybe
that's
a
commitment
only
tool.
A
They
can
do
it
up
the
terminal.
They
can
write
their
input
deck
in
the
file
editor
and
then
they
can
run
marble
in
a
notebook,
connect
and
visualize
and
do
data
analytics.
So
these
are
the
two
modes
we
see
so
in
batch.
We
want
to
go
to
do
something
called
it,
what
we
call
attachable
Diagnostics.
So
in
the
case
that
your
simulation
is
running
for
two
weeks,
you
might
want
to
connect
to
it.
Maybe
like
seven
days
then
pause
the
simulation
and
then
maybe
query
some
values.
A
Maybe
do
some
visualization,
you
could
modify
a
boundary
condition,
add
a
source
method
on
it
to
a
different
mesh
and
then
regenerate
the
primal
and
keep
running.
This
feature
doesn't
exist
for
applications
we
have
now.
You
could
take.
One
of
your
checkpoints
kill
your
job
restart
from
that
checkpoint,
while
changing
some
parameters,
but
that
isn't
as
clean,
especially
if
it's
difficult
for
you
to
get
an
allocation,
you
want
to
you,
keep
using
the
one
you
have
interactively.
A
A
Is
just
some
of
a
gift,
so
I
don't
do
live
demos
with
VPN
cuz.
They
probably
just
fail
miserably.
This
is
just
using
MATLAB
hooking
up
to
Marble,
so
we
can
do
Wendy.
Curves
people
are
interested
in
this,
so
they
want
to
see
the
evolution
their
time
step.
This
is
just
showing
grabbing
some
numpy
data
from
our
application
applauding
it
sort
of
basic.
This
is
another
example.
This
is
using
geo
viz,
but
a
widget
eyes,
javascript
version
of
it
to
do
in
my
analysis.
A
A
I'll
talk
on
this
more
if
there's
time
that
app
is
actually
transpiled
or
compiled
within
scripting
I,
mostly
plus
plus
OpenGL,
a
WebGL
webassembly
thing.
We
had
a
summer
student
work
on
that
it
works
pretty.
Well,
it's
pretty
amazing
what
you
can
do
so
brief
thomas
went
over
this
yesterday.
What
does
Jupiter
look
like
at
Ln
L?
A
We
have
Jupiter
hub
it
launches
notebooks
servers
on
login
nodes
security
is
important
as
he
went
over,
so
we
require
everything
to
be
UNIX
domain
sockets
so
that
other
people
on
the
login
node
can't
use
drop
cuz,
even
if
you're
on
the
same
logon
notes.
Maybe
you
are
on
the
same
units
group
seen
on
the
same
permission
to
view
data,
so
that's
important
to
us
so.
A
When
I
first
started
looking
at,
how
do
we
bridge
the
notebook
code
gap?
These
are
some
challenges
or
considerations.
I
came
up
with
so
for
us.
Basically,
our
jobs
are
not
local,
so
maybe
Marvel
is
massively
parallel
could
be
running
on.
It
could
be
running
single
core,
but
it
could
be
running
on
on
thousands,
hundreds
of
thousands,
millions,
of
course,
depending
on
who's
running
it.
Since,
since
our
notebook
servers
are
our
login
node
and
our
codes
in
our
compute
node,
we
need
a
way
destitute.
A
We
use
MPI
applications,
a
lot
of
or
all
of
the
existing
kernels
I've
seen
are
serial.
Maybe
they
use
threads.
Maybe
these
GPUs,
but
multi
process
in
the
MPI
sense
could
be
hard
to
integrate
for
some
of
the
kernels
security
very
important
as
I
went
over
as
Thomas
showed.
Yesterday,
you
can
eavesdrop
you
can't
like
connect
and
execute
code.
You
can
definitely
if
anyone
else
is
sending
data
over
TCP.
You
can
listen
to
that.
A
So
we
need
some
way
to
encrypt
messaging
if
we're
going
to
use
TCP
and
make
sure
that
no
one
can
connect
to
your
kernel
or
back-end
that
isn't
allowed
integration.
So
we
have
Intel
machines,
we
have
IBM
machines
and
so
portability
is
important
for
us.
One
of
the
big
reasons
we
use
Lua
is
because
it's
to
compile
type
on
an
IBM
machines.
A
A
Before
we
even
had
Jupiter
on
LLNL
I
was
opening.
I
was
running
a
jupiter
notebook
server
on
my
laptop,
creating
and
boarding
tunnel
to
our
you
know,
log
and
our
compute
node,
and
just
catching
over
that
and
the
jupiter
protocol
was
intimidating
to
me
and
there
wasn't
a
c++
kernel.
So
I
of
course,
wrote
my
own
small
thing,
not
as
feature
pool,
but
it
put
all
of
our
considerations.
A
This
is
a
screenshot
there'll,
be
a
gif
demo
on
the
next
slide,
but
basically
what
this
is
is
it's
a
widget
that
opens
and
looks
in
a1
on
location
for
a
list
of
backends
that
are
available.
So
here
we
have
some
marble
run
it's
using
TCP
encrypted
TCP
once
it
started.
Codename
connect
displays
a
lot
of
our
post
information.
You
can
do
some
stuff.
So
what
this
lets
us
do
is
the
connectible
Diagnostics.
So
you
can
run
your
code
in
a
bath.
Partition
could
be
running
for
a
while.
You
open
this
up.
A
You
say:
connect
you
list,
you
can
connect
to
it.
It'll
compile
your
simulation.
You
are
now
in
a
notebook
connected
to
your
simulation.
That
is
paused.
You
can
do
some
stuff
and
when
you
disconnect
the
code
keeps
running
yes,
untie,
tunneling
automatic
authentication,
encryption
and
good
support
for
MPI.
This
also
only
uses
open,
SSL
and
POSIX
libraries,
so
IBM
actually
has
open
SSL
on
if
I
default.
So
it
was
easy
to
bring
over
between
the
machines.
A
A
A
A
A
So
that's
the
time
can
put
right
there
if
you
saw
some
of
it,
so
the
good
attachable
Diagnostics
is
developed
with
MPI
in
mine,
so
it's
very
easy
to
integrate,
only
used
open
source
on
POSIX,
which
we
have
everywhere
bad,
as
we
saw
it
to
multi-step
workflows.
So
if
you
want
to
share
a
notebook
with
someone
it
sucks,
if
you
want
to
get
tutorial
it
sucks,
you
have
to
like
start
the
code
open.
This
notebook
up
widget
find
your
instance
connectivity
stuff.
A
So,
along
the
way,
some
people
were
complaining
that
they
don't
know
Lua
and
that
we
should
have
a
Python
shell
too,
because
a
lot
of
our
legacy
production
codes
are
Python
and
a
lot
of
our
users
use
Python
for
other
stuff.
So
I
looked
at.
How
do
we
get
Python
without
duplicating
all
our
Lua
source
because
we
use
not
really
low
and
not
just
for
input
problems,
but
also
for
some
of
the
user
interface
development?
So
there's
this
project
called
lupa,
which
are
just
bridges
between
Python
Aloha.
So
you
expose
your
luan
straight
to
Python.
A
You
expose
Python
objects
to
Luis,
so
in
Python
you
can
say
Louis,
State,
DOT,
simulation,
dot,
time,
step
or
glass,
kinetic
energy.
You
can
get
tab
complete.
So
it
looks
like
you
have
this
Python
dictionary.
Python
object,
that's
really
a
little
object
works
quite
well,
and
because
we
now
have
both
of
by
the
Lua
and
divided
Python,
we
can
use
I've,
hi
kernels
bed
kernel
to
actually
embed
a
native
Python
kernel
without
much
work
at
all.
So
this
worked
I,
don't
know
if
I
have
a
picture
here.
A
So
with
this
in
mind,
we
created
two
new
extensions
of
the
ipython
kernel,
just
an
MPI
Python
and
in
cataluña
kernel,
the
master
rank
runs
the
normal
kernel.
There's
every
other
rank
runs
a
little
loop
that
just
waits
for
you,
execute
or
evaluation
requests
and
execute
them.
So
it's
a
little
bit
hacky.
It
works.
Okay
or
I
haven't
run
into
issues,
but
a4c
people
trying
to
use
it
in
having
a
crash.
A
Every
time
the
user
runs
the
code
using
this
mode
of
kernels,
they
get
a
URL
punch
of
the
screen
that
they
can
copy
and
paste
into
their
browser,
which
opens
a
notebook
that
uses
another
extension.
We
developed
called
env
Connect
existing,
which
is
a
extension
for
the
notebook
server
to
connect
to
existing
kernels
automatically
opens
tunnels
that
you
need
them
and
we
also
get
access
to
widgets,
which
is
important
because
of
that
visualization
application.
I
showed
you
earlier.
A
So
good
and
bad
good,
normal
workflow,
so
you
can
actually
go
into
notebook
server,
application
or
Juber
lab
like
new,
like
marble
you're
there.
Of
course
it's
only
cereal,
but
it
works
for
tutorials.
It
works
for.
If
someone
wants
to
explore
the
code,
just
like
type
complete
look
at
you
know
the
structures
that
define
their
input
or
look
at
the
various
more
dynamic
structures.
Like
our
simulation
object,
we
have
a
documentation,
a
swing
space
documentation.
We
can
include
links
in
a
documentation
of
the
notebooks
that
you
can
say.
A
Oh
I
want
to
learn
about
how
our
events
are
a
simulation
or
r1d
curve
saving
objects
blur.
You
can
click
on
a
link.
It
opens
a
notebook
in
all
the
tutorial
better
than
a
big
block
of
text
and
static
code,
as
we
saw
we
use
map
out,
live,
and
so
it's
easy
to
inject
that
followed
in
everything
is
using
the
ipython
kernel.
So
even
the
lure
kernel
you
can
have
you
can
inject
into
the
lua
namespace
a
function
called
plot
that
just
uses
map
outlet
on
the
back
end
and
lupa.
A
Does
the
variable
conversion
for
you
and,
of
course,
widgets
bad
attachable
Diagnostics
is
hard
we
haven't.
Actually
I've
actually
found
a
way
to
do
this,
so
I
said
you
can
connect
to
non-local
jobs,
but
the
job
cannot
have
started
running
and
then
pause
on
connect
we're
using
the
ipython
kernel.
Where
interrupts
our
signal
based
and
we
would
really
need
to
use
message
based
interrupts
for
non-local
jobs
unless
we
have
some
weird
SSH
tunnel
signal
command
that
runs,
so
that
is
very
limiting
factor.
A
Mpi
support
is
a
bit
happy,
like
I
said
we're
just
having
two
very
separate
blocks
of
code,
so
it
can
be
hard
to
reason
about
it
and
we've
had
people
either
when
building
our
code
or
running
our
code,
some
local
Python,
something
in
their
local
Python
path
or
something
in
their
local
library,
and
that's
things
up
a
lot
of
legacy
codes
will
have
you
on
script
that
will
set
up
Python
top
in
a
weird
way.
Also,
we
need
how
to
run
our
stuff
that
crashes
so
either
doesn't
build
it
or
doesn't
run
so.
A
Inclusions,
what
did
people
think
so
we
wanted
to
embed
notebooks
or
use
notebooks
as
a
front-end
to
get
a
rush,
much
richer
set
of
features,
and
then,
when
we
saw
you
for
your
lab
we're
thinking,
oh
this
could
be
like
a
one-stop
shop
for
a
workflow.
Like
I
said
you,
users
sometimes
need
to
run
command
line
tools
that
don't
have
or
don't
foresee,
any
support
for
a
Jupiter
front.
End
users
might
need
to
allocate
their
jobs
in
a
very
special
way.
A
So
maybe
we
could
think
about
that
to
honor
that
I'm,
not
someone
on
the
like
the
service
side,
I,
don't
have
a
lot
of
say
in
that
the
connecting
to
existing
kernels
can
be
tricky.
So
if
you
don't
have,
if,
in
the
demo,
everything
works,
but
if
you
don't
have
SSH
key
set
up
sales
and
some
users
don't
understand
what
a
association
keys
are,
how
to
slip
in
love
and
we
don't
have
a
lot
of
users.
So
a
lot
of
our
users
are
also
developers
or
there
some
sort
of
developers.
A
So
they
might
have
a
little
bit
more
experience
with
actually
coding
in
Python,
actually
coding
in
C++,
and
so
they
might
they're
more
flexible
and
willing
to
step
through
the
problems
that
they
encounter.
When
we
have
users
who
are
just
like
physicists,
we
just
want
to
use
our
code.
Maybe
they
won't
be
as
flexible.
Maybe
they
won't
want
to
even
change
their
workflow.
So
it's
to
be
seen
whether
or
not
people
will
adopt
this
future
work.
A
We
want
something
that
does
the
attachable
diagnostics
like
the
bridge
kernel
and
the
I
just
want
to
start
a
kernel
on
the
notebook
server
or
Jupiter
lab
interface,
so
maybe
juice
kernel.
It's
in
C++
I
feel
much
more
confident
about
packing
on
that
and
adding
FBI
support.
It
supports
widget
protocol,
so
we
could
just
add
a
suite
plus
plus
sir
colonel
implementation
or
widget,
and
because
we
wouldn't
be
building
with
Python.
We
wouldn't
worry
about
user
environment
screwing
up
our
builds
and
screwing
up.
Our
startup
here
are
some
references
thanks.
B
So,
thank
you
very
interesting
as
somebody
who
works
at
it.
Beauties
I
really
got
promoted
out
ways
for
you
just
to
check
in
on
their
talents
each
other
yeah,
two
quick
questions.
You
you're
not
using
Tunisia
to
dinner,
fits
correctly
identify
weird
but
but
you're
not
intercepting
the
eyeball,
yeah.
Okay,.
B
A
A
B
A
So
there's
a
project
called
a
scent
Cyrus
Harris
and
Matt
Larsen.
If
you
know
those
names,
they
are
using
part
of
the
fridge
kernel
stuff
with
their
Python
interface
to
ascend,
so
that
any
code
that
uses
a
scent
can
request
the
code
to
stop
and
then
give
information
back
to
an
oppa
and
that's
limited
to
stuff.
A
scent
knows
about
so
it's
mostly
visualization,
actually,
updating
the
run
state
would
be
just
yes,
see,
yeah,
there's
a
link
in
the
references.
If
you
want
to
look
at
it.
A
I
was
want
to
go
over
this.
This
is
something
that
could
be
useful
to
some
of
you
who
have
C++
applications.
You
want
to
be
widgets,
we
had
a
summer
student
come
in.
Let
me
go
over
deal
this
first
so
deal
vis
into
OpenGL,
X
Windows,
C++,
visualization
application
I
talked
about
earlier.
It
does
server
based
visualization,
so
you
run
a
server.
A
You
connect
your
application
to
the
server
you
send
requests
and
it
performs
visualization
over
x
windows
blast,
which
is
our
arbitrary
lagrangian,
Alerian
hydro
package
uses,
higher-order,
l
meshes
or
fuller
grant
wishes
and
the
thing
about
geode
is
it
uses
if
you've
heard
of
em,
but
it
natively
renders
high
order
fields
meshes.
So
you
don't
have
to
project
it
onto
a
little
order.
A
Representation
of
the
object.
That's
nice,
my
demo
I,
don't
think
sure,
showed
the
nice
curved
elements,
but
do
that
so
we
had
a
summer
student
last
December
call
named
max
gang
who
basically
took
this
super
sad
location
used.
Opengl
m54
operated
a
to
2.1,
because
inscription
doesn't
work
so
well.
Fixed
function,
pipeline
added
some
inscription,
specific
things
and
just
transpiled
it
and
she
will
live
assembly
library
and
that
way
the
assembly
library
was
just
wrapped,
like
you
would
a
normal
JavaScript
library
as
a
widget.
The
only
tricky
thing
is.
A
It
feels
like
a
one
tricky
thing,
because
the
descripton
javascript
object
wants
to
remove
some
all
keys.
You
have
to
make
sure
that
you
tell
it
only
except
like
listen
to
he's
in
the
cell
that
which
it
is
in
it's
a
open
story.
It's
hasn't
been
developed
a
whole
lot,
I
plan
to
do
some
more
Python
side
stuff,
but
yeah
it
was
successful,
except
I
mean
he
had
to
modify
like
6,000
lines
of
code.
So
it
was
very
impressive
on
that
front.