►
From YouTube: Jupyter demo video
Description
Presented by Matthew Henderson (with Shreyas Cholia and Rollin Thomas)
June 3, 2020
Large scale "Superfacility" type experimental science workflows require support for a unified, interactive, real-time platform that can manage a distributed set of resources connected to High Performance Computing (HPC) systems. Here we demonstrate how the Jupyter platform plays a key role in this space - it provides the ease-of-use and interactivity of a web science gateway while providing scientists the ability to build custom, ad-hoc workflows in a composable way. Using real-world use cases from the National Center for Electron Microscopy (NCEM) we show how Jupyter facilitates interactive analysis of data at scale on NERSC HPC resources.
Jupyter Notebooks combine live executable code cells, with inline documentation and embedded interactive visualizations. This allows us to capture an experiment in a fully contained executable Notebook that is self-documenting and incorporates live rendering of outputs and results as they are generated. The Notebook format lends itself to a highly modular and composable workflow, where individual steps and parameters can be adjusted on the fly. Additionally, the Jupyter platform can support custom applications and extensions that live alongside the core Notebook interface.
We will use real world science examples to show how we create an improved interactive HPC experience in Jupyter including:
- Improvements to the NERSC JupyterHub Deployment
- Scaling up code in a Jupyter notebook to run on HPC resources through the use of parallel task execution frameworks
- Demonstrating the use of the Dask task framework as a backend to manage workers from Jupyter
- Enabling project-wide workflows and collaboration through sharing and cloning Notebooks, and their associated software environments
We will also discuss related projects and potential future directions.
B
A
Thanks
everyone
for
joining
us.
This
is
the
fifth
in
our
series
of
demonstrations
from
the
super
facility
project
this
this
late
spring
early
summer.
So
for
those
of
you
who
are
unfamiliar
with
the
super
facility
project,
this
is
a
project
to
coordinate
work
and
that's
going
on
in
the
computing
sciences
area,
at
berkeley
lab
to
support
experimental
science
on
running
on
a
large-scale
networking
and
computing
resources.
A
So
we've
been
holding
a
series
of
demos
to
show
off
and
advertise
and
solicit
feedback
on
some
of
the
tools
and
capabilities
that
have
been
developed
as
part
of
this
project.
This
has
included
data
management
tools,
super
facility,
api
networking
capabilities
and
this
and
edge
computing
capabilities,
and
so
this
demonstration
today
is
around
jupiter
and
I'm
going
to
pass
it
off
now
to
matt
and
the
rest
of
the
team
to
introduce
themselves
and
introduce
the
topic
thanks.
C
Some
of
it's
experimental
and
in
sort
of
a
range
of
projects,
including
light
sources.
Microscopy
telescopes
we've
been
primarily
interfacing
with
als
and
ansem,
which
I'll
talk
about
a
little
bit
more.
But
a
key
takeaway
on
this
slide
is
that
there
was
a
survey
conducted
and
where
projects
were
asked
to
self-identify
what
their
primary
role
was
and
about
35
percent
of
them
identified.
C
Some
role
with
experimental
data,
and
so
that's
what
we're
trying
to
address
is
how
to
serve
their
needs.
So
ensign
stands
for
the
natural
national
center
for
electron
microscopy
they're
part
of
the
molecular
foundry
next
time,
your
own
lab.
These
are
the
buildings
that
they're
located
in
the
incentives,
primarily
materials,
imaging
and
analytical
characterization
of
materials.
So
they
look
for
things
like
defects,
nano
structures,
thin
film
surfaces,
things
like
that
and
then
at
the
ls.
C
That's
a
very
broad
range
of
activities
that
are
supported
there,
but
it's
a
synchrotron
which
is
a
particle
accelerator
and
they
support
a
lot
of
different
science,
including
x-ray,
soft
x-ray,
hard
x-ray,
infrared
ultraviolet
and
other
wavelengths
of
e-mines,
and
just
to
give
you
an
overview
if
you're
not
familiar.
This
is
what
a
process
is
like
for
a
researcher
who's
traveling
to
a
place
like
the
als
they
design
their
experiments.
Based
on
a
hypothesis.
C
They
submit
a
proposal
to
that
facility
to
request
instrument
time.
They
then
get
a
reservation
of
instrument,
time
for
those
experiments
and
then
they
travel
there
and
then,
in
the
case
of
als,
you
actually
bring
your
equipment
as
well
as
your
samples
or
things
that
you
want
to
measure
for
something
like
end
stem
you
might
you
would
only
bring
the
samples
because
they
have
the
equipment
you're
going
to
use,
but
you
arrive
at
the
facility.
C
You
set
everything
up.
You
do
a
test
of
your
apparatus
in
the
als
case,
and
then
you
perform
your
experiments
in
the
window.
Time
do
you
have
and
then
later
you
analyze
your
data
and
usually
that's
not
done
on
site.
When
you
collect
the
data,
you
haven't
just
enough
time
to
get
a
bunch
of
data,
and
then
you
take
it
home
with
you
a
lot
of
times
and
you
look
at
it
later.
The
problem
with
that
is
that
you
may
not
collect
it.
C
If
you
need
it-
and
you
won't
know
until
it's
too
late,
you're
already
gone,
so
we
want
to
try
to
improve
that
process
and
make
that
a
little
bit
better.
And
so
how
are
we
doing
that?
We
are
trying
to
move
the
standard
towards
a
real-time
experimental
data
analysis,
which
means
being
able
to
quickly
process
that
data
visualize
the
results
and
then
also
have
some
type
of
interactive
analysis
that
includes
human
in
the
loop
and
so
human
in
loop.
Just
means
that
you
have
the
ability
to
change.
C
What's
happening,
it's
not
just
a
script
and
set
activity.
You
actually
go
in
and
modify
it
and
so
how's
this
going
to
help.
This
is
going
to
help
you
by
identifying
whether
or
not
you
got
that
data
the
critical
data
you
needed.
It
gives
you
feedback
that
you
can
use
to
either
conduct
more
experiments
or
get
something
done.
You
didn't
anticipate
when
you
started.
C
So
let
me
talk
a
little
bit
about
jupiter
who
doesn't
know
what
jupiter
is.
The
short
description
of
jupiter
would
be
to
describe
it
as
a
framework
for
interactive
computing.
But
if
you
want
to
break
it
down
in
components,
then
you
would
first
start
thinking
about
notebooks,
which
are
the
electronic
notebook
version
within
jupiter
they're
documents.
They
contain
code
and
other
things
like
images
or
markdown
or
other
rich
text
elements.
C
Then
there's
something
called
the
jupiter
notebook
app,
which
is
really
it's
sort
of
a
combination
of
the
front
end
and
also
the
server
that
controls
the
execution
of
the
kernel.
So
it
starts
and
stops
the
kernel
it.
It
handles
the
input
and
output
for
the
kernel,
and
then
it
also
handles
writing
and
reading
the
actual
notebook
files
themselves.
C
So
an
example
of
a
notebook
app
and
the
one
that
you
if
you've
used
the
nurse
jupiter
you've
seen,
is
jupiter
lab.
This
is
the
most
recent
incarnation
of
the
jupyter
notebook
app.
The
previous
one
is
referred
to
now,
as
the
classic
notebook,
and
in
this
interface
you
have
a
combination
of
features.
So
it's
not
just
notebooks.
You
can
sort
of
combine
a
lot
of
different
activities
in
the
same
window.
C
It
makes
it
more
like
a
development,
integrated
development
environment
where
you
can
have
terminals
text
editors,
many
other
things,
and
one
of
the
advantages
is
that
there's
a
large
improvement
in
the
extensibility
of
this
system
compared
to
the
previous
system.
Where
you
can
now
modify
the
ui,
you
can
create
dashboards.
You
can
create
more
interactive
widgets
that
were
hard
to
do
before,
and
so
why
are
we
using
jupiter
and
why?
C
There's
a
configurable
spawner
type
that
jupiter
hub
supports
nurse
uses
the
ssh
spawner,
the
auth
is
configurable
and
the
apps
run
on
login
notes
by
default.
Although
you
can
request
access
to
other
resources,
so,
for
instance,
you
could
run
a
notebook
on
a
single
node.
You
can
get
a
dedicated
node
that
will
run
your
notebook
server
and
you
can
also
request
access
to
gpu
resources
if
you're
doing
machine
learning,
deep
learning
or
some
other
activity
like
that.
C
But
there
are
challenges
if
it's
not
without
without
any
challenges
here.
So
primarily,
the
big
item
is
that
we
have
a
batch
q
system
and
this
is
not
designed
for
interactive
work.
The
nurse
q
policy
also
isn't
always
super
compatible
with
some
of
the
tools
out
there
and
how
they're
intended
to
work.
So
an
example
of
that
would
be
job
key,
which
we'll
talk
about
a
little
bit
later,
and
you
do
have
things
like
the
interactive
queue
now
with
jupiter.
C
C
C
One
of
the
students
that
work
with
us
he's
now
apple.
He
worked
on
a
couple
of
enhancements
that
you've
probably
seen
there's
a
favorites
and
a
recent
extension
that
were
added
that
allow
you
to
more
easily
navigate
when
you're
in
jupiter,
and
this
is
actually
kind
of
important,
because
the
favorites
one
in
particular
there
was
problem
navigating
to
the
home
directory.
C
These
are
notebook-based
python
pieces
of
code
where
you
either
have
a
long-running
notebook
cell,
or
you
have
some
repetitive
tasks
where
you're
processing
a
lot
of
data
using
the
same
functions
over
and
over
again,
and
we
would
like
to
be
able
to
have
something.
That's
easy
to
use.
It
doesn't
require
significant
refactoring
that
has
an
improved
interactivity
for
deeper
users,
it's
interoperable
with
other
tools
in
the
ecosystem.
C
C
C
But
there
are
some
limitations
relative
to
nurse
and
how
jobq
was
developed.
It
seems
like
it's
a
bit
more
oriented
towards
smaller
cluster
sizes
roland
and
I
attended
a
desk
developer
workshop
and
after
talking
to
people,
it
seemed
like
that
was.
The
more
common
use
case
was
that
people
who
are
using
that
strategy
tend
to
be
doing
smaller
things.
So
it
really,
if
you're,
if
you're
doing
something
big
or
you
want
a
larger
gas
cluster.
That's
job.
C
And
then
we
have
the
dash
cheaper,
lab
extension,
which
you
may
also
have
seen.
It's
currently
deployed
on
the
jupiter
hub,
the
nerf
jupiter
hub,
and
it
allows
you
to
connect
to
a
das
cluster
visualize,
what's
happening
with
taskrap,
do
some
visual
debugging
and
you
can
either
run
a
local
cluster
with
it
directly
and
launch
one
or
you
can
configure
it
to
use
desktop
key.
C
The
caveat
there
is
that
desktop
2
is
not
actually
installed
into
the
default
python
environment,
so
you
have
to
have
your
own
condo,
but
you
can
do
it
and
it's
also
possible
to
access
a
cluster.
That's
been
started
via
batch
job
separately,
so
for
scaling
up
hyphen
desk.
I
was
going
to
show
a
video
that
rollin
actually
recorded
cosmology
example,
and
then
I
was
going
to
talk
a
bit
about
desk
and
desktop
queue
and
then
show
the
dashboard
so
you'd
give
a
like
demo
there.
C
C
C
C
There
we
go
okay,
so
he's
got
a
cluster
started
up,
he's
we're
connecting
to
the
dashboard,
which
is
actually
going
to
be
in
a
separate
browser
window.
Now
you
can
actually
do
some
of
this
with
the
live
extension,
but
it's
also
nice
sometimes
to
be
able
to
get
a
bigger
view
in
a
separate
window
of
the
task
graph
and
other
components
of
desk.
C
So
we're
looking
at
now
is
a
task
stream
and
a
task
stream,
and
once
he
starts
executing
you'll
see
some
of
this
being
populated.
So
it's
reading
in
data
and
then
after
the
data
is
being
read,
we'll
start
seeing
the
tasks
being
submitted
and
the
task
graph
will
start
being
built.
It'll,
look
a
bit
more
complicated
and
this
the
nice
thing
about
looking
at
this
is
that
you
can
see
if
there
are
issues
in
the
processing.
C
C
You
can
see
the
a
profile,
so
I
think
there
you
go
yeah,
so
you
can
get
a
plain
graph
profile
of
the
execution
where
you
can
see.
Percentages
of
different
function.
Calls
that
tell
you
how
much
time
was
spent
at
a
particular
step,
and
so
that
will
help
you,
if
you're,
trying
to
understand
why
something
is
taking
longer
than
you
would
expect
it
to
or
if
you're,
just
trying
to
understand
how
your
workflow
is
running.
Is
it
running
efficiently
or
early
problems.
C
The
challenging
part
of
using
that
job
is
when
you
don't
have
a
previous
reservation
and
you're
just
sort
of
waiting
on
the
queues
to
pick
up
workers
and
there's
there's
a
mismatch
between
how
long
you
have
to
wait
for
the
workers
to
start
versus
how
much
work
you
can
actually
get
down.
So
let
me
go
ahead
and
run
this.
C
C
C
C
C
C
E
C
C
C
There
we
go
okay,
all
right,
so
so
here's
the
job
script
that
it
wrote
out
based
on
the
config
that
I
passed
it,
and
so
it's
writing.
On
an
sbatch
script,
that's
going
to
call
python
and
run
desk
to
run
desk
workers.
I
scaled
the
cluster
to
two
workers
essentially
or
two
two
instances,
so
we're
gonna
wait
for
that
to
start
out
it
shouldn't
take
too
long,
which
I
may
have
already
started.
C
Yeah,
I'm
sorry!
So,
first,
I'm
I'm
creating
this
rosenbrock
plot
here
and
then
just
to
show
just
a
quick
demonstration.
So
what
happened?
Was
these
points
were
being
generated
for
each
task
and
that's
what
actually
ran
it
ran
really
fast.
So
I
mean
we
missed
the
actual
run.
But
if
I
go
to
go
back
out,
I
can
connect
to
the
actual
plus
the
verified.
It's
a
bit
clunky
still,
but
you
can
put
in
the
proxy
information
for
this.
D
C
D
C
Yeah,
I
mean
it
really
it's
actually
very
simple
compared
to
like,
for
instance,
if
you
were,
if
you
wanted
to
develop
your
code
with
api,
there's
a
lot
more
involved
typically
than
here,
where
you
can
basically
take
your
vanilla
python
code,
as
is
do
some
very
small
modifications
and
you're
running
in
parallel.
That's
the
advantage
of
using
task
here.
The
clunkiness
that
I'm
talking
referring
to
is
is
connecting
to
the
extension
when
the
cluster
wasn't
actually
started
by
the
extension.
There's
still
there's
still
some
some
user
stuff
to
fix
there.
Yeah.
E
All
right
so
so
everybody's
a
little
bit
different
and
everybody
has
different
opinions
about
how
it
should
work
and
that's
job
queue
is
a
perfect
example
of
this.
When
we
say
oh
we're
at
hbc
center
and
we're
trying
to
run
desk,
everybody
says:
oh,
you
need
to
run
das
job
queue.
It's
just
the
wrong
thing
for
us.
A
manual
startup
is
a
lot
better
waiting
around
for
workers
to
start
up
for
interactive
data
analytics.
It's
just
waiting
for
the
cue.
That's
not
gonna
work
here.
E
E
I
will
say
that
what
what
matt's
doing
here
or
actually
what
people
can
do
right
now
on
jupiter,
not
like
not
interacting
with
the
batch
queue
you
can
still
run
desk.
You
just
can't
run
it
really
big
for
really
long
there,
but
we're
working
on
trying
to
figure
out
which
of
these
tools,
because
there
are
a
number
of
different
choices
out
there
desk
being
won,
that's
probably
being
the
one
that
most
people
have
heard
about
and
like
and
has
a
lot
of
developers
behind
it
for
a
while.
E
E
D
C
C
That's
right,
and
so
it
could
be
it
could.
The
cluster
could
have
been
started
by
job
queue
or
it
could
be
something
we
started
manually,
but
the
actual
the
rest
of
the
code
doesn't
really
change
right.
You
have
a
class
client,
it's
connected
to
a
cluster.
However,
you
started
it
and
then
you
just
submit
work
to
it
right
and
that's
the
beauty
of
it.
It's
actually
fairly
simple
to
get
going
so
for
the
nsam
use
case
we
had
after
we
talked
to
ansem.
C
That
was
actually
quite
long,
and
it
was
basically
it
was
running
in
serial
and
it
was
effectively
an
embarrassing
parallel
task
where
you
were
running
the
same
function
across
many
different
slices
of
the
same
data
set,
and
so
this
was
a
fairly
natural
pattern
to
use
something
like
desk,
and
so
you
have
to
process
all
the
patterns.
It
doesn't
matter
what
order
they're
in
a
65
gigabyte
data
set
for
them
on
their
workstation
took
about
11
hours
to
complete.
C
So
it's
not
super
interactive
when
your
notebook
takes
11
to
12
hours
to
run,
and
then
the
300
gigabyte
data
set
that
they
had,
which
was
the
largest
they
had
collected.
They
hadn't
tried
that
on
the
work
station
because
they
were
assuming,
it
was
going
to
take
a
really
long
time
and
they
knew
that
they
needed
to
do
something
about
it.
So
we
we
developed
a
parallel
implementation
and
improved
the
run
time
for
the
65
gigabyte
data
set.
C
So
now,
with
a
modest
number
of
hpc
nodes,
you
can
get
that
down
to
a
couple
of
minutes
from
the
original
11
hour.
So
it's
pretty
substantial
improvement
for
them.
You're,
definitely
going
from
I'll
wait
till
tomorrow
to
I'll
have
it
ready
in
a
few
minutes,
and
so
this
is
enabling
them
to
be
able
to
reduce
the
data
that
they
hand
off
to
the
user.
So
it's
already
been
pre-processed
for
them.
They
don't
have
hard
drives
full
of
data
and
they
can
take
it
home
and
and
do
their
further
analysis
with
it.
C
So
what's
happening
here
for
enzyme
is
they're
doing
something
called
scanning
transmission
electron
microscopy,
which
basically
is
they've,
got
a
focus,
speed
of
electrons
they're
scanning
it
over
the
sample
and
it's
a
in
the
case
of
40
stem
they're
using
the
2d
grid
to
sample
all
all
the
way
across
the
sample
and
each
sample
point.
You
get
these
diffraction
patterns
here
and
the
computational
expensive
step
is
essentially
computing.
The
peak
once
you
compute
that
you
can
determine
things
like
the
structure
of
the
material.
C
This
is
a
chart
showing
their
data
rates,
so
the
old
school
camera
using
film
was
one
gigabyte
per
hour
and
now
they're
at
with
their
custom.
40
stem
detector
they're
at
about
200
terabytes
per
hour,
and
so
realistically,
an
experiment
is
probably
a
few
minutes
at
a
time,
but
the
data
collection
rate
and
the
size,
the
amount
of
pixels
that
you're
generating
is
increasing
dramatically
and
so
they're
already
having
a
problem
here,
which
is
the
current
data
set
it's
using
this
k2
detector
and
that's
at
four
and
a
half
terabytes
per
hour.
C
C
C
So
you
have
something
to
compare
to
you,
get
a
line
profile
and
then
you
would
just
peek
at
some
of
the
diffraction
patterns
from
the
data
with
the
sample
just
to
make
sure
your
parameters
are
okay,
you
run
it
on
a
little
bit
of
data
to
get
an
example
of
what
that's
going
to
look
like
and
then
we're
going
to
go
ahead
and
run
the
entire
data
set.
That's
already
executing,
let's
see
where
we're
at
so
we're
0.3
complete.
It
probably
started
like
a
minute
ago.
C
C
Task:
okay,
so
I'm
at
the
parallel
step
now
so
what's
happening
here
is
the
code
is
refactored
a
little
bit
to
incorporate
a
desk
implementation
of
the
function
that
was
performing
slowly
in
serial
fashion
and
what
we
did
was
we
kept
the
same
interface
to
the
code,
but
now
you
can
basically
say:
okay,
I'm
using
das.
I
want
to
run
this
in
parallel
versus
running
it.
The
way
you
always
did
before,
which
would
just
be
a
serial
run.
C
So
the
only
thing
really
that's
happening
here
and
I'll
show
the
serial
version.
So
here's
the
serial
version
you
pass
in
your
parameters
that
you
reset
and
then
the
das
version
is
identical
other
than
that
you're
also
passing
in
some
information
about
how
to
connect
to
task
where
to
write
the
data
things
like
that,
and
so
we'll
go
ahead
and
start
this.
C
This
time,
inversion,
you're
just
entering
through
all
the
different
slices
in
the
parallel
version
based
on
the
number
of
workers,
we're
dividing
up
the
amount
of
data
as
equally
as
possible
and
then
spreading
that
out
to
the
different
nodes
so
that
the
work
can
happen
in
parallel,
and
that
makes
a
pretty
dramatic
improvement
in
the
run
right
now.
I
think
we're
still
waiting
for
some
of
the
data
to
load.
C
Yeah
it'll
take
it
will
still
take
a
little
while
it'll
take.
It
takes
a
couple
of
minutes
to
run
so
I'll
I'll.
Do
I'll
go
for
a
couple
minutes
so
now
let
me
just
get
into
the
the
idea
of
sharing
notebooks
and
what
the
different
mechanisms
are.
So
notebooks
are
files,
so
you
can
share
your
notebooks
using
any
mechanism
that
you
would
normally
share
a
file.
So
there's
nothing
special
there
in
that
aspect
of
it.
C
C
They
don't
have
a
lot
of
coding
experience,
but
they
do
understand
the
data
and
they
do
understand
how
to
set
parameters,
and
so,
if
you
give
them
a
pre-work
analysis
where
all
they
have
to
do
is
plug
in
the
parameters
and
the
data,
then
they
can
run
that
and
perform
their
data
analysis
and
so
in
instant,
bulk,
instem
and
als.
These
are.
C
These
are
one
of
the
use
cases
that
we're
exploring
and
then,
in
addition
to
that,
you
have
the
sort
of
I
want
to
share
or
publish
my
analysis
to
the
world
or
to
other
science
scientists
out
in
the
community
beyond
just
nurse
so
like,
for
instance,
you
might
put
it
now
once
you've
shared
the
notebook.
What
does
it
take
to
actually
run
that?
We,
of
course,
have
the
notebook
file
itself,
but
then
you
also
need
a
compatible
jupyter
kernel
which
executes
the
code.
C
C
When
you
visit
one
of
these
you'll
get
a
fully
rendered
notebook,
assuming
it
was
saved
with
the
outputs
that
you
can
preview
before
you
do
anything
with
it,
and
we
talked
to
peter
parente
and
dan
allen
from
brookhaven.
They
would
have
both
used
this
for
different
purposes
and
sharing
notebooks,
and
so
we
took
that
work
william.
C
So
it's
running
alongside
the
jupiter
hub
deployment,
and
in
order
for
this
to
work,
you
of
course
have
to
have
a
pre-built
python
environment,
so
you
have
to
have
your
content
set
up
and
it's
public
or
available
for
the
people
you
want
to
share
with.
But
at
that
point
you
can
point
them
to
this
and
tell
them
where
your
notebook
is
at
with
the
kernel
spec
and
they
can
just
copy
it
over
and
it
will
work.
So
that's
pretty
nice
and
so
let's
go
ahead
and
take
a
look
at
that.
F
C
So
from
here
there's
a
services
menu
item.
Now,
where
you
can
access
nviewer,
it
takes
you
here.
I
can
input
a
github
repo
that
I
have
set
up
with
a
notebook,
that's
ready
to
be
flown
from
notebooks.
So
this
is
a
science
notebook.
This
is
my
fork.
The
original
is
here
and
it's
a
frequency
domain,
em
experiment.
C
D
C
C
And
you
know
it's
in
super
typical,
jupiter
notebook:
it's
got
some
plots
in
it.
There's
some
interactive
widgets
that
are
included
here.
So
I
can.
I
can
change
this
a
bit
and
look
at
some
different
aspects
of
it.
It
works
right.
I
didn't
have
to
do
anything
else.
So
that's
the
nice
thing
about
notebooks.
Let
me
jump
back
in
real
quick
before
we
run
out
of
time,
because
we're
getting
close,
then
let
me
talk
a
bit
about
binder.
C
So
binder
is
another
tool
from
jupiter
project
that
allows
you
to
share
your
notebooks
in
this
case
with
a
custom
environment.
So
you
would
store,
typically
store
your
notebook
and
get
get
repo
either
get
over
something
else,
and
then
you
would
add
files
to
that
repo
with
the
notebook
that
define
the
dependencies
that
are
needed
to
run
your
network.
C
So
this
would
be
like
a
requirements
of
text
or
it
would
be
a
environment.yaml
for
condo
or
or
maybe,
if
you
have
additional
packages
you
need
to
install
into
like
linux
packages,
you
could
include
an
app.text
that
would
be
installed
into
a
docker
image.
That's
built
from
that,
and
so
there's
a
site
called
mybinder.org,
which
is
a
public
resource
where
you
can
do
this,
but
it
has
limited
compute
and
data
allowances
and
then
the
docker
image
that
it
creates
from
your
repo
is
stored
there
within
the
mybinder
environment.
C
C
So
what
I've
been
working
on
as
a
in
a
separate
project-
it's
actually
not
funded
by
this,
but
through
oscar,
but
is
related.
So
that's
why
it's
here
you
have
a
prototype
that
allows
you
to
do
something
almost
identical
and
run
the
same
repo
that
you
would
normally
run
in
my
binder
at
nurse,
and
so
the
unique
thing
there
compared
to
phone
notebooks
is
that
it
gives
you
more
flexibility.
C
You
can
include
things
that
are
hard
to
do
with
phone
notebooks,
for
instance,
if
you
have
a
custom,
jupyter
lab
extension
that
you
want
to
work
on
for
your
project
and
share
with
other
users
right
now.
That's
kind
of
hard
to
do
with
the
current
jupiter
lab
set
up
the
way
it
is,
but
it's
much
easier
to
do
with
this
sort
of
recipe
right.
You
can
set
up
your
repo
that
has
all
the
dependencies,
including
your
extensions.
C
C
C
She
works
with
fernando,
and
so
here
I'm
running
on
my
binder,
I
forked
for
repo,
and
I
made
a
couple
of
modifications,
so
that
would
work
inside
jupiter
lab
because
by
default
my
binder
still
works
with
the
classic
input,
and
then
I
can
run
that
and
what's
what's
different
compared
to
the
other.
One
that
I
ran
is
that
this
one
so
right
here
what
you're
seeing
down
here?
C
C
You
would,
on
your
laptop,
build
a
docker
image,
push
it
to
the
nurse
image
registry
documentary
registry
and
then
use
shifter
to
actually
run
it
with
jupiter
hub,
and
so
that
allows
you
to
take
anything,
that's
built
to
run
on
mybinder
and
it
will
work
on
nurse
with
the
caveat.
Then
you
might
have
to
do
something
special
to
make
sure
that
it
works
in
jupiter
lab
if
there's
some
widgets
or
extensions
that
are
needed.
D
C
D
C
G
Matt,
you
may
have
said
something
about
it.
I
missed
it.
Is
there
any
thought
about
sort
of
turn,
turning
kind
of
making
it
easy
to
create
the
artifacts
that
you're,
showing
with
the
binder
piece
so
like?
If
you
had
a
functional
kind
of
set
of
analysis,
to
make
it
easy
to
kind
of
create
a
repo
from
that
and
push
it?
And
so
it's
now
ready
to
share
with
others
through
that
mechanism
or
something.
C
Yeah
yeah
so
well
also,
so
there
are
questions
about
how
to
deal
with
large
data
if
you're
putting
in
data
right.
That's
a
separate
problem
from
what
I'm
doing
here.
I
do
think
that
that's
an
issue,
though
that
has
to
be
explored.
How
do
you
deal
with
large
data
if
you're
sharing
it
out
with
other
people?
Do
they
access
a
nurse?
Do
you
know
what
do
they
do
right?
That
is
a
question
sort
of
an
open
question.
I
have,
I
don't
have
a
solution
for
you.
D
In
fact,
it's
the
other
way
around
right.
One
would
actually
point,
I
think
the
reason
we
developed
this
is
that
the
so
binder
sort
of
works,
typically
in
the
cloud
where
you
don't
really
have
direct
access
to
your
data,
you
just
have
small
instances
of
data
if
you
do
need
to
run
it
adjacent
to
your
data.
The
idea
is
that
you
would
have
some
a
tool
like
this
that
can
actually
launch
this
at
nurse
square
data
sitting,
so
you
have
direct
access
to
it.
B
D
There's
a
tool
called
vola
which
we're
starting
to
prototype
in
the
jupiter
environment
and
that
that's
basically
what
you
said,
which
is
a
front
end
that
that
hides
all
the
code
and
only
exposes
the
visual
elements,
and
it's
it
isn't
it's
something
we're
playing
with.
I
think
part
of
it
is
it's
so
so
we
have
an
early
use
case
for
that
and
we'll
be
prototyping
it
and
it's
if.
F
One
little
tidbit
that
I
might
offer
I
was
experienced
working
with
spot
is
that
a
lot
of
the
work
that
a
lot
of
the
visiting
scientists
wanted
to
do
was
was
pretty
simple
or
pretty
straightforward.
They
were
pretty
happy
with
something
that
could
just
do
some
near
real-time
analysis
and
give
them
feedback
as
to
how
their
samples
are
going
and
one
direction
we
were
trying
to
go
and
but
never
got
to
is
kind.
F
Of
their
analysis
that
they
wanted
to
do
as
somewhat
equivalent
to
like
you
know
the
sample
that
they
would
bring
to
the
beam
line
or
even
some
of
the
equipment
and
work
with
these
scientists
ahead
of
time.
I
think
there's
a
real
opportunity
here
with
a
lot
of
the
tools
that
you
guys
are
using
to
do
that,
where,
if
you
could
you
know
ship
them
out
a
notebook
ahead
of
time
and
say,
here's.
F
F
C
D
And
I
I'd
actually
say
that
I
think
some
of
this
was
also
the
function
of
the
the
sort
of
demos
themselves
where
they
were
code,
heavy
demos
we
could
more
or
less
do
the
same
things
where
we
hide
a
lot
of
that
from
you,
and
so
I
think
some
of
that
is
just
on
our
choice
of
examples.
Maybe
and
then-
and
I
think
what
you're
saying
keith
is
actually
still
true.
I
think
we
can
do
all
these
things
and
we
could.
F
C
A
F
D
A
All
right,
I'm
gonna,
have
to
wrap
things
up,
I'm
afraid
we're
getting
into
the
the
next
hour.
So
I
want
to
thank
our
presenters
again
and
thanks
to
matt
and
shreish
and
rolling
and
as
well
for
a
nice
demo,
and
so
this
is
the
last
demo
in
the
current
series
of
super
facility
demonstrations,
but
look
out
for
demos
coming
up
in
the
future.
We're
going
to
be
scheduling
more
later
this
year,
as
we
have
new
functionality
to
demonstrate
for
you.
So
thanks
everyone
and
have
a
good
afternoon.