►
From YouTube: 9. Terraforming the Cloud to Teach HPC with Jupyter
Description
June 12, 2019 Jupyter Community Workshop talk by Felix-Antoine Fortin, Universite Laval
A
Yes,
sir,
good
morning,
everyone
I
am
from
Mestalla
city,
as
you
might
tell,
by
my
accent:
I
am
french-canadian
out
to
yubi
foreign
club
teach
hpc
with
Jupiter,
so
my
presentation
so
feel
free
to
ask
me.
Questions.
I
have
not
put
enough
technical
details
in
my
presentation
talking
about
some
mental
stimulation
for
you
this
morning.
I
have
a
question
so
earlier
I
reported
in
in
Lords
of
the
ring.
So
like
huge,
you
actually
think
about
that
questions.
All
I
present
to
you
my
context
eventually
I
try
to
answer
that
question.
It
is.
A
It
is
related
to
Jupiter
in
what
we're
doing
I
assure
you
I'm
so
context.
So
as
I
said,
I'm
Canadian
I
have
something
called
pew.
Canada
she's,
an
organization
actually
liked
exceed
that
is
in
charge
of
providing
research,
cyber
research
infrastructures
to
every
Canadian
researchers
in
Canada,
so
I
have
access
to
our
high-performance
computing,
centers
and
free
ads
provided
by
government
centers.
A
There
and
warning
back,
and
we
have
sites
also
in
British
Columbia,
that
is
dedicated
to
cloud
solutions.
All
of
that
so
previously
on
the
on
the
map.
Smokers
are
actually
sites
that
provide
research
supports
or
the
big
dots
are
actually
HPC
sites
and
the
smoke
step
on
how
to
use
the
HPC
resources.
Now
part
of
our
mission.
We
have
training
researchers
and
how
to
use
our
HBC
our
infrastructures
so
hundred
fifty
course,
a
year
across
Canada.
A
Okay,
most
most
of
these
cultures
require
access
to
HPC
clusters
and
in
order
to
get
access
to
those
HPC
cluster,
you
need
a
compute
Canada
account,
while
those
accounts
are
free,
some
paperwork's
to
fill
and
it's
a
process.
So
normally
people
when
they
come
in
and
do
our
training.
They
don't
necessarily
have
an
account.
A
So
what
we
actually
operate
was
that,
well,
you
need
some
gas
accounts,
but
in
order
to
get
gas
accounts,
I
would
have
to
ask
assistant
min
two
weeks
in
advance
for
accounts
on
my
on
my
cluster
and
it's
not
like
it
was
a
too
long
process,
so
it
started
thinking.
Well,
you
actually
need
a
real
cluster
in
order
to
teach
users
about
I
mean
and
about
out
to
huge
HPC
and
slurring,
or
should
I
like
make
it.
A
A
That's
just
pure
all
right,
so
so
mightier
that,
what's
missing
in
Lords
of
the
Rings,
our
wizardry
schools
in
our
schools
that
teach
children
all
to
use
magic
while
HPC
for
new
users,
this
kind
of
magic
right
so
I
have
a
proposition
and
what
I'm
probably
see
workshop
platform-as-a-service
in
order
to
teach
new
users
how
to
use
rich
basic
services.
What
up
I
think
to
get
them
access
directly
on
our
HPC
cluster,
so
see.
B
A
A
A
B
A
B
A
Cluster
blended
moment,
I
assure
you
right
so
would
in
compute
Canada
we
have
a
as
I
said
in
British
Columbia
have
a
OpenStack
clap
when
I
did
through.
My
fault
eventually
talked
to
some
of
my
project
that
eventually
talked
to
OpenStack
I'm,
just
way
to
show
you
this
in
a
moment.
So
if
I
can
access
my
tab
area,
so
this
was
the
state
of
OpenStack
project
like
a
few
minutes
ago.
We're
actually
calling
my
my
computer
kind
of
wizard
I
have
a
big,
wizard
I.
Think
team
are
going
in
that
presentation
now.
A
A
A
So
what
is
there
for
who's
familiar
in
the
room
always
already
familiar
with
with
terraform?
Okay,
so
be
ready
to
be
amazed
for
the
scripting
language
to
define
data
center
infrastructure
with
eye
level
language.
So
this
is
an
open-source
software.
It
is
it's
quite
efficient
in
same
kind
of
language.
You
can
talk
with
all
sorts
of
REST
API
as
the
rest
of
the
eyes
that
are
provided
by
different
commercial
cloud
providers,
OpenStack
providers,
so
on
terraform
what
I've
built
something
called
magic,
app
obsession
with
magic
and
castle
I'm.
A
Sorry,
it's
just
so
series
of
terraform
script
that
actually
wrap
into
four
modules
that
we
are
going
to
see
in
the
next
slide
that
first
create
instances
so
route
HPC
cluster.
What
is
an
HPC
cluster
compute?
No
a
log,
so
the
terror
group
is
also
in
charge
of
actually
creating
the
volumes
of
where
the
storage
is
being
the
network
are
the
networks
they
see
apps
or
the
fire.
All
the
thing
eventually
also
create
all
of
the
certificates.
A
So
when
our
Jupiter
is
are
being
put
online,
it's
already
have
HTTPS
certificates,
it
created
the
password,
a
random
password
for
the
gas
accounts
and
create
Jessica.
It
creates
true
in
20
minutes
all
of
the
essential
things
that
we
need
in
order
to
teach
users
each
pcs
and
all
of
that
done
directly
to
input
parameters.
So
one
of
us,
talking
to
my
fault
I,
was
just
actually
giving
some
of
the
input
parameters
that
can
be
provided
to
my
modules
in
order
to
get
an
introducing
cluster.
A
So
what
you
don't
always
have
to
talk
to
your
phone
in
order
to
create
an
HPC
cluster
I'm,
going
to
show
you
first
alright.
So
how
does
it
look
technically
when,
when
it
comes
to
the
actual
infrastructure
of
adh
PC
cluster,
so
you
have
your
users
coming
from
the
area
and
it
can
lock
it
into
a
loggin
out
and
that's
the
e
say
a
free
way
of
describing
you
know.
That's
a
very
multi
usage
argue
note.
So,
on
this
logging
ugly,
app
SSH,
you
have
Jupiter
hub.
A
Yes,
the
users
can
log
in
directly
on
your
Drupal
instance,
but
that's
fine.
This
is
for
workshop.
This
is
not
production
usage
and
our
create
over
sent
one.
So
in
our
HPC
workshops
we
are
actually
teaching
users
how
to
use
our
Globus,
so
I,
clicked
Rudy
due
to
to
terraform,
provide
provisioning.
We
are
registering
automatically
a
new
global
standpoint
that
can
be
used
during
our
workshops,
and
this
is
a
Oh,
our
different
axes,
and
it's
did.
A
B
A
And
effects
all
of
our
stuff,
our
firm
cpld,
all
that
the
animation
and
then
you
have
a
series
of
notes,
so
you
can
create
as
many
computer
as
you
like,
as
long
as
you
have
enough
resources.
So
just
our
firm
demon
na
jupiter
single
users,
when
it
is
being
launched
to
the
batch
burner
on
the
trigger
so
pretty
structure,
would
provide
to
the
users,
is
a
terraform
module
that
it
essentially
three
components.
So,
first,
it's
a
we
provide
source
or
a
cloud
providers
so
magic
castle.
A
Actually,
yes,
it
can
create
to
PC
cluster
in
an
open
step,
but
so
do
it
in
Microsoft,
Azure,
aw,
wet
and
in
Google
Cloud,
so
the
actual
top
writers
and
available
truth,
their
form
can
be
and
I
just
by
actually
spawn
an
HPC
cluster
on
then
we
are,
can
provide
veneering
input.
So
now
we
want
to
actually
configure
HPC
cluster
and
then
a
different,
specific
infant's.
B
A
Of
note
that
we
want
to
build
the
number
of
users,
the
size
of
the
storage
that
we
want
to
use
and
eventually
to
public
key
pass
through
to
ssh
as
an
admin
on
the
cluster
Chua
to
do
our
stuff.
So
all
of
these
firm
units
are
shared
between
all
the
cloud
providers
and
it's
all
of
the
same
code
in
order
to
actually
provision
the
clusters.
A
So
it
is
not
out
specific
those
are
independent
and
then
the
that
are
specific
to
a
cloud
provider
so,
for
example,
the
image
virtual
machine
image
that
you
want
to
use
the
kind
of
instances
that
you
want
to
use.
Eventually,
if
you
already
have
different
parameters
from
cloud
providers,
certainly
out
of
the
case
of
OpenStack
ilk,
a
floating
IDs
and
associate
them
directly
with
you
long
enough.
A
B
A
Just
calling
to
treat
all
the
instances
all
this
of
the
resources
and
at
the
end
it
outputs
you
per
meters
that
weren't
generated
so
the
name
of
the
user
and
the
backwards
very
clean,
I'll
put
it
in
the
industry,
so
my
Google
assistant
cannot
provide
it,
but
it
could
be
look
back
in
my
in
my
cell
phone.
So
since
we
are
positioning
from
scratch,
it
takes
to
be
under
or
around
20
minutes.
This
is
not
containers.
This
is
virtual
machine,
so
it's
submit
long
and
third,
but
still
it
provides,
looks
up.
A
You
know,
so
we
are
coming
from
inter
tools
that
is
directly
on
your
laptop
that
talks
them
to
any
of
the
compromise
of
you
choosing
and
finally,
the
actual
private
of
the
notes
are
done
to
puppet
modules.
So
why
puppet?
Why
not
ansible,
with
using
puppet
to
provision
armor
to
provision
our
clusters
and
I
would
actually
be
able
to
reuse
some
of
those
modules.
I
was
not
but
yeah
I'm
able
to
contribute
back
some
of
that.
Some
of
that
stuff,
so
I
a
provisioning
is
being
done
through
a
masterless
puppet
architecture.
A
A
So
the
base
configuration
just
come
with
a
slum
cluster,
but
as
you
ition
that
you
you're
progressively
adding
stuffs
or
global
standpoint
and
eventually
the
whole
shebang
with
with
Jupiter
Hall
and
what
makes
it
actually
a
compute
Canada
cluster
is
it
software.
So,
as
I
said,
all
of
these
administrative
software's
are
being
installed,
but
the
two
key
parts
are
the
that
would
be
compute
Canada,
CDM
offense.
A
So
most
of
you
like
using
a
kitchen
sink
problem,
is
that
Cuba
installing
up
software
on
place
and
then
distributed
to
all
of
our
cluster
to
something
called
C
VMFS.
So
it's
a
virtual
file
system
that
was
bill
at
CERN
in
order
to
deploy
this
way
that
file
system
can
actually
be
mounted
anywhere
since,
essentially
it's
an
inch
tt
endpoint
are
using
that
directly
in
Magic
Castle
and
we
are
to
actually
provide
over
600
softwares
and
four
thousand
I
claim
kind
of
mismatch
of
compilers
and
libraries
directly
on
our
cluster
after
20
minutes.
A
That's
scientific,
see,
VMFS,
architectures
and
module
is
going
to
be
presented
at
pert
19
next
month
and
compute
counter
is
eventually
in
the
process
of
making
that
repo
available
publicly.
So
anyone
could
actually
mom
dad
CBM
ffs
software's
in
that
people
being
made
available
as
our
open
source
software
that
our
research
analysts
have
built.
A
Once
you
have
access
to
all
of
that
software's,
you
got
it
true.
We
give
it.
We
get
access
to
that
software
to
do
something
called
modules
or
alman
and
to
interact
with
true
jupiter
extensions.
Come
originally
now
mod
allow
YouTube
March
to
interact
different
modules
directly
in
Jupiter,
modified
environments,
árbol
and
then,
when
you
launch
notebooks
your
inner,
it's
those
environment
variables
and
you
have
access
to
all
of
the
different
scientific
software-
is
that
you
have
loaded
through
a
mod.
So
you
can
either.
If
you
want
your
terminal,
you
have
access
to
all.
A
B
A
A
Tension
I
built
a
full
API
for
that
will
allow
you
to
actually
load
my
Python
directly
in
your
noble
and
finals,
but
then
it
would
require
some
programming,
but
we're
not
modifying
your
noble
content.
I
also
are
able
to
modify,
and
some
of
the
Jupiter
developer
might
not
have
liked
me
for
that.
But
we
are
able
please
I've
the
kernels
that
up
display
billable
kernels.
So,
for
example,
if
you
only
have
a
pipe
entry
by
default,
you
Elma
sanshin,
you
look
refresh.
You
now
have
the
by
country
kernel
notebook
available.
A
A
To
ask
for
so
I
started
a
class
year
since
then
we
have
been
using
it
for
three
of
our
summer
schools
and
all
of
our
workshops
are
not
directly
top
of
that,
instead
of
asking
for
guest
accounts
on
our
on
our
clusters,
so
we
are
all
sort
of
thing
introduction
to
each
PC,
servers,
interaction,
programming
and
etc,
but
this
is
from
the
user
perspective
benefits
from
the
admin
side.
So
since
we
are
pledged
our
kind
of
HCC
environments,
we
can
also
teach
admins
how
to
interact
with
slurm
and
where
our
environment,
so
here
are.
A
He
had
an
intern,
actually
loving,
all
of
the
software
code
for
the
different
cup.
However,
that
would
not
OpenStack
and
he
talked
on
himself.
All
HPC
actually
worked
from
the
inside
by
developing
that
and
this
summer
we
currently
up
you.
Can
you
have
a
wish
to
actually
able
to
provide
user
feedback
on
the
usage
of
their
jobs,
their
live
once
their
job
is
finished.
We
would
like
to
send
them
an
email
saying
what
you
know
they're
a
course,
but
you
only
use
one.
A
You
could
not
be
better
than
that
so
developing
some
plugins
that
I
use
metrics,
true
Prometheus
and
eventually
end
up
in
any
mouth,
so
you
say
bolt
you
actually
do
that
directly:
Patrick
Castle
and
funny
file
setup
as
as
you
like,
so
key
takeaways.
If
you
didn't
know
our
tera
forms
nope
you
to
build
complex
things,
but
this
could
be
simplified
by
a
single
modules
and
Magic
Castle.
Is
that
teaching
and
develop
a
table
that
can
be
used
at
large
and
where
Jupiter
is
a
first-class
citizen?
Adam
Hydra
Castle
is
not
yet
fully
source.
A
B
A
A
B
A
Actual
sum
I
talked
about
it,
but
I
haven't
done
anything
in
that
regard
like,
for
example,
saving
that,
if
modules
name
in
the
notebook
in
order
to
reload
it
once
you're
starting
over
I
thought
about
it.
But
since
for
now,
I
think
the
only
one
of
the
few
actually
are
using
the
Ahmad
extension
I'll
leave
feedback,
but
yeah.
B
A
So
the
module
yes
so
all
is
like
transparent
for
the
users
underneath
it's
using
auto
af-s.
So
when
you
actually
access
the
file
is
being
fetched
and
it's
quite
fast,
a
small
latency,
the
first
time
that
you
actually
access
the
software's.
But
then
there's
a
local
cache
on
the
node
and
you
can
actually
determine
the
size
of
your.
B
B
A
B
A
A
So
if
I
would
like
to
expand,
you
can
note
five
to
do
is
going
to
modify
to
ten
doing
apply
yours
going
to
create
the
eighth
notes
that
are
missing
a
slow,
so
I'm
do
some
black
magic
slurm,
because
normally
you
have
to
be
finding
the
number
of
nodes
that
you
are
going
to
use
so
I'm
just
telling
him
well
250
nodes,
even
if
I'm
lying
to
him
and
I
actually
learn
calm.
Our
swim
is
not.
This
is
not
production
and
it
works.
A
B
A
I'm,
not
the
Steve
EMFs
next
boot
I'm,
just
like
benefiting
from
all
of
the
art
work
that
has
been
done
by
by
the
guys
are
who
are
going
to
present
it
at
Burke
next
month.
So
there
you
have
a
full
page
papers
that
were
accepted.
I've
heard
so
look
at
this.
If
you,
if
you'd
like
to
know
more
or
contact
me,
I,
can
put
you
in
contact
with
them.