►
Description
Presented by Gabor Torok, Cory Snavely, & Bjoern Enders (NERSC)
May 20, 2020
The Superfacility API aims to enable the use of all NERSC resources through purely automated means using popular development tools and techniques. An evolution of its predecessor, NEWT, the newly-designed API adds features designed to support complex, distributed workflows such as placing future job reservations and registration of API callbacks for asynchronous processes. It will also allow users to offload tedious tasks such as large data movement via simple REST calls.
While the Superfacility API is designed for non-interactive use, this demonstration will use a Jupyter notebook to step through a working example that calls the API to conduct a simple workflow process. Discussion will include additional information on planned API endpoints and authentication methods.
A
The
super
facility
project
for
those
of
you,
unfamiliar
with
it,
is
designed
to
help
support
experimental
scientists
who
need
to
use
high
performance
and
network
and
computing,
and
it's
a
collaboration
of
people
in
crd
in
es
net
and
at
nursk
berkeley
lab.
So
there
have
been
two
demos
already
in
the
series.
B
Hi
there
thanks
for
coming
everyone,
so
I
wanted
to
just
give
some
appreciation
to
the
other
speakers
here
and
actually
or
if
you
want
to
share
the
presentation
slides.
So
we
have
so
gabor
who's
who's
sharing.
There
is
the
developer
of
the
api,
and
so
he's
going
to
give
the
demo
and
talk
about
some
of
the
technical
detail
there
and
then
also
speaking,
is
byron.
Henders
who's.
One
of
our
science
engagement
leads
within
the
super
facility
project,
and
so
he
helps
us.
B
You
know
basically
liaise
with
scientific
projects
and
and
determine
needs
and
requirements,
and
things
like
that
and
his
input
has
been
really
instrumental
in
developing
this
api,
and
my
name
is
corey
snavely
and
I'm
the
lead
for
the
infrastructure
services
group
at
nurse
square.
The
development
of
the
api
is
is
centered,
we've
got
some
plants
in
the
audience
too.
B
With
that
I'll
go
ahead
and
get
started,
I'm
just
going
to
kind
of
give
a
brief
introduction
of
what
it
led
us
to
develop.
A
functionality
like
this
and
and
we'll
talk
through
a
couple
use
cases.
Bjorn
will
talk
about
that
and
and
then
gabor
will
give
the
demo,
and
then
we
hope
to
leave
ample
time
for
questions
that
you
may
have
and
we'd
also
kind
of
like
to
hear
whether
there
are
people
in
the
audience
who
feel
like
they
would
need
some
different
functionality
from
what
we
already
have
planned.
B
B
This
is
kind
of
a
summary
of
the
the
types
of
work
and
some
of
the
characteristics
of
these
experimental
observational
projects
that
we
that
we
drew
out
of
our
engagement
and
and
sort
of
generalized
across
these
the
different
needs,
and
so
you
may
have
seen
some
reference
to
some
of
these.
This
is
this
is
sort
of
the
umbrella
of
what
we
consider
to
be
the
super
facility.
So
I
won't
go
through
this
in
detail,
but
you
can
see
the
common
threads
here
and
they
may
relate
to
projects
that
you're
working
on
yourself.
B
So
you
know
a
focus
on
being
able
to
work
with
the
scheduler
actively
to
place
future
reservations
and
to
have
some
notion
of
resiliency
for
continuous
workflows
software
defined
networking.
If
you
talk,
if
you
came
to
the
sense
talk
previously,
you've
got
some
detailed
information
about
what
we
have
in
mind
for
automation,
automation,
around
networking,
also
data
movement
and
just
visibility
into
data,
and
that
includes
the
the
dashboard.
B
Next
week,
at
this
time,
slot
I'll
be
talking
about
the
spin
containers
as
a
service
platform,
and
that
fits
in
as
a
place
to
run
services
that
can
sort
of
sit
outside
hpc,
but
complement
that
that
and
workflows
and
then
the
and
then
what
we're
talking
about
today
is
api,
the
api
and
federated
identity,
and
what
we
really
see
this,
as
is
a
kind
of
a
piece
of
glue
that
helps
all
of
the
functionality
that
we
envision.
All
of
the
interaction
with
with
nurse
with
these
workflows
need
to
be
done
in
an
automated
way.
B
So
how
do
we
project
folks
using
the
api
as
part
of
their
project?
So
one
of
the
big,
the
big
use
cases,
that's
common
across
all
of
the
projects
we've
talked
to
is
facilitating
unattended
operation
minimizing
a
human
in
the
loop
for
projects
either
because
they
need
to
move
rapidly
or
because
the
volume
of
different
atomic
operations,
like
the
number
of
jobs,
for
example,
to
submit
and
track
is
just
too
high
to
realistically
handle.
B
There
is
also
the
notion
of
building
more
intelligence
into
real-time
processes,
such
as
adjusting
job
parameters
and
having
that
stuff
run
in
an
automated
fashion
by
scripts
workflow
managers,
because
it
just
isn't
time
in
a
lot
of
cases
as
we'll
discuss
to
to
make
those
kinds
of
adjustments
or
to
handle
failures
that
happen,
maybe
at
3
am
so
really
anywhere.
Automation
makes
sense
is
where
the
facility
api
fits
in
and
that's
what
we
have
as
our
vision.
B
So
here
are
the
actual
endpoints
and
the
api
that
we
have
prototyped,
and
so,
if
anyone
here
is
familiar
with
the
first
iteration
of
api
called
newt
at
nurse,
some
of
these
are
inherited
from
newt
and
have
some
compatibility
in
their
call
interface
and
some
of
the
capabilities
are
new
and
were
brought
about
by
the
super
facility
project
and
the
planning
and
the
and
the
our
science
project
engagements
there.
So
I'll,
let
you
read
the
the
the
endpoints
there.
So
we
do.
B
B
There
are
callbacks,
there's
a
capability
to
register
callbacks
for
asynchronous
operations
or
situations
where
something
needs
to
complete
and
and
call
back
to
a
workflow
manager
in
order
to
continue
an
operation
methods
for
dealing
with
files,
uploads
downloads
checking
the
health
of
the
center
or
the
resources
at
the
center.
So
you
can
know
whether
to
fire,
workflow
off
or
wait
or
plan
around
scheduled
maintenance.
B
C
B
B
So
here
is
kind
of
a
colorful
and
busy
diagram
that
tries
to
show
the
authentication
and
access
model.
So
one
of
the
things
that
we
want
to
do
with
the
api
is
embrace
modern
design
patterns
for
code
so
that
we
sort
of
meet
developers
where
they
are
in
terms
of
implementation,
and
so
this
is
a
reconceived
authentication
model
that
is
based
on
oauth2
and
json
web
tokens,
which
are
popular
ubiquitous
now
for
these
kinds
of
needs,
and
so
you
know
this
is
all
available
after
the
talk.
B
But
just
very
briefly
what
happens
and
we'll
show
this
in
the
demo
too.
Is
that,
like
most
apis,
you
can
retrieve
an
api
key
that
functionality
is
built
into
iris,
which
is
sort
of
the
user
portal
at
nurse,
if
you're
not
familiar
with
that,
you
can
create
a
refresh
jot
token,
their
json
web
tokens
are,
if
you've
never
seen
one.
B
It's
an
opaque
string
like
you'd,
expect
with
an
api
key,
so
you
can
log
into
iris
and
create
one
of
those
and
it
has
a
certain
long
lifetime
associated
with
it
and
a
certain
scope
or
permission
associated
with
it
and
then
the
mode
of
access.
Is
you
store
that
in
the
workflow
manager
as
a
credential
and
the
so
your
workflow
manager
or
script
or
whatever
system?
B
B
You
know
the
vision
of
the
api
is
really
to
provide
an
automated
means
for
interacting
with
nurse
so
between
the
different
endpoints
that
prototyped
and
an
authentication
model
where
that
kind
of
fits
modern
standards
and
can
make
use
of
common
libraries
that
are
out
there.
We're
really
trying
to
do
is
just
ease
access
to
some
of
the
most
common
operations
in
an
automated
way.
C
B
B
B
B
So
if
there's
an
opportunity
to
change
some
of
the
coding,
here's
a
cleaner
way
to
do
it
and
we're
not
necessarily
taking
away
some
of
those
legacy
capabilities,
but
here's
an
opportunity
to
simplify
code
down
the
road
and
then
for
new
projects.
It
makes
a
much
easier
on-ramp
and
then,
like
I've
mentioned.
What
we
want
to
try
to
do
too
is
help.
The
software
developers
end
up
with
cleaner
code
that
you,
like
better
so
fitting
design
patterns
rather
than
fighting
them,
shared
libraries,
api
calls
and
modern
standards
and
modern
authentication
and
security
models.
D
Right
thanks
corey
yeah,
so
we
are
covering
two
use
cases
here,
although
there
are
a
lot
more
within
the
super
facility
framework,
so
one
that
I'm
particularly
involved
is
light
sources.
I
apologize
that
lcs
is
not
on
the
map
over
here,
but
the
the
general
story
is
that
either
from
the
nature
of
the
light
sources
or
the
upgrades
are
coming
they're,
basically
taking
more
and
more
correlated
data,
so
they
have
to
invest
in
detectors
that
increase
the
overall
data
rate
because
they
are
area.
D
Or
they
measure
at
higher
frame
rates,
and
that
leads
to
like
a
very
high
data
volume
and
the
computation
required
to
analyze,
the
data
has
become,
has
become
increasingly
demanding
to
do
on
site.
So
looking
for
an
oscar
collaboration,
so
the
the
basic
thing
with
eod
facilities
that
the
workflow
isn't
doesn't
look
too
complex
at
first.
So
it
means
that
you
have
to
get
the
data
where
the
compute
facility
is.
D
This
also
relates
to
the
fact
that
they
that
the
work
at
light
sources
is
also
exploratory,
which
means
that
your
current
analysis
will
inform
your
your
experiments.
So
what
kind
of
steps
you're
doing
next
and
then,
of
course,
they
don't
want
to
be
surprised
that
you
know
nurse
is
unavailable
and
they
don't
want
to.
D
So
they
don't
want
to
have
you
know,
nurses,
exclusive
process
of
the
data
or
maybe
a
primary,
but
not
necessarily
exclusive,
and
they
want
to
move
to
different
hpc
facility
or
other
resources,
so
they
wanted.
The
nurse
should
be
available
in
their
workflow
really
as
an
accelerator
and
there's
also
other
aspects
that
come
with
license
workflows.
D
D
Here
so
that's
like
a
whole,
a
whole
lot
of
new
users
that
need
to
be
taken
care
of
in
the
in
this
frameworks,
and
they
also
want
the
data
to
move
in
and
out
pretty
quickly.
I
think
this
is
already
pretty
well
established
with
esnet
and
science
dmz.
But,
yes,
all
these
kind
of
aspects
tied
together
in
in
the
requirements
for
eod
workflows
at
hpc
facilities.
D
D
Then
they
want
to
push
the
data
to
nurse
when
they
experiment
its
life
and
they
want
to
run.
You
know
their
jobs
cause,
I
think,
with
data.
So,
let's
you
know,
submit
the
jobs,
api
or
check
if
your
reservations
up
and
maybe
have
make
some
you
know,
correlating
work
so
that
or
that
you
know
one
step
depends
on
the
other
and
then
yeah.
They
also
want
to
get.
D
You
know
feedback
from
the
from
the
work
they're
doing
so
they
might
want
to
transfer
something
out
but
still
running,
or
they
want
to
do
arbitrary
commands
and
yeah.
Once
you
are
done
with
everything
you
need
to
store
it
somewhere
and
you
need
to
distribute
the
data
and
send
it
out
to
collaborators
all
these
points
again
touch
the
transfer
and
the
file
api
and
then
there's
a
lot
of
other
things
within
the
super
facility
framework
like
hosting
services
and
spin,
and
so
that's
also
important.
D
So
these
all
these
kind
of
things
need
to
time
together
to
enable
these
workflows
thanks,
so
the
just
to
show
another
science
engagement,
so
we're
not
only
targeted
for
like
a
specific
user
base.
So
the
other
thing
is:
where
is
the
ruben
observatory
dark
energy
science
collaboration,
which
is
like
a
900
member,
strong
collaboration?
D
D
Is
that
if
you
have
a
that,
they
don't
only
want
to?
You
know,
look
at
the
data,
but
they
also
want
to
say
you
know
you
want
to
run
simulations
to
better
understand
the
instrument
or
the
physics
they
want
to
combine
it
with
data,
that's
not
not
only
from
that
observatory
and
yeah,
and
they
want
to
also
have
like
a
sharing
platform.
So
if
you
have
the
the
facility
external
to
the
the
place
where
the
data
is
is
kept
in
the
first
place,
it
enables
you
to
do
all
these.
D
So,
just
to
show
you
how
how
similar
the
dark
energy
science
collaboration
workflow
is
to
what
the
light
source
is
doing
is
the
same
in
the
same.
In
the
same
way,
they
have
to
figure
out
if
they
want
to
do
if
they
want
to
run
the
compute
campaigns
or
their
jobs,
did
you
figure
out
if
nurse
gets
up
and
if
they
can
actually,
if
they
can
actually
do
their
work,
and
then
there's,
I
think,
within
the
supernova
project,
they
want
to
basically
measure
all
these.
D
D
So
they
need
to
transfer
the
data
in
from
the
central
storage
location,
and
then
they
have
to
submit
the
right
compute
jobs
on
top
of
that,
and
because
it's
a
very
large
collaboration,
they
also
want
to
you
know,
stay
ahead
of
all
these
processes
and
services,
so
they
want
to
have
like
all
these
checking
and
control
of
user
quarters.
In
addition,.
E
A
E
E
E
So
this
is
the
iris
ui.
It's
a
new
section
in
the
iris
interface
called
super
facility,
api
charts.
We
got
a
little
bit
cut
off
sorry,
so
if
you
click
new
job,
you
get
a
dialog
where
you
can
fill
out
some
details
of
how
you
want
to
use
this
token,
the
more
restrictions
you
add
the
longer
lifetime
your
job
can
have,
and
if
you
click
create
job,
it
will
go
over
to
the
super
facility
api
site,
where
you
may
be
authenticated
again
and
will
return
with
this
encrypted
text
and
that's
your
job.
E
E
E
E
E
E
That
you
can
set
on
it
we'll
show
you
later,
where
the
documentation
as
you
can
see,
but
you
can
tell
you,
can
see
in
the
output
that
the
job
is
now
running.
It's
running
on
the
real-time
queue,
so
it
runs
relatively
quickly,
but
I'm
still
going
to
fast
forward
a
little
bit
because
it
takes
a
couple.
E
E
E
E
E
Another,
so
that's
that's
a
basic
workflow
that
you
can
do
today
using
the
super
facility
api
and
as
far
as
there
are
a
few
people
acknowledged
at
the
end,
but
I'd
like
to
also
add
shreyas
for
helping
out
with
the
new
questions.
Shane
worked
with
me
on
the
ssh
proxy
chalk.
Integration
and
mark
dave
was
very
helpful
with
the
shivan
issues.
E
So
I
mentioned
earlier
that
we
have
documentation,
the
documentation
actually
comes
from
swagger
annotations
in
the
source
code,
so
so
it
should
always
be
up
to
date,
and
it's
also
dynamic
and
interactive.
So
you
can.
If
you
go
to
this
url,
you
can
try
all
the
api
calls.
Some
of
them
have
not
are
not
implemented.
So
callbacks
are
not
going
to
do
much
and
reservations,
don't
work,
but
all
the
others.
You
should
be
able
to
interact
with.
B
Yeah,
so
I
want
to
leave
a
lot
of
time
for
questions
and
and
discussion.
I
see
there's
some
chat
here
about
sharing
the
sharing
this
notebook,
so
we
we
do
recognize
it's
a
little
bit
of
irony
to
to
give
a
whole
lead
into
the
talk
about
how
unattended
operation
and
not
having
a
human
in
the
loop
is
is
the
kind
of
the
driving
impetus
here
and
then
we
just
gave
our
demo
in
in
jupiter,
but
you
know
for
anyone
else,
who's
in
love
with
jupiter.
You
know
it's.
B
It's
really
good
as
sort
of
a
classroom,
presentation
tool,
so
imagine
that
same
code,
logic
running
you
know
overnight
or
you
know,
while
you're
doing
other
things
is
the
idea
so
and
then,
as
far
as
timing
too,
so
we
we
do
have
prototypes
of
the
api
that
are
running,
we're
doing
an
internal
review
of
the
system
at
nursk
and
making
some
final
final
tweaks
to
the
authentication
model
to
make
sure
that
it
sort
of
fits
well
with
other
libraries
that
are
available.
So
it's
easy
to
develop
against.
B
So
so
we
don't
have
it
rolled
out
just
yet,
but
we're
working
on
those
final
internal
reviews
and
putting
together
our
rollout
plans
where
people
could
actually
try
it
out.
So
just
it's
just
an
internal
testing.
Right
now
and
let's
see
I
see,
there's
some
questions
popping
up
on
the
chat.
I
think
we
should
be
able
to
share
in
the
notebook.
Like
I
said
in
the
chat,
it's
it's
a
pretty
thin
wrapper
and
most
of
that
block
at
the
top.
That
just
divide
defines
some
kind
of
convenience.
B
E
B
Yeah,
which
is
a
perfect
segue
into
the
latest,
question
that
just
popped
in
there
what's
the
benefit
of
using
this
api
instead
of
the
regular
ssh
session,
so
yeah,
probably
a
little
bit
of
subjectivity
in
the
answer
there
and
I'll,
let
others
chime
in.
But
one
of
the
big
things
I
think
is
from
a
security
standpoint.
B
B
And
so,
when
you
see
this
on
on
other
systems
too,
like
a
one
of
the
examples
that
we
kind
of
discussed
in
this
was
with
github.
So
you
can
create
api
tokens
with
github,
but
if
you
reduce
the
privilege
that
the
api
token
has
making
it
read
only
so
you
can't
make
any
actual,
you
know
code
changes
or
do
commits.
B
E
Sure
yeah
I
I
would
encourage
people
to
go
to
the
url,
which
is
not
visible,
but
maybe
I
can
go
back
to
that
url
and
then
see
what
the
transfer
api
does.
It's
essentially
a
wrapper
over
the
globus
scripts
that
lisa
wrote
that
are
available
on
corey,
so
it
lets
you
initiate
the
transfer
using
globus
and
then
the
other
call,
I
believe,
lets
you
check
on
the
status
of
the
transfer.
B
B
If
there's
something
simpler
and
better,
so
I
think
our
view
is
that
there
are
situations
where
ssh
is
the
tool
you
need,
and
there
are
situations
where
ssh
is
the
only
tool
you
have
and
some
alternative
would
be
better.
That's
the
gap
that
we're
trying
to
fill
with
the
api.
So
if
you're
coding
something
up,
then
you
know
making
it
work
against
an
api
that
kind
of
fits
modern
conventions.
We
hope
is
easier
for
for
people
to
use.
I
don't
know,
does
anyone
else
want
to
chime
in
on
on
that?
B
E
Yeah,
so
in
the
demo
there's
only
one
jot
there.
There
is
no
refresh
and
access
token
that
there's
only
an
access
access
token,
but
in
in
the
future
we
will.
You
know
we're
heading
towards
having
two
of
these
tokens
and
using
them
according
to
the
oauth
2
specification,
but
in
the
current
version
they
don't
exist
yet.
B
C
A
B
B
B
B
B
As
you
see
the
version
number
on
the
api
right
there,
a
big
shiny
v1,
and
so
we
we
do
know
that
maintaining
stability
there
is
is
critical
and
we
also
recognize
we
can
more
easily
do
that
if
we
kind
of
draw
a
line
of
demarcation
at
the
api
and
its
contract
versus.
If
we
extend
into
you,
know
a
provided
stk.
D
B
B
So,
right
now,
when
you
log
into
iris
you're
using
the
nurse
login
page,
essentially
it's
using
saml
and
we
are
working
on
also
as
part
of
the
super
facility
sort
of
umbrella
of
projects,
the
ability
to
support
federated
authentication
via
a
set
of
other
identity
providers,
a
home
institution.
So
if
you
use
other
distributed
systems
where
you
essentially
choose
your
login
page,
that's
the
kind
of
functionality
that
we
want
to
introduce
as
part
of
our
federated
identity
initiative.
B
So
those
authentication
mechanisms
are
all
built
with
web
protocols
in
mind
and
and
human
interaction
in
mind.
So
the
strategy
in
order
to
be
compatible
with
that
is
number
one.
It's
would
be
the
refresh
token
to
be
delivered
via
a
human
interface
like
iris,
but
to
sort
of
compensate
for
that,
we
have
these
capabilities
to
scope
the
refresh
jaw
so
that
it
can
have
a
long
life.
So
the
kind
of
pattern
that
we
see
evolving
is
yes,
you
log
in
as
a
as
a
human
or
someone
else
on
the
project.
B
B
These
are
good
questions,
I'm
curious.
If,
if
there's
anyone
who's,
you
know
seen
the
talk
here
and
is
thinking
about
well,
something
that's
missing.
You
know,
like
my
workflow,
does
something
else,
then
I
don't
see
anything
in
here
native
to
handle
that
we
would
be
really
interested
to
know
what
kind
of
functionality
we
should
be
thinking
about
for
future.
B
E
B
B
B
I
mean,
I
think
so
as
we
as
we
start
to.
I
don't
have
an
answer
for
you
at
the
ready,
but
as
we
start
to
roll
this
out,
we
would
expect
to
take
questions
via
the
normal
channels,
and
so
I
think
what
will
probably
happen
initially
is
some
of
those
questions
will
either
feed
into
our
process
to
enhance
the
documentation
or
if
there
are
actual
problems
or
bugs
or
oversight
in
some
of
the
how
the
functionality
works.
Then,
questions
like
that
could
get
dispatched
into
my
group.
B
For
you
know,
people
like
moore
to
look
at
or
some
of
the
other
folks
in
the
super
facility
project,
so
yeah
we
do
intend
to.
You
know,
provide
some
level
of
support
for
working
against
this,
because
we
also
want
to
hear
the
feed
back.
As
far
as
you
know
whether
things
are
working
well
or
what
are
there,
problems.
B
Oh
heather,
we
don't
quite
have
a
rollout
date
yet
because
we're
doing
some
internal
review,
but
I
would
expect
we
should
be
able
to
make
some
of
these
endpoints
available
like
over
this
summer,
they're
the
one
particular
endpoint
that
is
maybe
useful
to
folks.
Even
now
that
doesn't
require
authentication.
So
it's
less
of
an
issue
to
make
public
is
the
health
api
which
you
can.
You
can
tell
the
status
of
individual
resources
or
the
sender
and
aggregate,
and
also
see
scheduled
maintenance
ahead
of
ahead
of
when
it's
currently
happening.