►
From YouTube: 2022-04-26 - Juan Pablo Dorsch - FirecREST, RESTful HPC
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title:
FirecREST, RESTful HPC
Speaker:
Juan Pablo Dorsch, HPC Software Engineer & Lead for the Innovative Resource Access Methods, CSCS Swiss National Supercomputing Center
Abstract:
FirecREST is a RESTful API to HPC that empowers scientific communities to access compute and data HPC services and infrastructure through a web interface. This API supports and enhances the development of scientific portals that allow web developers and HPC users to adapt their workflows in a more flexible, secure, automated, and standardized way. In this talk, we will present FirecREST and provide an introduction to its capabilities.
Bio:
Juan Pablo Dorsch is a software engineer and lead for the Innovative Resource Access Methods at the CSCS Swiss National Supercomputing Centre. His areas of expertise include microservice architecture design, IAM, web development and RESTful services. Before joining CSCS, Juan held the position of HPC engineer with the Computational Methods Research Centre (CIMEC), and the position of scientific software developer with the International Centre for Numerical Methods in Engineering (CIMNE). He was also previously a degree professor at the National University (UNL) of Littoral in Santa Fe, Argentina. He holds a degree in Informatics Engineering with an emphasis on scientific applications from UNL.
Host of Seminar:
Jonathan Skone, Advanced Technologies Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
Today
we
have
with
us
juan
dorsch
who's
with
cscs
swiss
national
supercomputing
center
in
switzerland,
and
he
currently
is
a
software
engineer
there
and
he's
been
working
with
the
firecrest
project,
which
is
the
api's
hpc
service
and
he's
going
to
give
us
a
update
about
efforts
along
those
lines
and
give
some
but
general
background
as
to
the
the
capabilities
that
it
enables.
So
with
that
go
ahead,
one.
B
A
B
B
B
So
you
can
see
that
we
have
around
2003
users
or
projects
we
run.
We
help
institutions
like
eth
series,
the
polytechnic
from
lausanne,
the
university
of
students
burn
bustles.
You
need
basically
all
the
universities
in
in
switzerland
and
we
have
different
research
fields
and
all
the
applications
have
this
kind
of
research
field,
chemistry,
materials.
B
B
So,
first
of
all,
I
will
show
you
an
example
of
a
project
that
I
was.
I
was
working
since
the
beginning
in
in
cscs,
since
2019.
the
project
is
called
celveda.
It
stands
for
service,
large
scale,
volume,
experiment,
data
analysis,
a
long
name,
and
we
would
interface
it's
a
collaboration
between
the
or
share
institute
ps9,
as
I
mentioned
before,
and
cscs
is
a
swiss
university.
A
B
This
is
basically
an
infrastructure
branch,
a
machine
that
analyzes
surface
of
objects
in
a
really
high
resolution
to
get
information
about
the
structures
and
the
properties
of
the
materials
that
compose
the
surface.
So
they
can
have
like
a
protein
crystals
that
are
important
for
pharmaceutical
research
or
magnetic
properties
for
new
source
technologies
and
so
on
and
so
on
and
so
on.
B
Is
analyze
objects
in
a
in
a
pipeline
right?
They
call
the
beam
line,
so
they
have
a
thing
called
the
typography
scan
that
takes
image
of
different
resolutions
and
they
do
experiment
with
that
information.
Those
those
images.
They
are
really
really
big,
we're
talking
about
sizes
of
gigabytes
and
television.
B
So
the
thing
is
that
people
from
bsi
designed
if
they
have
cut
they
have
a
window,
the
time
frame
time
to
use
the
beamline,
because
they
all
want
to
use
the
typography,
but
they
don't
have
the
time
to
do
it.
So
they
need
to
do
the
analyze
of
the
objects
in
the
machine
and
then
the
past
people
experiment
in
an
hpc
system
that
they
have
really
fast
in
order
to
find
in
these
images
region
of
interest,
to
do
another
integration
with
typography
in
order
to
get
crosstalk
right.
A
B
They
need
to
do
this
as
fast
as
possible
to
get
the
time
window
from
the
from
the
beam
line,
but
the
problem
is
that
the
computer,
the
hpc
system
they
have
in
psi,
is
not
fast
and
not
powerful
enough.
They
need
a
super
computer,
so
that
is
when
cellular
starts
to
contact.
Yes,
yes,
so
when
psi
starts
to
contact,
cs,
cscs
and
cellular
starts
so
serverless
is
the
project
that
will
provide
the
workflow
between
the
psi,
beamline
and
cscs
computational
resource.
B
B
With
this
first
of
all,
that
and
users,
they
need
to
integrate
hpc
into
into
their
scientific
work
into
their
advances
and
difficult.
It's
not
just
submitting
a
job,
they
need
to
do
a
lot
of
different
things,
and
this
is
challenging
for
users,
because
we
don't
have
a
development
standard
to
work
with
hpc.
B
B
That's
just
you
know,
but
you
that
you
can
understand
and
and
at
the
same
time
you
will
create
your
own
workflow
and
if
another
guy
has
the
same,
if
another
project
won't
do
the
thing
you
will
have
to
create
another
one
and
that
will
be
growing
and
growing
when
people
want
to
use
more
and
more
advanced
workflows
in
hpc.
So
that's
a
problem
because
we'll
have
multiple
growth
loops
that
will
lead
to
multiple
support
approaches.
B
Always
have
a
problem
supporting
it
so
that
that's
not
a
scale
right
at
the
same
time
to
allow
this
advanced
and
this
workflow,
we
need
a
modern
interface
for
what
we
have
in
hpc
resources.
So
it's
not
the
same
for
for
users
to
connect
to
one
cluster
to
another.
Maybe
they
have
a
basket.
Maybe
they
have
another
way
to
connect?
Maybe
they
need
to
use
ssh
credentials?
Maybe
they
they
allow
to
use
just
public
keys.
B
I
don't
know
there
is
different
way
to
access
platforms,
so
we
need
to
abstract
the
access
to
the
classes
and
we
need
to
extract
also
the
object
and
the
resource
of
the
class
the
same
for
job
schedulers.
It's
not
the
same
for
a
user
to
execute
a
command
in,
learn
that
in
torque,
maui
and
pbs
or
oge
there's
a
lot
of
different
tools.
B
A
B
B
Have
another
inversion,
you
have
another
different
hpc
centers,
so
the
idea
is
to
standardize
this
component,
this
npc
resource
and
also
interfacing
with
what
we
already
have
in
our
hpc
services
and
new
projects.
We
want
to
create
something
that
uses
what
we
have.
What
do
we
already
have
in
in
our
infrastructure
avoid
to
build
something
avoid
to
reinvent
the
the
and
be
as
lightweight
as
possible?
B
We
got
there,
we
have
to
address
all
those
issues,
some
problems
and
and
try
to
do
something
that
provides
us
of
a
solution
for
advanced
work,
and
we
came
with
this
definition
of
cycles
at
the
end
of
of
developing
a
lot
of
fibers.
But
fireplace
is
a
wave-enabled
api
to
hpc
resource,
so
this
tries
to
present
a
standard
programming
interface
that
is
based
on
the
concept
of
red
api,
the
standard
of
rest
api.
So
you
have
here
the
concept
of
a
standard,
http
communication.
B
B
B
The
idea
is
to
translate
web
requests
into
hpc
business
logic
into
hpc
command,
hpc
data
mover
and
everything
that
is
need
to
be
done
for
doing
this.
This
scientific
workflow
at
the
same
time,
that
response
is
a
web
friendly
format
like
when
it's
not
json
or
xml
or
someday
some
response
data
format
that
can
be
understood
and
can
be
easily
used
by
a
developer
to
pass
the
data
in
the
web
application
at
the
same
time
with
cyclists.
We
want
to
provide,
by
using
this
standard
program,
interfacing
web,
providing
web
interface
for
classic
hpc.
B
We
now
can
have
a
lot
of
different
devices,
support
from
multiple
devices
to
face
hpc
resources,
and
this
is
really
interesting,
because
we
can
have
now
hpc
communication
with
devices
like
smartphone
and
like
tablets,
and
this
is
something
that
brings
more
people
to
use
hpc.
So
that's
another
of
the
of
the
objectives
of
this
kind
of
work.
A
B
B
A
B
A
B
B
B
Center
on
the
system,
so
you
can
see
all
of
this
is
pretty
standard.
So
a
developer
can
see
that
we
can
provide
that
you
can
provide
the
methods,
which
is
the
get
methods
that
you
have
parameters.
So
you
have
a
machine
name.
You
put
here
the
machine
name
and
you
will
get
the
information
in
the
response,
and
you
see
that
the
response
are
all
standard
ones
like
200.
B
And
you
see
that
the
different
different
verbs
that
are
used,
useless
and
methods
are
used.
They
all
are
related
to
the
actions
that
they
perform.
So
you
can
get
is
for
listing
something
you
don't
do
any
action
more
than
get
the
status
of
something
like,
for
instance,
the
director.
So
you
will
get
information
you
will
get.
You
will
get
this
kind
of
response
and
in
the
type
of
a
list,
because
you
are
listening
directory
and
you
will
have
this
information
in
order
to
program
your
advanced
workflow,
your
web
application
over
hpc.
B
We
use
spots
for
creating
things.
Some
everything
that
is
is
used
by
a
post
put
is
a
verb
for
changing
the
status
of
a
resource.
In
this
case,
let's
say:
changing
the
the
permission
of
files
and,
of
course,
delete.
If
you
want
to
give
it
something,
this
utilities
endpoint
is
used
for
a
small
non-blocking
post,
posix
unix
operation,
and
you
can
see
that
they
are
mapped
to
specific
command.
B
B
You
should
assume
my
theories,
of
course,
the
job
submission
and,
let's
say,
scheduler
workload,
manager,
endpoint
and
the
idea
here
is
almost
the
same
for
submitting
a
job,
creating
something
that
runs
in
your
scheduler,
and
you
see
here
that
if
we
use
the
example
of
a
local
smart
files,
this
is
an
abstraction.
So
we
can
map
with
this
idea.
We
can
map
different
schedulers.
This
is.
B
That
we
are
working,
you
can
list
the
information
of
the
of
the
jobs
and,
of
course
you
need
to
cancel
a
job
and
get
accounting
information.
Such
we
have
two
different
ways
of
managing
storage.
One
is
for
the
internal
okay.
This
is
moving
data
from
one
health
system
to
another
five
system
chain,
hpc
environment,
that
you
can
see
that
can
be
used
with
different
commands.
Here
then,
we
have
the
external,
which
is
when
you
want
to
transfer
data
from
your
from
a
remote
server
to
your
laptop
of
vice
versa.
B
B
B
A
B
We
have
in
this
bottom
half
in
the
left
side
the
parameters
of
the
problem
like,
for
instance,
how
many
nodes
we
want
to
use
just
how
many
steps
this
job
will
have
and
the
base
name.
That
will
be
appended
later,
the
name,
the
number
of
step,
of
course,
and
the
partition
in
this
and
the
constraints
we
have
in
the
gpu
and
multicore
so
cb
or
mc.
You
can
use
one
of
them
and
when
you
click
the
button
submit
your
all,
these
jobs
are
spawned.
B
B
B
B
So
this
visual
table-
and
if
you
know
some
html,
can
is
basically
mapped
into
a
html
table
where
we
have
all
the
headers,
with
the
columns
of
the
data
that
we
want
to
show
you
right.
So
all
these
are
related
to
our
data
fields
that
we
have
to
go
from
a
data
url,
and
so
you
just
have
to
program
in
your
backend
this
this
url-
and
this
is
actually,
if
you
know,
boost
cloud
table.
This
is
just
a
booster
table,
so
automatically
helps
you
to
to
refresh
the
data
from
a
url
automatically.
B
B
It
passes
the
name
of
the
machine
in
a
header,
because
that's
how
the
specification
of
the
api
tells
you
to
do,
and
then
you
get
the
information
cycles
here
in
this
request.
This
is
I'm
just
using
python.
So
if
you
want
to
know
how
is
this
done?
It's
a
flask
library
which
I
really
like
to
use
in
is
the
same.
If
you
do
this
in
jboss
with
java,
if
you
do
this
with
php,
if
you
do
this
with
bash,
you
will
have
to
change
some
specific
things.
B
So
what
it
actually
do
is
calling
flightcrest
saying
data
that
cycle
gives
you
in
a
json
is
mapped
every
15,
updated
and
in
the
15
seconds,
moved
and
populated
here
in
the
table.
So
you
see
that
how
easy
and
how
I
don't
know
it's
easy,
but
you
have
to
follow
some
specific
steps
and
these
steps
are
really
all
standard
and
the
abstraction
are
quite
clear
for
understanding
what
we
want
to
achieve
with
cyprus
and
the
specific
web
portal
in
this
case.
B
B
B
B
A
B
So,
first
of
all,
this
is
the
psi
point
of
view,
so
the
scientific
point
of
view,
etc.
Scientific
point
of
view.
They
will
create
the
web
portal
that
interacts
with
the
psi
beamline
at
the
same
time
with
the
flycross
api.
So
for
them
you
will
be
sending
the
object
to
be
to
be
analyzed
in
the
typography.
Getting
all
the
information
and
all
the
all
the
images
and
then
send
the
images
to
parkours
and,
like
a
black
box,
automatically
turn
the
analysis
in
a
url,
so
they
will
download
like
a
third
flat
file.
B
B
You
know
errors
and
success
methods
and
of
course
we
can
provide
that
because
we
have
http
responsible
and
http
responsive
status,
but
they
have
this
information
and
they
could
make
all
of
these
pipelines
to
work
and
once
the
data
is
stored
in
they
need
to
start
the
analyze.
So
this
will
do
the
job
with
the
compute
endpoint
they
just
use.
B
The
analysis
of
the
data
will
be
the
original
interest
that
they
need,
so
they
triggered
the
download
and
what
ficus
does
is
moving
the
data
already
analyzed
that
the
user
needs
to
a
station
area,
object,
storage,
station
area
like
a
temporary
storage
place
and
then
micros
will
inform
automatically
that
the
download
temporarily
download
your
url
is
exposed
for
the
site.
So
the
scientist
will
have
the
portal
and
at
some
point
they
will
have
like
a
message
saying
that
your
url
is
here.
B
But
all
of
this
is
really
nice
and
all
everything
works
like
smoothly,
let's
say,
but
we
have
to
understand
how
packets
work
internally,
to
give
approach
of
you
know
some
security
concepts
and
this
kind
of
things
like
the
station
area,
and
I
will
show
you
because
this
is
the
most
boring
part
of
the
presentation.
Yes,
there
is
a
more
boring
part
of
the
presentation
this
one,
so
the
very
simplified
approach
that
we
have
with
with
cyclists
is
that
you
have
as
a
developer.
B
So
this
is
a
very
simplified
approach
as
it
says
here,
but
you
might
first
think
okay,
but
I
can
execute
something.
But
how
can
I
actually
access
files?
How
does
cycles
authenticate-
and
this
is
really
interesting,
because
spiker
doesn't
do
all
the
authentication
we
rely,
as
I
said
before,
on
an
iam.
A
B
B
B
They
can
be
used
only
if
we
trust
in
the
source
of
the
token-
and
we
don't
have
to
do
any
authentication
with
the
with
the
with
the
application
we
just
trust
in.
In
the
token,
in
this
case,
the
of
course,
the
the
the
developers
and
the
scientific
application.
What
they
have
to
do
is
register
their
application
on
on
oidc
service.
In
this
case,
we
have
key
club.
This
will
depend
on
your
ydc
server
in
in
your
in
your
sender,
and
you
can
rely
on
external,
also
external
services.
B
What
we
only
ask
is
a
jwp
token,
with
a
specific
configuration,
which
is
the
standard
configuration
of
the
token
and
yeah-
that's
pretty
much
it,
because
we
also
configure
ficus
in
a
way
that
we
actually,
we
accept,
only
token
from
a
specific
server.
In
this
case,
we
should
only
drag
in
our
kickbox
server,
but
at
the
same
time
we
have
a
specific
configuration.
So
if
you
have
another
token
from
the
same
key
club
server
that
we've
done,
we
don't
allow
anything.
We
just
load
the
ones
that
are
for
specific
cycles.
B
How
how
parker's
know
that
I'm
executing
something
in
games,
for
instance,
or
in
any
hpc
system?
Well
in
this
result,
we
have
issued
translation.
We
take
the
token
we
take
the
information
from
the
token
that
is
verified
and
we
extract
user
information
to
create
user
credentials
that
are
capable
of
x
being
executing
hptp.
B
In
this
case,
we
use
ssh
certificate,
so
we
have
certificate
authority
and
the
essay
certificate
has
really
interesting
features
that
we
like
like
this,
that
we
like
from
there,
which
is
that
we
can
create
for
a
short
period
of
time.
We
just
have
one
minute
the
credential
chart
is
the
certificate
is
created
for
the
value
of
one
one
minutes
and
we
can
inject
a
specific
comma
with
electric
business.
So
if
you
for
some
reason
slowest
certificate
here,
we
will
have
a
really
big
problem
here.
B
But
let's
say
that
there
is
something
there
that
can
can
get
a
specific
certificate.
Then
you
just
can
use
it
for
a
specific,
specific
command
and
specific
parameters,
and
these
comments
are
written
like
by
fact,
and
the
execution
is
not
from
you
to
fight
yourself
from
the
cyclist
service
to
the
machine.
But
on
your
website,
so
we
don't
do
any
pseudo
or
root
or
any
kind
of
execution
of
that
kind.
The
certificate
has
your
name,
so
you
are
executing
the
code.
So
if
you
execute
anything
there,
you
cannot
execute.
B
B
Of
data
that
needs
to
be
transported
from
external
server
to
our
internal
service
and
we
have
to
provide
a
non-blocking
response
to
the
to
the
call.
So
what
we
do
is
detach
the
the
communication
for
the
api
one
side
and
the
transfer
in
another
channel
and
for
doing
that.
What
we
do
is
moving
data
from
external
and
from
internal
things
to
a
staging
area.
B
B
We
don't
have
anything
against
them,
but
this
this
will
drive
the
people
to
have
to
install
this
kind
of
software
in
their
laptops
or
in
the
servers
where
the
web
application
runs,
and
we
don't
want
to
do
it.
So
what
we
want
is
that
everything
will
be
as
http
as
possible
as
a
standard
as
possible,
and
we
achieve
that
with
with
this.
B
B
The
graphic
and
we
have
to
use
it
because
it's
it's
really
interesting.
The
idea
of
cycles
at
the
beginning
was
removing
from
the
bottom
half.
You
know
this
disaster
when
we
have
different
workflows
for
different
users
that
are
executed
in
different
ways,
and
we
have
to
manage
in
different
ways
in
the
hpc
provider
side
to
a
more
advanced
integration
for
different
workloads.
B
A
A
A
A
B
B
B
Provides
of
course,
the
link
to
the
github
repo,
which
is
not
only
the
code,
but
you
have
a
demo
environment
there
when
you
can
with
docker,
compose
or
kubernetes.
If
you
have
mini
cube
micro,
kubernetes
or
if
you
have
a
kubernetes
cluster,
you
can
deploy
ficres
fully,
I
mean
with
all
the
components,
even
including
key
clock,
cluster
chpc
system
and
also
a
minimum
as
an
object,
storage
and
a
lot
of
components.
The
demo
application
you
have
test,
you
can
check
check
it
out.
It's
really
interesting
and,
of
course,
the
documentation.
B
B
The
definitions
of
the
method
and
the
possible
status
cost
that
http
response-
and
you
have
here,
for
instance,
examples
in
dash
and
in
python,
how
you
can
check
some
endpoints
and
get
information
like
police
and
listing
files
and
stuff
like
that,
and
also
we
have
fly
fighters,
which
is
a
python
client,
because
we
know
that
sometimes
during
these
kind
of
things
in
in
hdb,
like
requests
here,
it's
a
little
bit
tedious.
So
we
use
a
wrapper,
a
python
wrapper
when
for
leasing
a
file.
B
You
just
have
to
do
this,
something
like
clients,
dot.
These
files
send
the
the
directory
here
and
the
other,
but
the
first
parameter
is
the
system
that
you
want.
You
want
to
see
and
you
will
have
all
the
information
of
the
of
the
output.
I
think
it's
easier
to
use
than
this.
I
use
it
a
lot
and
I
really
like
pipefire.
So
if
you
want
to
see
it,
please
check
it
out
and
that's
it.
That's
it
from
my
side.