►
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title:
funcX: Federated FaaS for Scientific Computing
Speaker:
Ryan Chard
Data Science and Learning Division
Argonne National Laboratory
Abstract:
Exploding data volumes and velocities, new computational methods and platforms, and ubiquitous connectivity demand new approaches to computation in the sciences. These new approaches must enable computation to be mobile, so that, for example, it can occur near data, be triggered by events (e.g., arrival of new data), be offloaded to specialized accelerators, or run remotely where resources are available. They also require new design approaches in which monolithic applications can be decomposed into smaller components, that may in turn be executed separately and on the most suitable resources. To address these needs we present funcX—a distributed function as a service (FaaS) platform that enables flexible, scalable, and high-performance remote function execution. funcX's endpoint software can transform existing clusters and supercomputers into function serving systems, while funcX's cloud-hosted service provides transparent, secure, and reliable function execution across a federated ecosystem of endpoints. We demonstrate the use of funcX with several scientific case studies and show how it integrates into the wider Globus ecosystem to enable secure, fire-and-forget scientific computation.
Bio:
Ryan Chard joined Argonne in 2016 as a Maria Goeppert Mayer Fellow and then as an Assistant Computer Scientist in the Data Science and Learning Division. He now works with Argonne, UChicago, and Globus to develop cyberinfrastructure to enable scientific research. In particular, he works on the Globus Flows platform to create reliable data analysis pipelines and the funcX service to enable function serving for HPC. He has a Ph.D. in computer science and an M.Sc. from Victoria University of Wellington, New Zealand.
Host of Seminar:
Jonathan Skone, Advanced Technologies
Bjoern Enders, Data Science Engagement Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
It's
my
pleasure
to
introduce
ryan
shard
who's
works
for
globus
labs.
As
a
computer
scientist,
he
was
previously
a
argons
maria
gopert
mayor
fellow
and
then
became
an
assistant
computer
scientist
with
them,
and
he's
he's
had
affiliations
with
argon
uchicago
and
his
current
company
globus
and
he's
primarily
worked
in
this
cyber
infrastructure
development
space
to
further
enable
and
help
folks
in
the
scientific
computing
area
enable
their
research.
A
B
C
A
little
bit
sick
over
the
last
few
days
and
was
contemplating
whether
or
not
I
should
do
this
one,
but
I
was
really
looking
forward
to
giving
another
talk
here,
so
this
has
worked
out
well,
so
yeah
yeah.
So
as
jonathan
was
saying,
I
work
quite
closely
with
lobus
labs.
I'm
actually
I'm
a
contractor
back
to
the
lab
at
the
moment
based
in
new
zealand.
So
I'm
probably
about
5000
miles
away
from
you
guys,
but
I
continue
working
on
projects
with
globus
and
globus
labs
in
particular.
C
So
just
for
a
little
background
there,
globus
labs
is
sort
of
a
research
arm
of
globus,
so
you're,
probably
familiar
with
globus.
The
data
management
service
that
ian
foster
and
rational
co,
lead
and
yeah
global
slab
sort
of
our
mission
is
to
research
and
develop
different
data
management
services
to
enable
scientific
computing,
and
then
our
work
often
feeds
back
into
the
globus
product
line.
C
So
I've
got
two
examples
in
this
talk
today,
one
is
relating
to
funkex
and
then
a
little
bit
later
I'll
get
to
sort
of
how
this
fits
into
the
globus
ecosystem
and
bring
up
our
pipelining
tool.
Our
sort
of
automation,
service
called
globus
flows,
which
is
another
effort
that
we've
prototyped
and
then
turned
into
a
product
at
globus.
C
So
just
quickly,
my
my
role
in
the
funkyx
team
is
essentially,
I
built
the
initial
prototypes
of
funkyx,
so
I'm
pretty
familiar
with
how
it
all
clips
together
after
sort
of
showing
some
examples
of
success
with
this
and
plugging
it
into
various
scientific
cases.
Around
argon
around
the
lab,
we
managed
to
get
a
nsf
grant
which
allowed
us
to
employ
some
developers
to
come
and
fix
what
I
had
built
so
yeah.
C
Now
we've
got
a
fairly
substantial
team
of
people
working
on
this,
bringing
it
together
to
a
production
service
that
can
service.
Many
thousands
of
users
is
where
we're
heading
and
yeah
just
meet
the
scale
and
load
of
our
various
use
cases.
So
that's
where
I
fit
in
apparently
space
doesn't
work.
I've
got
a
quick
overview.
I
don't
really
like
overviews,
but
I
figured
there
was
quite
a
bit
going
on
in
here,
so
just
to
give
some
context
to.
C
What's
going
on,
I've
got
a
quick
backup,
background
introduction
into
sort
of
serverless
computing
and
how
that
works.
I've
been
introduced,
funkex
and
how
that
sort
of
fits
into
all
of
us.
I've
got
a
live
demo
that
I'll
I'll
show
you
how
we
can
work
through
this
and
you
guys
can
follow
along
if
you
like,
it's
hosted
in
binder,
so
you
can
actually
execute
it
all
in
your
your
web
browser
and
it'll
run
through
we'll
dive
into
some
of
the
implementation
details
which
I'm
more
than
happy
to
stop
and
feel
questions.
C
Very
brief
evaluation,
just
showing
that
things
run
pretty
quickly
and
then
sort
of
highlight
how
this
fits
into
the
globus
ecosystem,
with
some
example
use
cases,
and
then
the
lessons
learned
so
without
further
ado,
we'll
just
jump
straight
into
how
how
all
of
this
comes
together.
So
I'm
sure
most
of
you
are
probably
familiar
with
serverless
computing
and
this
concept
that's
really
taken
off
and
the
cloud
space.
C
Recently
I
say
recently,
I
think
lambda
functions
came
out
about
2011
so
10
years
ago,
but
this
is
just
a
model
for
providing
compute
capacity
where
the
user
doesn't
need
to
understand
the
infrastructure
or
be
responsible
for
managing
any
of
the
actual
underlying
resources.
So
they
can
simply
focus
on
the
actual
application
we're
trying
to
deliver
without
having
to
worry
about
halfing
scale
and
that
sort
of
thing
behind
the
scenes.
So
there's
two
real
benefits
to
this
is
one
you
don't
have
to
worry
about
the
upside.
C
You
can
save
some
money
by
not
doing
the
upside,
but
cloud
does
that
for
you
and
the
other
side
is
that
you
can
basically
just
oh
man,
I'm
completely
blank
on
where
I
was
going
just
here,
but
it
allows
you
to
really
just
focus
on
the
application,
so
you
don't
have
to
worry
about
all
this.
There's
huge
cost
savings
inside
alongside
it.
C
This
is
really
a
way
of
defining
these
lightweight
computations
or
very
small
computations,
typically
such
as
a
python
function.
You
write
this
function
code,
which
can
just
be
a
a
little
python
function
with
a
couple
of
supporting
modules
or
something
like
that
yeah,
you
register
it
with
the
cloud
provider
and
then
you
run
it
on
demand.
So
you
don't
care
about
provisioning,
the
the
actual
computers
for
this
to
execute
on
you
just
write
your
function
register.
It
then
run
it
and
you
can
run
it
at
almost
any
scale.
C
So
yeah,
it's
just
a
really
nice
easy
way
to
use
compute
capacity.
The
benefits
of
this
are
there:
it's
typically
extremely
low
latency,
so
you
can
support
websites
with
this.
It's
completely
on
demand,
so
you
run
it
whenever
you
want
to
run
it
and
the
cloud
will
sort
out
provisioning
your
your
infrastructure
to
actually
execute
it
and
it
will
elastically
scale.
So
you
can
run
it
once.
You
can
run
it
a
million
times.
The
cloud
will
sort
it
out.
Your
jobs
will
come
back
in
a
nice
way.
C
C
So
how
does
function
as
a
service
fit
into
the
the
scientific
world?
So
we
sort
of
see
three
real
advantages
to
it.
One
is
that
it
provides
a
way
to
support
this
sort
of
new
class
of
data
intensive
research
if
it
wants
to
do
on-demand
compute.
So
we
have
a
lot
of
users
that
generate
a
bunch
of
data
with
things
like
a
scanning
electron
microscope
or
a
synchrotron,
and
they
really
want
to
do
this
sort
of
event,
driven
or
stream
processing
sort
of
model
of
data
created.
C
So
this
is
talking
about
from
your
lap,
your
laptop
or
your
laboratory
cluster,
all
the
way
through
to
super
computers
and
clouds.
We
can
place
those
jobs
sort
of
as
needed
right,
so
we
can
put
them
wherever
it
makes.
The
most
sense-
and
we've
got
this
picture
on
the
right
here,
of
an
iceberg
which
is
really
just
trying
to
show
sort
of
from
the
scientists
perspective.
If
we
can
provide
this
sort
of
serverless
infrastructure,
they
only
need
to
care
about
that
that
top
10
percent
of
what's
actually
going
on
they.
C
They
just
need
to
pick
where
their
compute
goes
and
provided
it's
like
set
up
by
someone.
They
don't
need
to
care
about
the
infrastructure
underneath
it.
They
don't
need
to
worry
about
provisioning
nodes
on
a
on
a
supercomputer
or
making
sure
tasks
are
reliably
delivered
over
to
this
or
authenticated
and
everything's
securely
transferred.
They
just
care
about
their
simple
interface,
a
fire
off
a
job,
and
it
gets
done
so.
We
already
have
a
quite
a
number
of
examples
of
use.
Cases
that
are
that
are
moving
in
this
granular
direction.
C
Right,
like
I've,
got
a
few
examples
here
and
I'm
sure
you
guys
can
think
of
many
others
just
quickly.
Looking
through
these,
so
the
first
one
is
tabular
file
extraction.
So
this
is
a
metadata
extraction
task
and,
as
you
can
see,
with
these
latencies,
these
tasks
typically
take
less
than
a
second.
However,
we
want
to
apply
them
at
an
enormous
scale.
So
essentially
we
want
to
apply
them
to
all
scientific
literature
or
something
like
that,
and
it's
just
a
a
fairly
lightweight
computation.
C
We
just
need
the
tool
to
actually
execute
on
a
file,
so
it
runs
pretty
quickly.
So
it's
a
nice
way
of
sort
of
granularly
defining
these
things.
Tyler
schlosarczyk
the
student,
the
phd
student
working
on
this
stuff
he's
got
about
10
of
these
different
file
extraction
tools
like
image,
extractor,
so
on
and
so
forth
and
yeah.
He
can
define
each
of
these
as
a
function
and
then
run
them
on
demand,
as
it
reads
them.
Another
really
compelling
use
cases
with
machine
learning,
inference
tasks.
So
this
is
m
prediction
again.
C
It
takes
on
the
order
of
a
second
or
two
and
again
it's
something
that
you
can
spin
up
and
down
as
you
need
it,
because
you're
probably
not
doing
digit
prediction
constantly
these
last
three
ones
are
actually
use
cases
from
the
advanced
photon
source,
the
layer
bar
synchrotron,
the
first
one
is
dial
still
processing.
This
is
a
a
serial
crystallography
experiment.
C
It's
determining
the
diffraction
patterns
from
blasting
protein
crystals
with
x-rays
and
then
trying
to
work
out
the
structure
of
different
proteins,
so
that
takes
only
a
second
or
two
to
actually
process
these
tools,
processes,
data,
tomographic
reconstruction.
This
is
much
more
computationally
intensive,
where
we're
now
on
the
order
of
like
five
to
ten
seconds
and
then
xo
correlation
spectroscopy
takes
about
a
minute
which
is
a
bit
longer
and
to
keep
up
with
sort
of
real-time
generation
of
these
data.
C
So
I'm
sure
you
guys
are
probably
fairly
familiar
with
the
barriers
to
entry
for
high
performance
computing,
so
I've
got
a
picture
of
aurora
here,
which
obviously
has
some
pretty
good
barriers
to
entry.
But
all
joking
aside,
these
super
computers
often
have
a
pretty
big
task
for
users
to
actually
become
familiar
with
them
in
such
a
way
that
they
can
use
them
as
they
need
them.
So
I
I'm
mostly
familiar
with
the
computers
at
the
leadership
computer
facility
argon,
but
these
have
various
different
cues
associated
with
them.
C
The
computing
facility
is
actually
using
basically
there's
this
impedance
mismatch
between
short
duration
function,
workloads
and
existing
research
computing
infrastructure,
which
we
want
to
address
with
funkex.
We
want
to
provide
a
way
that
users
can
leverage
these
computing
resources
in
an
on-demand
manner
without
having
to
learn
all
of
this
stuff
themselves.
C
The
other
sort
of
side
component
of
this
is,
with
the
end
of
moore's
law,
rapidly
approaching.
We
really
need
to
establish
this
entire
continuum
of
computing
resources
and
provide
ways
that
you
can
flexibly
or
fluidly
compute
across
them,
so
you
can
deploy
jobs
to
your
edge
devices
or
your
laboratory
machines
or
your
hpc.
Really.
C
So
I've
got
an
example
here.
Jonathan
was
asking
if
we
work
with
these
guys-
and
the
answer
is
no-
but
this
is
a
really
cool
example
that
ian
came
across
and
we
we
use
as
just
this
motivational
sort
of
demonstration
of
why
this
stuff
is
so
important.
So
this
is
a
group
from
fermilab
they
do
top
quark
jet
tagging
and
neutrino
event,
classification
based
on
a
resnet
and
basically
they
as
neutrino
events
are
detected.
C
But
because
this
thing
is
so
much
quicker,
they
can
actually
outsource
their
data
and
their
computation
to
this
fpga
and
see
40
times
acceleration
to
actually
run
this
analysis
at
a
different
place.
So
this
is
just
really
highlighting
the
need
for
this
ability
to
provide
little
computation
so
just
quickly.
None
of
us
is
like
terribly
new.
C
C
Allow
you
to
use
your
credentials,
your
university
credentials
to
authenticate
at
different
laboratories,
computing
resources
and
people,
trust
that
you
are
who
you
say
you
are,
and
we've
got
containers
which
this
technology
that
really
allow
us
to
package
up
an
execution's
runtime
all
of
our
dependencies
and
modules,
and
move
that
somewhere
else.
So
it
gives
us
this
way
to
actually
abstract
where
this
thing
is
running
or
allow
it
to
run
in
different
places.
C
So
with
all
of
that,
I'm
now
at
funk
x.
So
funkyx
is
our
function
serving
ecosystem
that
we've
been
working
on.
Basically,
it
follows
a
similar
model
to
things
like
lambda,
where
you
would
register
your
function
with
the
the
service
and
then
the
differences
in
the
in
the
lambda
world.
You
would
then
not
know
where
that
execution
actually
runs
right,
like
you
would
register
your
function
and
then
invoke
it
and
it
runs
somewhere
in
the
aws
cloud
in
our
model.
C
You
register
your
function
with
the
service
and
you
then
invoke
that
thing
and
specify
an
endpoint.
You
want
it
to
run
on
so
that
endpoint
could
be
an
endpoint
you've
deployed
on
your
laptop.
It
could
be
an
endpoint
sitting
on
a
supercomputer
somewhere
else.
However,
you've
got
the
same
interface
from
your
perspective
of
register.
Your
function,
invoke
it
with
whatever
payload
and
just
specify.
If
you
wanted
to
run
it,
your
laptop
you
just
put
in
your
laptop's
uid.
C
If
you
wanted
to
run
up
the
super
computer
you
put
in
the
super
computer's
uid,
so
these
functions
register
once
run
anywhere
at
any
time.
It's
our
sort
of
goal.
We
really
want
to
provide
this
sort
of
fire
and
forget
sort
of
reliable,
secure
execution
model
where
things
will
be
buffered
at
the
cloud.
They'll
be
buffered
at
the
end
point
we'll
make
sure
that
that
task
actually
gets
executed.
C
We've
got
endpoints
that
are
deployed
on
these
remote
resources,
so
they
can
really
run
on
anywhere
any
machine
that
supports
python.
You
can
work,
and
these
endpoints
will
dynamically
provision
up
resources
as
you
need
them,
you
can
configure
them
to
talk
to
different
queues,
etc,
etc,
and
you
can
tell
them
to
use
containers
I've.
I've
got
an
asterisk
somewhere
around
that
I
can
talk
about
container
management
and
functions
a
little
bit
more
later,
but
yeah.
C
So
you
can
tell
these
endpoints
to
use
containers
and
then
they
will
manage
the
execution
of
these
functions
inside
those
different
containers.
The
funky
service
allows
you
to
register
endpoints.
So
when
you
start
an
endpoint
you're
forced
to
log
in
with
globus
login
using
the
globus
native,
auth,
client
flow
native,
app
client
flow,
and
that
verifies
who
you
are
your
identity,
then,
when
your
endpoint
connects
it
registers
itself,
it
passes
along
your
tokens
and
then
it
it
securely
associates
that
endpoint
with
you,
so
we
can
lock
down.
Who
has
access
to
that
endpoint?
C
We
technically
can
share
endpoints,
so
we
can
associate
a
globus
group
with
an
endpoint.
I
mean
anybody
in
that
globus
group
can
execute
on
that
endpoint.
However,
we're
a
little
bit
concerned
about
security
implications
of
that.
So
we
don't
really
publicly
advertise
that
it's
more.
You
send
us
a
message.
We
explain
this
is
somewhat
worrying
and
then
we'll
turn
it
on
for
you.
D
C
Yeah
good
question,
so
the
way
we
do
this
in
the
current
model
is
everything
is
sort
of
single
tenant,
so
it
runs
in
the
user
space.
So
you
as
a
user,
would
connect
to
nurse
and
then
start
your
own
endpoint
and
that
would
be
running
under
your
users.
So
every
execution
would
be
run
on
your
behalf.
So
it's
really
like
the
globus
connect
personal
model
where
you
can
start
a
gcp
endpoint
and
that
does
not
use
root
permissions
that
just
uses
your
user
permissions.
C
We
do
have
a
path
toward
providing
multi-tenant
endpoints.
That's
on
our
roadmap,
we're
probably
a
year
away
from
actually
having
that
sort
of
deployed,
which
is
why
we
support
endpoint
sharing
at
the
moment.
But
our
long-term
goal
is
to
facilitate
multi-tenant
endpoints,
where
an
administrator
would
set
up
a
funk,
exam
point
for
the
entirety
of,
say,
fata
or
something
like
that
in
a
model
similar
to
how
globus
connect
server
works.
So
gcs
does
that
answer
your
question.
It.
C
Okay,
so
just
quickly
going
through
this
stuff,
so
basically
you've
got
a
couple
of
different
machines.
You've
got
your,
I
think.
That's
a
cluster
at
alcf,
which
supports
singularity
and
you've
got
a
laptop
or
a
kubernetes
cluster,
or
something
maybe
running
mini
cube
on
your
your
laptop.
You
can
start
an
endpoint
there.
I
can
show
how
this
actually
works
and
then
that
will
go
off
and
register
with
the
funkex
service
as
your
endpoint.
So
again
it
passes
your
tokens
along.
C
The
user
then
comes
along
and
registers
a
function
plus
dependencies
again.
I've
got
an
asterisk
there,
but
I
don't
actually
have
an
asterisk,
but
I
probably
should
but
yeah
they
register
some
function.
In
this
case.
It's
just
funkex
sum,
which
takes
some
arg
items
and
returns
to
some
of
them.
That
process
will
give
you
back
a
uuid
for
that
function
and
I
actually
skipped
over
that
part.
But
when
you
register
an
endpoint,
you
get
back
a
uuid
of
your
endpoint
as
well,
where
you
uuids
go
everywhere
in
the
system.
C
So
you've
now
got
a
uuid
of
your
endpoint
and
a
uuid
for
your
function,
and
then
you
can
come
along
through
whatever
client
you're
using
if
it's
a
jupiter
environment
or
a
jupiter
notebook
or
some
python
script.
You
say
I
want
to
execute
f
of
x
on
the
payload
one,
two,
three
four.
You
send
that
off
to
funkex
and
you
specify
what
endpoint
you
wanted
to
execute
on
and
then
that
routes
it
down
to
the
end
point
for
execution.
C
So
this
is
so
I
don't
know
how
familiar
you
guys
are
with
parcel.
So
I'll
just
quickly
give
a
one
minute
overview
of
what
parcel
is
puzzle's
a
parallel
scripting
library
created
their
uchicago
argon.
It
actually
builds
on
a
tool
called
swift.
I
don't
know
if
you
guys
are
familiar
with
swift.
That's
been
around
for
mike
wilde
for
about
15
years,
so
parcel
is
kind
of
like
the
python
implementation
of
swift,
but
really
how
we
use
parcel
in
this
system
is
hustle
is
essentially
underpinning
the
entire
endpoint.
C
So
when
you
submit
a
job
to
the
funkyx
endpoint,
it
uses
parcel
to
go
and
acquire
compute
nodes,
which
is
using
one
of
these
configurations,
so
this
in
this
case
we're
using
a
slam
provider
to
acquire
five
nodes
at
a
time
for
30
minutes,
and
when
the
node
starts
it's
going
to
use
the
function,
point
function,
condo,
environment
and
then
it
uses
a
pilot
job
model
to
start
a
worker
on
each
of
those
compute
nodes
that
connect
back
to
your
endpoint.
That
then
pull
jobs.
C
So
when
you
submit
a
function
in
for
execution,
one
of
these
workers
pulls
it
and
then
executes
it
locally
on
that
compute
node.
So
the
way
you
start
a
funky
x
endpoint
is
you
pip
install
our
function,
software,
that's
actually
a
title
that
should
be
funk
x,
dash
endpoint.
If
you're
starting
an
endpoint,
you
then
do
funkyx
endpoint
configure,
which
will
ask
you
to
log
in
if
you're
not
already
logged
in
with
globus
on
that
system,
and
then
you
define
some
configuration.
So
we
give
you
one
default
out
of
the
box.
C
If
you
create
an
endpoint
you're
going
to
get
a
config
there,
but
it's
just
going
to
use
local
threads.
So
it's
only
going
to
execute
locally
on
the
machine.
You
can
overwrite
that
config
with
any
one
of
these
configs,
that's
provided
so
we've
got
various
examples,
I'll
just
quickly
jump
over
to
that.
Hopefully,
alt
tab
works.
So
these
are
some
of
the
examples.
There's
one
for
film
matter.
Where
we've
got
this
high
throughput
executable,
you
give
it
whatever
machine
address.
C
You
want
to
execute
on
what
provider
you
wanted
to
use
and
then
some
configuration
information
about
like
how
long
you
wanted
to
hold
nodes.
How
many
nodes
you
wanted
to
require
what
queue
you
wanted
to
use.
So
this
is
using
a
gpu
partition
and
we've
got
these
for
many
different
machines,
so
you
can
basically
copy
paste
one
of
these
and
just
modify
it.
C
C
So
all
of
this
coming
together
really
gives
us
this
ability
to
code
for
computing
continuum
with
funk
x.
So
I've
got
this
example
here
of
how
this
actually
works.
We
can
just
quickly
walk
through
that,
where,
after
you've,
pip
installed
funk
x
in
this
case,
because
we're
using
the
client
sdk,
you
can
create
a
funcs
client.
So
this
is
just
a
wrapper
or
basically
a
communication
layer
over
our
rest,
api
of
our
service
running
in
the
cloud
you
can
then
define
a
function.
So
you
can
see.
C
This
is
purely
just
a
python
function
that
you
specify
makes
it
really
easy
to
take
existing
functions
and
then
register
them.
We've
got
our
fxc.
This
is
our
client
register
function.
You
point
it
at
the
python
function.
You
want
it
to
register
and
then
that
gives
you
back
a
uuid
for
that
thing.
Actually,
so
you
can
invoke
it
onto
one.
You
then
specify
some
payload
if
you
have
payloads
in
this
case
we're
taking
a
an
items
field
as
part
of
that
or
an
item's
argument
as
part
of
that
function.
C
So
we
specify
just
one
argument:
payload.
You
then
call
fxc.run
the
payload,
the
endpoint
id
and
the
function
id
that
gives
you
back
a
task.
Uuid
really
are
uuids
everywhere,
and
then
that
will
give
you
back
some
some
result
pretty
quickly.
So
the
whole
round
trip
time
on
all
of
us
is
we
we're
aiming
for
about
100
milliseconds
we've
just
added
in
some
more
sort
of
task
management
stuff.
C
Probably
the
only
the
coolest
thing
about
all
of
us
is
how
we're
dealing
with
args
and
keyword
arcs.
You
can
have
whatever
function
you
want
here.
It
can
take
numpy
lists,
numpy,
arrays,
all
sorts
of
different
fields,
different
objects,
different
whatever,
and
you
just
specify
them
as
input
to
your
your
functions
run
command,
and
then
we
will
pack
that
we
will
serialize
it
with
pickle
and
pickle,
fails,
we'll
use
dell
and
then
we'll
route
that
over
to
your
endpoint,
get
the
result
back
and
then
return
it
back
out.
C
So
I
see
I'm
already
at
half
past
the
hour,
so
I'm
not
going
to
go
through
the
demo
of
setting
up
the
endpoint,
but
you
can
see
just
how
this
works.
So
you
pip
install
function.
Point
you
in
it
to
to
do
the
login.
You
configure
an
endpoint
and
you
start
an
endpoint
and
that'll,
give
you
a
uuid
running
on
your
machine.
You
can
do
that
right
now.
We
also
have
this
binder
example.
So
if
anybody
wants
to
play
with
this,
just
need
to
move
this
thing.
C
If
you
go
to
funkex.org,
I
assume
you
can
see
my
browser
and
then
scroll
down,
there's
a
tri-func
exponent.
So
if
you
just
click
that
it'll
launch
a
binder
example
where
this
will
all
just
run
in
the
browser,
so
I'm
just
going
to
quickly
loop
through
some
of
this
once
our
finder
instance
starts
here
we
go
nice
and
quick.
C
So,
as
you
can
see
just
going
through
the
the
same
code
as
we
had
before,
we
create
our
client.
We
click
go.
That's
going
to
ask
me
to
log
in
because
this
is
not
running
on
my
machine,
which
I'm
already
logged
in,
and
this
is
running
on
binders
instance
that
it
provisioned
for
me,
so
I
am
not
logged
in
on
this
machine,
so
I
copy
the
code
back
paste
it
in
I
mean
that'll
create
my
client.
C
We
have
a
public
tutorial
endpoint
that
is
accessible
to
anybody,
so
you
can
go
and
run
whatever
you
like
there
a
little
bit
dangerous.
Please
don't
run
too
many
bad
things
there
yeah,
I
don't
know.
We
should
probably
do
something
about
making
that
a
bit
more
secure,
but
you
can
do
whatever
you
like
there.
You
can
run
whatever
function.
C
You
just
hit
execute
that
goes
off,
gives
you
a
task
id
and
then
you
grab
the
result
back
and
it
gives
you
that
so
the
nice
thing
is
it's
just
pure
python
right,
so
you
can
import
platform
return
spell
platform.unname,
re-register
that
it
gives
you
an
id
and
then
run
it
and
you'll
get
back.
This
is
running
on
our
tutorial
endpoint.
This
is
running
on
a
kubernetes
cluster
with
and
so
on
and
so
forth.
So
you
already
saw
that
you
can
pass
in
args
keyword
ads.
That
gives
you
back
15..
C
You
can
see
just
how
quickly
this
does
actually
run.
So
that
was
a
run
command
and
then
immediately.
We
call
get
status
or
get
result,
and
that
gives
us
back
the
result
you
can
put
arbitrary
dependencies
in,
as
you
just
saw
where
I
imported
platforms
so
provided
these
dependencies
exist
on
the
machine
that
it's
executing
in
so
you've
pip
installed
them
over
there.
C
It
will
be
able
to
use
them.
Your
worker
will
be
able
to
import
them
and
actually
execute
them
utilize
them.
You
can
do
things
like
p,
open
or
sub
process
out
various
commands,
so
you've
got
a
lot
of
control
over
how
what
you're
actually
executing
over
there.
You
can
catch
exceptions,
so
this
is
actually
really
nice.
This
is
probably
one
of
my
favorite
features
where
an
exception
generated
by
your
worker.
C
It's
just
returned
to
you
as
a
typical
exception,
so
we
serialize
it
up
then
return
it
back
to
you
and
you
get
your
exception,
saying,
deterministic
failure
and
similarly,
if
you
import,
gibberish
you'll
get
the
exact
same
thing
where
it'll
say
this
module
was
not
found-
and
this
is
the
exception
raised
by
your
worker-
on
that
remote
endpoint
in
the
interest
of
time,
I'll
just
sort
of
skip
through
the
stuff.
It's
just
running
sort
of
a
more
intensive
example,
and
then
we've
got
this
asynchronous
execution,
so
this
stuff
is
really
cool.
C
C
So
you
can
actually
grab
what
your
endpoint's
up
to
how
many
workers
it's
got.
How
many
are
idle,
etcetera,
etcetera?
How
much
work
is
done,
so
that's
all
I
have
for
the
demo,
I'm
gonna
carry
on
in
the
interest
of
having
a
chat
about
some
of
this
stuff,
so
how
this
works
under
the
hood.
Basically,
there's
three
components
that
you
just
solve
is
the
sdk
client
sort
of
side
that
you
register
your
functions
with.
C
You
invoke
your
functions
with
there's
a
web
service
which
is
just
a
rest
web
service
hosted
in
eks,
so
the
elastic
kubernetes
cluster
on
aws.
This
is
fully
globus
off.
So
every
request
or
interaction
with
the
web
service
has
to
have
some
sort
of
error.
Token
on
it
that
we
will
verify
who
that
user
is
so
every
task
invocation
task
status,
task,
endpoint,
registration,
etc.
We
check,
if
you're,
allowed,
to
be
doing
what
you
do.
C
C
C
We
have
one
user
eric
jonas,
who
is
he's
a
professor
at
uchicago
he's
a
great
guy
he's
really
hammered
our
platform,
so
we
had
a
problem
where
he
kept
filling
our
retire
store
with
too
much
data,
because
we
were
pumping
results
back
into
register
for
the
user
to
collect
them
as
well.
So
we've
now
added
s3
support
to
store
results
out
there,
just
because
eric
was
destroying
our
our
lettuce
store.
C
So
that's
all
right
and
then
there's
this
folder
component,
which
is
responsible
for
taking
tasks
off
the
endpoints
redis
queue
and
giving
it
to
the
endpoint
so
yeah.
I
just
said
there
were
three
components
for
the
client
sdk,
the
web
service
and
the
endpoint
right.
So
this
endpoint
is
basically
just
a
python
package
that
layers
over
the
top
of
a
parcel.
So
that's
how
we
do
all
of
our
compute
in
members
this
layer.
C
On
top
of
that,
which
does
all
of
the
communication
up
with
our
web
server,
so
we
use
a
0mq
to
do
low
latency
communication
between
our
endpoint
and
the
folders.
This
is
all
secured
with
curved
cmq,
so
it's
an
encrypted
channel
where
part
of
the
registration
gives
you
back
a
key
and
then
the
other
thing
here
is
that
we
connect
out
from
the
endpoint
to
the
folder.
So
this
actually
runs
on
basically
any
login
node
that
you
have
internet
access.
C
I'm
not
terribly
interested
in
the
evaluation
side
of
us
probably
should
be,
but
just
just
a
high
level
sort
of
highlight.
What's
going
on
here,
we
we've
done
tests
on
various
machines
and
basically
just
shown
that
we
can
scale
to
many
thousands
of
workers.
We
scale
pretty
well,
so
we've
got
131
000
concurrent
containers
running
on
cory.
I
believe,
and
we
can
push
millions
of
tasks
without
too
much
issues.
So
this
is
just
really
a
evaluation
of
how
parcel
works
so
yeah.
C
We
do
elastic
scaling
so
because
we
have
such
fine
grained
control
over
what
that
endpoint
is
doing,
we
can
tell
it
to
spin
up
nodes.
We
can
tell
it
to
destroy
nodes
as
needed.
If
nodes
are
idle,
you
can
control
how
long
you
want
them
to
be
idle
for
before
the
end
point
releases
them.
This
works
especially
well
along
kubernetes
clusters,
because
you
can
just
spawn
pods
and
delete
pods
as
needed,
but
yeah
funkish
gives
you
this
elastic
scaling
over
whatever
cue
you
want,
because
we
use
this
pilot
job
model.
C
So
it's
not
actually
one
function
per
endpoint.
It's
one
function
is
executed
by
a
worker.
That's
persisting
on
that
that
node,
so
I
see
the
endpoint
before
that
node
we've
got
this
multi-layered
fault,
tolerance
where
our
cloud
service
tracks
tasks
and
our
endpoints
track
tasks.
So
if
your
endpoint
dies,
we
can
recover
from
that.
Your
tasks
will
get
executed.
C
Once
your
endpoint
comes
back
online,
it
will
start
returning
results
that
it
hadn't
already
returned,
or
it
will
start
executing
tasks
that
were
queued
up
for
it
in
the
cloud
and
then
this
is
just
sort
of
an
interesting
side
of
things
where
we
can
optimize
performance.
So
this
is
this
graph
on
the
left
here.
I
know
I'm
going
super
fast
with
these
graphs.
I'm
sorry
if
anybody
was
really
interested
in
looking
at
the
pretty
pictures
just
just
quickly
this
free
fetching
one.
C
This
is
kind
of
a
this
cool
feature
that
we
can
control
on
an
endpoint
by
telling
the
workers
how
much
work
to
pull
at
any
given
time.
So
really,
the
aim
of
this
is
just
to
minimize
the
amount
of
communication
between
the
worker
and
the
endpoint
itself.
So
if
the
tasks
are
very
small,
we
can
tell
that
worker
to
take
100
jobs
at
once.
Instead
of
one
job
at
once,
and
then
we
can
drastically
increase
the
throughput
of
those
those
nodes.
C
This
one
on
the
right
is
just
about
batching,
so
obviously,
with
funky
experience,
a
cost
to
submitting
a
task
and
getting
a
result,
because
we
send
that
task
up
to
the
cloud
and
all
of
its
payload
and
then
we
send
that
down
to
the
endpoint
and
then
the
result
goes
vice
versa.
So
we
can
batch
tasks.
We
can
send
a
thousand
tasks
in
one
request
and
then
yeah,
basically
just
get
rid
of
a
lot
of
the
overhead
of
communicating
with
the
service.
C
C
This
one
is:
we
took
a
a
set
of
different
machines
from
raspberry
pi's
desktops
cloud
instances
gpus,
I
think
a
super
computer
in
there-
maybe
not
a
super
computer
in
this
one,
but
mn
took
some
different
sorts
of
jobs,
so
matrix
multiplication
map
produce
file.
I
o
and
then
just
told
it
to
run
in
different
models
so
round
robin
obviously
distributes
work
pretty
evenly
fastest.
C
Endpoint
just
always
uses
the
best
endpoint
for
each
job,
and
then
this
sort
of
cool
one,
this
smallest
eta,
where
rohan's
system
it
was
called
delta,
would
learn
how
jobs
execute
on
different
machines
and
then
make
the
decision
of
where
to
actually
place
that
job.
Based
on
what
it's
learned
so
doing
the
smallest
eta.
It
would
actually
utilize
these
raspberry
pi's
with
very
lightweight
jobs.
Just
so,
we
can
reduce
some
of
the
load
on
the
bigger
machine
to
get
a
a
smaller
full
execution
time
on
everything.
C
Thanks
man,
there
we
go
okay,
so
that
really
sums
up
our.
What
is
funk
x,
how
funky
x
works.
I
just
wanted
to
quickly
dive
into
sort
of
how
this
fits
into
the
wider
globus
ecosystem
and
how
we
use
this
to
support
scientific
computing,
in
this
case
the
advanced
photon
source
so
very
quickly.
I
just
wanted
to
shout
out
to
this
project
called
gladium,
which
is
a
globus
architecture
for
data
intensive,
experimental
research.
C
It's
a
great
background.
I
I
don't
even
know
if
we're
going
backwards
with
a
made-up
word,
but
that's
all
right,
so
this
project
is
really
to
sort
of
define
a
framework
for
how
you
would
approach
taking
many
different
global
services
like
transfer
off
search
flows,
which
is
a
newer
service
and
actually
build
these
pipelines
to
accomplish
science.
So
if
your
instrument,
your
sem,
is
generating
data,
what
sort
of
framework
could
you
use
to
take
all
those
global
services
and
actually
accomplish
that
science
in
an
automated,
reliable
and
fast
way?
C
So
that's
what
gladio
is
doing,
and
it's
kind
of
nice,
because
it
highlights
how
funkyx
plugs
into
all
of
these
different
services
and
how,
with
us,
using
blubus
auth
as
this
this
layer
on
top
of
our
entire
web
service,
how
we
can
just
click
funk
x
into
any
of
these
processing
pipelines,
so
you're,
probably
not
familiar
with
globus
flows.
This
is
a
recently
added
beta
service
data
service
on
the
globus
app.
C
So
if
you
go
to
globus.org
amp.globus.org,
you
can
go
to
slash
flows
and
you'll,
see
your
flows,
which
you
probably
have
very
few
of
them
just
quickly.
This
is
it's
a
pipelining
tool
built
on
top
of
amazon
step
functions,
so
what
you
can
do
is
you
can
define
these
states
using
json?
This
is
doing
a
it's
a
pipeline
defined
in
json
with
multiple
different
states.
C
C
So
some
examples
of
doing
this
around
the
lab.
I've
got
far
too
many
examples,
so
I'm
gonna
basically
skip
them,
but
that's
okay.
So
we've
deployed
this
at
the
aps.
The
advanced
photon
source,
argon
synchrotron,
so
you
can
see
from
this
picture
on
the
right
globus
connects.
So
globus
transfer
service
is
widely
adopted
at
the
aps.
It's
at
many
of
the
different
beam
lines:
funky
eggs
less
so,
but
we're
slowly
going
to
take
over
I'm
excited.
C
We've
got
function,
points
on
their
local
processing
cluster
or
first
we've
got
them
there,
lcrc
our
laboratory,
computing
research
center
and
we've
got
them
at
the
leadership
computing
facility
as
well.
So
this
gives
our
researchers
at
the
aps
opportunities
to
build
these
flows
that
leverage
any
of
these
resources
from
aps
clusters
to
their
local
cluster
local
servers
through
to
alcf
resources
as
well
so
very
quickly.
C
A
couple
of
examples
of
how
this
actually
works,
we've
defined
those
globus
flows,
pipelines
for
a
few
different
of
these
aps
groups,
so
we
work
with
them
to
define
them.
This
is
serial
crystallography
to
solve
protein
structures
for
covert.
So
we
were
doing
this
last
year
was
it
last
year
it
must
have
been
last
year
where
we
took
cover
19
structures,
put
them
in
the
aps
and
then
kicked
off
this
flow.
C
Every
time
a
new
image
was
collected,
we
would
start
a
flow
that
would
do
this
analysis
loop,
so
we
would
transfer
the
data
to
alcf.
We
would
do
analysis
on
those
data
we
visualize
them
into
these
sort
of
crystal
maps.
I
don't
know
if
you
can
see
that
under
this
thing,
then
we
would
do
analyzing
of
the
results
which
would
refine
the
structures,
publish
them
into
a
catalog,
which
you
can
see
here.
C
This
tiny
catalog
that
shows
you
sort
of
in
real
time
or
near
real
time,
while
the
experiment
is
running,
the
results
coming
out
of
these
machines,
which
was
really
nice
to
guide
like,
is
this
chip
actually
any
decent?
Or
should
we
move
on
to
the
next
sample
of
this?
Has
this
one
dried
out
already
or
whatever,
and
then
we
did
occasionally
actually
solve
the
structure
from
this
using
functions
as
well.
C
C
Exophoton
correlation
spectroscopy-
this
is
another
aps
use
case
where
we've
defined
this
analysis
pipeline
that
moves
data,
as
it's
collected
from
the
aps
over
to
alcf
and
then
runs
their
eigen
correlation
tool
and
then
plots
results.
So
both
of
these
tasks
are
done
with
funk
x.
Our
flow
uses
functions
to
run
this
correlation
tool
on
the
compute
nodes
and
then
plots
results
and
publishes
them
using
the
login
nodes,
because
we
need
internet
access
to
do
the
publication
steps.
C
So
we
have
two
endpoints
deployed
on
the
on
the
fader
logger
node
one's
talking
to
the
queue
and
one's
talking
to
just
a
local
executor
using
threads
to
actually
do
the
publication,
because
we
need
that
internet
access-
and
I
skipped
these
ones
just
because
I
want
to
have
a
chance
to
chat
yeah.
This
is
just
a
typography
example
where
they
ran
three
thousand
flows
again
using
funkex
extensively
to
do
2d
and
3d
reconstructions,
and
this
is
a
neural
network
training
pipeline
that
zhengchan
has
worked
on.
C
So
the
goal
of
this
is
to
train
models
using
alcf's
machine
learning,
resources.
We've
got
dedicated
reverse
machines
and
that
sort
of
thing
to
do
machine
learning,
training
at
scale
and
then
the
plan
here
is
to
now
put
a
funk
x,
endpoint
on
the
each
device
and
actually
do
the
inference
again
so
redeploy
that
model
back
to
the
edge
and
then
do
the
inference
as
data
collected
at
hdmv
one.
C
And
then
this
is
another
hddm
energy,
xl
diffraction
microscopy
use
case
this
one's
actually
really
cool.
So
herman
is
a
user
and
he's
a
developer
at
two
at
sector.
Two
and
he's
built
this
midas
tool.
That
does
analysis
so
he's
built
some
funkyx
functions
to
basically
just
use
midas
at
whatever
machine
he's
at
so
he's
defined
configurations
to
use,
fader
or
or
polaris
or
his
author's
cluster,
and
then
he
kicks
off
a
flow,
and
at
this
invocation
time
of
that
flow,
he
specifies
what
globus
end
point.
C
The
data
is
getting
transferred
to
and
what
function
point
the
execution
should
be
sent
to.
So
he
can
define
at
the
inputs
of
the
flow
basically
where
this
computes
and
he
can
control
running
it
on
alcf
or
polaris.
Just
by
changing
a
field
of
use.
This
config
use
that
config
etc,
and
the
cool
thing
about
this
is
he's
now
going
to
his
users
and
saying,
if
you
want
to
run
this
at
your
home
institute,
just
deploy
a
function
point
and
everything
else
will
get
sorted
out.
We
can
send
the
data
there.
C
C
So
wrapping
up
lessons
learned,
so,
basically,
our
users
really
like
that
you
can
abstract
for
resources
and
run
on
their
laptop
or
the
super
computer
with
the
exact
same
interface.
The
simplicity
of
automatic
scaling
in
this
one
interface
is
really
our
big
selling
point.
In
addition,
with
this
sort
of
flexible
web-based
authentication
model
that
they
understand,
so
they
don't
need
to
like
really
understand
how
these
different
machines
are
all
working
under
the
hood.
They
just
make
sure
they're
authenticated,
make
sure
they've
got
access
and
then
they
go
from
there.
C
We
found
a
great
deal
of
success,
but
that's
probably
over
selling
things.
We
found
good
success
with
event-based
processing,
so
this
is
really
our
bread
and
butter
for
where
people
are
using
funk
x,
sort
of
in
production
by
defining
these
event-based
processing
pipelines
and
globus
flows,
pipelines
trigger
as
data
recaptured
and
do
analysis
on
different
machines
increases
portability
between
sites
sounds
believable.
Resources
can
be
used
efficiently
and
opportunistically.
C
However,
it's
not
this
sort
of
like
amazing
solution
for
everything.
S
definitely
doesn't
meet
every
scientific
computing
problem.
If
your
job
is
like
super
dependent
on
data,
we
don't
yet
have
this
a
really
great
model
for
how
you
get
your
data
to
different
places
if
your
jobs
are
really
like
interwoven
like
your
your
functions,
are
big
mpi
jobs,
we
sort
of
just
step
away
and
say:
well,
you
can
use
funkex
to
start
your
mpi
jobs,
but
you're,
probably
better
off
just
using
mpi
to
do
all
of
that
sort
of
enterprises.
C
C
Of
course,
allocation
models
are
still
a
bit
of
a
pain
point
just
because
these
research
computing
resources
aren't
necessarily
designed
to
work
this
way
and
we're
we're
trying
to
fix
that,
but
we'll
see
how
we
go
and
decomposing
30
year
old,
fortran
applications
isn't
always
easy.
It's
an
expensive
task
both
for
time
and
resources.
Somebody
needs
to
be
employed
to
do
that.
C
So
it's
it's,
not
a
perfect
match.
So
yeah
it's
basically
our
lesson
zone.
There
next
steps,
I
sort
of
I
mean
I've-
pointed
this
out
a
few
times,
but
our
containerization
and
management
of
runtimes
is
not
yet
like
fully
fledged,
so
we're
working
with
university
of
wisconsin
to
build
out
a
better
container
service
and
then
support
in
our
endpoints
to
stage
like
singularity
containers.
C
Currently
we
can
we
do
the
staging
of
docking
containers
and
that
sort
of
thing,
because
kubernetes
will
do
that
for
you,
so
we
don't
really
do
it
at
all,
but
singularity.
We
need
those
containers
to
already
exist
on
the
machine,
so
you're
sort
of
responsible
for
putting
that
run
time
where
you
want
that
runtime
and
our
goal
is
to
automate
that
process.
C
S3
storage
for
task
results
that
should
be
shipped.
I
think
that
gets
released
this
week,
so
that'll
be
nice,
aws
mq.
I
didn't
really
go
into
how
our
communication
layer
works
terribly.
Well,
we
use
emq,
but
it's
sort
of
a
pain
point
for
both
scaling
and
reliability.
It's
a
bit
of
a
beast
that
only
a
couple
of
us
know
how
actually
works.
So
we've
decided
we're
going
to
move
towards
amazon's
message
queue,
so
this
is
a
rabbit
and
q
service
and
use
that
instead,
just
it
just
seems
easier
for
moving
forward.
C
C
So
that's
it
that's.
My
summary
funkyx
is
a
federated
fast
system
designed
to
meet
the
requirements
of
scientific
computing.
It
enables
fluid
execution,
it
doesn't
do
it
for
you,
but
it
does
give
you
the
tools
to
do
this
fluid
execution
and
make
decisions
about
where
jobs
are
placed
with
a
relatively
simple
interface
and
it
can
be
leveraged
for
reliable,
distributed
data
management
and
pipelines,
and
that
sort
of
thing.
C
So
thanks
a
lot,
you
can
check
it
out
at
funkyx.org.
You
can
play
with
our
binder
example
there
or
there's
a
bunch
of
documentation
on
our
read
for
docs.
We
also
have
a
slack
channel
if
you
go
to
funkyx.org
support
you'll,
see
join
us
on
slack
button,
which
will
give
you
an
invite
to
join
our
slack
channel
and
we're
very
active
on
there.
Always
sharing
yeah
would
be
most
welcome
to
have
you
join
yeah.
That's
all.
I
got
thanks
a
lot.
A
Thanks
ryan,
so
I
I
guess
we'll
kick
off
the
questions
with
since
byron
and
I
are
hosting
I'll,
let
byron
since
he
asks
in
the
chat.
Ask
the
first
question
anybody
else
that
has
a
question
just
raise
your
hand
and
then
we'll
we'll
go
according
to
order.
So
byron
had
asked
if
the
you
know,
if
you
could
compare
this
with
balsam's
workflow
tool,
how
does
it
you
know,
compare
contrast.
C
Yeah,
it's
a
good
question,
so
we
work
closely
with
misha
and
tom
amish
is
now
left,
but
we
work
closely
with
the
boston
team.
We
had
a
joint
project
with
them
to
do
a
different,
aps-b
miner
powder
powder
diffraction.
So
the
way
we
worked
with
a
beer
was
funkix
was
sort
of
doing
the
communication
over
the
sort
of
wider
networking
men
invoking
balsam
at
the
edge
so
at
the
computing
resource.
C
I
know
they're
now
working
on
a
layer
to
do
this
themselves
for
the
sort
of
remote
execution,
so
we're
not
terribly
involved
there.
I
think
the
the
main
differences
are
that
we
plug
entirely
into
the
globus
ecosystem
of
auth
they've
got
their
own
authentication
model
built
on
top
of
lcs.
Oh
that's
my
understanding,
I
mean
yeah,
so
we
yeah,
I
think
they
their
focus
is
more
on
the
enabling
large
scale
analysis
and
our
focus
is
really
on
enabling
many
small
scale
analysis
tools.
If
that
makes
sense.
A
C
C
C
To
be
honest,
we're
we're
more
focused
on
those
smaller
tasks
like
there's,
nothing
stopping
us
from
doing
those
longer
tasks,
but
I
think
our
it's
a
easier
model
to
support
for
us
for
those
smaller
tasks,
so
I
would
think
anything
running
in
less
than
like
10
minutes.
15
minutes
is
probably
ideal
for
us.
We
do
have
use
cases
that
run
for
longer.
There's
an
alpha
folder
use
case,
I'm
working
on
at
the
moment
that
does
about
two
hours
of
computation,
but
yeah
generally,
I'm
mostly
interested
in
those
sort
of
shorter
ones.
C
E
Hey
great
talk,
I
I
really
enjoyed
this,
so
I
I
have
two
questions
that
are
associated
about
with
the
function
registry.
So
one
is,
can
you
use
a
function
that
someone
else
registered
like
if
you
were
gonna,
do
a
like
a
model?
You
know
machine
learning
model.
You
know
sharing
across
multiple
people.
C
Yeah,
so
the
answer
here
is
you
can,
if
they
share
it
with
you
or
make
their
function
public.
So
when
you
try
to
invoke
a
function,
we
go
through
a
series
of
tests.
Are
you
the
owner
of
this
function?
Is
for
function
public
or
are
you
in
a
group
associated
with
this
function?
If
it's
associated
with
that,
if
any
one
of
those
passed
you're
allowed
to
run
it?
If
none
of
them
pass,
then
we
reject
you
saying
you
don't
have
access
to
this
function.
E
C
Yeah,
so
I
would
say
that
side
of
things
is
outside
of
the
globus
realm,
so
we
use
globus
as
our
our
authentication
layer
for
our
creating
bearer
tokens
and
passing
those
bearer
tokens
and
ensuring
those
bearer
tokens
belong
to
different
users.
The
actual
security
of
our
system,
while
we
do
run
in
a
globus
account,
the
production
function,
service
is
deployed
in
a
globus.
Aws
account
we're
really
just
dependent
on
aws
is
our
security
layer.
There
I
mean
everything
sits
in
a
virtual
private
cloud.
C
You
can't
access
reddit
unless
you're
from
one
of
the
vpcid
instances,
so
yeah
we've
basically
just
gone
best
practices
on
how
aws
does
security
and
try
to
follow
that.
So
we
work
closely
with
the
aws
doe
team,
so
yeah.
They
definitely
provide
us
input
on
how
we
should
be
doing
things
and
getting
feedback.
That
way.
F
C
Yeah
yeah,
so
I
assume
you
mean
the
endpoint
side,
where
that
does
just
run
in
user
space
as
the
user.
So
if
I
start
an
endpoint
on
alcs
fader,
that
is
just
my
that's
my
endpoint.
My
executions
happen
on
my
behalf.
If
I
then
share
that
with
somebody,
which
is
why
we're
we're
sort
of
hand
waving
about
the
ability
to
share
we,
don't
we
don't
publicly
allow
people
to
share
endpoints
where
you
have
to
go
through
one
of
us
to
actually
use
an
admin
to
set
that
field.
D
C
So
we
haven't
had
this
happen.
We
definitely
would
well
yeah
we'd,
be
very
concerned
about
doing
that
for
just
a
user
of
the
system,
like
we've,
we've
done
this
for
an
admin
at
ncsa's
blue
waters,
machine
where
they
wanted
to
make
an
implant
public,
and
we
spent
a
long
time
talking
to
him
about
this
is
scary,
so
we
in
the
end
didn't
make
it
public.
We
made
it
accessible
to
a
globus
group,
but
they
managed
so
they
could
choose.
Who
was
in
that
globus
group
and
we
went
through
that
way.
Yeah.
C
Yeah
yeah
yeah
completely.
I
mean
it's
a
major
concern
to
me
yeah
I.
That
would
be
scary
if
we
made
it
public
yeah.
So
I
wasn't
following
check,
but
I
see
now
suresh
asked
about
the
performance
limits
on
data
going
through
the
service.
So
this
is
another
really
great
question.
So,
with
our
eric
jonas
user,
we
got
quite
strict
on
the
amount
of
data
we
were
passing
through
the
web
service
because
we
were
running
out
of
space
in
our
register
and
which
is
why
we've
now
switched
to
s3.
C
I
believe
the
latest
limits
are.
You
can
only
push
five
migs
as
input
for
a
batch
of
tasks
and
the
result
of
a
single
task
can
only
be
500k,
so
we
really
encourage
people
to
use
transfer
or
something
like
that
to
move
larger
data
with
our
s3
backing,
we'll
probably
bump
the
return
result
back
up
to
a
mig
or
two,
because
500k
is
pretty
restrictive
but
yeah.
Definitely
where
we
try
to
limit
that.
For
obvious
reasons,.
A
D
I
did
have
another
question,
so
you
mentioned
that
you
were
running
in
sort
of
a
a
short
queue
in
argonne
and
we
actually
are
kind
of
really
interested
in
getting
a
preemptible
cue,
fully
populated
at
nursk,
and
so
I'm
you
know
I
just
wanted
to
put
this
like.
We
might
be
open
to
having
those
kind
of
workflows
at
nurse.
So
I
assumed
you
partnered,
like
with
with
users
that
were
already
on
the
system
to
set
these
things
up.
Is
that
where
do
you
have
folks
that
are
looking
for
more
extremely
short-term
resources.
C
So
the
way
we
accomplished
that
was,
we
partnered
with
mike
pepker,
the
director
of
lcs,
saying
we
need
a
a
way
to
acquire
20
nodes
as
opposed
to
128,
because
our
aps
use
cases.
We
were
driven
by
the
xpcs
use
case,
which
we
use
frequently
in
ssx
for
crystallography
use
case
where
we
don't
need
128
nodes
to
meet
their
sort
of
on-demand
load,
but
those
nodes
are
often
via
idle.
C
They,
if
you
check
the
status
page
on
lcfc,
there's
typically
50
70
nodes
sitting
there
so
yeah
we
talked
to
mike
about
it
mike
agreed
to
give
us
a
backfield
queue.
He
already
had
a
backfield
queue
for
another
group,
so
we
just
worked
with
that
and
then
threw
that
in
and
it's
yeah.
It's
worked
great.
D
G
Yeah.
Thank
you.
This
is
a
great
talk
and
impressive
work,
so
I
had
a
multiple
questions,
but
let
me
start
with
the
my
my
first
one.
I
will
just
ask
two
questions.
So
one
is.
I
saw
in
your
example.
You
showed
one
aps.
Workflow
kind
of
there
is
like
speed
up,
and
you
said
from
several
weeks
down
to
like
a
few
hours,
I'm
I
was
interested
in
knowing
where
the
speed
up
come
from,
because
my
understanding
is
like
you
are
more
like
providing
the
convenience
than
kind
of
performance.
In
my
understanding.
C
Yeah
yeah
good
point
yeah,
so
I
kind
of
glossed
over
it.
So
the
way
they
used
to
do
analysis
was
at
the
1990,
the
crystallography
beam
line.
They
had
a
server
called
bc
brock,
which
was
a
powerful
wish
server.
It
had
32
cores,
they
would
run
their
experiment
for
an
entire
day,
collect
probably
eight
ish
or
maybe
maybe
ten
chips
worth
of
data.
C
So
that's
a
on
the
order
of
half
a
million
images
and
then
they
wouldn't
do
any
analysis
during
the
actual
beam
time,
because
they're
busy
doing
their
their
beam
time,
they
would
check
that,
like,
as
images
went
through
they'd
check.
Yes,
they
were
the
fractions,
we
think
that's
good,
but
they
wouldn't
actually
do
any
analysis
or
reconstruction.
C
So
what
we
did
was
we
came
in
and
developed
that
pipeline
with
them
using
globus
flows,
and
then
we
integrated
functions
to
actually
use
alcf,
so
their
problem
with
bc
frock
was
they
would
collect
all
of
their
data
and
kick
it
off
running.
But
that
would
take
like
many
many
days,
if
not
weeks,
to
actually
do
the
analysis
because
it
was
just
one
machine.
So
we
scaled
to
hundreds
of
nodes
on
alcf
and
then
that
actually
allowed
them
to
run
all
the
analysis
during
moving
time.
G
E
G
C
C
The
plan
is
to
get
multi-tenant
where
an
administrator
would
do
that
sort
of
heavy
lifting
have
set
up
the
endpoint
and
then
they
would
just
be
able
to
use
it
in
terms
of
running
it
from
their
local
machine
or
running
like
if
you're
logged
into
therma
and
then
just
want
to
execute
you're.
Probably
I
mean
there
would
be
some
benefits
to
running
it
through
funkyx
purely
that
we
would
give
you
the
reliability
of
making
sure
that
task
completes.
So
we'll
give
you
sort
of
that
fine
and
forget.
C
H
C
C
We
start
running
into
issues
with
how
much
data
did
you
actually
just
write
out
to
that
file,
because
it's
too
much
we
will
reject
you
sending
that
file
actually
as
input
or
payload
back
through
the
service.
As
part
of
that
result,
the
way
we
often
do
this
is
we'll
write
a
lot
of
results
locally,
like
in
the
alpha
fold.
C
Example
we'll
write
out
the
entire
logs
and
all
of
that,
and
then
we
will
simply
return
the
location
of
that
file,
so
the
path
name
of
that
file,
or
perhaps
a
globus
url
to
that
file
as
a
result
of
the
function
so,
rather
than
sending
the
actual
file
contents,
we
just
return
a
pointer
to
that
file.
Does
that
make
sense?
Okay,
so
yeah.