►
From YouTube: 2020-09-11 - D. Ahn & S. Herbein - Flux: Overcoming Scheduling Challenges for Exascale Workflows
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Flux: Overcoming Scheduling Challenges for Exascale Workflows
Dong Ahn & Stephen Herbein (Lawrence Livermore National Laboratory)
Abstract: Many emerging scientific workflows that target high-end HPC systems require complex interplay with the resource and job management software (RJMS). However, portable, efficient and easy-to-use scheduling and execution of these workflows is still an unsolved problem. In this talk, I will present Flux, a next-generation RJMS designed specifically to address the key scheduling challenges of modern workflows in a scalable, easy-to-use, and portable manner. At the heart of Flux lies its ability to be seamlessly nested within batch allocations created by itself as well as other system schedulers (e.g., SLURM, MOAB, LSF, etc), serving the target workflows as their “personal RJMS instances”. In particular, Flux’s consistent and rich set of well-defined APIs portably and efficiently support those workflows that can often feature non-traditional execution patterns such as requirements for complex co-scheduling, massive ensembles of small jobs and coordination among jobs in an ensemble. As part of this talk, I will also discuss Flux’s graph-based resource data model, Flux’s response to needing to schedule increasingly diverse resources, and how this model is becoming the center of our industry co-design efforts: for example, multi-tiered storage scheduling co-design with HPE and Cloud resource co-design with IBM T.J. Watson and RedHat OpenShift.
A
This
has
this
workflow
has
high
mission
relevance
for
livermore,
but
being
able
to
run
their
workflow
on
sierra
was
not
without
a
challenge
also,
so
specifically,
they
need
to
run
lots
of
lots
of
data,
collect
lots
of
lots
of
data
to
build
a
machine
learning
model.
In
fact,
their
initial
target
was
to
run
one
billion
on
short-run
jobs.
A
So,
in
fact,
about
three
years
ago,
we
went
ahead
and
surveyed
those
emerging
workflows,
including
you
know,
those
users
who's
using
their
own
ad
hoc
workflow
scripts,
while
those
users,
you
know
those
workflows,
are
from
different
domains
of
science.
We
found
that
about
four
distinct
workflow
challenges
are
being
shared
by
them,
so
I
list
them
one
by
one.
A
A
And
finally,
there
is
a
good
old
portability
challenge.
Users
want
to
write
their.
You
know
core
logic
once
and
then
be
able
to
run
that
across
systems
with
different
batch
systems,
but
each
system
comes
with
different
syntax
and
semantics,
and
maintaining
that
complexity
in
and
of
itself
can
be
a
heroic
effort
and
our
users
identify
that
as
a
big
challenge
as
well.
A
A
So
flux
is
a
large
open
source
project
that
is
in
active
development
at
github
and
looking
at
some
of
the
early
code
commits
they
are,
you
know,
dated
back
to
2013,
so
it's
almost
like
seven
years
now
and
to
manage
the
complexity
associated
with
the
large
project
like
flux.
We
have
our
github
organization
called
flux
framework,
and
then
we
separate
out
different
components
of
flux
as
a
separate
project
repo
under
that
organization.
A
One
is
what
we
call
single
user
mode,
whereby
a
individual
user
can
instantiate
a
flux
instance
and
use
it
as
an
individualized
resource
management
scheduler,
and
we
mature
that
functionality
first
and
then
that
has
been
used
in
production
for
about
three
years
now
and
then
a
lot
of
things
that
I'm
gonna
talk
about
is
actually
the
single
user
mode
and
we
do
have
a
multi-user
mode
in
addition
to
the
single
user
mode.
We
added
securities
and
resiliency
and
things
like
that
and
then
you
know.
A
A
So
the
resource
allocation
can
be
created
by
flux
itself.
If
the
facility
like
lc
or
nurse,
uses
it
as
a
system,
level,
scheduler
or
the
allocation,
you
know,
resource
allocation
can
be
created
by
other
batch
systems
and
resource
managers
like
slurman
lsf.
So
what
it
means
is
that
users
don't
need
to
wait
for
the
facility
to
to
adapt
and
deploy
flux
as
the
system
resource
manager
to
benefit
from
it
because
it
takes.
A
You
know
a
lot
of
effort
to
for
the
facility
to
be
able
to
do
that
and
then,
once
you
get
a
resource
allocation,
you
can
instantiate
your
personalized
flux,
instance
or,
generally
speaking,
a
hierarchy
of
flux.
Instances
like
this
and
then
you
can
customize
flux,
instances
tailored
to
the
performance
characteristics
of
your
workloads
and
the
hierarchy.
D
A
So
flux
is
architected
specifically
to
embody
the
generalized
multi-level
scheduling
model
that
I
talked
about
so
on
the
left.
We
have
flux's
architectural
diagram
and
in
the
interest
of
time
I'm
going
to
just
highlight
only
four
distinct
features
of
flux
pertaining
to
our
subject
here.
So
you
notice
that
flux
is
a
software
framework
and
then
a
software
framework
is
underpinned
by
a
scalable
tree
based
communication
overlay
network.
A
A
So
scheduler
specialization
maps
to
the
scheduling,
co-scheduling
challenge
that
I
was
talking
about.
So
when
it
comes
to
scheduling
policy
specialization,
the
traditional
batch
job
schedulers
don't
offer
much
in
this
space.
Each
site
typically
enforce
one
scheduled
policy
like
conservative
backfill
for
old
jobs,
and
that's
it.
They
don't
support
users
to
use
this
same
scheduler
to
schedule
their
tasks
and
let
alone
allowing
them
to
change
the
policies
to
you
know
to
tailor
to
their
performance
the
performance
characteristics
of
their
workloads.
A
By
contrast,
flux
enables
both
system
and
user
level
scheduling
under
the
same
common
infrastructure
and
interfaces.
What
I
mean
by
that,
so
users
will
use
the
same
flux,
commands
and
interfaces
that
sysadmin
will
use
to
enable
batch
batch
jobs.
So
you
don't
have
to
use
a
separate
library
or
separate.
A
So
so,
specifically,
they
can
change
queuing
policies
and
resource
match
policies
and
hierarchical
policies.
They
look
to
their
workloads,
and
here
is
just
one
simple
example.
So,
in
this
case,
users
are
running
a
flux
instance
under
slurm,
so
you
get
a
location
from
slurm
like
this,
and
then
you
start
flux
using
flock
start,
but
using
the
native
s
run
to
instantiate
it
typically.
A
The
default
scheduling
policy
here
is
first
come
first
serve,
but
they
can
be.
You
know
easily
altered
by
setting
an
environment
variable
like
this
and
then
set
that
to
an
easy,
backfill,
capable
scheduling.
There
are
other
ways
you
can
actually
specialize
it,
but
this
is
one
easy
way:
okay,
so
the
second
technique
is
a
schedule
of
parallel
parallelism
that
maps
pretty
well
to
the
throughput
challenge
that
I
was
talking
about
again.
The
traditional
approach
doesn't
offer
much
in
this
space
other
than
a
few
research
products
that
we
know
of
the
centralized.
A
Further
within
our
allocation,
you
can,
you
can
instantiate
a
deeper
levels
of
you
know,
flux
instances
like
this,
so
in
this
in
this
graph
within
our
location,
you
see
you
have
allocation
level
flux
instance.
It
spawns
four
flux
instances,
each
of
which
deals
with
the
node
within
this
allocated
resources,
and
then
it
then
spawns
many
core
level
schedulers,
as
many
as
you
have
cores
on
the
node.
So
obviously,
if
you
deepen
the
schedule
hierarchy,
you
get
the
obvious
boost
in
the
parallelism
and
the
performance.
A
So
if
your
jobs
are
pretty
trivial,
like
sleep,
zero
being
the
most
trivial
jobs
and
you
see,
as
you
deepen
the
schedule
hierarchy,
you
get
a
drastic
job.
Throughput
improvement
scheduling
up
to
1
million.
But
if
your
jobs
are
a
little
bit
involved,
and
then
you
know
especially
the
ones
that
are
pounding
on
some
shared
resources,
such
as
memory
stream
with
the
case,
the
a
little
bit
flatter
hierarchy,
works,
looks
better,
we
found,
and
then
the
but
the
point
here
is
flux.
Provide
a
capability
for
you
to.
A
You
know,
choose
affordable
level
of
scheduler
parallelism
that
you
need,
and
then
the
third
technique
is
a
rich
set
of
well-defined
apis
and
this
maps
well
to
the
job,
coordination
and
communication
challenge
and
job
scene.
Ensemble
based
or
couple
based
simulations
often
require
close
coordination
and
communication
with
the
scheduler
as
well
as
amongst
themselves.
A
We
have
two
categories
of
apis:
one
is
primitives
event,
pops
up
and
rpc
requested
reply,
send
and
receive
patterns,
and
then
we
do
provide
apis
for
our
high-level
services
too,
which
include
things
like
kbs
api
and
job
api,
and
you
name
it.
And
if
you
want
to
take
a
look
at
those
apis,
we
have
a
pretty
well,
you
know
documented
api
set.
So
if
you
go
to
this
website,
read
the
docs
you'll
see
you
know
the
apis
from
false
core,
at
least
at
this
point.
A
So
you
as
a
you
as
a
scientist,
will
write
your
scientific
workflow
on
this
api
set
once
and
enjoy
the
portability
to
the
platforms
where
fox
has
been
ported
so
but
at
the
same
time,
that
also
puts
a
lot
of
burdens
to
the
flux
community
to
port
and
optimize
flux,
on
as
many
platforms
as
as
possible.
So
we
judiciously
chose
the
apis
that
we
depend
upon
and
we
require
linux
and
then
for
single
user
flux
mode.
A
A
The
patch
tunnel
membrane
and
you
know
in
book
actually
in
queue
those
most
interesting
patches,
jobs,
which
then
gets
executed
as
gpu
becomes
available,
and
then
but
the
thing
is,
as
the
macro
model
proceed,
the
score
of
those
patches
constantly
change.
That
means
that
they
also
need
to
dequeue
some
of
the
jobs
if
the
priority
changes.
A
And
then
you
know
to
hide
that
complexity,
they
use
the
maestro
vicero
workflow
manager
and
how
they
how
they
and
then
they
develop
a
flux,
maestro
adapter
to
be
able
to
handle
the
you
know,
starting
and
stopping
of
those
jobs
within
our
location.
And
then
they
also
specialize
the
flux
scheduling
to
do
non-node
exclusive
scheduling
so
that
you
could
do
core
level
scheduling
and
gpu
level
scheduling
as
needed
to
map
those
different
components
across
different
sets
of
resources
on
a
location.
A
And
then
this
is
actually
you
know
globally
how
they
put
different
components.
So,
for
example,
the
macro
level
ddft
continuum
based
model
runs
at
about
500
nodes
of
2000
nodes
which
uses
cpu
only
and
then,
for
example,
the
md
simulations
utilize
entire
set
of
gpus
on
our
location
and
so
on
and
so
forth.
So
if
you
crack
open
a
a
node,
then
you
will
see
this.
You
know
many
components
are
mapped.
A
You
know
very
sophisticated
like
this,
so
the
cpu
only
jobs
such
as
ddft
the
continuum
based
model
and
data
brokers
and
and
whatnot
are
using
about
20
cores
for
each
of
the
socket
and
then
each
md
simulations
are
mapped
to
a
gpu
of
four
on
sierra,
along
with
a
core
that's
closest
to
gpu
and
then
in
situ
data
analytics
are
mapped
to
some
other
cores
like
this.
So
and
then
also
you
know
those
md
simulations
exposed.
A
A
And
then
you
know
little
more
uncertain
qualification,
pipeline
workflow,
actually
refactor
its
ensemble
manager,
layer
around
flux
and
then
produce
a
component
out
of
uqp
that
can
be
used
not
only
by
itself,
but
also
by
other
workflow
tools,
targeting
not
only
youtube
but
also
vmd
and
parameter
and
sensitive
studies,
and
things
like
that.
And
then
they
demonstrate
the
unifi
unifying
workflow
and
system.
Scheduling
using
flux
can
provide
significant
benefit
with
respect
to
performance
as
well
as
of
their
complexity.
A
A
And
then
one
thing
that
happened
in
the
past
few
months
was
that
they
they
used
this
merlin
workflow
to
support
the
what-if
modeling
scenario
for
kobe
19
spread.
So
one
scientist
commented
that
with
flux
they
were
able
to
model
one
scenario
with
uq
for
the
entire
country.
In
you
know,
five
minutes
on
a
few
sierra
notes.
Last
thing
is
an
open
size
era
near
real
time
feedback,
and
now
this
workflow
has
been
ported
to
not
only
livermore
but
also
oak
ridge
and
and
nurse.
A
A
So
what's
next,
so
working
with
the
users
last
few
years,
I
realized
there
are
a
lot
of
workflow
manager
software
systems
out
there.
Even
you
know,
across
the
doe
complex
alone.
One
survey
shows
that
we
have
about
over
200
some
some
tools
and
then,
while
each
of
those
tools
work
very
well
for
their
domains,
their
effectiveness
tend
to
go
down
when
that's
applied
to
the
other
other
domains.
Perhaps
more
importantly,
you
know
they
are
designed
as
a
single
tool
so
that
they
cannot
be
easily.
C
A
A
So,
regarding
this
a
little
more
know,
it's
not
an
exception.
That
has
been
the
workflow
ecosystem.
So
far,
so
distinct
workflow
functionalities
have
been
implemented
across
multiple
tools.
For
example,
workflow
description
was
implemented
in
both
uq
pipeline
as
well
as
maestro,
and
then
workflow
runtime
orchestration
has
been
implemented
across
uqp,
maestro
and
merlin,
and
hpc
resource
allocation
and
scheduling
are
implemented
across
lsf
flux
and
slurm.
A
But
ideally
what
we
really
want
to
do
is
we
want
to
allow
users
to
pick
and
choose
and
compose
functionalities
from
multiple
tools
in
a
way,
that's
mostly
effective
for
their
for
their
job,
so
one
might
wanna
just
use
you
know
one.
You
know
one
functionality
from
maestro
for
a
workflow
description
and
then
use
merlin
as
a
you
know,
orchestra
layer
and
then
use
flux
as
the
the
portable
hpc
allocation
scheduling
layer.
A
So
but
in
fact
the
lnl
teams
have
been
collaborating
together
with
a
common
vision
to
create
a
sort
of
plug
and
play
a
workflow
ecosystem.
So
you
know,
I
believe,
what
merlin
team
did
with
regard
to.
That
was
quite
instrumental,
so
when
users
actually
complained
about
the
workflow
description
interface
of
merlin,
what
they
did
was
they
co-designed
the
description
interface
with
the
merlin
and
I'll
carve
out
semantics.
C
A
You
know
on
extra
scale
systems
for
you
to
use
so
so
I
started
like
three
minutes
after
so
I
want
to
spend
a
little
bit
more
time
before
I
give
a
torch
to
steven.
Hopefully
that's,
okay,
so
so
I'll
talk
briefly
about
how
flux
addressed
the
other
main
scheduling
problem,
which
is
called
the
resource
challenge.
So
this
challenge
arises
as
more
diverse
set
of
compute
and
other
resources
are
coming
to
our
way.
A
So
unless
our
schedulers
can
consider
them
as
additional
constraint,
the
system
can
run
a
risk
of
performance
losses
both
at
the
system
as
well
as
workflow
levels.
So
from
my
perspectives,
this
problem
really
goes
back
to
the
simplicity
resource
model
that
the
traditional
approach,
like
slaron
pbs,
pro
lsf,
used.
A
They
were
designed
20
plus
years
ago,
and
you
know
when
our
hpc
systems
were
much
simpler,
more
or
less
a
homogeneous
set
of
compute
nodes
and
then
so
they
they
used
a
relatively
simple
data
model
to
optimize
their
scheduling,
but
then,
but
then
we
believe
that
their
model
is,
at
this
point
outpaced
by
the
the
level
of
resource
diversity
that
we
see,
and
instead
flux
uses
a
graph
graph
based
model
and
each
vertex
in
our
graph
represents
a
resource
and
each
edge
represents
the
relationship
between
connecting
vertices.
For
example.
A
In
this
case,
a
cluster
is
modeled
as
a
resource,
and
a
rack
is
modeled
as
a
resource.
You
know
in
the
same
way
that
we
model
compute
core
and
compute
node
as
a
resource,
and
with
this
graph
we
can
actually
add
additional
constraints,
like
rack
level,
locality,
constraint
and
cluster
on
what
have
you
so
it
turned
out.
A
So
we
felt
pretty
good
about
it,
and
flux
is
also
driving
our
interactions
with
the
cloud
computing
players
too,
so
ibm
tj
watson,
you
know,
create
a
prototype
called
kuflux
to
use
function,
to
build
hpc
scheduling,
policies
into
kubernetes
and
recently,
red
hat.
Open
shift.
Team
also
came
to
us
expressing
their
interest
to
use
function
and
flux,
to
define
kubernetes
standard,
scheduler
interface,
and
they
said
it's
becoming
important,
as
their
commercial
customers
are.
You
know,
have
workloads
that
need
to
pay
attention
to
hpc
as
well
high
performance.
A
You
know
performance
critical,
so,
overall,
this
partnership
will
strengthen
the
flux's
position
with
respect
to
all
four
planets
of
how
we
envision
to
embrace
hpc
plus
cloud,
but
for
interest
time.
I'm
gonna
stop
it
here
and
for
those
who
are
interested
in
our
directions.
We're
gonna
have
our
talk
at
super
computing
20
as
part
of
state
of
the
practice
november
17th,
but
we
don't
have
the
actual
time
slot
yeah.
So
if
you're
interested
in
drama
line,
then
I'll
send
you
the
the
link
as
as
the
schedule
gets
pumped
up.
C
A
Well,
so
there's
two
things
right,
so
you
can
pick
and
choose
what
scheduling
policy
you
want
statically
right,
so
you
can
choose
different
queuing
policies
like
you
know.
First
come
first
serve
or
different
kind
of
backfield
scheduling.
You
know
from
conservative
backfield
to
hybrid
to
easy,
and
then
you
can
also
change
match
policies
right.
Sometimes
this
match
works
better
and
sometimes
other
matches
work
better
and
you
can
actually
write
the
match
policies
and
put
that
into
the
scheduler
and
the
hierarchical
problem.
A
A
D
A
Yeah
we
have
been
talking
with
leave
ensemble
and
so
one
area,
one
potential
collaboration
area
between
lib,
ensemble
and
excel
works.
Is
that
as
we
create
a
component,
is
it
possible
to
carve
out
something
from
lib,
ensemble
and
plug
in
you
know
some
excel
works
component
in
it
right,
but
with
respect
to
flux
and
leap
ensemble,
we
kind
of
you
know
sketch
how
those
two
tools
can
leverage
with
one
another
at
a
supercomputing.
A
A
You
can
you
can
do
that.
In
fact,
there
was
a
researcher
who
wanted
to
do
that,
and
we
actually
did
some
back
of
the
calculation
about
you
know.
Can
we
actually
do
better
than
what
they
use?
They
basically
use
a
glue
make
as
a
way
to
do
that,
and
then
the
back
of
the
calculator
calculation
was
that,
with
respect
to
that
particular
workload,
we
couldn't
do
that
because
of
the
overhead
of
python.
Like
you
know,
flux
mini
iran
and
things
like
that.
A
But
if
this
is
a
kind
of
significant
enough
use
case,
we
can
think
of
ways
to
specialize
our
execution
engine
such
a
way
that
we
strip
out
some.
You
know
heavy
weightedness
from
the
execution
engine
so
that
you
can.
You
can
do
that
so
with
a
scheduler
you're,
not
gonna,
you
know
over
schedule
or
under
schedule
your
resources
right,
but
with
the
power
build
and
make
you
can
easily
lead
to
over
scheduling.
A
So
so
you
know,
with
the
you
know,
avoidance
of
over
scheduling.
We
gain
some
performance,
but
with
the
you
know
like
overhead
coming
from
python,
and
things
like
that,
you
add
some,
so
we
are
on
par
but
as
you
know,
if
we
add
more
thing
to
service
specialization,
I'm
sure
we
can
actually
do
do
better.
A
E
All
right
looks
like
zoom
is
responding,
so
hopefully
everyone
can
see
my
screen.
Let
me
know
if
the
font's
not
big
enough,
I
can
switch
to
a
different
terminal
so
right,
so
this
is
running
on
starting
out
on
a
login
note
of
corey
and
it's
just
using
the
the
basic
default
modules.
So
all
the
cray
stuff
is
loaded,
no
need
to
mess
with
any
of
that.
E
If
you
don't
want
to,
but
what
we
will
do
is
we
will
pull
in
the
module
files
for
flux
that
we
have
installed
in
global
common
software
and
then
we'll
go
ahead
and
do
a
load
of
one
of
our
dependencies,
czmq
and
flux,
and
shout
out
to
brandon
cook
for
getting
us
the
packages
installed,
so
that
this
was
much
simpler
and
cleaner.
So
we
now
have
our
environment
set
up.
You
can
see.
Flux
is
in
the
path,
and
now
we
can
move
on
to
getting
allocation
smart.
So.
E
Of
the
single
user
mode
that
don
referred
to,
where
we're
going
to
get
the
nodes
from
slurm
and
then
bootstrap
flux
within
that,
so
we'll
just
grab
two
nodes.
There'll
be
haswell
for
this
demonstration,
but
it
also
works
just
fine
on
knight's
landing.
It'll
just
be
a
different
number,
of
course,
per
node.
Obviously,
and
then
we'll
grab
an
interactive
session.
E
So
then
we're
going
to
do
an
s
run
like
dong
showed
and
flux
is
going
to
bootstrap,
just
like
mpi
would
using
the
pmi
interface,
and
then
we
could
do
a
quick
sanity
check.
Where
we'll
ask
flux,
what
resources
did
you
auto,
detect
using
the
hw,
lock
library,
and
we
can
see
that
it
found
both
nodes?
It
found
all
64
cores
across
those
two
nodes,
as
well
as
the
hardware
threads,
and
so
now
that
we
have
our
little
flux
cluster,
that's
spun
up
in
single
user
mode.
We
can
start
walking
through
the
cli
basics.
E
So
the
simplest
thing
you
can
do
in
flux
is
through
the
flux
mini
command,
so
it
has
a
couple
of
sub
commands,
one
of
which
is
run
so
we'll
do
flux
mini
run.
You
can
see
this
is
kind
of
modeled
after
each
learn
we're
trying
to
keep
the
arguments
relatively
the
same.
So
we'll
do
two
nodes.
Two
tasks
cross,
those
two
nodes
and
you'll
see.
We
got
two
different
host
names
because
it
ran
in
parallel.
E
We
can
tweak
that
and
say:
hey,
let's
run
hostname
eight
times
and
we'll
get
eight
outputs
four
from
each
node,
and
if
we
run
the
help
you
can
see
there's
a
whole
bunch
of
options
you
can
pass
to
it.
You
can
ask
for
gpus,
you
can
ask
for
standard
out
and
standard
error
to
be
labeled,
with
the
rank
that
it
came
from.
You
can
redirect
standard
out
standard
error
to
a
file.
You
can
manipulate
environment
variables,
all
that
fun
stuff.
E
E
So
the
other
command
you
can
do
is
flux,
mini
submit
and
the
difference
between
run
and
submit
is
submit
just
immediately
shoots
the
job
off.
In
the
background
to
run
returns,
the
job
id
doesn't
block
waiting
for
output.
So
you
can
see
it's
running
in
the
background
right
now
and
that's
going
to
run
forever.
So
we
can
kill
that
off
by
pasting
in
the
drop
id
and
we'll
see
that
because
we
killed
it
off,
it
had
a
negative
exit
code,
and
so
now
it's
it's
red
in
the
job
listing
with
an
failed
status.
E
So
then
the
the
third
type
of
mini
sub
command
is
flux,
mini
batch,
and
so
this
is
very
similar
to
what
you're
probably
used
to
with
slurm
with
s
batch
and
sram.
So
we
have
a
mpi
hello
script
and
an
mpi
hello
world
binary,
and
if
we
dump
the
contents
of
that
script,
it's
just
a
simple
bash
script,
that's
doing
some
module
swaps
and
then
doing
a
flux
mini
run
of
that
mpi
binary.
So
again,
this
is
really
similar
to.
E
Is
it
actually
redirects
the
standard
out
to
a
file?
So
now
that
it's
completed,
we
can
check
and
see
that
okay
completed
successfully,
and
we
can
look
at
the
file
system
to
see
there's
our
out
file.
We
can
peek
at
that
to
see
that
all
64
ranks
reported
something
and
they
did
and
then
we
can
actually
peek
at
the
top
of
it.
To
see
the
contents
and
yeah
it's
working,
just
fine
mpi
was
able
to
bootstrap
and
print
out
the
ranks
so
take
a
little
detour
into
the.
What's
underpinning
all
of
this.
E
It's
a
thing.
We
call
flux
job
spec.
So
when
you
run
flux
mini
anything
under
the
hood,
what
it's
doing
is
it's
generating,
what
we
call
job
spec,
which
is
a
yaml
file
that
describes
exactly
what
you
want
to
run.
So
here's
the
json
form
of
our
sleep,
zero
example-
and
so
you
can
see
here
up
at
the
top-
are
the
resources
that
are
being
requested.
E
We're
asking
for
a
task
slot
with
a
single
core
inside
of
it
and
then
we're
asking
that
a
sleep,
zero
command
be
run
inside
of
that
slot,
and
then
we
have
some
metadata
about
the
job
like
the
current
working
directory.
How
long
we
requested
zero,
meaning
infinite,
and
then
we
stripped
out
the
environment
with
end
remove
just
so
it
fit
on
the
screen,
but
you
would
see
all
the
environment
variables
there.
E
If
we
didn't
do
that,
and
then
we
can
tweak
this,
we
can
ask
for
two
nodes
and
multiple
tasks
and
you'll
see
that
the
node
resource
now
pops
up
there
and
the
number
of
slots
is-
has
gone
up
to
four
per
node.
This
is
a
multiplicative,
so
not
that
you
ever
really
need
to
generate
any
of
this,
but
just
want
to
kind
of
give
you
a
peek
into
what's
happening
under
the
hood.
You
can
see
the
gpus
now
pop
up.
E
One
of
the
really
cool
things
about
having
this
the
file
be
generated
is
that
it's
stored
in
flux's
key
value
store,
and
so
you
can
actually
go
back
and
grab
the
job
spec
that
was
used
for
any
job
that
was
submitted
in
the
past,
so
we
can
grab
our
mini
batch
job.
The
f2k
cg
paste
that
in
and
say
flex,
job
info,
job
id
job,
spec
and
so
that'll
dump
the
job
spec
that
was
used
to
run
that
job.
E
So
here
you
can
see
the
resources
kind
of
fell
off
the
screen,
but
in
the
attributes
you'll
see
that
the
standard
out
was
requested
to
be
redirected
to
a
file
you'll.
Also
see
here,
the
batch
script
that
we
asked
to
run
was
read
in
and
stored
in
the
job
spec.
So
now
you
have
full
prominence
of
your
job
exactly
what
the
script
was
that
was
being
run
and
what
resources
requested
for
it.
E
You
can
also
grab
what
we
call
r,
which
is
the
concrete
set
of
resources
that
the
job
ran
on
and
so
for
that
mini
batch
job.
We
ran
on
both
nodes
and
all
32
cores.
If
we
pick
one
of
the
smaller
jobs,
you'll
see
the
it's
limited
to
a
subset
of
cores,
and
so
this
is
really
nice.
If
you
want
to
see
you
know,
oh,
we
had
a
run
that
was
slow.
Maybe
it
was
the
nodes
that
we
were
given
that
were
acting
up
or
we
hadn't.
E
We
had
this
one
job
crash
out
of
the
big
ensemble.
Could
it
have
been
the
resources
that
it
ran
on
there's
a
whole
lot
of
things?
You
can
do
with
that
information
all
right,
but
off
that
rabbit
trail
on
to
the
python
api.
So,
as
don
mentioned,
we
have
this
really
nice
python
api
as
an
alternative
to
our
cli,
and
we
have
an
example
in
the
flux
framework
organization
on
github.
We
have
these
flux,
workflow
examples,
there's
a
whole
bunch
of
them.
I
encourage
you
to
check
them
out.
E
The
one
I
wanted
to
highlight
was
the
async
bulk
job
submit,
and
so
in
this
we
have
just
a
simple
python
script
and
at
the
top,
all
it's
doing
is
it's
going
to
iterate
over
the
arguments
that
we
passed
in,
read
them
in
as
files,
and
it's
going
to
assume
that
there's
job
spec
json
in
there
and
just
pass
that
json
through
to
the
scheduler
through
the
submit
async
interface,
and
so
all
that's
kind
of
happening.
Right
now
is
the
tutorial.
The
demo
is
just
creating
a
sample
job
spec
to
submit.
E
So
if
we
continue
looking
at
the
code,
there's
a
asynchronous
submission,
it
uses
features
inside
of
flux
with
reactive
programming,
so
this
callback
will
be
run
once
that
job
is
submitted,
and
then
we
just
enter
a
while
loop
waiting
for
jobs
to
complete.
So
we
went
ahead
and
copied
that
job
spec
511
times
so
now
the
script
is
going
to
read
each
one
of
those
json
files
submit
them
off
and
then
wait
for
them
all
to
complete,
we'll
get
a
nice
little
progress
bar
as
those
jobs
finish.
E
So
it's
a
really
simple
example
of
just
submitting
a
job.
The
point
is
just
to
illustrate
that
we
do
have
a
python
api,
that
you
can
do
a
lot
of
things
with
you
can
submit
jobs.
You
can
query
their
status.
You
can
wait
on
their
status.
You
can
cancel
them.
You
can
access
the
key
value,
store,
submit
messages,
submit
events,
there's
all
sorts
of
things
you
can
do
from
python.
E
So
going
into
the
scheduler
specialization
that
dong
was
talking
about,
we
have
a
little
demo
here
of
we're
going
to
submit
a
onenote
job
that
runs
for
an
hour
and
then
we're
going
to
submit
some
larger
jobs
and
show
how
the
first
come
first
serve
can
have
an
effect.
So
we
see
that
our
onenote
job
just
ran
perfectly
fine,
we're
going
to
submit
a
two-door
job,
which
we
expect
won't
run
immediately,
because
we
only
have
two
nodes
and
one
job
is
sucking
up.
E
One
of
them
currently
and
then
we'll
submit
another
one
node
job,
and
we
would
anticipate
that
this
would
run,
but
because
by
default
flux
uses
first
come
for
serve.
It's
actually
going
to
be
blocked
by
that
two
node
job,
and
so
what
we
can
do
now
is
we
can
actually
reload
our
queue
manager
module
with
a
different
queuing
policy.
E
So
it's
really
quick,
actually
just
show
you
the
modules
that
are
liv
flux.
So,
as
don
was
saying,
these
are
all
modular
micro
services
that
collectively
make
up
flux,
and
you
can
see
that
we
have
a
key
value
store
and
a
resource
and
a
job
manager
and
the
one
that
we're
going
to
be
reloading
is
the
scan,
flexi
and
q
manager.
E
So
let's
go
ahead
and
cancel
all
of
those
jobs,
get
a
clean
slate
and
then
we
can
move
on
to
high
throughput
via
hierarchical
scheduling.
So,
as
dong
mentioned,
one
of
the
key
things
that
flux
has
is
it
can
nest
itself,
and
so
we
have
a
little
utility
called
flux
tree.
That
makes
it
easy
for
you
to
spin
up
those
hierarchies
and
distribute
jobs
across
them
rather
than
having
to
use
a
whole
set
of
scripts
and
mini
batch
and
stuff.
E
So
we
type
out
flux
tree,
tell
it
to
use
two
nodes
and
32
cores
per
node
and
then
that
capital
t
is
the
topology,
so
we're
asking
it
to
create
eight
subschedulers
underneath
the
current
one
and
then
we're
asking
it
to
do.
512
jobs
across
those
eight
and
our
job
is
just
flux,
mini,
submit
sleep,
zero
and
so
that'll
run
due
to
the.
I
think
we
sped
this
up
in
the
recording,
so
it
shouldn't
take
the
full
20
some
seconds
but
yep
there.
It
goes.
E
E
So
the
other
thing
you
can
do
is
you
can
actually
spin
up
multiple
levels,
and
so
we
can
do
two
schedulers
underneath
the
current
one
and
then
eight
below
those.
We
can
also
quickly
customize
the
queuing
policy
to
be
easier.
First
come
first
serve.
We
can
customize
the
parameters
that
are
passed
to
the
the
queueing
manager,
including
the
queue
depth
or
the
match
policy.
E
Pretty
easily
and
then
finally,
what
we
can
do
is
you
can
actually
customize
the
queuing
policy
and
the
parameters
per
level,
so
you
could
say
like
easy
at
the
two
level
and
then
first
come
first
serve
at
that
leaf
eight
level
within
the
flux
tree
utility,
so
just
a
nice
little
script
to
make
it
easy
to
take
advantage
of
that
power
scheduling,
and
so
with
that
I
will
turn
it
back
over
for
q.
A.
A
C
A
Yeah,
so
so
our
design,
actually
we
design
our
protocols
to
support
elastic
growth
and
shrink
like
that,
and
then
we
implemented
the
partial
functionality
of
that
and
in
in
our
design.
We
have
a
way
to
do
that.
It
just
has
not
been
the
focus
for
us
yet
right.
Our
current
focus
is
to
make
sure
some
of
the
advanced
workflow
can
be,
can
be
run
effectively
in
production
and
then
next
making
our
system
instance.
A
You
know
the
system
instance
of
our
linux
systems
and
then,
probably
in
two
years
down
the
road
you'll
start
to
see
more
elastic,
scheduling
and
dynamic
resizing.
You
know
popping
up
in
our
production
production
software.
We
we
have
developed
our
software
stack
for
a
while,
but
you
know
with
a
large
project
like
this.
We
have
to
have
a
sense
of
priorities
and
then
definitely
with
the
cloud-
and
you
know
elastic
workloads.
C
Yeah,
there's
a
a
more
immediate
use
case
that
even
kind
of
without
the
cloud
kind
of
model
where
you
know
we
might
have
a
workflow
engine
that
needs
to
submit
things
in
and
then
you
know
that's
running
continuously
and
then
we
add
add
resources
sort
of
over
time
based
on
how
much
is
queued
up
or
something
like
that.
Yeah
I
mean
maybe
there's
other
ways
to
achieve
it,
though.