►
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title:
Pegasus Workflow Management System
Speaker:
Karan Vahi, Information Sciences Institute, University of Southern California
Abstract:
Workflows are a key technology for enabling complex scientific computations. They capture the interdependencies between processing steps in data analysis and simulation pipelines as well as the mechanisms to execute those steps reliably and efficiently. Workflows can capture complex processes to promote sharing and reuse, and also provide provenance information necessary for the verification of scientific results and scientific reproducibility. Pegasus (https://pegasus.isi.edu) is being used in a number of scientific domains doing production grade science. In 2016 the LIGO gravitational wave experiment used Pegasus to analyze instrumental data and confirm the first detection of a gravitational wave. The Southern California Earthquake Center (SCEC) based at USC, uses a Pegasus managed workflow infrastructure called Cybershake to generate hazard maps for the Southern California region. In 2021, SCEC conducted a CyberShake study on DOE systems Summit that used a simulation-based ERF for the first time. Overall, the study required 65,470 node-hours (358,000 GPU-hours and 243,000 CPU-hours ) of computation with Pegasus submitting tens of thousands of remote jobs automatically, and managed 165 TB of data over the 29-day study. Pegasus is also being used in astronomy, bioinformatics, civil engineering, climate modeling, earthquake science, molecular dynamics and other complex analyses. Pegasus users express their workflows using an abstract representation devoid of resource- specific information. Pegasus plans these abstract workflows by mapping tasks to available resources, augmenting the workflow with data management tasks, and optimizing the workflow by grouping small tasks into more efficient clustered batch jobs. Pegasus then executes this plan. If an error occurs at runtime, Pegasus automatically retries the failed task and provides checkpointing in case the workflow cannot continue. Pegasus can record provenance about the data, software and hardware used. Pegasus has a foundation for managing workflows in different environments, using workflow engines that are customized for a particular workload and system. Pegasus has a well defined support for major container technologies such as Docker, Singulartiy, Shifter that allows users to have the jobs in their workflow use containers of their choice. Pegasus most recent major release Pegasus 5.0 is a major improvement over previous releases. Pegasus 5.0 provides a brand new Python3 workflow API developed from the ground up so that, in addition to generating the abstract workflow and all the catalogs, it now allows you to plan, submit, monitor, analyze and generate statistics of your workflow.
Bio:
Karan Vahi is a Senior Computer Scientist in the Science Automation Technologies group at the USC Information Sciences Institute. He has been working in the field of scientific workflows since 2002, and has been closely involved in the development of the Pegasus Workflow Management System. He is currently the architect/lead developer for Pegasus and in charge of the core development of Pegasus. His work on implementing integrity checking in Pegasus for scientific workflows won the best paper and the Phil Andrews Most Transformative Research Award at PEARC19. He currently leads the Cloud Platforms group at CI Compass, a NSF CI Center, which includes CI practitioners from various NFS Major facilities(MF’s) and aims to understand the current practices for Cloud Infrastructure used by MFs and research alternative solutions. https://www.isi.edu/directory/vahi/
Host of Seminar:
Hai Ah Nam, Advanced Technologies Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
Thank
you,
hi.
Everyone
thanks
for
coming
to
the
nurse
data
seminar.
We're
really
excited
today
to
have
Karen
vahi
from
the
information
Sciences
Institute
at
USC
School
of
Engineering
Karen
is
a
senior
scientist
computer
scientist
at
the
science
automation,
Technology
Group
at
the
USC
information
Sciences
Sciences
Institute
he's
been
working
in
the
field
of
scientific
workflow
since
2002
and
has
been
closely
involved
in
the
development
of
the
Pegasus
workflow
management
system,
he's
currently
the
architect
and
lead
developer
for
Pegasus
as
and
is
in
charge
of
the
core
development.
A
He
also
currently
leads
the
cloud
platforms
group
at
CI
Compass,
which
is
an
nsfci
Center,
including
practitioners
and
major
NSF
facilities,
aiming
to
understand
the
current
practices
for
cloud
infrastructure
and
and
research,
alternative
Solutions
I,
so
I
got
to
meet
Karen
because
he's
also
a
2022
bssw,
better
scientific
software,
Fellowship
fellow
and
I'm
the
coordinator,
so
I'm
was
really
excited
about
his
work
and
knew
that
we
should
definitely
hear
about
it
at
nurse.
So
with
that,
I'll
give
it
to
you
Karen.
B
Yeah
thanks
a
lot
hi
for
the
introduction
and
also
thank
you
for
making
this
opportunity
available
to
formula
to
talk
about
Pegasus
workflows.
You
know
nurse
has
always
been
an
attractive
resource
for
us
to
do
more
Summit
and
run
some
workflows
in
general
and
I'm,
hoping
that
today's
talk
can
spark
conversations
about
how
to
support
Pegasus
workflows
at
nurse
better
for
the
general
user
community.
So
keeping
that
in
mind
I'm
going
to
structure
my
talk
for
about
40
45
minutes
and
leaving
last
15
minutes
for
discussion.
B
So
so
before
I
talk
about
Pegasus
is
a
short
Preamble
about
what
are
scientific,
workflows,
scientific
workflows.
In
general,
they
conduct
a
series
of
computational
tasks.
Resources
can
be
distributed
across
the
internet.
You
can
chain
tasks
together.
Reason
why
you
want
to
use
scientific
workflows
is
ease
of
use,
ability
to
give
non-verbial
access
to
sophisticated
codes.
It
provides
a
framework
to
host
or
assemble
Community
set
of
applications
and
multi-disciplinary
workflows
can
even
promote
broader
collaborations.
B
On
the
right
of
the
slide
are
simple:
workflow
building
blocks.
You
know.
Most
of
the
scientific
scientific
workflows
in
the
community
are
tags,
and
you
know
the
simple
graph
constructs
using
these.
You
can
pretty
much
create
a
pretty
complex
workflows,
so
single
jobs
can
be
considered
a
workflow
series
of
jobs,
producing
some
outputs
and
then
a
merge
step
is
one
popular
pattern:
splitting
a
large
input
data
set
and
working
on
it
in
parallel
or
running
things
in
sequence.
All
those
are
simple
examples
of
scientific
workflows.
B
So,
given
there
is
this
thing
here,
scientific
workflows,
why
would
a
user
consider
using
Pegasus
to
manage
their
workflows?
So
Pegasus
allows
automation
of
complex,
multi-stage
processing
pipelines.
It
enables
parallel
and
distributed
computations.
One
of
the
key
tenets
of
Pegasus
is
the
data
management
aspect,
so
it
does
automatically
execute
data
transfers
for
your
workflows.
B
The
description
of
in
which
you
describe
your
Pegasus
workflows
is
a
reusable
description.
It
is
reproducibility
basis,
does
a
fairly
good
job
in
recording
how
the
data
is
produced,
especially
the
runtime
provenance
of
the
data
we
handle
failures
with
automatic
job
retries
to
provide
reliability
and
Pegasus
does
keep
track
of
data
and
files
and
also
ensures
data.
B
B
B
You
need
to
be
able
to
also
deal
with
this
notion
of
the
resources
and
software
change
over
time
and
by
that
I
mean
the
underlying
middleware,
if
you're
running
large
workflows,
whether
you
are
able
to
easily
debug
and
monitor
them
and
overall
ease
of
use
So
within
Pegasus,
you
know,
we've
had
a
clear
focus
on
what
our
strengths
are.
So
one
of
the
things
which
we
focused
on
since
the
Inception
is
like
a
clear
demarcation
between
the
workflow
description
and
workflow
execution.
B
So
in
Pegasus
there's
a
notion
of
a
workflow
complaining
step
which
compiles
your
workflow
for
your
Target
execution
resource
as
your
tasks
get
executed.
We
do
monitor
how
they
are
running
and
grab
and
capture
a
lot
of
provenance
data
which
we
make
available
to
our
users
via
a
series
of
command
line
tools
and
a
web
dashboard,
and
we've
also
done
a
lot
of
work
recently
to
provide
additional
assurances
that
the
scientific
workflow
is
not
accidentally
or
maliciously
tampered
with
during
its
execution.
C
B
So
one
of
the
biggest
uses
of
Pegasus
is
at
University
of
Southern
California
in
the
Southern
California
earthquakes
in
Center,
so
there
they
do
a
sandwich
application
where
they
try
to
generate
a
hazard
map
for
the
whole
of
California
now.
So
these
workflows
are
a
mix
of
MPI
single
core
jobs
with
a
mix
of
CPU
and
GPU
codes,
so
we've
been
supporting
them,
since
2005.
B
skek
usually
has
been
on
the
Forefront
of
using
National
CI
resources
from
Terra
grid
to
exceed
and
now
access,
and
they
are
also
heavy
user
of
Poe
resources
that
ORNL
we've
done
some
work
to
get
some
skate
codes
running
at
perm
looter
in
the
past
year
or
so
another
big
user
of
Pegasus
is
ligo
where
they
run
a
one
of
the
biggest
pipelines
for
gravitational
wave
detections,
which
is
a
high
throughput,
Computing
workload
using
Pegasus
on
open
science
grid
and
like
those
own
clusters.
So
we've
been
working
with
ligo
since
2001.
B
B
You
know
like
Escape
workflows,
are
a
traditional
hvc
workflows
where
there's
a
healthy
mix
of
single
core
jobs,
but
they're
predominantly
multi-core
jobs,
MPI,
and
they
end
up
running
doing
these
studies
every
year,
where
the
attempting
to
build
more
detailed,
Hazard
Maps
for
the
California
region
in
general,
and
these
Hazard
maps
are
of
great
use
to
USGS
and
the
insurance
industry
in
California.
So
it's
a
it
has
a
definite
scientific
pickup
as
a
result
of
what's
created
so
depending
on
where
they're
running.
B
Fashion
on
open
science
grid
and
other
resources
which
are
captured
from
their
own
captive
ligo
clusters,
so
they
use
a
combination
of
pilot
jobs
which
are
managed
by
Glidden
WMS
on
OSG
and
run
these
workflows.
So
the
ligo
workflows
have
complicated
data
movements
just
because
of
the
target
execution
environments
distributed
setup
and
they
use
Pegasus
to
seamlessly
do
the
do
the
data
transfer
for
the
workflows
leverage,
all
the
caches.
We
are
necessary.
B
And
then
recently,
in
the
past
one
year
we
worked
with
the
USC
HPC
cluster,
where
they
run
a
bit
slow
cluster
and
USC
recently
got
a
delivery
of
cryo
em
microscopes.
So
these
are.
This
is
electron
microscopy
and
we've
worked
with
them
to
build
a
system
that
automatically
does
data
processing
using
Pegasus
of
all
the
data
that
is
coming
out
of
these
microscopes,
so
the
workflows
get
launched
in
real
time
as
a
new
data
set
sorted
new
images
are
acquisitioned
through
these
microscopes
and
you
know.
B
On
the
NIH
side,
there
is
a
big
Center
called
NIMH,
which
is
stands
for
National
Institute
of
medical
health,
so
they
have
a
repository
in
the
genomics
results
called
nrgr
and
there
we
Pegasus
is
used
to
curate
data
sets
that
are
submitted
by
various
psychiatric
researchers.
So
as
part
of
that
there's
a
web
interface
where
the
various
NIH
researchers
from
universities
all
over
the
US
and
worldwide
Summit,
the
phenotypic
data
sets
for
curation
and
harmonization.
So
again,
like
the
processing
is
built
on
top
of
basis.
There.
B
B
So
some
key
Pegasus
Concepts.
So
as
I
mentioned,
you
know,
Pegasus
maps
workflows
to
infrastructure,
so
Pegasus
uses
Condor
dagman,
as
our
workflow
executor.
So
Pegasus
will
take
a
high
level
description
of
your
workflow
and
translate
in
into
a
condo
attackman
description
that
can
then
be
run
that
submitted
and
run
on
your
resources
in
Pegasus.
Workflows
are
tags.
By
tags
we
mean
directed
a
cyclic
graph
of
task,
so
nodes
indicate
jobs,
edges
indicate
dependencies,
the
dependencies
can
be
controlled
flow
or
data
flow.
B
We
don't
support
while
Loops
or
conditional
branches
out
of
the
works.
Jobs
are
Standalone
executables.
So
that's
the
only
requirement
in
Pegasus
planning
occurs
ahead
of
execution.
I
will
get
into
it
in
my
coming
slides.
So
the
thing
the
way
to
think
about
this
planning
process
is
like
a
workflow
compiler.
So
it
takes
this
high
level
description
and
generates
an
executable
workflow
that
can
actually
be
executed
on
your
resource.
B
So
what
do
we
mean
by
that?
So
on?
The
left.
I
have
the
a
portable
description,
which
is
like
the
Pegasus
input
specification,
which
now
is
yaml
formatted
and
there
a
user
describes
to
Pegasus
what
tasks
they
want
to
be
executed
for
each
task.
They
refer
to
the
task
with
logical
identifiers
or
Transformations,
which
help
Pegasus
identify
what
executables
to
invoke
when
a
particular
task
runs
for
each
task.
B
The
user
describes
their
input
and
output
data
sets,
which
are
represented
as
files
and
again
with
these
when
they
are
describing
these
data
sets,
they
use
logical
identifiers
for
it.
So
if
you
were
to
think
about
it,
this
description
is
very
similar
to
what
you
would
discuss
with
your
colleague
on
a
whiteboard
when
describing
your
pipeline.
You
don't
worry
about
any
of
the
execution
details
you
just
sketch
out
or
describe
how
your
com
computations
look
like
and
what
arguments
each
task
needs
to
be
invoked
with.
B
So,
when
you
give
this
high
level
description
to
Pegasus-
and
you
say
to
Pegasus
I
want
to
run
my
workflows
on
these
clusters.
What
Pegasus
will
do
is
it
will
look
up
certain
data
catalogs
to
figure
out
where
your
data
sets
reside
and
then
automatically
annotate.
The
underlying
tag
with
additional
jobs
in
the
workflow,
so
some
of
the
jobs
which
we
add
for
the
workflow
are
oriented
around
data
management.
B
So
to
do
this
mapping.
What
sort
of
information
Pegasus
requires,
so
Pegasus
will
refer
to
three
catalogs
and
again
these
catalogs
in
the
most
default
configuration
our
yaml
formatted
files.
So
a
replica
catalog
is
a
simple
mapping
which
describes
where
your
data
sets
are
stored.
So
it's
lfnpf
and
mapping
by
elephant
naming
logical
file
names.
Pfl
is
physical
file
names
like
the
paths
to
your
data
sets
and
on
what
clusters
or
sites
those
data
sets
exist.
B
Transformation
catalog
allows
a
user
to
describe
your
executables
as
to
what
architecture
there
are
compiled
for
where
they
reside,
whether
they
need
to
be
executed
using
a
application
container
and
so
forth,
and
then
the
side
catalog
describes
your
execution
resource.
So
you
know,
for
example,
in
case.
A
D
B
Where,
for
example,
in
open
science
field,
we
are
running
pilot
jobs
or
like
not
like
the
end
Pegasus
user,
but
the
open
sign
script
Community,
they
look
at
the
job
queues
and
then
launch
pilots,
and
then
they
overlay
a
condor
pool
in
case
of
open
science
grade
and
using
what
workers
spin
up
and
are
available.
You
can
do
that.
B
B
So,
as
I
mentioned
in
terms
of
like
tools
to
generate
the
subtract
workflow,
you
know,
Pegasus
now
provides
a
very
powerful,
simple,
easy
to
use.
Python
3,
workflow
API,
where
you
can
describe
your
jobs
and
out
here.
I
have
a
simple
hello.
World
example
like
two
jobs
specified
apparent
and
a
dependent
job,
so
I
specify
that
I'm
creating
a
workflow.
D
B
Requires
there's
a
file
called
web
page
I
have
a
cool
job,
some
arguments
and
I
add
outputs,
there's
a
count
job
that
counts
the
number
of
lines
in
the
job.
You
can
connect
them
together
the
dependencies
and
write
it
out
and
if
you
look
at
this
description,
it's
also
fairly
easy
to
you
read
and
reason
about
for
anybody,
so
everything
is
being
identified
by
logical
identifiers
similar
to
our
analogy
of,
like
you
know,
you
describe
your
workflows
in
a
similar
way
to
how
you
would
describe
a
workflow
to
your
colleagues
on
a
wife.
B
So
Pegasus
5-0,
we
released
it
a
couple
of
years
back,
and
that
was
a
big
undertaking.
At
our
end,
where
you
know,
we
decided
to
focus
heavily
on
a
Python
3
interface
for
users
to
describe
their
workflows
so
python,
the
Python
3
bases
API,
is
complete
in
terms
of
like
it
allows
you
to
compose
workflows,
submit
them,
monitor
them,
configure
all
the
catalogs.
B
B
You
can
submit
your
workflows
to
run
on
one
or
more
compute
sites,
compute
sites
in
Pegasus
generally
align
well
with
clusters
at
a
higher
level.
So
if
you
have
different
clusters,
you
could
refer
to
them
as
different
compute
sites,
or
you
know
when
you're
running
in
open
science
grid.
You
know
the
pilots
are
grabbing
resources
from
Malta,
like
tens
or
hundreds
of
clusters
and
there's
one
Global
Condor
pool
that
has
been
created.
B
Users
have
an
option
of
Distributing
their
input.
Data
sets
at
multiple
servers,
so
your
input
data
says:
don't
need
to
be
co-located
with
where
your
jobs
are
going
to
be
run.
Pegasus
will
take
care
of
the
data
transfers,
depending
on
the
configuration
in
which
you
are
running
your
workflows.
Their
Pegasus
has
a
notion
of
a
data
staging
site
which
is
used
to
store
the
intermediate
data
products.
B
Have
them
co-located
if
required,
and
an
output
site
is
a
site
where
you
want
the
generated
data
products
to
be
made
available
as
your
workflows
line.
So
going
back
to
the
traditional
statement,
I
said,
I
have
a
workflow
I
need
to
run
it
on
one
or
more
compute
sites,
and
these
are
my
servers
to
use.
B
Yeah
also
like,
as
Jobs,
run
the
default
distributed
case.
All
jobs
get
wrapped
by
lightweight
Pegasus
light
instances
which,
when
a
job
starts
up
on
a
worker,
node
figures
out
what
directory
on
the
open
order
job
needs
to
run
and
how
to
pull
in
the
data
sets
just
for
that
particular
job
as
if
there's
any
checksums
to
be
done,
it
does
it
informs
check,
serves,
generates
the
outputs
and
then
pushes
them
to
the
data
staging
site
and
things
get
kicked
out.
B
B
So,
in
terms
of
a
higher
level
architecture
as
to
how
Pegasus
gets
deployed
in
multiple
user
communities,
so
a
lot
of
our
users
also
do
you
interface
with
Pegasus
directly
using
that
apis.
So
we
do
have
apis
in
Java,
R
and
python,
with
the
python
one
being
most
widely
used,
and
now
the
most
complete
one
in
terms
of
allowing
you
to
submit,
monitor
your
workflows
and
so
forth.
Pegasus
is
also
incorporated
into
workflow
gateways
such
as
hubzero
cybers,
science
gateways
and
portals
such
as
on
open
on
demand.
B
That's
something
which
I'll
get
done,
get
into
a
bit
more
detail
as
part
of
the
new
Access
program,
which
is
the
successor
to
exceed
where
Pegasus
has
a
small
part
in
the
support
part
and
through
Pegasus.
You
can
run
your
workflows
on
campus
clusters,
local
clusters,
open
Spirit
access
resources.
You
can
set
up
your
workflows
to
run
in
Cloud
environments,
and
you
can
use
a
variety
of
different
mechanisms
in
how
you
retrieve
your
data.
B
So
from
end
user
perspective,
Pegasus
does
also
make
a
user
dashboard
available
which
allows
users
to
monitor
their
workflow
executions
status,
shows
the
status
of
the
workflows
jobs
and
it
is
a
drill
down
dashboard.
So
you
have
a
home
page.
Where
you
see
all
your
workflows
running,
it's
color-coded
blue
indicates
a
workflow
is
running
red
indicates
failure
green
indicates
success,
and
then
you
can
click
on
a
workflow.
B
Look
at
all
the
jobs
and
drill
down
on
each
job
to
get
further
details,
and
we
also
provide
useful
statistics
for
the
whole
workflow
when
your
workflow
is
complete.
So
this
allows
you
to
debug
and
troubleshoot.
You
also
have
a
rest
API
available
for
to
create
your
provenance
information
if
people
want
to
integrate
it
in
a
high
level.
Web
API,
the
same
set
of
information
is
available
made
available
to
the
users
through
command
line
tools.
So
all
the
information
gets
populated
into
a
sqlite
database
at
runtime
in
your
workflow
Summit
territory.
B
B
There
is
a
command
called
Pegasus
statistics,
which
is
very
useful
for
data
end
users
to
estimate
the
usage
requirements
for
the
workflows,
so
it
tells
you
workflow,
Of,
All
Times.
How
much
CPUs
did
you
use
and
so
forth,
and
also
a
breakdown
of
job
succeeded
field
and
detailed
information
about
every
job?
B
So,
when
you're
running
your
workflows
in
a
distributed
environments,
failures
are
bound
to
happen,
so
in
Pegasus
all
jobs
have
associated
with
job
retries.
So
that
helps
with
transient
failures.
You
can
set
the
number
of
retries
per
job
by
default.
We
set
it
to
two
retries.
If
you
know
after
retrying
the
job
your
workflow
is
still
fails,
we
end
up
creating
a
rescue
tag
which
allows
you
to
resubmit
the
workflow
from
where
you
left
off
and
that's
pretty
useful.
B
Let's
say
you
have
a
long
running
workflow
and
your
cluster
goes
down
for
maintenance,
so
your
workflow
will
eventually
fail,
but
then
you
can
submit
the
rescue
tag.
If
your
jaws
generate
checkpoint
files,
then
you
know
you
have
an
option
to
mark
your
files
as
checkpoint
files
and
then
on
job
retries.
We
do
make
the
checkpoint
file
available
the
last
checkpoint
file
available
on
the
job
D
track
and
then
a
PostScript
gets
Associated
by
with
each
job
which
allows
us
to
reliably
detect
looking
at
the
job
logs
or
what
the
exact
code
was.
E
B
So
instead
of
the
user
logging
into
each
of
the
access
resources,
they
log
into
the
central
node
submit
their
workflows
and
launch
pilots
from
the
various
access
resources,
and
you
can
launch
pilots
from
multiple
access
resources
at
the
same
time
and
create
a
overlay,
condo
pool
and
then,
depending
on
the
job
that
are
in
your
queue
and
what
nodes
are
available.
Jobs
will
get
mapped
to
run
on
various
access
resources,
so
that
goes
back
to
highest
original
question
couple
of
minutes
back
about.
B
D
B
And
one
of
the
reasons
why
we
are
able
to
work
in
such
a
diverse
set
of
environments
is
that
we
have
our
own
data
transfer
tool
called
Pegasus
transfer,
which
has
the
support
for
a
number
of
different
protocols,
and
we
have
this
notion
of
one
two
one
tool
that
rules
all
so
using
this
tool.
We
create
directories
for
the
jobs
we
remove
files.
We
can
do
two-stage
transfers.
B
So,
let's
say
you're
running
your
workflows
in
a
cloud
environment
where
you
would
want
to
use
S3
as
your
data
staging
site,
but
your
input
data
sets
might
be
in
red
FTP.
We
can
allow
users
to
run
in
this
configuration
we'll
just
do
the
two-stage
transfer
internally
bring
down
the
data,
sets
locally
from
a
great
FTP
server
and
then
push
them
to
S3
and
vice
versa.
For
example,
Pegasus
transfer
has
a
notion
of
parallel
transfers,
automatic
retries,
credential
management.
B
So,
in
terms
of
data
staging
configurations,
depending
on
what
your
execution
environment
is,
Pegasus
workflows
can
be
run
in
different
modes.
So
with
Pegasus
fiber,
we
changed
our
default
data
configuration
to
be
a
pure
distributed
computing
setup
such
as
a
local
Ponder
pool
or
open
science
grid.
So
there
we
use
leverage
internal
Condor,
built
file,
transfer
protocols
to
transfer
the
data
sets,
so
the
workers
just
get
launched
on
the
various
resources
and
your
jobs
when
they
start
they
can
pull
in
data
from
the
summit
host
using
Condor
file
IO.
B
A
more
General
version
of
this
is
the
non-shared
file
system
setup,
which
can
be
used
in
a
cloud
environment
or
open
science
grid
where
you
can
use,
for
example,
in
clouds
like
S3
as
your
data
staging
area.
So
when
your
workflows
run,
Pegasus
will
create
a
parking
or
use
a
bucket
or
create
a
bucket
if
it
doesn't
exist,
or
your
data
staging
and
push
your
outputs
as
they
are
created
into
that
and
also
bring
in
the
inputs
to
that
in
the
traditional
hvc
environment,
where
there's
a
shared
file
system
available
across
all
the
nodes.
B
Staging
servers,
for
example,
ligo
relies
on
it
also
in
Pegasus.
We
do
have
a
tool
called
Pegasus
MPI
cluster,
which
allows
users
to
run
large
fine-grained
workflows
on
HPC
systems.
So
what
that
allows
us
to
do
is
that
you
can
have
a
workflow
composed
of
thousands
of
tasks
which
are
all
single
core,
but
when
they
are
run
on
a
HPC
resource,
Pegasus
translates
the
part
of
the
workflow
into
or
the
whole
workflow
into
an
API
job
which
can
be
run
using
Pegasus
API
cluster.
B
So
it
runs
the
tasks
internally
in
a
simple
master
worker
Paradigm.
So
that
allows
us
to
fit
fine
green
workflows
onto
hvc
systems.
Pegasus
does
have
good
support
for
containers.
In
terms
like
of
you
know,
users
have
an
option
of
specifying
in
the
transmission
catalog
a
container
in
which
they
want
their
jobs
to
run
in.
There
are
a
variety
of
ways
they
can
use.
Containers
they
can
refer
to
container,
they
want
to
use
and
Pegasus
will
just
stage
in
their
executables
into
the
Container.
B
So
let's
say,
like
you
know,
you
have
a
base
container
that
you
want
to
use
for
all
your
jobs.
You
can
do
that
and
the
Pegasus
will
stage
in
your
executables
into
the
Container,
or
you
can
also
tell
Pegasus
that
my
codes
are
already
packaged
into
an
application
container,
which
is
what
I
want
to
use
when
my
job
runs
internally
on
a
node
when
a
job
lines
up
Pegasus
will
automatically
pull
the
image
start.
B
The
container
Mount,
whatever
directories
are
required,
set
the
job
environment
stage
in
the
inputs
execute
your
code
stage
of
the
output,
stop
the
container
and
clean
up
so
in
terms
of
support,
so
does
Pegasus
supports
Docker,
Singularity
and
shifter,
so
all
three
are
supported.
What
Pegasus
does
is
it
treats
containers
as
input
data
dependency?
So
you
know,
containers
are
like
any
other
input.
Data
set
a
job
requires
they
can
be
staged
to
compute
nodes.
If
they
are
not
present,
I
users
can
use
images
which
are
populated
into
Docker,
Singularity
house
internally.
B
What
Pegasus
does
is,
if
you
describe
a
in
image
in
a
Docker
Hub
was
a
singularity
Hub.
You
know
as
one
of
the
steps
in
the
workflow
it
gets
exported
as
a
tar
file
and
then
just
gets
moved
around,
like
any
other
input.
Data
sets-
and
this
helps
us
to
scale
up
for
larger
workflows,
as
let's
say
like
if
you're,
using
the
same
image
for
all
of
your
jobs
in
the
workflow,
we'll
put
the
image
once
and
that
has
pricing
consideration
since
now
Docker.
B
On
terms
of
performance,
Pegasus
has
a
notion
of
job
clustering,
so
that
allows
us
to
fit
jobs.
Better
I
mentioned
one
way
is
like
using
the
BMC
tool,
or
you
can
also
cluster
jobs,
short-rently
jobs
together
and
they
get
launched
on
a
remote
node
in
a
sequent
shift
fashion.
But
you
do
optimize
on
the
scheduling
delays.
B
Pegasus
can
scale
up
to
hundreds
of
thousands
of
jobs
in
a
workflow.
In
fact,
like
you
know,
the
large
scale
like
overflows
for
the
production
workflows,
they
are
easily
above
hundreds
of
thousands
of
jobs,
and
they
also
leverage
your
concept
known
as
hierarchical.
Workflows
with
a
job
in
the
workflow
can
be
another
workflow
itself,
but
that
allows
them
to
connect
different
pipelines
together
and
do
a
production
run
in
one
course.
B
Next,
as
I
mentioned
at
the
start
of
my
presentation,
their
Pegasus,
when
you
describe
overflow,
you
tell
it
what
data
sets
in
each
of
your
job
requires.
So
that
means
that
we
can
also
do
data
reuse
where,
like.
If
your
job's
output
is
already
in
output
data
catalog,
we
can
reason
about
it
at
the
graph
level
and
promote
jobs
that
are
no
longer
required.
So
do
a
full
Pro
deduction
and
run
the
subset
of
your
workflow.
B
So
modern
I.T
systems
are
not
perfect.
Every
screen
at
the
large
big
data
sizes
we
have
started
to
see
checksums
break
down.
There
can
be
threat
of
intentional
changes
a
lot
of
times.
Users
perceive
that
they
may
be
already
protected
by
TCP
check
sums,
encrypted
transfers
and
so
forth.
But
when
you
are
running
in
a
distributed
environment
as
a
user,
you
don't
know
whether
all
the
data
staging
Ops
are
predicted
or
Not
So
within
Pegasus.
We
have
built
in
automatic
Integrity
checking
where
you
know,
users
can
specified.
D
B
We
have
a
variety
of
support
channels
available
juices.
Meeting
list
dedicated
support
me
list.
A
lot
of
our
users
now
do
use
slack,
so
you
can
request
an
invite
to
try
to
join
PC
userslack.com
in
slack
app
I'll
give
a
YouTube
channel,
and
we
also
have
a
pegasus
and
Five
Minutes
video
and
we
do
have
officers
every
couple
of
months
to
address
user
questions
in
a
price,
the
community
of
new
developments.
A
F
B
So
if
you
don't
know-
and
that's
a
good
point
as
I
mentioned,
you
know
like
when
you
describe
your
workflow,
so
you
describe
your
input,
data
sets
and
your
output
data
sets
and
there
you
have
an
option
to
tag
a
file
for
a
job
as
a
checkpoint
file
and
then
what
Pegasus
does
is
if
a
job
fails
and
the
file
that
you
mentioned
is
a
checkpoint
file,
then
it
has
slightly
different
semantics
on
the
data
life
cycle.
For
that
particular
file.
You
replicate
available
on.
F
B
So
I
think
so
originally
we
started
with
Docker
and
Singularity.
Then
we
put
in
support
for
shifter
So
within
the
Pegasus
planner.
We
have,
you
know,
abstracted
out
our
internal
interfaces
to
allow
us
to
put
in
support
and
then
there's
some
support
that
goes
into
Pegasus
light
overall
I
think,
like
our
core
interfaces,
support
edition
of
new
technologies.
We
just
haven't
gotten
around
to
using
pod
man.
B
D
B
I
was
hoping
somebody
would
ask
that
question.
So
one
of
the
things
which
I
want
to
really
clarify
about
Pegasus
dependency
on
Condor
is
that
Pegasus
by
default
is
using
Condor
as
a
submit
as
a
way
to
submit
jobs.
So
what
that
means
is
so
let's
say
you
have
your
own
local
HPC
cluster.
So,
for
example,
in
case
of
nurse
can
query,
you
only
need
to
install
Pegasus
and
Condor
on
the
workflow
login
node,
where
we
have
access
to
slow
on,
so
that
Contour
can
submit
jobs
to
slow.
B
They
just
deployed
Pegasus
and
and
Condor
on
one
of
their
login
nodes
and
user,
submit
their
workflows
from
there.
So
my
hope
is
that
you
know
this
presentation
starts
over
the
discussion
with
relevant
nurse
suicide
means
to
convince
them
to.
You
know,
make
a
workflow
login
node
available,
where
you
know
Pegasus
and
condo
can
be
made
available
to
the
users.
D
B
Like
when
I
went
through
my
access
slides,
yes,
that's
how
that
would
be,
but
the
way
to
think
about
it
in
a
default
HPC
cluster
case
would
be
a
push
right
like
so.
The
workflow
is
running
through
condo
their
jobs.
Those
jobs
need
to
be
run
on
slums.
Some
resources,
Condor
internally,
does
just
as
a
s
batch
command
to
submit
the
job,
and
that's
it
okay,
so
it
just
so
all
the
jobs.
Then
you
know
eventually
end
up
appearing
in
the
Sloan
queue.
E
B
Users
feedback
when
they
run
on
locals,
learn
clusters
is
you
know
we
can
run
workflows
with
thousands
of
tasks?
You
try
to
run
the
workflow
in
thousands
of
tasks
and
dump
all
those
jobs
in
a
slum
queue.
Often
that
creates
a
lot
of
issues,
but
that's
not
an
issue
at
our
end
because
you
know
UV
can
control
how
many
jobs
get
submitted
to
slow
them
at
any
given
time
in
line
with
use
of
policies.
A
C
B
Available
so
and
then
literally,
it's
yum
installed
for
two
packages,
and
that
should
create.
So
you
know
thing
to
keep
in
mind
is
like
Pegasus
itself
runs
into
user
space,
but
since
Condor
is
also
like
has
Zips
demons
Etc,
so
Contour
does
do
user.
Switching,
so
Condor
needs
to
be
started
as
root
through
the
system.
Ctl
standard
practices
like
the
same
way,
you
would
run
a
Patcher
or
anything
else.
E
B
B
B
E
B
It
they've
done
it
in
the
past,
but
they
mainly
do
it
based
on
what
you
know
like
how
much
the
demand
is
for
that
and
like
the
way
I
framed
my
answer
that
does
not
get
rid
of
contour
for
you
directly,
because
if
you
know
like
Pegasus
does
build
on
top
of
condor
this,
we
have
a
lot
of,
like
our
you
know,
provenance
information,
all
the
monitoring
stuff
built.
On
top
of
that.
So
you
know
as
a
proof
of
concept
use
case.
We
could
remove
that
dependency,
but.
C
Oh
yes,
I
have
a
question
about
how
many
tasks
so
in
your
slide,
you
showed
Pegasus
can
handle
like
a
hundreds,
thousands
of
tasks.
I
suppose
you
mean
those
are
either
simultaneously
or
kind
of
lined
up
tasks,
so
I
would
know
like
within
the
Learning
System.
Do
you
know,
like
any
example
of
I,
mean
how
many
s
runs?
You
were
able
to
run
and
run
smoothly.
B
So
yeah
so
when
I
say
number
of
tasks,
I
refer
to
number
of
tasks
in
the
workflow
so
which
means
number
of
tasks
in
the
underlying
graph.
So
that's
very,
like
end
user
specific
in
terms
of
how
many
jobs
run
at
any
given
time,
because
you
know
a
job
can
only
be
released
when
all
the
parent
jobs
have
succeeded
successfully.
B
So,
for
example
like
if
I
were
to
go
back
and
go
back
to
ligo,
so
in
ligo
for
the
production
runs,
you
know,
they're
running
hierarchical,
workflows
with
and
each
of
the
sub
workflow
has
I
would
say
around
70
to
90
000
jobs,
and
then
sometimes
they
use
jaw
clustering
in
Pegasus
to
reduce
the
number
of
jobs
that
gets
submitted
to
sran
I,
like
picking
upon
your
analogy,
or
they
can
be
individual
subjects.
But
you
know
again,
like
the
whole
reason
about
workflows.
B
Is
that
you
know
your
workflow
want
time
and
your
whole
workflow
CPU
time
they
scale
up
in
a
different
way.
So
you
know
your
workflow
volt
time
relatively
remain
short.
But
if
you
have
a
lot
of
resources
available,
you
can
get
a
lot
of
work
done,
based
on
the
inherent
parallelism
exposed
by
your
workflow
structure.
C
Yeah,
okay,
sorry
I
have
one
more
question,
so
I
think
you
had
some
examples
show
on.
Like
a
you
know,
all
crazy
machines
and
something
like
that,
then
I
wonder
we
all
live
in
a
certain
like
batch
limit.
So
when
certain
the
markets
allow
the
time
yeah
I
mean
I
want
to
know
how
many
human
time
will
be
involved
with
those
failed
jobs
because
of
the
work
so.
B
You
know
that
goes
back
to
my
earlier
question
like
where
you
know
like
having
the
control.
Cube
goes
useful,
so
we
are
able
to
control
like
as
a
user,
specify
that
how
many
jobs
we
need
somebody
to
know
particular
firm
resource
at
the
time
or
not
right.
So
your
workflow,
you
can
launch
a
workflow
with
100
000
tasks,
but
let's
say,
like
you
know
that
on
a
particular
cluster,
the
enforcement
is
only
5
or
10.
Jobs
can
be
running
per
user
at
a
given
time.
We
can
set
that
configuration
in
Condor.
B
So
even
though
the
jobs
will
be
ready
to
run
in
the
control
queue,
they
will
not
be
submitted
to
slow
for
execution.
So,
for
example,
like
means
Escape
transfer
on
olcf
resources.
They
are
cognizant
of
that.
So
the
and
you
know
like,
depending
on
what
their
workflow
structure
is.
Sometimes
they
use
pilot
jobs,
or
sometimes
they
use
a
push-based
mechanism.
B
A
Great
so
we're
reaching
the
top
of
the
hour
and
I
think
there's.
You
know
certainly
more
discussions
we
had
to.
If
there
are
any
less
questions
we
can
address
them,
or
we
can
also
take
this
offline
and
have
some
additional
questions
about
some
of
the
more
harder
requirements
and
how
this
actually
gets
deployed.
B
B
Thank
you
so
much.
Thank
you
very
much
yeah
and
like
from
our
end,
I
think
like
bone
tutor
is
a
very
attractive
resource
address.
That's
coming
up,
for
example,
you
know
skegly
reach
out
to
us
a
couple
of
months
back
to
see
if
they
could
use
form
looter.
So
you
know
this
definite
interest
at
RN
to
use.