►
From YouTube: 2022-01-13 - Gregory Kurtzer - The Next Generation of High Performance Computing: HPC-2.0
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title:
The Next Generation of High Performance Computing: HPC-2.0
Speaker:
Gregory Kurtzer, CEO of Ctrl IQ, Inc. and Executive Director of Rocky Enterprise Software Foundation / Rocky Linux
Abstract:
We’ve been using the same base architecture for building HPC systems for almost 30 years and while the capabilities of our systems have increased considerably, we still use the same flat and monolithic architecture of the 1990’s to build our systems. What would the next generation architecture look like? How do we leverage containers to do computing of complex workflows while orchestrating not only jobs, but data? How do we bridge HPC into the 2020’s and make optimal use of multi-clusters and federate these systems into a larger resource to unite on-prem, multi-prem, cloud, and multi-cloud? How do we integrate with these resources in a cloud-native compatible manner supporting CI/CD, DevOps, DevSecOps, compute portals, GUIs, and even mobile? This isn’t a bunch of shoelace and duct-tape on top of legacy HPC, this is an entirely new way to think about HPC infrastructure. This is a glimpse into HPC-2.0, coming later in Q1 of 2022.
Bio:
Gregory M. Kurtzer is a 20+ year veteran in Linux, open source, and high performance computing. He is well known in the HPC space for designing scalable and easy to manage secure architectures for innovative performance intensive computing while working for the U.S. Department of Energy and joint appointment to UC Berkeley. Greg founded and led several large open source projects such as CentOS Linux, the Warewulf and Perceus cluster toolkits, the container system Singularity, and most recently, the successor to CentOS, Rocky Linux. Greg’s first startup was acquired almost 2 years ago and now he is working on software infrastructure, including Rocky Linux as well as building a cloud native, cloud hybrid, federated orchestration platform called Fuzzball.
Hosts of Seminar:
Jonathan Skone, Advanced Technologies Group
Glenn Lockwood, Advanced Technologies Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
B
A
Pleasure
to
virtually
introduce
greg,
kurtzer
who's,
a
the
current
ceo
of
control,
iq
and
many
of
you
know
him
from
his
past,
exploits
at
lbnl
and
elsewhere
in
the
open
source
software
space,
both
as
a
developer
and
founder
for
several
projects,
including
singularity
werewolf,
rocky
linux,
perseus
among
others.
So
today,
greg's
going
to
give
us
a
high-level
view
of
what
he
sees
as
the
future
for
hpc,
involving
the
orchestration
and
management
of
disparity
resources
across
various
resource
centers.
A
B
Thank
you,
jonathan,
and
it's
great
to
see
everybody.
I
see
a
lot
of
familiar
names
here,
so
it's
great
to
see
y'all
again
at
some
point,
we're
all
gonna
have
to
get
together,
as
I
come
back
up
to
the
lab
and
I'm
still
in
berkeley.
So
at
least
part
time,
I'm
in
berkeley
so
it'd
be
great
to
see
you
all
again
face
to
face
at
some
point.
B
You
know
I'm
an
engineer.
First
and
foremost,
my
my
new
role
is
basically
just
basically
playing
with
excel
spreadsheets
and
powerpoints
at
this
point,
but
I've
now
been
coined
and
tagged
as
coming
off
a
little
bit.
Marketing
and
sales
wise,
which
is
just
hurts
my
hurts
my
heart
to
be
called
that,
but
I
don't
mean
this
to
be
a
marketing
talk.
This
is
more
about
just
really
how
we're
thinking
about
this
and
what
got
us
to
this
point.
B
So
if
you
do
have
technical
questions
at
the
end,
absolutely
please
hammer
me
with
them.
I'm
not
at
all.
Thinking
that
this
is,
you
know
the
end-all.
This
is
just
how
we're
thinking
about
this,
and
what
and
and
again
I'm
going
to
kind
of
walk
you
through
how
we
got
to
this
place
and
I
would
love
feedback.
B
We've
been
saying
this
in
hpc,
for
as
long
as
I've
been
in
hpc
that
enterprise
and
cloud
needs
to
well
cloud
wasn't
existing
when
I
first
got
an
hpc,
but
enterprise
needs
to
be
taking
over
a
lot
of
these
hpc
resources,
and-
and
this
is
going
to
translate
directly
to
what
enterprise
is
doing
and
and
whatnot
yeah.
That's
that's
so
far.
That's
never
happened,
but
I
think
now's
different.
B
Of
course
everyone
always
thought
that
previously.
That
now
is
different,
but
I
do
think
now
is
different
and
I'll
explain
why
in
just
a
moment,
but
this
was
originally
kind
of
started
off,
being
kind
of
hpc
for
non-hp
sears
and
what
I've
learned
since
then
is
most
hpc.
Sites
are
thinking
about
this
as
well
and
they're.
B
Thinking
that
we
need
to
be
modernizing,
we
need
to
be
changing
and
rethinking
kind
of
the
base
structure,
the
fundamental
structure
of
high
performance
computing
and
I'm
kind
of
going
off
script
a
little
bit
now,
just
because
I'm
riffing
on
this,
but
that
was
what
we
were
founded
to
do.
That
was
the
beginning
of
this.
Now,
as
we
looked
at
the
the
portfolio
of
projects
that
I've
been
associated
with,
we
realized
that.
Well,
it's
not
just
about
the
middleware.
B
So
werewolf
kind
of
became
part
of
the
planned
go-to-market
strategy
for
what
it
is
that
we
were
building,
and
I
also
as
as
jonathan
mentioned,
I
created
a
singularity
and
I
created
singularity
when
I
was
still
back
at
lbl,
and
I
collaborated
quite
a
bit
with
with
doug
and
shane
and
absolutely
appreciated
the
relationship
that
I
have
with
with
with
them
fantastic.
I
think
I
saw
I
think
I
saw
doug.
I
think
I
saw
you
here.
B
So
we
put
together
a
vote
and
ended
up
with
apptaner
and
then
lastly,
rocky
linux,
I'm
also
a
founder
of
centos,
which
is
used
obviously
by
a
lot
of
high
performance
computing
sites,
among
others
in
the
ecosystem,
and
when
red
hat
decided
to
pull
the
plug
on
that
and
pivot
the
project
you
know
rocky
linux
was
born.
I
was
associated
with
that
and,
as
a
result,
a
lot
of
people
now
have
asked
us
to
also
do
support
for
rocky
linux.
B
So
this
is
kind
of
what
we
do
at
a
at
a
high
level
and
now
that
I'm
past
that
I'm
going
to
jump
right
into
kind
of
the
tech
of
fuzzball
and
and
what
got
us
here.
So
I'm
assuming
that
pretty
much
everyone
in
the
room
is
familiar
with
the
traditional
beowulf.
It's
an
almost
30
year
old
architecture
and
it's
served
their
needs
extraordinarily
well.
Almost
every
hpc
system
sold
and
used
and
architected
is
based
on
this
model.
B
We've
done
a
lot
to
be
able
to
scale
this
model
up
and
be
able
to
do
additional
things
with
this.
But
the
core
model
is
a
relatively
legacy
architecture
at
this
point,
and
it's
made
it
very
difficult
to
integrate
with
a
lot
of
capabilities
that
have
been
accelerated
in
cloud
hyperscale
and
enterprise.
Things
like
orchestration,
ci,
cd
and
there's
a
number
others
that
I'll
show
in
in
a
later
slide.
B
But
this
monolithic
kind
of
legacy
architecture
has
made
it
very
difficult
to
incorporate
a
lot
of
those
those
capabilities
and
any
pink
floyd
fans
in
the
room.
You
know
we
haven't
really
innovated
very
much
on
the
base
architecture
side,
but
we've
absolutely
innovated
on
everything
else,
but
the
base
architecture
could
definitely
use
a
little
bit
of
rethinking,
as
I'm
as
I've
been
considering.
B
B
I
think
containers
and
the
desire
to
integrate
with
containers
all
of
a
sudden
caused
the
hpc
community
to
actually
look
at
what
the
rest
of
the
enterprise
and
cloud
was
doing,
and
I
think
that
became
kind
of
the
gateway
drug.
So
it's
pandora's
box
or
pandora's
container.
I
think,
really
started
that,
but
in
addition
to
that,
we're
also
seeing
again
we're
seeing
a
lot
more
job,
diversity
and
types
of
jobs
coming
in
and
eight
traditional
hpc
centers
are
now
being
tasked
to
come
up
with
solutions
for
non-traditional
high-performance
computing
applications
usage
models.
B
When
I
was
talking
about
building
a
cluster
with
many
enterprises,
a
common
theme
would
be
well.
Ssh
is
for
system
administrators,
not
users.
As
a
matter
of
fact,
even
most
of
the
the
bigger
enterprises
that
I
spoke
to
said
actually
ssh
isn't
even
used
by
system
administrators
anymore.
Everything
should
be,
you
know,
deployed
and
provisioned
and
managed
via
some
other
form.
B
So
usage
models
have
shifted
considerably,
but
we're
still
using
this
in
in
traditional
high
performance
computing,
we're
seeing
a
lot
of
issues
with
security
and
we're
trying
to
make
things
easier.
You
know
we
used
to
have
a
joke
that
almost
every
hpc
system
we
built
was
serial
number
one
in
the
sense
that
every
system
was
kind
of
custom
built
and
it
was
there
was
not
a
huge
amount
of
of
cookie
cutter
and
scale
for
for
the
commodity,
sweet
spot,
high
performance
computing.
It's
been
a
number
of
initiatives
to
try
to
do
that.
B
B
All
right
so
just
to
kind
of
demonstrate
this
I'm
going
to
walk
everyone
through
a
story
and
this
this
happened.
This
is,
and
it's
not
quite
verbatim,
but
you
know
the
names
have
been
changed
to
protect
the
innocent
and
and
whatnot,
but
the
general
gist
of
this
is
exactly
kind
of
how
the
conversation
went.
B
And
so
I
was,
I
was
contacted
by
one
of
the
biggest
social
media
companies,
I'm
not
going
to
name
them,
but
you
can
probably
guess
in
the
world
who
basically
came
to
me
and
brought
me
into
their
office
and
said:
hey
greg,
we're
going
to
be
the
biggest
hpc
system
in
the
world
and
of
course
you
know,
I
ask
how
all
right,
but
we're
going
to
take
a
rack
and
in
that
rack
we're
going
to
make
it
really
dense.
We're
going
to
put
like
100
compute
nodes
in
there
we're
gonna
put
disaggregated
memory.
B
You
know
composable
hardware,
storage
system,
network
subsystem
and
a
control
surface
and
that's
gonna
be
our
base
system,
so
this
is
kind
of
what
they're
talking
about
you
know.
Some
of
us
in
high
performance
computing
have
been
thinking
about
this,
but
this
is
really
more
of
kind
of
an
ocp
type
way
of
thinking
of
the
world
or
a
hyperscaler
way
of
thinking
of
the
world
and
and
they're
really
kind
of
moving
towards
a
lot
of
composable
resources
and
disaggregated
memory
is
an
interesting
one.
B
You
know
remote
gpus,
pci
switches
and
switching
and
all
that
sort
of
composable
infrastructure,
and
so
it's
definitely
really
kind
of
a
neat
neat
system.
But
you
know
it's,
it's
really
not
big,
but
their
response
to
me
was
well
we're
hyperscalers,
that's
just
one
system
we're
gonna
put
together
thousands
of
these
and
so
yeah.
I
start
thinking
well,
there's
one
of
these
there's
25
of
these
at
2
500
nodes.
There's
a
hundred
of
these
at
10
000
nodes
and
yeah.
I
did
math
in
my
head
really
really
good
and
quick
like
yeah.
B
B
We
typically
think
of
it
from
the
other
perspective
of
let's
make
a
giant
flat
system
and
just
make
that
as
big
as
possible.
So
it's
a
kind
of
it's
a
subtle
but
a
different
way
of
thinking
about
this,
and-
and
there
was
more
right.
Obviously
you
can't
put
this
all
in
one
place.
I
jokingly
say:
you'd
need
a
nuclear
reactor
to
power
it
in
an
ocean
to
cool
it
and
and
where's
your
data
right.
B
So
it
has
to
be
scattered
all
over
the
world
or
at
least
geographically,
but
the
data
that
they
were
talking
about
is
exabytes
in
size,
and
that
is
sharded
all
over
the
world
which
should
be
you
know
any
anyone
we
all
should
be
probably
thinking
well,
how
are
you
going
to
match
up
than
where
the
workloads
are
running
to
which
cluster
onto
which
data?
And
where
is
that
data?
And
how
do
you
manage
that
data
right?
You
don't
want
to
mirror
exabytes
of
data
because
they
keep
their
data
mostly
in
object
store.
B
So
you
don't
want
to
mirror
that
onto
spinning
disks,
so
you
have
to
be
thinking
about
where.
How
are
you
going
to
manage
the
data
as
part
of
this
they've
got
massive
numbers
of
users
and
I
should
say,
they're,
not
just
all
users.
Many
of
these
are
sources
of
different
types,
but
they
do
have.
You
know
tens
of
thousands
of
of
users.
Even
I
already
talked
a
little
bit
about
job
diversity
event
driven
all
right.
Not
you
don't
always
want
to
submit
a
job
by
hand
right.
B
A
lot
of
this
needs
to
happen
automatically
based
on
various
hooks,
maybe
pipelines.
It
needs
to
be
absolutely
federated
right.
We
we
need
to
completely
distribute
this
and
manage
all
that
distribution
of
resources
and
distribution
of
of
jobs
and
workflows
in
a
way
that
is
a
function
of
resources.
Where
are
those
resources,
physical
resources,
the
cost
of
running
on
those
resources
and
data,
and
when
we're
thinking
of
data,
we're
thinking
of
data,
locality,
data
mobility,
gravity
and
security?
B
B
Obviously
this
is
you
know
the
p
and
hpc
is
performance.
We've
got
to
be
performant,
you
know,
just
as
as
you
look
at
this
from
an
enterprise
perspective.
Most
people
are
just
thinking
about
this.
Well,
let's
just
throw
kubernetes
to
be
our
our
whole
job,
resource
management
system
and
whatnot,
the
people
that
have
done
this
that
I've
spoken
with
most
of
them
have
said
yeah,
but
it's
not
really
great
on
the
performance
side
and
it
kind
of
gets
in
the
way.
B
So
we've
got
to
build
something,
that's
doing
better
than
kubernetes
and
built
design.
For
this
we
have
to
be
able
to
hit
massive
scale
and
not
just
horizontal
and
vertical
scaling,
but
also
the
the
meta
orchestration
of
the
federated
scaling,
which
maybe
it's
almost
like
a
z-axis
scaling
all
right.
We
need
to
be
thinking
about
data
management,
composable
workflows,
I'm
just.
I
don't
want
to
read
off
a
wall
of
text,
because
that
just
feels
weird
to
me.
So
I'm
looking
through
here
and
you
can
kind
of
see.
B
I
mean
cloud
integration,
there's
one
that
you
know
since,
since
I
was
even
back
at
lbl
since
about
2005
2006
there's
been
initiatives
to
move
hpc
into
the
cloud
and
and
we've
always
been
asked
by
management
about
moving
hpc
into
the
cloud.
And
what
are
we
going
to?
How
are
we
going
to
do
that?
The
current
hpc
model
makes
that
extraordinarily
difficult,
it's
possible,
but
but
it's
very
difficult
and
it's
a
little
clunky.
B
But
if
we
can
do
this
as
a
first
class
citizen
in
this
in
the
system
right
that
kind
of
changes,
the
paradigm
I'm
going
to
move
on
I'll
share
these
slides.
So
if
anybody
wants
to
come
back
to
them
afterwards,
so
many
various
student
people
in
hpc
have
been
able
to
build
and
to
build
in
some
of
these
capabilities
on
top
of
traditional
hpc,
definitely
possible
building
all
of
them
in.
I
would
challenge
anyone
to
to
do
and
I
would
actually
challenge
the
logic
of
it.
It's
it's
to
me
it.
B
It
seems
like
it's
overly
complicated,
trying
to
shoehorn
in
a
series
of
capabilities
around
a
legacy
architecture.
It
just
seems
easier
and
and
better
just
to
rethink
that
architecture
and
see
you
know.
Is
there
a
better
way
of
doing
this?
That
is
better
aligned
and
there's
a
lot
of
capabilities
that
have
come
out
of
enterprise
cloud
and
hyperscale
that
historically
have
been
distinct
and
separate
from
what
we're
doing
in
high
performance
computing.
B
I
think
if
we
start
looking
at
how
do
we
build
a
more
capable
and
compatible
infrastructure?
All
of
a
sudden
we
can
start
cross-pollinating
and
further
innovating
and
then
leveraging
each
other's
kind
of
major
value,
ads
and
innovations,
and
I
think
that
that's
the
way
that
we
need
to
be
thinking
about
this
moving
forward.
B
B
This
particular
workflow
and
resources
may
be
different
for
each
individual
job,
that's
running
within
a
workflow,
and
so
we
need
to
be
able
to
manage
that
accordingly
and
with
all
of
that
kind
of
inclusively,
we
can
now
start
to
create
composable
hpc
resources
for
each
particular
job
and
workflow.
That
needs
to
run,
and
so
I've
got
a
bunch
of
examples.
I'm
going
to
start
with
a
example
workflow
and
I'm
going
to
go
into
the
architecture
and
I'm
going
to
show
some
more
workflows.
B
Excuse
me,
so
this
is
a
workflow
that
we
always
just
kind
of
seem
to
come
back
to
because
it
describes
a
number
of
different
things
within
it
at
the
top.
You
can
see
in
the
image
you
know,
abstractly
we're
basically
taking
some
input
data.
We
run
it
through
a
pipeline
of
containers
and
then
we
persist
the
output
data
from
that
to
another
source
so
again
bring
ingest
of
of
data,
run
it
through
your
your
graph
pipeline
and
then
persist.
What
you
need
again
to
another
could
be
an
object
store.
B
It
could
be
a
data
lake.
It
could
be
the
same
place
that
you
got
the
original
data
source
from,
but
what
we're
creating
is
a
way
of
describing
the
entire
workflow
in
a
way
that
that
workflow
could
run
anywhere
and
we
leave
it
then
up
to
the
system
to
decide
based
on
the
requirements
of
the
workflow
and
what's
inside
the
workflow,
we
can
make
logical
decisions
as
to
where
that
job
or
workflow
should
run
so
just
to
kind
of
run
through
this
real
quick.
B
B
This
volume
is
ephemeral,
so
this
volume
will
be
blown
away
as
soon
as
the
job
is
done.
We're
going
to
ingress
a
few
sources
now,
one
of
the
things.
Even
though
this
volume
is
ephemeral,
we
have
the
ability
to
cache
the
sources,
so
subsequent
jobs
that
require
the
same
sources
could
have
an
affinity
to
land
on
the
same
resources.
B
B
So
in
terms
of
how
this
operates?
We
first
create
the
volume
we
bring
in
the
ingress.
Egress
happens
when
the
job
pipeline
is
done,
so
I'm
going
to
jump
over
to
jobs
first,
job
that
we're
going
to
do
is
just
to
untar
this
data.
You
can
see
we're
pulling
over
a
tarball
here,
putting
it
at
input.tar.gz
and
we're
mounting
this
volume
at
slash
data,
so
we're
basically
going
to
take
that
input.tar
z,
that
we
wrote
right
here
and
we're
just
going
to
splat
it
out
to
slash
data.
B
The
next
job
is
basically
a
ufs
model
notice,
we're
running
in
a
different
container
for
each
one
of
these.
So
this
ufs
model
is
an
mpi
application.
We're
just
pulling
this
straight
out
of
docker
hub
we're
going
to
basically
specify
you
know
what
you
know
where
the
volume
is
getting
mounted
it's
getting
mounted
in
a
different
location.
Here,
it's
an
mpi
job
and
this
by
the
way
is,
is
very
simplistic.
B
There's
a
lot
more
capabilities
and
resources
that
you
can
specify
associated
with
a
particular
job
like
what
are
the
specific
node
requirements.
How
many
cores
do
you
need?
How
much
memory
do
you
need?
How
long
is
the
job
going
to
run?
What
are
policies
you're
going
to
associate
with
this
job
and
so
on
and
so
forth?
B
So
there's
a
lot
of
capabilities
in
here,
but
this
is
a
very
simplistic
example
just
to
kind
of
demonstrate
the
fundamental
concept
and
we're
going
to
run
this
weather
model
through
an
open,
mpi,
well,
mpi
implementation
and
run
it
across
128
notes
and
at
the
end,
once
that
job
is
done
and
by
the
way
you
can
see.
There's
a
job
pipeline,
but
via
the
requires
tag,
so
ncl
requires
ufs.
Ufs
requires
ontar.
B
We
have
no
whatsoever,
I'm
looking
for
no
religion
in
terms
of
the
format
of
this
file,
it's
parsed
by
a
particular
piece
of
the
of
the
the
system
that
can
be
easily
modified
and
or
completely
replaced.
So
if
there's
a
different
organization
or
or
api
that
we
want
to
implement
here,
it's
actually
it's
very
easy.
B
So
I'm
going
to
start
with
the
management
cluster
here
in
the
bottom
left
we're
going
to
run
some
basic
operating
system,
we're
going
to
run
kubernetes
and
we
run
kubernetes
only
in
the
management
cluster,
because
what
we've
created
with
fuzzball
is
in
fact
a
microservice
platform.
So
we
want
to
run.
On
top
of
you
know
a
microservice
orchestrator.
B
You
could
use
kubernetes
use
any
version
of
kubernetes
that
you
want.
We
don't
care,
we
have
a
fantastic
relationship
with
rancher
and
soos,
so
we
would
definitely
you
know
like
it
if
somebody
wants
to
run
ranchers
sues,
but
you
can
run
anything
as
I
said
anything
that
you
want.
We've
packaged
up
a
version
of
kubernetes
just
to
make
it
easier
for
people
that
doesn't
have
another
kubernetes,
but
whatever
you
want
to
do
here
is
absolutely
fine.
Fuzzball
orchestrate
will
come
via
helm,
sit
on
top
of
kubernetes
and
create
kind
of
the
management
subsystem.
B
Now
the
resource
cluster
is
very
lightweight
each.
At
least
each
node
is
very
lightweight.
What's
installed
in
the
node
again
we're
gonna
do
some
sort
of
provisioning.
Whatever
you
want
to
use
here
is
fine.
Whatever
operating
system
you
want
to
use
here
is
fine,
we're
not
bound
by
any
of
the
kind
of
underlying
fundamentals.
You
can
do
anything
you
want
substrate,
fuzzball
substrate
is
called
substrate
specifically
because
it
is
the
foundation
of
where
all
jobs
and
workloads
will
run.
B
It
is
kind
of
the
the
init
process
in
a
matter
of
speaking
of
all
proc,
all
other
compute
processes.
What
fuzzball
substrate
actually
is.
It
is
a
api
driven
container
hypervisor
and
runtime
it.
The
api
is
specific
around
leases
and
you
have
the
ability
to
ask
substrate
for
a
lease,
and
you
can
do
this
either
by
hand
via
a
cli
or
it
will
connect
into
orchestrate.
So
what
could
happen?
Is
you
can
say?
I
need
two
gpus.
I
need
this
number
of
cores
on
this
many
sockets
or
these
particular
sockets.
B
Orchestrate
is
a
set
of
micro
services
that
will
control
fuzzball's
substrates,
so
fuzzball
orchestrate
has
microservices
like
a
workflow
engine,
there's
a
data
mover
image,
service
volume,
manager,
job
scheduler
and
so
on
and
so
forth.
So
when
a
workflow
comes
in
from
a
user
or
from
a
ci
pipeline,
it
will
automatically
be
ingested
by
orchestrate
and
orchestrate
will
start
setting
up
the
composable
resources
to
to
satisfy
this
job
requirement.
B
It
will
basically
again
it'll
move
the
data,
and
you
can
kind
of
see
this
outlined
here
on
the
right
right.
It'll,
ingress
data
create
a
volume,
create
a
job
pipeline
of
containers,
sit
on
top
of
that
and
then
egress
your
data
out
of
there
anything
you
want
to
persist,
and
it
manages
all
of
this.
The
user
api
for
this
is
coming
in
via
grpc
there's
no
ssh.
B
Every
api
in
the
system
is
governed
by
an
im
policy
engine
that
we
wrote
and
we
designed
to
basically
manage
policies
of
what
every
user
is
allowed
to
do
for
every
step
of
the
process.
So
basically,
this
workflow
comes
in
everything
has
been
defined
by
system
administrators
and
by
organizational
policies
on
what
each
user
is
allowed
to
do.
What
are
the
defaults
for
each
user
and
then
how
this
whole
thing
operates?
We've
had
a
number
of
requests
of.
B
B
So
imagine
you
have
one
of
those
clusters
up
in
aws.
You
have
one
in
azure:
maybe
you
have
one
cluster
on-prem?
Maybe
you
have
another
site?
Another
cluster
on
another
site,
maybe
there's
another
organization
that
you're
working
with
and
they've,
got
a
cluster
and
so
on
and
so
forth
via
the
third
piece
of
fuzzball,
which
is
called
federate,
and
what
federate
allows
us
to
do
is
to
create
organizational
policies
based
on
where
should
applications
land.
B
So
as
the
as
the
workflow
comes
in,
we
now
get
to
make
scheduling
and
policy
decisions
that
are
kind
of,
as
I
alluded
to
before,
they're,
unlike
typically
the
the
types
of
policy
decisions
we've
made
before
in
the
past.
We
get
to
make
decisions
based
on.
Where
are
those
resources
right
is
this?
Is
this
workflow
coming
in?
Does
it
require
arm?
Does
it
require
power?
Does
it
require
x86?
B
Does
it
require
particular
gpus
that
maybe
one
side
has
and
another
site
does
not?
What
is
the
cost
of
running
at
one
site
versus
another
and
again,
if
you're
talking
about
on-prem
on-prem
sites?
Maybe
it's
not
that
big
of
a
deal,
but
if
you're
talking
about
cloud
right,
there's
definitely
a
factor
there
that
we
need
to
be
considering
or
if
you're
talking
about
running
on
a
collaborator
site.
You
know
maybe
there's
a
cost
associated
with
that
and
then,
of
course,
the
data
right
and
we
have
data
locality.
B
I
think
it
was
mobility,
gravity
and
data
security
that
we
need
to
be
considering.
So
all
of
those
decisions
are
now
being
made
at
the
fuzzball
federate
level.
So
when
a
workflow
comes
in,
we
now
can
say:
okay
well,
the
data
for
this
exists
over
here,
but
the
compute
requirement
exists
over
here,
but
maybe
we
can
run
it
up
in
the
cloud
and
do
this
and
that
whole
policy
can
be
defined
by
the
organization
users.
Using
this
system
may
never
even
have
access
or
even
knowledge
to
what
the
policy
is
from
their
perspective.
B
Everything
is
completely
transparent.
They
just
submit
this
workflow
and
it
runs
where
it
needs
to
run
all
right.
I'm
going
to
keep
blowing
through
this
real
quick
get
through
this
and
then
I'm
hoping
that
there's
going
to
be
some
cool
dialogue
and
q
a
afterwards.
So
I'm
going
to
touch
real
quick
on
here.
Here's
just
what
some
of
the
cli
looks
like
you
create
a
context,
the
context,
so
a
fuzzball
cli
can
actually
be
connecting
to
many
separate
fuzzball
instances.
B
B
You
can
submit
a
workflow
to
the
system
again.
This
is
running
on
a
laptop
or
on
a
workstation,
not
on
the
system.
We
are
not
ssh
into
the
system,
so
you
can
start
a
workflow.
You
can
monitor
a
workflow.
You
can
attach
to
workflows.
You
can
exec
a
you
know,
a
shell
in
a
workflow
and
something
I'll
show.
In
just
a
moment.
B
B
Persistent
volumes
is
another
type
of
volume
that
we
have.
In
addition
to
ephemeral,
persistent
volumes
is
basically
could
be
used
when
a
particular
system
or
or
resource
has
data
that
is
associated
with
that
and
already
kind
of
pre-populated.
So,
on
the
lbl
side,
when
I
was
when
I
was
there,
we
worked
with
the
astrophysicists
with
a
a
project
called
sloan,
which
was
a
digital
sky
survey
and
they
had
massive
amounts
of
data
associated
with
a
particular
cluster,
and
they
were
they
were
mirroring
that
data
to
multiple
sites.
B
Princeton
was
a
site,
and
there
was
there
was
a
few
other
sites.
Princeton
had
a
whole
a
whole
mirror
of
the
of
the
data
other
sites
just
had
small
pieces
and
small
chunks
of
that
data.
Well,
if
we
were
to
be
able
to
use
something
like
fuzzball
for
this,
we
can
say
that
each
one
of
these
sites
have
a
whole
set
of
different
persistent
data
data.
That's
already
associated
with
that
particular
site,
and
this
could
be
genomics.
B
It
could
be
whatever
fuzzball
will
now
make
sure
that
this
job
or
the
workflow
will
land
where
that
persistent
data
exists.
So
if
you
have
and
there's
one
big
assumption
that
we
make
and
not
to
get
too
far
into
persistent
volumes,
but
there's
one
big
assumption
we
make,
which
is
nomenclature
of
the
persistent
volume
is
very
important.
So
if
you
have
two
clusters
that
have
the
same
nomenclature,
we're
going
to
make
an
assumption
that
those
are
mirror
copies
of
each
other.
B
So
this
workflow
can
land
on
either
of
those,
and
then
we
can
make
decisions
based
on
other
attributes
of
the
workflow
of
where
that
should
land
but
persistent
volumes,
and
you
can,
by
the
way,
via
im
policies.
You
can
specify
what
users
are
allowed
to
to
mount.
This
up
with
read,
write
versus
read
only
access,
and
we
can
also
govern
that
all
the
way
down
to
the
posix
user
security
side
all
the
way
when
the
job
actually
runs,
here's
an
a
benchmark
that
we
did,
which
is
just
the
hplai
benchmark
from
nvidia.
B
We
did
this
on
some
dgxs
and
we
got
the
obviously
this.
I
think
this
should
go
without
saying,
but
just
to
state
it
state
the
obvious.
We
got
the
exact
same
performance.
Actually
we
didn't
get
exactly
the
same.
We
got
a
little
bit
better
performance,
but
I
think
that
was
just
because
when
we
ran
it
on
bare
metal,
we
didn't
peg
cpu
affinity
or
anything
fuzzball
automatically
does
that
so
for
all
equivalent
purposes.
It
ran
exactly
the
same
as
was
expected.
B
Let's
see
oops
there
we
go.
Gromacs
just
demonstrate
gpus
blast
this
one's
really
interesting
and
again,
I'm
going
to
send
this
out
to
you.
So
you
all
can
read
through
this,
but
this
one's
really
interesting,
because
we
actually
use
a
job
in
the
workflow
to
actually
do
pre-population
of
data
inside
of
the
volume
you'll
see
the
volumes
created,
but
there's
no
ingress
and
that's
because
the
ingress
is
done
here.
B
B
We
were
able
to
change
where
the
job
landed,
based
on
simply
changing
the
container
that
we're
leveraging
and
the
gpu
specified.
Specifically
the
cuda
here-
change
that
to
rockum
we're
able
to
put
a
different
container
here,
that's
built
for
rockum
and
without
changing
anything
else,
we
can
do
an
exact
back
to
back
comparison.
B
B
All
the
way
down
to
your
end
system,
very
much
like
an
ssh
tunnel,
so
it
makes
it
real
easy
now
for
users
to
implement
this,
and,
of
course
we
can
wrap
this,
and
this
is
all
api.
So
you
can
wrap
this.
However,
you
want,
you
can
even
make
a
web
interface
for
all
of
this
if,
if
so
desired
or
integrate
with
open
on
demand
and
yeah
we're
currently
in
early
access
and
pocs.