►
From YouTube: 2022-06-07 - L. Bryant - Building a Platform for Operating Multi Institutional Distributed Services
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title:
Building a Platform for Operating Multi-Institutional Distributed Services
Speaker:
Lincoln Bryant, Research Engineer at the University of Chicago's Enrico Fermi Institute
Abstract:
Much of science today is propelled by research collaborations that require highly interconnected instrumentation, computational, and storage resources that cross institutional boundaries. To provide a generalized service infrastructure for multi-institutional science, we propose a new abstraction and implementation of this model: Federated Operations (FedOps) and SLATE. We will show the general principles behind the FedOps trust model and how the SLATE platform implements FedOps for building a service fabric over independently operated Kubernetes clusters. Finally, we will show how SLATE is being used to manage data and software caching networks in production across computing sites in the US ATLAS computing facility in support the ATLAS experiment at the CERN Large Hadron Collider.
Bio:
Lincoln Bryant is a Research Engineer in the Enrico Fermi Institute at the University of Chicago. He has over a decade of experience building and supporting High-Throughput Computing (HTC), distributed storage, and containerization/virtualization systems for both the ATLAS experiment at the Large Hadron Collider and other collaborations as part of the Open Science Grid Consortium. Lincoln is one of the primary contributors to the Services Layer At The Edge (SLATE) project and has been an active Kubernetes user since 2017.
Host of Seminar:
Jonathan Skone, Advanced Technologies Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
Okay,
so
it's
my
pleasure
to
introduce
lincoln
bryant,
who
is
a
researcher
at
the
enrico
affirmative
institute
at
the
university
of
chicago
he's,
also
part
of
rob,
garner's
group
at
the
university
of
chicago
and
he's
one
of
the
maintainers
of
the
of
the
atlas
tier
2
facility
at
the
university
of
chicago.
So
they
have
something
like
several
tens
of
thousands
of
cores
that
are
maintained
for
the
high
energy
physics
atlas
project.
A
So
he's
been
part
of
the
slate
ci
project
since
its
inception
and
he's
going
to
give
us
a
background
on
that
project
and
how
it
relates
to
providing
distributed
services
at
multiple
sites
and
any
updates
that
the
folks
in
the
slate
ci
project
have
embarked
on
in
the
past
year
or
two
go
ahead.
Lincoln.
E
B
And
so
the
motivation
behind
this
is
that
when
you
have
these
research
collaborations
that
span
multiple
institutions-
typically,
they
they
may
have
something
like
you
know,
experiment
facilities
that
are
gathering
data.
You
know
they
need
to
work
with
other
facilities
that
are
doing.
You
know
simulation
and
analysis.
They
need
a
task
system.
Over
top
of
that,
that
can
coordinate
that
that
submission
to
different
sites,
and
then
they
need.
You
know
good
good
networks
and
and
tools
to
to
move
data
around
and
and
building
such
a
system
is
kind
of
daunting.
B
B
B
In
my
little
cartoon
here
I
have
you
know
four
different
sites
that
are
running
three
different
versions
of
some
data
lake
software,
and
so
any
clients
that
talk
to
those
data
lakes
would
then
have
to
be
compatible
with
all
the
different
versions
and-
and
ultimately
you
know,
monitoring
the
whole
thing-
responding
to
outages,
responding
to
vulnerabilities
things
like
that
can
be
really
difficult,
and
just
to
show
that
I'm
not
making
this
totally
up.
It's
not
all.
Just
in
my
head.
B
B
Centers
that
you
know
have
you
know,
tape,
copies
of
data
and
they're
doing
all
this
batch
processing.
Then
we
have
another
layer
of
this
onion
where
we
have
tier
two
centers
that
are
doing
simulation
and
analysis.
They
also
have
large
attached
data
stores.
You
know
the
order
of
10
petabytes,
and
then
you
have
sort
of
the
classic.
You
know
cluster
in
a
closet,
what
we
call
it
a
tier
three
institution,
you
know
which
are
running
these
small
clusters
that
are
in
support
of
their
local.
B
You
know
physics
group,
and
then
we,
of
course,
have
these
large
hpc
centers
like
nurse
which
can
provide
us
really
substantial
amounts
of
computing.
But
then
we
have
to
adapt
our
workloads
to
fit
on
their
super
computers,
and
so
somehow
we
have
to
be
using.
You
know
essentially
all
the
same
services.
We
have
to
run
approximately
the
same
software
and
we
have
to
do
it
all
within
a
single
global
namespace
for
data,
and
so
how
do
we
address
these
challenges?
B
A
B
B
And
so
what's
changed
to
make
this
possible?
You
know,
I
think,
historically,
it
would
have
been
really
challenging
and
and
in
many
cases
not
possible
to
give
users
the
ability
to
run
some
kind
of
application,
especially
one
that
needed
privileged
ports.
Things
like
that
to
to
run
those
applications
somewhere
in
your
infrastructure,
because
you'd
have
to
give
them
root
access
somewhere.
B
You
know
that
might
be
root,
access
on
a
piece
of
hardware
that
might
be
root
accessing
vm
or
whatever,
and
I
think
what's
changed
fundamentally
as
things
like
kubernetes
and
docker,
and
those
sorts
of
things
have
come
along
that
there
are
now
sort
of
these
a
la
carte,
containerization
and
capability
subsystems,
specifically
in
linux
and
and
using
those
you
can
sort
of
grant.
You
know
particular
pieces
of
minimal
privilege
to
your
users
to
let
them
stand
up
applications.
B
You
know
it's
now
kind
of
open
the
door
to
building
multi-institutional
federation
of
services,
and
so,
while
we,
you
know
realize
that
you
know
what
we
want
to
do
is
now
technically
possible
in
many
places.
There's
still
some
significant
policy
work
that
has
to
be
kind
of
you
know,
thought
through
and
standardized
in
particular.
B
What
we
found
with
working
with
organizations.
Is
that
the
you
know,
so
the
security
cyber
security
groups
within
these
organizations
really
voice
that
they
need
the
ability
to
scrutinize
and
kind
of
introspect
things
that
are
running
on
their
network,
and
so
that
means
that
that
you
know
any
kind
of
service
that
is
hosting
containers
or
whatever
needs
to
provide
some
specific
tools
and
policies
for
cyber
security
people
to
be
able
to
do
things
like
traceability
and
auditability
like
if
there's
incidents,
you
know
how
do
incidents
get
responded
to?
B
B
They
ultimately
want
to
be
able
to
control.
You
know,
what's
being
run
on
their
network,
who's
running
it
and
for
how
long,
and
so,
with
the
slate
team,
we
worked
with
trusted
ci
in
2018
and
2019,
to
establish
this
groundwork
for
a
trust
framework
and
set
policies
for
implementing
what
ultimately,
we
would
call
federative
operations
and
we.
B
D
B
B
And
you
know
caches
of
various
flavors
and
that
kind
of
thing,
and
what
we
would
try
to
do
then,
is
to
make
it
very
simple
to
deploy,
upgrade
and
maintain
those
applications
at
various
sites
that
are
part
of
our
fabric,
and
we
would
build
out
a
policy
process
to
sort
of
you
know
establish
what
it
means
to
be
a
trustworthy
application
and,
and
overall
we've
been
taking
a
declarative
approach
to
application
deployment.
We
really
want
users
to
not
think
about
things
in
terms
of
you
know,
kubernetes
objects
and
things
like
that.
B
B
B
Essentially,
when
you
set
up
a
slate
cluster,
you
install
a
component
called
the
federation
controller,
that's
a
little
operator
for
for
kubernetes.
That
gives
the
slate
api
the
ability
to
call
into
your
cluster
and
gives
it
the
minimum
privileges
needed
to
create
namespaces
on
behalf
of
user
groups
that
you
have
granted
to
use
your
cluster
and
so,
for
example,
if
you
you
know,
I'm
in
the
with
the
atlas
experiment,
and
so
I
want
to
run
this
application
called
xcache.
B
And
so
you
know
I
have
this
atlas
cache
group,
you
know
I
I
email
you
or
whatever.
As
a
as
a
cluster
administrator-
and
I
ask,
can
you
please
add
my
group
to
your
cluster,
so
I
can
deploy
this
application
and
we
give
enslave
a
very
fine-grained
ability
to
say:
okay,
yes,
you
can.
You
can
deploy
this
application
and
only
for
your
group
and
then
the
api
server
exposes
essentially
a
restful
api
for
clients
interact
with
it
and
the
api
server
links
together.
Users
who
come
in
through
our
portal.
B
They
come
in
and
sign
in
with
their
institutional
identity.
It
associates
those
users
to
groups
and
it
tracks
running
application
instances,
and
so
any
user
that
comes
into
slate
can
do
things
like
four
clusters
on
which
they
have
access.
They
can
launch,
update
and
delete
instances.
They
can
do
things
like
control
the
amount
of
replicas
for
a
particular
instance
that
they're
running
they
can
look
at
configurations.
B
B
So
the
api
server
is
granted
access
only
to
its
own
subset
of
namespaces,
so
slate
can't
really
reach
out
and
touch
other
name
spaces
and
importantly,
slate
isolates
users
from
each
other
just
by
using
the
sort
of
native
kubernetes
namespace
functionality,
and
that's
all
done
by
this
component
that
runs
on
the
cluster
that
we
call.
The
federation
controller
also
called
the
nrp
controller.
B
Essentially,
what
slate
is
trying
to
do
is
make
it
possible
for
resource
providers
to
share
access
to
their
clusters
with
well-known
groups,
and
essentially
all
resources
and
objects
in
slate
are
associated
with
the
group,
and
so,
as
a
cluster
owner,
you
can
go
and
register
a
cluster
and
and
slate
will
give
you
the
ability
to
launch
applications
on
your
own
cluster.
As
you
know
that
administering
group,
I
think
that
yeah
that's
over
here.
B
So
if
you're,
some
organization,
you
federate
a
cluster
with
the
slate
platform,
the
only
thing
that
say
allows
you
to
do
sort
of
right
off.
The
bat
is
to
allow
other
members
of
your
organization
to
deploy
applications
on
your
cluster,
so
it's
kind
of
an
opt-in
model
rather
than
an
opt-act,
and
so
then,
if
another
organization
comes
along
and
they
say
oh,
I
would
like
to
deploy
applications
on
your
cluster.
B
B
C
B
There's
not
really
any
active
directory
necessarily
involved
in
in
slate,
so
we're
just
pulling
the
the
relevant
identity
information
from
your
institution
by
way
of
ci,
logon
and
so
yeah.
We
I
mean
we
do.
We
do
keep
a
record
of
like
if
you,
if
you
log
into
the
slate
portal,
we
keep
things
like
you
know.
This
is
your
your
institutional
identity.
This
is
your
globus
identity.
We
keep
some
some
attributes
and
then
we
have
our
own
sort
of
internal
organization
of
groups
and
things
like
that.
B
C
So
an
another
just
I'm
just
a
quick
last
question,
so
let's
say
I
have
an
application
deployed
from
my
group
on
institution
level
and
then
I
want
to
share
it
to
another
group
at
another
institution
so
like
do
I
do.
I
have
to
expose
it
for
toward
from
the
slate
and
like
do.
I
also
need
to
do
some.
I
would
say
codes
some
work
to
to
share
that
on
the
slate
side.
C
B
B
B
So
essentially,
you
have
a
platform
administrator,
which
is
the
the
person
or
group
of
persons
that
run
in
our,
in
our
case,
the
the
slate
software,
the
api
server,
the
catalog
on
the
backing
store
for
all
of
that
stuff,
you
have
an
edge
administrator.
Who
is
somebody
that
is
running
a
cluster?
That's
participating
in
the
federation?
B
You
have
an
application
developer,
who
is
actually
putting
apps
into
the
catalog,
a
reviewer
that
just
checks
to
see
that
those
applications
coming
into
the
catalog
are
consistent,
see
consistent
with
the
policy
and
quality
standards
of
that
catalog.
They
may
choose,
for
example,
to
bounce
that
application
back
to
the
developer,
to
make
some
changes
that
sort
of
thing
and
then
ultimately
the
you
know
if
all
goes
well,
the
application
is
merged
into
the
catalog,
and
then
anybody
who
who
grants
access
to
using
an
application
can
use
it
on
clusters
in
the
federation.
B
B
You
know
they
they
want
to
do
something
to
run
a
global
store
or
a
person
or
test
point
or
whatever,
and
so
we've
also
tried
to
take
a
very
kind
of
security,
conscious
approach
to
packaging
applications
for
slate.
We
found
that
you
know
there's
a
number
of
different
deployments
for
setting
up
kubernetes
clusters.
B
You
kind
of
have
these
very
open
and
sort
of
liberal
tools
like
cube,
adm
right
where
not
a
lot
of
security
policies,
or
things
like
that
are
set
up
by
default.
It's
kind
of
you
know
you
build
a
cluster,
it
has
some,
you
know
reasonable
defaults
and
then,
if
you
want
to
add
in
security,
you
have
to
you
know
fold
all
that
in
on
the
other
side
of
the
spectrum,
you
have
things
like
okd,
which
are
very,
very
security.
Conscious
out
of
the
box.
You
can't
do
things
important
things.
B
B
So,
for
example,
one
thing
that
if
you,
if
you
just
go
out
there
on
into
the
into
the
world
and
look
for
a
helm
package
for
something
innocuous
like
postgres,
you
might
see
that
you
know
you
can
you
can
pull
down
the
postgres
configuration
and
it
gives
you
all
these
options
for
the
database
and
then
it
also
allows
you
to
do
things
like
change
the
container
image
in
the
slate
world.
You
know
things
like
that.
B
Can't
work
without
introducing
very
large
holes
into
the
model,
so,
for
example,
if
you
downloaded
postgres
and
you
you
changed
postgres's
container
image
to
some
bitcoin
miner
or
something
like
that,
you
could
imagine
folks
would
go
out
there
and
then
deploy
that
you
know
on
a
bunch
of
clusters
that
are
registered
in
slate
and
so
yeah.
That's
obviously
an
unstarter,
and
so
that's
part
of
the
reason
why
we
keep
this.
F
B
Okay.
So
what
does
it
actually
look
like
to
deploy
an
app
in
slate?
So
this?
This
is
an
example
for
globus,
and
so
here
on
the
on
the
left.
This
is
kind
of
what
a
user
sees
when
they're
trying
to
deploy
global
store
through
slate.
They
can.
They
have
a
few
configurables
here
that
they
can.
They
can
poke
around
with.
They
can
change
like
what
paths
can
be
read.
They
can
change
port
ranges
that
sort
of
thing
they
can
say
you
know.
B
So
you
know
if
you're
familiar
with
kubernetes,
this
shouldn't
look
too
terribly
different.
What
you
do
in
slate
is
you,
you
create
your
secrets
that
are
needed
to
instantiate
your
global
store.
For
example,
you
get
some
configuration
you
edit
it
appropriately
for
whatever
you
want
to
do,
and
then
you
can
run
the
slate
app
install
command.
B
You
give
it
your
configuration
file,
what
cluster
you
want
to
run
it
at
and
and
what
group
it's
going
to
run
under
and
then
in
five
to
ten
minutes
you
know
slate.
Will
you
know
essentially
have
your
global
store
running
it's
going
to
contact
the
kubernetes
cluster
that
you
want
to
deploy
on
it's
going
to
instantiate
the
objects
and
download
the
container.
All
this
happens
at
the
kubernetes
level,
and
then
the
pod
starts
up,
and
you
know
users
can
activate
the
global
store
vcl
if
they
ever
want
to.
B
You
know
update
upgrade
to
the
latest
and
greatest
version
of
that
application.
You
know
they
can
do
something
like
a
slight
instance
restart
and
get
the
latest
version,
and
so,
in
general,
we're
trying
to
build,
build
up
sort
of
trust
relationships
with
various
organizations
to
build
containerized
applications.
So
we
don't
have
to
do
all
of
the
building
of
the
containers
in-house,
and
so
you
know,
helm
charts
that
are
in
the
slate
catalog
can
also
refer
to
images
that
are
in
trusted
repositories.
B
You
know
such
as
those
operated
by
like
the
open
science
grid
or
cern,
or
whatever
one
of
the
things
that
we
found
while
using
kubernetes
for
a
while
is
that
using
some
kind
of
version
control,
specifically
using
some
kind
of
implementation
of
git
ops
is
actually
hugely
important.
We
think
it's
it's.
It's
super
necessary
to
be
able
to
track
like
when
changes
are
made.
You
know
to
have
people
be
able
to
make
like
pull
requests
against
the
configuration,
especially
if
you're
working
with
like
junior
members
of
a
team,
so
they
can.
B
You
know
have
that
reviewed
and
get
feedback,
so
you
can
do
things
like
roll
back,
bad
deployments
and
and
and
overall
so
you
can
have
kind
of
a
single
source
of
truth
for
your
cluster
state.
You
know
if
you
aren't
using
something
like
git
ops.
It
can
be
kind
of
hard
to.
You
know,
know
sort
of
the
provenance
of
where
things
are
coming
into
your
into
your
cluster,
and
you
know.
B
You
know
that
the
sources
of
their
containers
could
be
possibly
compromised
and
when
they're
running
services
or
running
applications,
you
know
they
could.
They
could
possibly
open
the
door
to
malicious
activity,
and
so
there's
a
number
of
things
that
were
we're.
Looking
at
here,
like
you
know,
container
images
are
kind
of
less
immutable
than
they
first
appear.
If
you're
running
you
know
some
application
tagged
with
version
one
out
of
docker
hub.
B
That
doesn't
necessarily
mean
it's
the
exact
same
container
image
tomorrow
or
likewise,
if
you
run,
you
know
an
application
with
the
latest
tag,
that's
constantly
mutating
and
becoming
something
different.
It's
also
doesn't
seem
to
be
all
that
common,
although
it's
becoming
more
common
that
the
you
know,
image
signing
things
like
that
are
are
not
super
popular
just
yet,
and
so
we're
building
a
service.
B
B
We
have
sort
of
harbor
running
at
the
university
of
wisconsin-madison
and
harbor
running
at
the
university
of
chicago
and
we're
you
know
hosting
this
service
for
folks
to
upload
containers
into,
and
it
can
serve
those
containers
as
a
container
registry,
and
it
will
also
do
some
container
analysis
via
these
sort
of
standard
tools
out
there,
like
trippy,
that
are
just
going
to
take.
Your
container
run
some
analysis
against
it
and
then
just
present
you
with
all
the
things
that
found
we're
not
going
to
necessarily
say
like
these
are
good
containers.
B
B
So
another
example
of
slate
in
the
wild
is
job
gateways,
so
in
the
open
science
grid.
There's
this
hosted
service
called
the
hosted
compute
entry
point
where
hosted
ce,
and
essentially
what
that
is,
is
a
microservice
that
lives
out
there
in
the
world
and
it
facilitates
batch
access
to
institutional
clusters
and
so
osg
is
using
slate
to
deploy
these
hosted
ces
to
join.
B
You
know
many
disparate
clusters
into
the
into
the
open
science
pool
and
also
to
do
this
for
various
collaborations,
and
so
I
believe,
there's
a
there's
a
hosted
ce,
pointing
to
nurse
that
I
think
that
the
cms
experiment
is
using.
We
did
the
you
know,
similar
things,
attack
lots
of
other
institutions
and.
B
So
what
does
that
look
like?
It
looks
really
similar
to
the
globus
example.
This
is
just
a
yaml
file
here
on
the
right.
This
is
just
the
the
home
template
you
just
plug
in
things
like.
Oh,
these
are
the
cache
directories
I
want.
I
want
to
put
the
metadata
on
this
ssd
over
here
high
in
the
watermarks
usual
caching
stuff
from
the
slate
side,
the
operator
uploads
and
encrypts
a
x
509
certificate
so
insulate
all
the
all.
The
secrets
are
encrypted
at
rest
and
they
deploy
an
instance
to
a
cluster.
B
So
xcache
is,
is
deployed
and
managed
via
gitops.
So
all
that
application
configuration
just
lives
in
this
in
this
github
repository,
that's
out
there
on
the
web,
and
so
whenever
somebody
makes
a
change,
or
rather
when
a
when
a
pull
request
comes
in
as
merged
or
somebody
makes
change
directly
to
the
repo
who's,
one
of
the
main
contributors
that
fires
up
the
github
action
and
then
slate
goes
and
and
updates
the
the
instance
that's
running
on
somebody's
kubernetes
cluster
somewhere,
and
it.
F
D
B
Changed
in
the
in
the
cluster
state
and
yeah,
and
so
when
site
operators
have
to
do
maintenance
or
whatever
they
can
and
do
make
pull
requests
against
this
repo.
When
site-specific
updates
are
needed,
there
was
a
pull
request.
I
was
just
looking
at
earlier
where
you
know
administrator
said:
oh,
this
disk
is
dead.
B
Computing
facility,
we've
equipped
all
of
our
tier
2
sites
with
kubernetes
a
small
kubernetes
cluster
and
we've
installed
slate
on
all
of
them
and
we're
doing
some
federated
service
management
on
those
clusters,
and
so
fundamentally,
each
of
the
sites
is
responsible
for
running
their
own
stuff
and
keeping
the
operating
system
and
kubernetes
up
to
date,
and
then
we
have
this
fedops
team
above
that
that
is
essentially
responsible
for
keeping
the
applications
up
to
date.
We're
running
two
applications.
B
One
is
squid
and
one
is
xcache
and
both
of
these
applications
are
configured
to
use
slate,
get
ops.
So
there's
a
single
github
repository
that
stores
all
the
configuration
that's
needed:
well,
a
single
github
repository
per
application,
that's
storing
all
the
config
needed
for
each
site
and
then,
ultimately,
the
slate
team
is
responsible
for
making
sure
helmed,
helm,
charts
and
docker
containers
and
all
that
stuff
are
reasonably
up-to-date.
D
B
That
they
don't
have
any
glaring
vulnerabilities,
and
so
ultimately,
you
get
the
same
management
interface
for
two
different
applications
operated
by
two
different
teams:
partitioned
in
name
spaces.
So
they
can't
interfere
with
each
other
on
these
kubernetes
nodes
and
we
shift
some
of
that
responsibility
up
to
of
a
higher
level.
B
And
so
our
traditional
tier
two
site,
you
know,
looks
like
something
on
the
left.
Where
you,
you
know,
you
have
a
bunch
of
hardware
and
install
the
os,
and
you
set
up
your
puppet
thing
and
then
you
have
to
figure
out
how
to
you
know,
run
all
these
applications
that
the
collaboration
has
asked
you
to
run,
and
so
under
this
new
model,
the
idea
is
that
the
site
admins
just
run
the
hardware.
B
B
And
then
you
have
things
like
slight
and
swateria
that
are
concerned
with
the
stuff
in
the
middle,
like
the
helm,
charts
and
the
containers
and
that
sort
of
thing
we
also
find
that
a
lot
of
applications
these
days
are
sort
of
going
cube
native
in
the
higher
energy
physics
community.
There's
been
a
lot
of
interest
in
building
what
people
are
calling
pythonic
analysis
platforms,
so
these
are
essentially
jupiter,
driven
with
really
easy
to
use.
Libraries
for
data
access
and
data
processing
and
these
platforms
have
kind
of
fundamentally
been
developed
with
kubernetes
in
mind.
B
It'd
be
really
actually
pretty
difficult
to
reproduce
them
outside
of
kubernetes,
because
they,
you
know,
have
a
number
of
things
going
on
like
they
have.
You
know
message:
queues,
object,
stores
and
databases
and
all
this
stuff
that
would
be.
You
know
a
pretty
significant
ask
if
you
were
to
go
to
a
site
and
ask
you
know
a
system
administrator
to
settle
the
stuff
up
for
you,
we're
supporting
two
of
those
today
not
directly
in
slate,
but
in
some
of
our
kubernetes
resources.
B
B
B
Our
focus
has
been
mostly
on
building
things
that
are
relatively
stateless,
so
we've
been
doing
a
lot
of
work
with
building
out
caching
applications
facilitating
file
transfer
through
things
like
globus,
we've
been
trying
to
integrate
things
like
personar,
plus
some
other
interesting
tools
like
open,
on-demand,
those
sort
of
things
and
as
a
person,
who's
working
with
the
atlas,
experiment
and
and
using
you
know,
nurse
machines.
As
an
end
user,
you
know
I
have
a
couple
applications
that
I
can
imagine
deploying
into
kubernetes
for
me,
of
course,
preferably
via
slate.
B
B
Things
like
this
xcash
software
that
I
mentioned,
where
we
want
to
just
kind
of
run,
a
bunch
of
jobs
and
then
have
the
cash
sort
of
pull
down
the
data
on
the
fly
and
and
cache
it
locally
to
improve
job
efficiency.
That
sort
of
thing
there's
a
number
of
ways
we
can
approach
this
one
way
is
that
we
could,
for
example,
had
slight
capability
to
kubernetes
cluster
at
nurse
we
could
also,
if
we
had
to
you,
know,
deploy
things
in
a
detached
manner.
B
As
I
mentioned,
you
know,
slate
applications
are
just
helm
applications
and
so
it's
possible
to
just
render
out
the
helm
templates
and
then
install
them
kind
of
in
the
usual
way
with
cube
ctl.
Then,
of
course,
from
a
user
perspective,
I
sort
of
lose
the
you
know
the
advantage
of
having
things
in
my
my
githubs
and
all
that
stuff,
but
it's
doable
for
data
transfer.
You
know
we're
mostly
sending
data
to
and
from
nurse
via
globus.
B
You
know
which
has
its
own.
You
know
things
that
we
have
to
do.
We
have
to
have
a
bunch
of
plugins
that
are
running
in
our
workflow
management
tool.
We
have
to
go
in
and
you
know
periodically
refresh
our
globus
credentials
and
all
that
stuff.
We
also
have
some
applications
that
we're
running
on
dtn's.
B
You
know
to
sort
of
insert
this
this
substrate
for
collaborations
to
be
able
to
build
their
applications,
and
we
think
that
you
know
if,
if
we
can
shift
some
of
the
application
operator
responsibility
to
developers
of
applications
rather
than
you
know,
putting
that
on
onto
the
sysadmin
at
every
site,
to
kind
of
learn
how
to
use
these
applications
and
join
them
up
and
maintain
them.
We
think
that
we
can
ultimately
lower
some
of
the
labor
costs
for
doing
these
things
and
yeah.
B
A
A
I
guess
in
the
meantime,
well
folks,
formulate
ideas
or
questions
I
I
would
wonder
if
you
could
elaborate
a
little
bit
more
about
the
capabilities
of
the
data
transfer
node
that
are
needed
for
the
science
gateways
that
you
see
that
are
not
available,
say
on,
say
something
like
spin
in
existence
today.
Is
it
just
simply
that
you
need
higher
bandwidth?
B
B
I
realize
that
there
are
some
challenges,
potentially
with
you
know,
doing
doing
user
accounts
and
things
like
that,
making
sure
that
the
the
accounts
running
in
a
container
are
going
to
match
up
with
the
accounts
and
groups
and
stuff
that
are
that
are
present
in
the
file
system.
So
there's
you
know
it's
not
without
challenges,
but
yeah
doing
that
sort
of
thing
would
be
would
be
really
useful.
Admittedly,
I
haven't
tried
to
do
any
kind
of
you
know
large
transfers
through
spin
things
like
that.
D
I
mean
I
think
it
could
be
useful
too.
I
mean
there's
probably
some
reasons
why
I
mean,
like
you
know,
there'll
be
a
different
security
model.
The
spin
is
currently
kind
of
protected
by
a
firewall
and
part
of
the
point
of
science.
Dmz
and
dtns
is
they're
a
little
more
open.
I
guess
for
bandwidth
and
performance
reasons,
I
would
just
say
on
your
on
the
workflow
that
you
described,
I
mean
if
you're
using
the
super
facility
api.
Then
you
can
initiate
via
that
api,
a
transfer
which
would
happen
with
globus
on
the
dtns.
D
F
D
B
Right
yeah,
I
mean
I,
I
would
be
very
interested.
I
I
don't
know
if
you
have
like
a
a
development
spin
in
a
production
spin,
something
like
that,
if
you
have
any
anything
like,
for
example,
I
know
that
it's
using
rancher,
we
haven't,
we
haven't
tried
things
with
rancher
in
quite
some
time,
so
there's
there's
there's
one
just
the
technical
thing
of
you
know
can:
can
we
install
the
slate
federation
controller
and
have
it
talk
to
your
cluster
and
then
there's
the
separate
kind
of
policy
thing
of
like?
B
F
Yeah
so
spin
uses
a
rancher
which
has
its
own
rbac,
so
you
know
that's
like
one
of
the
places
where
I'd
be
concerned,
I
mean
the
devils
are
in
the
details.
Here
I
mean
absolutely
definitely
there's
a
and
then
I
could
see
there's
some
other
other
pieces
that
look
like
they
would
be
probably
easier
to
integrate
more
than
on
the
helm
level
of
deploying
an
application,
but
again
we'd
want
to
look
at
it.
D
E
D
B
F
Yeah
we've
been
struggling
with
using
helm
just
for
mostly
for
because
we
would
have
to
basically
provide
a
catalog
of
like
better
vetted
applications
and
like
our
role
would
be
like
get
pull
and
then
like
change
a
couple
lines
and
then
get
pushed
to
our
own
repository
and
just
keep
that
in
sync
all
the
time,
and
that
was
not
something
we
could
handle
at
the
time.
It
is
something
we
are
discussing
a
lot
more
now,
but
I
think
having
another
catalog
where
people
already
did.
That
would
be
great.
F
It's
like
one
of
those
things
we
struggled
a
lot
with
with
helm,
is,
you
know,
helm,
just
kind
of
assumes
a
lot
most
of
the
applications
in
helm,
just
assume
that
you're
root
and
that's
okay
and
that's
not
something
we're
okay
with
so
we
had.
You
know
we
have
we
really
limit
capabilities
wherever
possible,
while
still
allowing
helm
and
then
just
lots
of
lots
of
applications,
just
don't
work
without
out
of
the
box.
So
we
have
to
like
do
some
deep
dives
and
to
fix
things
yeah.
That's.
B
We
haven't,
we
haven't
done
a
ton
of
work
here,
but
one
of
the
things
that
we've
thought
about
is
like.
We
should
build
our
helm
applications
assuming
the
most
restrictive
set
of
security
policies.
If
we
can
and
then
yeah.
If
it
works
in
that
restrictive
subset,
then
it
should
work
in
more
freewheeling
clusters.
F
B
F
Yeah,
I
think
it's
a
different
level
of
multi-tenancy.
I
mean
amazon
doesn't
really
care
too
much
if
your
application
gets
hacked
because
it's
contained
to
you
it's
contained
within
your
like
little
aws
realm,
and
so
it's
different
in
the
multi-tenant
environment
like
we
have,
and
amazon
and
google
aren't
really
tempted
to
to
solve
that
problem
because
they
make
more
money
if
they
sell.
F
F
E
B
D
B
Them
in
jbot-
and
you
know,
mount
them
to
some
well-known
location
and
then
communicate
that
to
the
the
team
running
the
xcash
application
of
like
all
these
things,
and
then
you
know
they
also
have
to
have
like
some
ssds
and
things
like
that.
So
I
mean
from
yeah.
So
from
a
resource
perspective,
you
know,
we've
asked
them
to
buy
a
reasonably
powerful.
You
know
dtn
sort
of
machine
with
some
built-in
disks,
and
things
like
that.
Does
that
answer
your
question.
B
Yeah
at
these
at
these
sites,
it's
kind
of
you
know:
we've
planted
the
seeds
everywhere
and
then
the
idea
is
that
now
we
have
the
the
substrate
out
there
and
then,
when
we
have
the
next
application
that
comes
along
right,
we'll
already
have
all
of
the
sort
of
initial
work
done
of
getting
these
clusters
up
and
running.
We
can
run
more
applications
there,
and
if
we
need
to
buy
more
hardware
we
can
we
can
grow
those
clusters.