►
From YouTube: Data Management Tools and Capabilities
Description
Presented by Lisa Gerhardt and Annette Grenier (NERSC)
May 13, 2020
The PI Dashboard is a web portal that will allow PIs to address many of the common permission issues that come up when dealing with shared files on the Community File System.
GHI is a new GPFS / HPSS interface that offers the benefits of a more familiar file system interface for HPSS. Often users want to store complex directory structures or large bundles in HPSS which can be difficult to do with the traditional HPSS access tool. GHI can be used to easily move data between HPSS and the GPFS file system with a few simple commands.
NERSC has written several command line data transfer scripts to users integrate data transfers into their workflows. We'll do a brief demo of these scripts.
A
All
right,
so
thanks
everyone
for
joining
us
today.
This
is
the
second
in
our
series
of
demonstrations
for
the
super
facility
project.
We
have
a
total
of
five
demonstrations
planned
this
month
on
topics
varying
from
data
movements
and
management
tools
like
we'll
be
talking
about
today.
Networking
tools,
as
we
talked
about
last
week
and
in
future
demonstrations
we'll
be
looking
at
edge
services
and
spin,
we'll
be
looking
at
jupiter,
jupiter,
notebooks
and
capabilities
at
nursk
and
we'll
also
be
looking
at
the
nurse
super
facility
api.
A
So
for
those
of
you
who
are
unfamiliar
with
the
super
facility
project,
this
is
a
project
that
was
started
a
little
over
a
year
ago
now
to
coordinate
some
of
the
work,
that's
being
done
in
the
computing
sciences
area
in
berkeley
lab,
including
work,
that's
being
done
at
esnet
at
nursk
and
in
the
computational
research
division.
This
is
all
work,
that's
being
designed
to
support
experimental
science,
experimental
science
teams
that
need
access
to
high
performance,
computing
and
networking.
A
So
these
demos
are
being
recorded
I'll
share
the
link
to
our
webpage
in
the
chat
window
shortly.
It's
also
in
the
calendar
invite
where
the
recordings
are
posted,
and
so
you
can
check
out
last
week's
demo,
and
you
can
also
see
today's
when,
if
you
want
to
recap
anything
so
I'll
now
hand
over
to
our
speakers
our
demonstrators
for
today,
and
we
have
lisa
gerhart,
annette
grenier
and
bjorn
enders
from
nurse
who
will
be
presenting
demonstrations
of
some
new
tools
that
will
make
it
easier
to
move
and
manage
data
at.
B
C
B
B
Okay,
so
the
main
point
of
the
of
interfacing
with
the
super
facility
project,
is
to
be
able
to
work
directly
with
science
groups
to
to
address
common
pain
points
related
to
data
management,
data
movement,
those
kinds
of
things-
and
the
idea
here
is
that
we
want
to
build
common
building
blocks,
that
our
our
science
partners
can
grab
onto
and
then
use
those
to
build
their
own
dedicated
portals.
So
we
don't
want
to
do
anything.
That's
that's
tailored
to
a
specific
group.
B
We
want
to
look
at
what
the
general
problems
are
that
are
experienced
by
all
by
most
science
groups
that
nurse
and
then
try
and
come
up
with
some
solutions
to
make
things
to
make
those
easier.
You
know,
and
some
common
issues
that
that
come
up
repeatedly
are
things
like
moving
data
between
the
layers
at
nurse.
You
know
we
have
three
layers,
maybe
four.
B
If
you
count
four,
if
you
count
the
burst
buffer,
that
you
need
to
move
your
data
between
and
curating
this
data
and
moving
it
between
the
layers
is
sometimes
often
a
full-time
job.
Particularly
difficult
is
getting
your
data
into
hpss
and
dealing
with
that
that
layer,
because
it's
a
it's
a
different
interface
and
it's
a
different
kind
of
it's
a
tape
archive,
so
it
takes
a
different
kind
of
thinking
to
deal
with
it,
and
so
that
that
requires
a
lot
of
time.
B
So
this
is
this
is
a
common
issue
across
the
groups
is
dealing
with
how
you're
moving
your
data
between
your
layers
and
getting
it
into
that
tape
archive
another
issue
that
comes
up
fairly
frequently
is
sharing
data,
and
this
is
both
internally
sharing
data.
Like
within
your
group.
You
have
someone,
you
have
a
bunch
of
data,
you
want
to
make
sure
that
it's
available
for
everyone
to
run
analysis
on
and
then
externally,
when
you
have
something
you
want
to
publish
or
you
have
a
set
of
refined
data
set.
B
And
one
thing
I
will
say
is
that
please
feel
free
to
to
ask
questions
during
the
demos
just
go
ahead
and
type
in
the
chat
or
unmute
and
ask
and
we'll
I'm
we'll
be
happy
to
walk
you
through
them
and
so
up.
First,
we
have
annette
she's
going
to
talk
about
the
pi
dashboard.
So
let
me
just
stop
sharing,
so
she
can
share.
D
Cool,
oh,
I
see
it's
just
not
doing
presentation
mode,
that's,
okay,
all
right!
So
the
pi
dashboard
is
an
important
piece
of
what
we're
putting
together
here.
It's
basically
meant
for
people
at
nurse
to
be
able
to
share
or
work
with
the
data
that
they
have
manage
the
data
for
their
group
and
be
able
to
share
it
amongst
each
other.
One
of
the
really
common
issues
that
we
find
happening
for
pis
in
particular,
is
and
actually
for
everybody
in
the
group.
D
D
D
F
D
All
right,
I'm
gonna,
be
looking
upward
for
a
bit
here,
but
that's
okay,
all
right!
So
mynersk
is
already
a
thing
that
we
have
here.
We
have
a
component
of
that
called
the
data
dashboard
that
we've
shown
to
people
before.
So
I'm
not
going
to
really
go
into
that,
but
this
is
kind
of
parallel
to
that
it
will
be
another
item
in
the
miners
menu
and
right
now
we're
calling
it
the
the
pi
dashboard
that
name
may
change,
but
we'll
see
so
log
in
to
my
nurse.
D
D
D
So
this
button
here
make
the
entire
project
directory
group
readable.
This
is
one
of
the
buttons
that's
you
know
set
up
to
do
a
really
common
task,
and
so
what
this
will
do.
If
I
click
on
it,
it's
going
to
change
the
group
of
all
the
files
and
directories
within
that
project
directory
recursively.
So
it's
going
to
get
everything
to
the
group
ownership
to
the
group.
D
So
I
can
click,
ok
and
the
request
is
pending
we're
saying
it
should
complete
in
the
next
few
hours.
Right
now
I
mean
these
things
are
going
really
fast
because
there's
a
little
script,
that's
running
every.
What
is
at
least
maybe
every
five
minutes
or
something
it's
it's
pretty
darn
quick
and
it
could
take
longer,
though,
if
somebody
else
has
started
a
process
that
happens
to
be
a
long
recursion.
That
will
take
a
long
time
so
just
in
case
we're
giving
people
more
time.
F
D
Of
course,
maybe
contradicting
each
other
they'll
be
done
in
the
order
in
which
they
came
in
okay
yeah,
so
you
can
see
this
actually
got
completed
already
and
there
there's
two
calls
here:
there's
one
to
the
the
changing
the
permissions
and
one
to
changing
the
group
readability
or
changing
changing
the
group
and
these
items
that
are
showing
up
in
the
recently
completed
stuff.
You
can
get
a
little
detail
about
it.
Usually
it's
just
the
request
completed.
D
D
B
C
D
Of
course,
we
don't
have
any
files
in
that
way,
but
this
path
updates
to
to
match
it
so
use
the
one
with
its
actual
files
in
it,
and
we
can
do
some
browsing
so
you
can
just
type
the
path
if
you
like,
typing
paths,
better,
go
through
it
that
way
or
you
can
click
around
go
up
and
down.
Just
like
a
typical,
you
know
graphical
file,
directory
traverser,
and
so
you
might
want
to
pick
some
files
to
operate
on.
D
You
might
want
to
select
a
lot
of
them,
although,
since
you
can
do
stuff
recursively,
you
shouldn't
really
mostly
need
to
select
a
lot,
but
I'm
going
to
just
upgrade
on
the
javascript
files
here
and
now
those
are
selected.
I
can
do
things
like
change.
The
group
of
the
selected
files,
so
this
will
figure
out
which
groups
I
have
permissions
on
and.
H
G
D
D
D
If
you
like
so
submit,
request,
should
look
familiar
and,
as
you
see
this
showed
up
as
pending.
Again
and
pretty
quickly,
these
will
start
to
show
up
completed
as
well.
So
that's
that's
basically
it
actually.
We
can
probably
start
to
see
some
of
the
changes
happening
as
well.
Oh,
I
went
to
das
anyway,
so
I'm
not
changing
anything,
but
so
that
that's
basically
what
this
interface
is
doing
right
now,.
F
D
A
D
D
G
B
C
A
All
right,
I
just
want
to
point
out
we're
aiming
to
launch
this
tool
in
the
beginning
of
june,
so
we
have
the
security
review
coming
up
soon
and
we're
planning
to
be
able
to
deploy
these
tools
for
users
within
a
couple
of
weeks.
So
we'll
be
updating.
We
can
update
you
all
about
that
in
the
nurse
weekly
uses
newsletter.
I
I
B
B
Okay
are:
were
there
any
other
questions
about
the
pi
dashboard
like
not
so,
but
feel
free
to
hit
us
up
in
the
chat
or
or
contact
us
directly?
So
we'll
move
on
to
our
next
capability-
and
this
is
this-
is
and
something
that
we've
just
recently
deployed
at
nurse
that
is
aimed
at
allowing
collaboration
accounts
to
use
globus
to
make
transfers.
B
So
just
in
case
you
don't
use
collaboration
accounts,
what
collaboration
collaboration
accounts
are
a
nurse
construct
and
what
they
are.
Is
there
a
mapping
of
many
users
in
one
group
to
one
user?
It's
frequently
used
for
managing
data
as
it
comes
in
installing
software
and
sometimes
running
jobs
as
a
group
and
what
it
does
is
it
lets.
You
share
the
burden
of
data
management,
software
installation
between
multiple
people
and
also
keeps
it
for
being.
B
But
up
until
this
point,
globus
transfers
into
nurse
with
collaboration
accounts
were
really
awkward,
because
it's
a
nurse
construct
that
you
couldn't
come
in
as
this
account,
so
you
would
transfer
it
as
yourself,
and
then
you
file
a
ticket
and
ask
to
have
all
the
data
that
you
just
moved
in
shown
to
the
collaboration
account.
You
have
to
wait.
B
So
it
creates
one
end
point
for
each
collaboration
account
that's
interested
in
this
and
then,
when
you
access
it
as
yourself,
you're
writing
to
the
nurse
system,
as
the
collaboration
account
and
the
the
access
to
this
endpoint
is
controlled,
such
that
only
nurse
users
who
have
a
globus
account
and
can
already
access
the
collaboration
account
can
see
this
endpoint
and
once
it's
once,
you
can
see
the
endpoint.
You
have
read,
write
access
to
all
the
files
and
directories
that
nurse
the
collaboration
account
had
access
has
access
to.
B
So
this
includes
globus
global
sharing
cfs.
This
is
not
yet
hooked
up
to
hpss.
Again,
that's
something
that
might
come
in
the
future,
but
not
right.
Now
we
don't
have
global
sharing
enabled
on
hpss,
but
this
would
be
at
least
scratch
and
cfs.
So
let
me
just
give
you
a
quick
demo
of
this.
So
hopefully
you
guys
are
seeing
the
global
screen
right
now,
and
this
is
fairly
familiar.
So
this
is
just
www.globus.org
already
logged
in,
so
you
guys
don't
need
to
enjoy
watching
me.
Do
my
password,
but
you
can
look.
B
This
is
I'm
logged
in
as
my
account.
So
if
we
go
over
here
we
go
to
the
nurse
dtn
and
then
we
can
go
to
this
test.
Endpoint
I've
created
here
this
is
data
dash.
There's
a
this
is
a
staff
collaboration
account
that
we
use
here
at
nursk.
This
is
a
specific.
This
is
a
shared
endpoint
that
lives
on
one
of
our
dtn
that
it's
a
shared
endpoint.
That's
served
out
of
one
of
our
dtm
nodes.
B
Here
I
am
looking
at
global
homes.
Data
attached
the
home
directory
of
data,
it's
pretty
much
empty,
so
I
can
come
back
to
my
home
directory.
I
can
just
transfer
one
of
these
text.
Files
over
should
go
fairly
quickly,
so
we
just
can
hit
refresh.
It
should
see
it's
already
there
and
then,
if
we
pop
over
to
the
shell
window
here
I
am
I'm
logged
in
as
data
dash,
and
you
can
see
there's
this
file
0
201.
That
has
been
written
to
the
account
as
the
data
dash
account.
B
F
F
B
So
you
wouldn't.
So
this
is
an
interface.
So
if
you
okay,
so
that
we
have
to
create
an
endpoint
specifically
to
do
this
so
right
now
you
can,
as
yourself
transfer
files
with
globus
into
the
into
this
project
directory,
and
it
would
have
right
as
you.
If
you
wanted
to
write
as
a
collaboration
account,
then
you
would
need
to
open
it.
J
J
B
Yeah,
so
I
think
that
you
could
do
that
now
with
with
an
spi
interface
as
long
as
that
user
is
you
know
that
data
is
already
in
a
group
that
you
are
a
pi
over?
You
should
be
able
to
tune
it
to
anyone.
That's
in
your!
When
we
have
shown
enabled
you
should
be
able
to
tune
it
to
anyone.
That's
in
your
project,
okay,.
B
B
Yes,
yeah
they're
not
going
to
cancel
each
other
out,
so
I
think
there's
still
going
to
be
needs.
You
know
the
the
globus
transfers
with
the
collaboration
accounts
were
a
lot.
There
were
a
lot
of
tones
from
there,
but
I
think
there's
still
going
to
be
a
lot
of
other
tones
that
need
to
happen
that
will
use
this
pi
dashboard
like
when
someone
leaves
there's
someone
trying
to
share
data
that
kind
of
stuff
and
that's
still
going
to
be
possible.
H
Hello,
I
just
had
this
is
stuart
campbell
from
the
brookhaven.
I
just
wanted
to
ask
as
well
for
is
there
any
plans
in
the
future
to
sort
of
connect
in
via
a
collaboration
account,
because,
of
course,
if
we're
a
facility,
that's
creating
data
that
we
want
to
send.
We
don't
want
to
run
systems
at
our
end
as
a
particular
person,
we
want
them
to
run.
As
you
know,
a
particular
light
source
account
or
a
service
account.
B
Yeah,
so
we
we
do
have
that
concept
of
people
needing
to
connect
in
and
be
be
collaboration
accounts
and
via
non-globus.
We
do
have
ways
to
support
that
right
now,
using
our
ssh
proxy
and
long
live
ssh
keys,
and
so,
if
I
think,
if
that's
something
that
you're
interested
in,
I
think
you
should
open
a
ticket
and
we'll
help
you
get
that
set
up.
H
B
B
B
So
what
these
are
is
it's
a
set
of
scripts
command
line
tools,
basically
that
are
based
on
the
globus
software
development
kit
that
you
can
use
to
manage
transfers
both
inside
and
outside
of
nurse
and
the
the
major
perk
of
this.
If
you
use
this
for
nurse
transfers,
is
it
understands
hpss
tape
ordering
so
right
now,
if
you
go
and
try
and
do
globus
transfers
out
of
our
hpss
endpoint,
maybe-
and
you
want
to
move
a
bunch
of
data
out
globus
will
not
do
this
in
an
ordered
manner
and
it'll
perturb.
B
The
system
you'll
probably
get
a
ticket
from
us
asking
us
asking
you
to
do
something
else,
but
if
you
use
these
command
line
tools,
it'll
actually
take
your
list
of
https
files,
you're
trying
to
read
and
feed
them
to
glow
this
in
sorted
order,
so
that
they'll
be
retrieved
in
the
correct
order
from
hpss
and
then
there's
also
some
scripts
that
you
can
use
to
check
on
the
transfer.
So
you
could
use
this
to
facilitate
data
data
staging
for
batch
jobs.
B
If
you,
if
you
were
interested
in
that
you
could,
you
know
fire
up
a
command
line
data
transfer,
it
gives
you
a
transfer
id
and
then
you
can
use
the
script
to
check
on
the
command
line
on
the
status
of
the
transfer.
So
you
know
when
it's
done,
you
can
submit
your
job,
and
these
are
these
tools
are
going
to
serve
as
the
building
block
for
incorporating
into
slurm
for
automatic
data
staging
probably
for
our
next
system
and
I'll
do
I'll.
B
Show
you
a
quick
demo,
but
these
are
also
outlined
on
our
website
on
the
globus
page.
There's
a
section
on
command
line
tools
that
you
can.
You
can
look
up
for
more
details
there
so
to
get
to
these.
So
here
this
is
just
me
on
corey,
you
do
module
load,
globus
tools
to
get
to
the
command
line
tools,
and
then
this
will
put
something
called
a
python
script
called
transfer
files
into
your
path.
B
You
can
say:
transfer
files
minus
h
to
find
out
how
to
use
it
and
basically,
what
you
do
is
you
tell
it
where
you're
coming
from
the
source,
the
target
where
you're
going
to
the
target
and
then
what
directory
you
want
these
files
to
land
in
at
the
target?
And
then
you
give
it
a
text
list
of
the
files
you
want
to
transfer
and
there's
some
optional
arguments.
B
If,
for
instance,
you
want
globus
to
preserve
timestamps
on
the
other
end,
you
can
add
this
flag
and
so
the
source,
the
source
and
the
target
are
source.
Endpoint
uuids,
which
are
you
can
get
from
you
can
get
from
globus.
If
you
just
go
to
this
endpoints
page
and
just
click
on
one
of
these
guys
and
then
down
here,
this
big
long
nasty
number
is
the
uuid,
but
if
you're
inside
of
nurse,
you
don't
need
to
know
that
it
understands
the
shortcuts
dtn
and
hpss
it'll.
B
It
knows
those
uuid,
so
you
don't
have
to
look
those
up.
So
I've
got
some
files
here
that
are
in
hpss,
so
I
want
to
get
them
out
and
put
them
in
my
scratch
directory.
So
I
would
just
say:
transfer
files
minus
s,
hpss
the
shortcut
for
that
and
then
the
target
is
going
to
be
the
dtn
node
dtn
endpoint,
because
that
has
query
scratch
on
it
and
then
my
scratch
directory
whoops.
B
B
B
B
B
And
then
si
is
transfer
list
and
it'll
go
and
submit
this
transfer
and
go
off
and
talk
to
the
globus
api
and
then
it'll
give
you
back
a
transfer
id
and
you
can
either
go
to
the
globus
page.
If
you're
more
familiar
with
that
or
you
can
use
the
script,
you
can
just
say:
what's
the
story
with
this
transfer,
you
can
just
give
it
your
transfer.
So
right
now
it's
thinking
about
it
and
it's
doing
this
transfer.
B
And
you
can
do
it
without
the
parsible
to
see
if
there's
some
kind
of
so
and
then
there's
a
minus
p
flag.
You
can
do
on
this
transfer.
So
if
you're
doing
this,
programmatically
you'll
get
some
nice
output
where
it's
just
it's
transfer
id
and
succeeded,
and
we
have.
I
have
this
this
little
script,
that
sort
of
shows
how
you
can
do
it.
It
takes
the
same
inputs,
it
submits
this
guy
and
then
you
can.
B
B
B
Now
it's
submitted
this
analysis
script
job,
so
you
can
see
it
in
the
queue
it's
running
right
now
doing
doing
the
analysis,
so
that's
a
way
that
you
can
incorporate
data
transfers
into
into
your
batch
scripts
programmatically
or
into
some
kind
of
calls.
You
can
use
these
helper
scripts
to
do
it
and
these
these
endpoints.
They
don't
have
to
be
near
scan
points.
They
could
be
any
globus
endpoint.
B
B
I
B
Okay,
so
the
first
time
that
you
run
this,
it's
going
to
look
so
globus.
Does
its
transfers
basically
gives
you
a
token
and
you
can
have
a
you-
can
have
a
long
live
token,
that
auto
activates
and
so
by
default.
Put
this
token
into
your
home
directory
when
you
run
it
under
a
special
directory
in
there,
the
first
time
that
you
run
it
it's
going
to
pop
up
a
web
page
that
will
take
you
that
you
have
to
paste
into
your
browser
that
will
take
you
to
globus.
B
Where
you
go
and
you
log
in
you
authenticate.
You
say
yes,
I
want
to
generate
a
token
to
do
this
and
then
it'll
give
you
this
big
long
string
and
you
paste
it
back
in
and
then
that'll
make
a
token
and
then
after
that,
it'll
just
check.
Do
you
have
a
token?
Does
it
work
and
if
it
does,
then
it'll
just
go
ahead
and
do
the
transfers?
If
not
then
it'll
make
you
activate
again
yeah.
B
B
Okay,
so
if
there's
no
other
questions,
then
we'll
go
on
to
the
the
final
thing
that
I
was
going
to
show
today-
and
this
is
whoops
excuse
me,
this
is
ghee
what
this
is.
A
new
prototype,
hpss
interface
ingee
stands
for
gpfs
hpss
interface
and
it's
a
way
to
use
the
hpss
tape
archive
using
a
familiar
file
system
interface.
So
what
happens?
Is
we
have
a
prototype
deployed
at
nursk?
B
So
what
you
do
is
you
interact
with
that
file
system
with
that
global
project
in
projectors,
directory
path
and
behind
the
scenes
is
tied
to
hpss,
and
this
lets
you
move
data
between
gpfs
and
hpss,
with
with
just
a
few
commands
and
behind
the
scenes,
gee
will
put
the
files
ultimately
optimally
into
hpss
on
your
behalf,
so
you
don't
need
to
worry
about.
You
know
each
turning
small
files
together
breaking
things
into
the
optimal
500
gigabyte,
size
chunks.
B
B
B
So
it's
it
really
helps
to
think
of
it
as
a
totally
separate
thing.
This
is
an
interface
for
the
tape
archive.
That's
that's
how
you
would
work
with
it.
It's
also
separate
from
the
way
that
you
talk
to
hpss
with
hsi
and
htar.
You
would
use
key
to
interface
interact
with
these.
You
would
not
use
hsi
and
htar
to
work
with
these
files
and
there's
some
documentation.
This
is
census
is
a
prototype
system.
B
You
know
we're
very
happy
to
have
you
kick
the
tires
on
it,
but
obviously
don't
put
any
unique
data
in
there
and
it's
in
our
dev
document
system
here
down
here,
to
show
you
how
to
use
it
and
some
use
cases
that
you
might
do
for
this.
Let's
say
you
have
thousands
of
files
ranging
in
size
from
a
few
bytes
to
hundreds
of
gigabytes.
B
You
know
in
aggregate
this
whole
directory
is
100
terabytes,
and
you
want
to
put
this
in
hpss
and
there
the
files
are
organized
in
a
complex
directory
structure
that
you
need
to
maintain
so
that
you'll
be
able
to
track
the
materials
in
the
files.
You
know
you
can't
archive
them
anywhere.
They
need
to
be
in
this
path.
So,
typically,
you
would
use.
B
Maybe
you'd
go
down
an
htar
each
one
of
these
subdirectories
or
break
them
up
into
pieces,
and
it
would
be
kind
of
a
little
bit
of
a
chore
to
get
them
into
hpss
with
key.
You
can
use
a
single
command
to
put
and
just
give
it
the
recursive
flag
and
it'll
put
the
entire
directory
into
hpss,
taking
care
of
the
bundling
and
whatnot
behind
the
scenes.
B
B
It's
just
got
a
bunch
of
you
know
kind
of
fake
files
in
here
you
know.
If
we
get
there's
a
subdirectory
2020.,
it's
got
subdirectory
of
more
files
in
there
and
there's
another
one.
There's
some
smaller
files
in
here
you
know.
So
it's
a
little
bit
of
a
like.
If
you
were
going
to
put
this
in
imagine,
there
were
many
more
files
and
they
were
much
bigger,
it'd,
be
kind
of
painful
to
put
this
into
hpss.
B
So
what
we
can
do
is
when
we're
on
this
special
system
on
the
special
file
system,
we
can
do
a
command
called
gls
and
it'll
tell
you
where
the
files
are
in
which
file
system
so
right
now
they
have
a
g
next
to
them,
which
means
that
they
are
only
on
the
gps
half
of
this
file
system
and
you
can
just
kind
of
noodle
around
and
look
at
these
other
ones.
You
know
and
they're
all
so
they're,
all
there
they're
all
just
sitting
on
the
gpfs
file
system.
B
So
let's
put
them
into
into
into
the
tape
archive.
So
right
now,
they're
just
in
gps,
I'm
going
to
say
gi
put,
so
that
means
put
it
into
hpss,
I'm
going
to
tell
it
minus
r,
which
means
do
it.
Recursively
and
dot
means
do
it
for
the
whole
directory,
and
so
it
takes
just
a
little
bit
of
time
to
run
this
because
it
has
to
go
down
and
check
that
each
file
is
on
the
right
file
system
and
that
it
has
the
right
permissions.
B
But
then,
when
this
is
done,
you'll
see
that
they're
all
going
to
be
moved
into
the
tape
system.
So
we
just
did
this,
so
we
can
say
gls
and
now
this
letter
has
changed
from
g
to
b
and
what
that
means
is
it's
no
longer
just
on
the
gpfs
system.
It's
now
on
both
gpfs
and
hpss,
and
so,
if
you
can
do
this
for
all
the
directories,
so
they're
all
in
hpss
now.
B
B
You
know
it's
already,
it's
already
may
so,
let's
get
rid
of
january,
so
we
can
say,
heat
punch
and
it'll
go
through
and
put
these
files
in
and
we
can
do
els,
and
now
you
see
that
all
these
files
have
an
h
next
to
them,
which
means
that
they're
only
in
the
tave
archive
system
they're
no
longer
on
the
gpfs
system,
except
for
a
tiny
stub.
That's
used
just
for
finding
the
file
in
hpss
and
conveying
the
metadata
about
the
file.
B
The
other
thing
you
can
do
with
key
is:
let's
say
you
have
a
really
important
file
that
lots
of
people
are
looking
at.
So
let's,
let's
do
els,
let's
say:
there's
something
really
exciting
in
data3
and
you
want
to
make
sure
that
that
doesn't
get
moved
off
off
of
the
spinning
disk,
because
it'll
immediately
get
pulled
back
by
whatever.
So
you
can
do
something
called
a
pin.
B
B
B
It
ends
up
writing
into
hpss
and
kind
of
this
really
yucky
looking
name
for
it,
and
so
you
know
you
it.
The
connection
to
this
file
is
is
contained
in
the
information
on
the
gps
file
system.
That
tells
it
where
to
look
for
it
in
hpss,
and
so
in
order
to
get
to
the
data.
That's
in
hpss
that's
linked
to
this
file.
You
need
to
go
through
the
gee
system.
You
can't
use
hsi
or
htar.
B
So,
let's
see
oh-
and
I
mentioned
earlier
before-
that
it'll
aggregate
files
for
you.
So
let's
see
so,
let's,
let's
look
at
these.
These
guys
were
small
files.
They
were
like
6k
and
if
you
look
at
this
file
name,
it's
got
this
ag
in
front
of
it
and
if
you
have
really
sharp
eyes,
I
don't
know
how
well.
This
shows
up.
You'll
see
this
actually
the
same
name
for
all
of
these,
where
this
is
in
in
hpss.
B
B
B
It's
a
little
bit
nicer.
Looking
so
it's
you
know
it's
about
12
terabytes!
So
if
you
go
back
here
so
we
have
some
big
files
here
and
if
I
d
punch
really
big
file,
that's
I'm
not
going
to
punch
the
three
terabrite
one,
because
that
seems
excessive,
but
I'll
punch,
the
300
gigabyte
one.
So
if
we
punch
this
and
it
thinks
about
it
for
a
minute
and
then
we
go
back
and
we
run
our
quota
check-
oh
it
was
already
punched.
B
B
C
B
B
Our
threshold
would
be
below
68
gigabytes,
so
you
wouldn't
feel
those
effects.
So
we
would
set
up
the
smaller
files
to
bundle
and
it
would
automatically
aggregate
those
things
for
you
and
you
wouldn't
run
into
that
issue.
Nor
did
you
run
into
the
issue
of
a
really
long
path,
name
in
htar,
because
again
it
does
the
bundling
for
you
underneath
the
covers
it
actually
does
ishtar,
which
is
like
related
to
htar.
B
A
B
Yeah:
apologies
to
bjorn,
sorry,
but
anyway,
if
anyone's
interested
in
trying
out
these
services,
just
please
feel
free
to
reach
out
to
us
either
directly
or
open
a
ticket.
Especially,
you
gave
us
some
really
good
feedback
already,
but
if
there's
anything
else
missing
that
you'd
really
like
to
see.
Definitely
let
us
know.
B
A
C
E
Had
a
a
comment
sort
of
back
on
the
the
pi
dashboard
kind
of
pieces,
was
there
any
thought
about,
like
within
a
file
system
area
like
a
community
directory
to
have
like
a
top
level
directive
that
you
could
just
someone
the
pi
could
say
like?
This
is
how
I
want
the
permissions
to
be
like
make
sure
all
these
directories
are
owned
by
this
account
or
something
like
that.
D
D
E
D
B
Yeah,
so
that
that
runs
now
for
four
or
five
groups
that
has
sort
of
had
had
this
come
up
over
and
over
and
over
again
so
yeah.
That
is
something
that
that
definitely
we
want
to
think
about,
for
the
future
is
some
way
for
them
to.
Let
us
know
that,
like
I
want
this
to
be
happening
I'll,
you
know
to
do
this
and
keep
it
in
place.
You
know,
I
think
that's
yeah
just
have
to
think
about
how
we
would
do
it
in
a
in
a
easy,
easy-to-use
way.
J
B
J
Well,
I
was
thinking
since
there's
a
quota
in
place.
You
might
want
a
management
system
that
decides
when
you
had
to
decide
to
keep
quota
and
you
had
to
pull
a
really
big
file.
You
have
to
push
something
back.
You
may
want
to
pick
the
oldest
thing
and
then
push
that
back,
and
so
that
way
you
have
the
only
the
freshest
last
sets
of
files
available
on
disk
at
any
given
time.
C
B
B
B
Yeah,
so
I
think
that
if
we
were
to
do
the
globus
interface
with
hpss,
this
seems
like
the
the
way
that
the
key
interface
is
the
way
that
globus
is
most
used
to
dealing
with
with
hpss.
And
so
I
think
that
with
some
modifications,
for
instance,
if
you
were
doing
a
read
via
globus
to
a
ge
thing,
you'd
want
to
do
a
stage
first
to
bring
the
files
up.
You'd
want
to
incorporate
that
into
the
globus
itself,
but
it's
it's
the
most.
That's
the
way
that
globus
is
built.