►
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Speaker:
Kjiersten Fagnan
Joint Genome Institute (JGI)
Lawrence Berkeley National Laboratory
Title:
JGI Computing - the future is looking cloudy
Abstract:
The DOE Joint Genome Institute produces high-quality omics data from fungi, plants, microbes, and metagenomes. The computational infrastructure needed to support processing and analysis spans laptops to exascale. JGI has adapted to these needs through moving to a distributed network of computing and storage resources. In this talk I'll describe those resources, what's run where, and the software infrastructure we're building to maintain a high level of usability for JGI's staff and User community.
Host of Seminar:
Nicholas Wright
Group Leader, Advanced Technologies Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
A
A
Amr
and
related
problems,
which
was
related
to
her
phd
work,
which
was
in
a
similar
area.
Then
in
2016.
She
moved
over
to
support
not
earlier
than
that
she
moved
to
support
the
gga
user
base
and
then
in
2016
she
moved
over
full
time
to
work
at
the
jgi
as
the
chief
informatics
officer
and
so
she's
responsible
for
deploying
and
executing
a
strategy
for
computing
and
their
analysis
of
the
ggi.
B
B
B
We've
actually
spent
less
time
in
that
building
than
we
have
in
shelter
in
place,
which
I
guess
gets
less
interesting.
The
longer
we
kind
of
do
this
working
remotely
thing.
It
also
houses
the
national
microbiome
data
collaborative
and
k
base.
So
jgi
is
a
user
facility
just
like
nurse
skinny,
snet
and
als.
B
We
provide
sequencing
capabilities
to
those
who
are
trying
to
do
weird
things
like
sequence
and
understand,
plants
and
fungi
and
metagenomes
yeah.
We
get
about
80
million
dollars
a
year
and
there
are
about
300
staff
that
work
at
jgi,
most
of
whom
are
remote
right
now.
This
is
the
history
of
the
jgi.
I
think
it's
pretty
cool
that
it
started
out
of
the
human
genome
project.
If
you
listen
to
people
now,
they're
like
it
shouldn't
have
been
called
the
human
genome,
it
was
just
a
human
genome.
B
B
This
is
kind
of
related
to
that,
and
also
looking
at
metabolomics
analysis.
So
if
you
take
a
sample
from
the
soil,
you
might
want
to
look
at
the
genome
the
blueprint
of
what's
there,
the
transcriptome,
what
genes
were
turned
on
and
then
also
the
metabolome
of
what
was
being
produced
in
the
soil
when
you
took
the
sample
yeah.
B
So
I
think
environmental
genomics
is
really
cool
and
if
you've
heard
a
little
bit
about
this
concept
of
the
bioeconomy
and
if
you're
thinking
a
little
bit
about,
I
don't
know
things
like
climate
change,
which
I
feel
like
should
be
on
everyone's
mind.
I
feel
like
this
is
a
safe
audience
for
this,
but
the
world's
getting
hotter
things
are
on
fire.
B
We
have
no
water
in
california,
I'm
not
sure
what
we're
all
still
doing
here,
except
that
it's
an
awesome
place
to
live,
but
a
lot
of
the
work
that's
going
on
in
synthesis
and
the
creation
of
biofuels
is
really
to
enable
replacement
products
for
things
that
typically
have
come
from
fossil
fuels,
and
so
I
think
that's
pretty
cool
and
you
can
check
out
this
slide
later.
If
you
want
to
jji,
has
lots
of
primary
users.
B
So
the
reason
I'm
talking
about
computing
and
informatics
in
this
talk
is
because
these
are
foundational
activities
across
jgi.
So
we
have
infrastructure
and
systems
that
support
everything
from
how
we
engage
with
our
users,
which
is
kind
of
this
proposal,
management
and
project
specification
to
bringing
in
their
samples
and
doing
the
sequencing
and
analysis
to
then
all
of
the
sequence.
Data
publication
that
comes
out
through
those
science
gateways
and
so
underlying
all
of
this
is
really
computing
and
informatics
infrastructure.
B
B
For
those
who
don't
know
this
is
a
cabinet
of
quarry,
that's
dedicated
to
jgi's
use.
Why
would
we
have
a
cabinet
dedicated
to
jgi
because
it's
a
user
facility
and
we
do
a
spectrum
of
work
that
ranges
from
being
able
to
qc
and
process
data,
all
the
way
up
to
data
analysis,
that
requires
super
computers
and
core
gene
pool
will
retire
in
2022.
So
one
of
the
things
that
we've
been
thinking
about
recently
is
how
do
we
replace
this
capacity,
we're
going
to
have
an
allocation
from
ber
on
nurse
perlmutter
system?
B
B
We
also
have
a
lot
of
services
that
enable
the
interaction
with
our
users,
like
the
science
gateways,
but
we've
got
a
bunch
of
small
services
that
also
handle
automation
and
workflow
execution
and
data
management
and
our
laboratory
information
management
system,
and
so
all
of
these
things
kind
of
combined
really
support
the
ability
of
jgi
to
function
and
operate,
and
over
the
past
couple
of
years
our
systems,
infrastructure
team
has
expanded
our
web
services
infrastructure
footprints,
we've
got
svms
and
lab,
and
we've
got
virtual
machine
infrastructure.
That's
going
into
igb
as
well.
B
They
like
some
ease
to
be
able
to
deploy
applications
and
services,
and
then
there
was
a.
There
were
many
many
many
comments
around
stability,
and
so
the
reason
I
included
this
slide
is
because
we've
got
kind
of
attention
within
jgi,
where
we
do
have
a
subset
of
staff
that
are
doing
research
and
development,
oriented
work,
they're
running
large
jobs
or
running
things
at
scale,
but
we're
also
a
user
facility
that
has
to
produce
products
for
our
users,
and
we
have
metrics
and
things
that
der
doe,
sorry,
the
office
of
biological
and
environmental
research
tracks.
B
So
that's
you
know
the
equivalent
of
oscar
that
funds
nurse
and
yes,
and
so
this.
This
stability
issue
comes
up
time
and
time
again,
because
we
have
data
that's
being
produced
consistently
every
day,
as
people
do.
Experiments
and
people
need
a
reliable
resource
for
being
able
to
process
that
data
and
the
other
thing
that
is
kind
of
an
interesting
quirk
when
it
comes
to
our
workflows
and
pipelines.
Is
that
they're
incredibly
dynamic?
B
So
we
also
need
an
environment
where
people
can
be
developing
and
incorporating
the
latest
tools
for
the
analysis
of
these
data
and
that
that
again
kind
of
drives
the
need
for
infrastructure.
That
looks
a
little
bit
different
than
what
you
would
find
on
cory.
So
this
is
just
some
fun
feedback.
We
get
from
vgsn
yeah.
B
I
would
I
don't
know
some.
Some
of
those
folks
are
probably
on
this
call,
and
I
probably
shouldn't
put
words
in
their
mouth,
but
I
think
that
what
I
would
what
I
would
get
is
just
yes,
we
would
like
all
the
things
and
mendel
was
really
stable,
so
why
can't
we
have
that
again,
and
so
I
think,
what's
needed
for
to
support
jgi
and
I'll.
B
The
other
thing
that
I
like
to
to
call
out
is
that
berkeley
labs
on
a
major
fault
line.
This
is
where
we
would
all
be
if
we
were
in
building
59
right
now.
I
don't
really
need
to
tell
you
guys
this,
but
most
samples
generated
used
to
generate
data
at
jgr.
You
need
to
be
replaceable,
so
we've
gotten
some
pushback
about
not
just
having
all
of
our
data
sit
at
berkeley
lab
on
a
fault
line.
A
lot
of
it
is
in
hpss.
We
have
about
14
petabytes.
There.
B
Our
current
infrastructure
looks
something
like
this.
Over
the
past
couple
of
years,
we've
moved
some
hardware
to
lab
iat
from
nurse
we've
moved
data,
the
irreplaceable
raw
sequence
data
to
oak
ridge
into
their
tape
archive.
We
also
have
storage
through
lawrencium
scratch.
We
have
a
bunch
of
storage
that
sits
at
nurse.
B
So
this
is.
This
is
a
slide
that
actually
was
somewhat
inspired
by
conversations
with
nick,
if
you
think
about
computing
along
one
axis
and
the
scale
of
data
on
another.
When
you
have
large
amounts
of
data,
you
might
still
have
small
amounts
of
computing
that
you
need
to
do.
We
have
you
know
some
pipelines
where
we're
going
to
be
processing,
terabytes
and
terabytes
of
data,
but
the
algorithms
themselves
that
people
are
comfortable
using
only
use
a
handful,
of
course,
and
then
we
have
some
algorithms
that
you
know
I'll
mention
later
like
metahitmer.
B
That
can
use
a
large
number
of
notes
and
can
make
good
use
of
a
supercomputer
and
are
also
ingesting
a
lot
of
data,
so
that
kind
of
sits
up
here
in
this
like
ideal
area
for
running
things
on
in
a
super
computing
environment,
and
then
we
have
this
more.
This
need
for
an
interactive
resource
where
people
are
doing
prototyping,
exploratory
analysis,
small
production
work
and
that's
really
kind
of
the
space
where
we
need
to
problem
solve
right
now
and
right
now
we
have
a
procurement
for
jgi.
B
B
When
we
were
looking
for
a
place
to
put
some
of
this
mid-range
computing,
you
know
we
were
looking
everywhere.
It
turns
out.
We
actually
don't
have
a
huge
amount
of
space
on
the
hill
or
a
huge
amount
of
infrastructure
where
we
can
actually
put
computing
and
data
storage,
and
we
were
looking
at
building
out
igb
and
saying:
hey.
B
You
know
the
lab
and
emsl
and
the
cloud.
So
the
cloud
computing
option
is
really
it's
kind
of
an
interesting
one,
because
if
we
were
able
to
have
persistent
storage
of
the
data-
and
we
could
probably
run
a
lot
of
jgi
analyses
there.
But
we
have
this
huge
archival
data
footprint
that
we
need
to
negotiate
and
then
one
of
the
deal
breakers.
There
is
actually
our
portal
infrastructure,
as
well
as
the
cost
and
the
time
of
getting
the
snap
ready
to
move
all
of
their
workflows
into
the
cloud.
B
B
But
until
we
figure
out
and
problem
solve
the
infrastructure
for
our
portals,
we're
a
little
cost
constraints
like
we
need
to
make
strategic
investments
to
make
sure
our
web
services
and
science
gateways
keep
running
and
in
parallel
with
that
figure
out
how
to
keep
our
computing
going.
And
so
we
just
can't
do
everything
at
the
same
time,
but
maybe
in
subsequent
years
our
next
investment
won't
be
for
mid-range
computing
that
sits
within
berkeley
lab,
but
it
could
be
moving
everything
to
the
cloud.
B
I'm
not
100
sure
and
I'd
be
happy
to
follow
up
on
that
later
and
so
we've
allocated
or
spent
about
150
000
on
aws
in
the
past
year
and
that's
primarily
been
for
running
the
metagenome
analysis.
Well,
assembly
pipelines
so
distributed
computing
is
hard.
This
is
one
of
the
things
that
we've
been
talking
about
a
lot
as
an
organization
is
how
ready
are
we
to
actually
move
jji's
workflow
between
different
resources,
and
one
slide
that
I
meant
to
include
that
I
didn't
when
we
were
talking
to
the
staff
about
requirements.
B
Is
I've
asked
and
I've
surveyed
jgi
staff
every
year
about
where
they'd
like
to
run
their
computing
and
I've,
given
them
the
whole
spectrum
of
options
from
nurse
to
lab
I.t
to
the
cloud
and
year
over
year?
People
are
selecting
nurse
because
it's
what
they're
comfortable
with
and
even
though
it
doesn't
provide
that
stability
that
they're
looking
for
to
nick's
earlier
point.
They
know
how
to
run
there.
They
know
what
it
looks
like
to
run
there.
B
Moving
to
another
location
requires
learning
some
new
things,
getting
new
passwords
and
accounts
and
figuring
out
how
to
move
data.
So
there's
a
huge
cost
to
switching
between
different
platforms
and
different
resources,
and
that
that's
you
know,
people
generally
don't
want
to
think
about
that
at
the
jgi.
B
So
jji's
approach
for
this
has
been
to
look
at
software
infrastructure
to
aid
and
the
execution
of
distributed
jobs.
So
we
have
the
jaws
system,
which
is
a
centralized
workflow
system.
That's
built
on
top
of
the
cromwell
workflow
manager,
which
is
a
workflow
manager.
Out
of
the
broad
institute
works
well
with
bioinformatics
pipelines,
there's
something
called
whittles.
The
workflow
description,
language,
which
is
really
similar
to
the
common
workflow
language
or
cwls.
B
B
The
community
is
producing
more,
there's,
not
really
been
a
slow
down
say
in
the
production
of
metagenomics
data,
and
what
people
are
also
interested
in
are
the
meta
transcriptomes,
metabolomes,
metaproteomes
and
all
of
the
ohms
from
a
different
environment
for
understanding
what's
happening
in
these
microbial
communities,
which
is
something
we
probably
all
want
right.
Everyone
here
has
probably
heard
about
microbiomes,
because
you
have
one
in
your
gut
you.
B
You
know
this
is
more
fun
when
we're
in
person,
but
I'll
just
say
that,
like,
if
you
think
about
it,
how
much
bacteria
do
you
have
in
your
body
about
five
to
seven
pounds
of
bacteria
in
the
average
person?
That's
a
lot
of
little
creatures
running
around
on
your
insides
like
helping
to
keep
you
healthy
and
safe,
and
mostly
I'm
just
saying
that
to
see
if
you
all
are
still
paying
attention,
because
I
know
zoom
talks
are
hard
and
email
and
everything
is
right
there
and
very
exciting,
but
that's
kind
of
crazy
to
think
about.
B
And
if
you
really
want
to
aggregate
all
of
that
information
about
that
five
to
seven
pounds
of
bacteria,
just
in
your
own
little
microbiome.
That's
a
lot
of
data
to
try
and
analyze.
It's
a
lot
of
data
to
try
and
reason
about,
and
what
we're
able
to
do
right
now
is
just
a
fraction
of
what
we'd
like
to
be
able
to
do
when
you
want
to
bring
all
of
those
different
types
of
measurements
together,
and
so
how?
How
can
we
actually
remove
these
barriers
to
data
access
and
analysis
at
larger
scales?
B
And
that
seems
like
something
where
we
should
be
leveraging
our
large-scale
computing
facilities?
And
so
what
I
think
is
really
cool
is
the
fact
that
meta,
hitmer
and
rob
is
on
the
call.
I'm
not
sure
anyone
else
from
the
x-biome
project
is
on
the
call,
but
this
is
really
awesome
and
really
exciting.
So
I
don't
know
the
meta
hitmore
project
started.
I
think
back
in
2014.,
so
think
about
that.
B
It's
like
been
like
seven
years
that
this
project
has
been
underway
and
now
it's
being
used
for
scientific
results
that
couldn't
have
been
done
any
other
way.
So
you
have
these
huge
metagenomic
data
sets
that
are
coming
into
jgi
and
typically
what
people
would
do
is
they
try
to
break
those
apart
into
subsamples
and
then
you
know
assemble
each
of
those
subsamples
separately
and
what
happens
when
you
do
that?
Is
you
actually
lose
out
on
some
of
the
like
lower
signal
in
that
sample?
B
And
you
don't
pick
it
up,
but
when
you
combine
all
of
the
samples
together,
you're
able
to
pick
up
the
signal
of
sort
of
lower
abundant
organisms
that
might
be
really
important
in
understanding
particular
dynamics.
You
know
in
a
subtropical
soil
or
after
there's
been
a
burn,
and
so
it's
pretty
exciting
that
now,
jgi
and
crd
have
put
together
a
tool
that
can
be
used
to
actually
help.
All
of
these
users-
I
guess
this
guy
is
at
berkeley
lab,
but
all
of
these
users
in
the
community
assemble
data
sets
and
analyze
data
and
interrogate
them.
B
Then
they
couldn't
have
been
able
to
do
this.
Otherwise,
but
I'll
tell
you
back.
In
2014,
people
were
real
skeptical
about
the
idea
of
using
a
super
computer
based
method
for
doing
metagenome
assembly,
because
there's
a
huge
focus
in
this
community
on
having
tools
that
people
can
run
on
their
laptops,
but
the
size
and
scale
of
the
data,
like
I
showed
in
that
last
slide,
really
demand
these
kinds
of
tools
and
really
demand
this
kind
of
development.
B
And
there
are
other
things
that
the
exobiome
project
is
working
on,
like
hip
mcl
for
looking
at
clustering,
some
different
types
of
deep
learning
methods
and
it's
all
really
going
to
drive
kind
of
a
discovery
in
a
different
way,
because
you
can't
escape
the
fact
that
these
samples
are
huge.
This
is
so
much
bigger
than
your
genome
or
my
genome.
You
know,
I
guess
it
might
be
comparable
if
we
took
our
gut
microbiomes
and
started
looking
at
them,
but
it's
kind
of
crazy.
B
And
then
there
was
this,
the
largest
ever
meta
genome
assembly
that
words
like
that.
Sound
really
cool,
but
for
the
same
reasons
like
the
science
coming
out
of
this
is
really
exciting,
and
so
they
actually
were
able
to
take
this
data
set
and
assemble
it
on
summit
and
it
had
to
be
summoned.
It
couldn't
be
korg
because
they
needed
that
much
memory
per
node
to
be
able
to
do
the
assembly.
B
If
you
have
questions
about
that
ping
rob
and
then
you
know
they
take
the
output
from
the
combined
assemblies
and
they're
able
to
do
bins
to
really
understand
what
organisms
are
present
in
that
sample,
and
so
that
was
an
eight
terabyte
run
or
tera
based
samples.
Sorry,
I
still
confuse
that.
I
still
say
bite.
B
Instead
of
base
that's
fine
anyway,
and
then
there's
this
25
terabase
sample,
which
would
be
really
awesome
to
be
able
to
assemble,
because
it's
a
huge
time
series
data
set
looking
at
what
microbial
communities
have
looked
like
in
this
particular
environment
over
20
years
and
so
that'll
be
really
exciting.
To
be
able
to
see
like
how
you
know,
those
communities
have
evolved
in
that
environment
and
it's
not
possible
if
we
don't
use
super
computers,
and
so
I
guess
this
is
my
way
of
saying
like.
Yes,
we
do
a
lot
of
processing.
B
Just
want
to
comment
a
little
bit
on
this
whole
fair
data,
sc
pure
data
resource
thing.
These
are
just
a
snapshot
of
statistics
and
a
summary
slide.
You
know
that
of
jgi
as
one
of
these
sc
pure
data
resources,
we
have
a
large
repository.
We've
supported
a
lot
of
publications
and
data
users
and
primary
users.
We
support
a
huge
number
of
downloads
and
a
lot
of
analysis
and
access
through
our
science
gateways,
and
you
know
what
is
jgi
doing
to
make
our
data
fairer.
B
I
don't
know
fair,
is
kind
of
an
interesting
topic.
It
sounds
really
nice
right.
We
want
everything
to
be
fair.
I
think
that
really
resonates
with
humans
is
think
when
things
are
fair,
so
that
stands
for
findable,
accessible,
interoperable
and
reusable,
and
this
I
guess
definition
of
fair
data
has
been
around
for
a
few
years.
It
came
out
of
europe
and
it's
really
gaining
momentum
and
the
idea
behind
it
in
case
you
haven't
heard
this
yet.
Is
that
we're
trying
to
make
data
accessible
to
machines,
not
just
humans
but
to
machines?
B
B
You
can't
like
assume
that
a
human
is
going
to
be
looking
at
it
and
understand
the
structure.
You
have
to
have
a
well-defined
structure,
well-defined
set
of
identifiers
such
that
you
could
send
in
a
bot
and
they
could
find
all
of
the
data
in
your
repository,
that's
kind
of
how
I
think
about
it
and
that
they
could
do
that
repeatedly
like
the
system's
not
going
to
change
over
time.
Unless
you
publish
and
let
the
world
know
what
those
changes
are.
B
So
it's
a
little
bit
weird,
but
it's
something
we
have
to
think
about,
because
we
are
producing
a
lot
of
data
at
jgi.
Like
I
said
already,
and
these
data
are
in
demand,
people
want
to
download
them,
they
want
to
reuse
them,
and
we've
got
this
large
data
repository
that
lives
at
nurse,
and
we
have
to
manage
that.
We
have
to
make
it
available,
and
so
in
the
past
year.
I'm
talking
about
this
because
I
think
this
is
a
really
cool
project.
It
took
a
lot
of
time.
B
It
actually
wasn't
just
the
past
year.
It
was
a
couple
of
years
where
we
worked
on
making
our
data
more
findable
and
accessible
in
the
community.
So
our
flagship
portals
microcosm
fico
cosm
now
for
algae
microcosms
for
fungi
phytosome
is
for
plants.
Img
is
for
microbes
and
metagenomes
have
done
an
amazing
job
serving
their
respective
communities
and
people
heavily
use
those
tools
to
do
analysis
of
their
respective
organism
of
interest.
B
So
we
have
these
use
cases,
kind
of
driving
the
need
for
new
infrastructure
and
rethinking
the
way
we're
doing
infrastructure.
And
you
know
we
have
this
fundamental
belief
that
sharing
data
is
going
to
make
for
better
science.
It
should
be
the
case
that
people
could
get
all
of
the
data
to
reproduce
and
experiment.
B
We
can
argue
about
whether
or
not
people
ever
will
reproduce
an
experiment,
but
the
fundamental
issue
is
like
they
need
to
be
able
to
access
the
data
and
we
need
to
provide
sort
of
clear
immutable
identifiers
so
that
people
can
come
back
and
find
it
again
and
again,
we
decided
in
2019
to
overhaul
our
major
data
access
points.
We
decided
to
actually
create
and
build
an
api
and
a
new
data
portal,
a
new
search
interface.
B
We
engaged
our
users
directly
and
we
simplified
access
to
jgi's
generated
public
data,
and
I
think
it's
really
important,
when
you're
building
a
user
facing
system
to
actually
engage
the
users
in
that
process.
This
is
a
big,
complex,
workflow
to
just
say.
You
know
we
iterated
through
this
process.
Over
the
last
year
and
a
half,
we
did
a
ton
of
user
research,
we
talked
with,
I
don't
know,
like
50
different
jgi
users
at
different
points
and
had
them
test
different
functionality
in
the
system.
B
This
is
a
snapshot
showing
kind
of
the
old
genome
portal,
where
everything's
organized
by
projects
and
in
the
new
data
portal.
Everything
is
now
organized
by
organism,
and
so,
if
you're
somebody,
that's
not
familiar
with
jgi,
if
you're
not
familiar
with
the
fact
that
we
organize
things
by
projects
because
pis
are
sending
us
samples,
you
know
when
you
come
to
this
interface
and
you
just
are
interested
in
say
e
coli.
B
You
know
it's
not
going
to
be
clear
to
you
necessarily
how
to
get
that,
and
that
was
what
we
heard
from
the
users
when
we,
when
we
did
interviews-
and
so
we
restructured
our
data
portal
to
be
focused
on
the
types
of
things
people
actually
wanted
to
search
for,
and
this
was
actually
a
huge
metadata
harmonization
problem.
Maybe
you
all
have
heard
this
phrase?
Maybe
you
haven't
but
metadata.
B
Harmonization
is
certainly
something
that
sounds
cool
right
now
right
and
that
means
kind
of
taking
your
metadata
and
making
it
sort
of
aligned
to
different
standards
and
getting
everyone
to
agree
that
if
there's
a
standard,
all
of
their
metadata
will
adhere
to
it
and
it's
a
huge
huge
pain
in
the
ass.
I
know
this
is
being
recorded
but
anyway,
and
why?
B
Because
it's
not
fun
to
sit
and
do
bookkeeping,
it's
not
fun
to
think
about
whether
or
not
this
id
that
you've
used
is
the
same
id
and
the
standard,
and
therefore
you
need
to
change
all
of
the
values
to
actually
match
what
the
set
like
what
the
controlled
vocabulary
is
and
like.
Are
you
using
ontologies
properly?
And
it's
just
it's
not
fun.
B
But
when
you
have
a
motivating
use
case
like
providing
a
new
access
point
to
jgi's
data
resources,
it
actually
catalyzed
an
effort
within
the
organization
to
align
on
a
few,
not
a
lot
of
common
metadata
fields,
but
a
handful
of
metadata
fields
that
power
that
organism-driven
search,
which
was
a
huge
effort
and
pretty
awesome
that
we
pulled
it
off,
and
this
clean
metadata
also
lets
us
have
a
cleaner
architecture.
So
we're
linking
everything
through
a
system
called
jamma
which
we've
talked
about
at
nurse
before,
but
for
those
who
haven't
heard
of
it.
B
B
That's
all
I'll
say
about
that,
but
it
can
be
a
place
to
kind
of
house
all
of
this
different
information
that
has
meaning
to
the
users,
but
is
not
necessarily
how
we
structure
and
organize
a
project
internally,
and
so
you
guys,
if
you're,
feeling
bored
someday
and
really
want
to
go,
find
your
favorite
microbe
or
find
your
favorite
plants
on
the
jgi
data
portal.
It's
data
yeah
anyway.
The
url
is
here.
B
It's
got
a
nicer
download
page,
and
one
of
the
things
that
we've
also
been
able
to
do
is
make
it
clear
when
data
is
on
tape
or
when
data's
on
disk
and
when
things
might
take
a
while.
We've
done
a
lot
of
work
actually
with
our
data
management
system,
and
by
way
I
mean
chris
b
croft
to
make
sure
that
we
don't
actually
abuse
the
hpss
system
at
nurse
which
actually
led
to
a
couple
of
fun
projects
but
yeah.
B
When
you
have
lots
of
different
identifiers
and
for
jdi,
they
can
either
get
data
from
jgi
or
they
can
get
it
from
the
the
sequence
read
archive
run
by
the
national
institute
of
health.
So
that's
like
the
main
sequence
data
repository
and
everyone
can
go
there
and
get
our
data
and
there
are
some
identifiers
associated
with
it,
but
they
might
not
cite
jgi
when
they
do
that,
and
so
we
have
kind
of
this
big
problem
of
trying
to
understand.
Really.
B
B
Yes,
why
is
it
challenging
because
there's
there
are
tons
of
identifiers,
and
so
we
took
metadata
from
jmo
and
we
worked
with
a
group
called
names
for
life
to
actually
take
all
of
those
identifiers
and
identify
publications
that
cite
jgi
identifiers,
that
weren't
actually
cataloged
in
our
resource,
and
so
we
actually
looked
at
these
results.
We
got
several
hundred
publications
back
with
a
98
validity
rate.
A
B
There
a
way
to
capture
this.
So
if
you
combine
the
names
for
life
work
that
we
did
with
this
tool
called
dimensions
out
of
all
metrics,
then
you
can
actually
find,
and
this
gets
kind
of
messy.
But
basically
you
can
identify
patents,
grants
and
other
publications
that
didn't
come
up
in
the
names
for
live
search.
Since
that's
just
looking
at
open
access
journals
and
the
takeaway
message
is
that
we
were
able
to
identify
an
additional
several.
B
B
When
you
want
to
think
about
the
impact
of
your
organization-
and
I
think
sometimes
you
get
a
little
too-
I
don't
know-
or
I
will
get
a
little
too
myopic
and
a
little
too
focused
on
like
what
jgi
is
doing
day
to
day,
and
I
think
it's
kind
of
cool
to
actually
be
able
to
extrapolate
beyond
that.
But
this
was
only
possible
because
we
actually
have
our
metadata
managed
in
a
centralized
system
and
because
we
have
this
collaboration
to
identify
these
publications
in
the
open
source,
literature.
B
And
then
there
are
existing
tools
like
dimensions
where
you
can
pay
for
a
license
and
get
all
this
information
back.
But
with
a
single
jamo
record,
we
were
able
to
identify
an
additional
82
publications
with
a
ton
of
citations,
and
so
this
gives
jgi
like
a
new
kind
of
data
resource
that
we
can
use
to
kind
of
look
through
and
understand
how
people
are
using
our
data.
It
lets
us
potentially
identify
new
data
users
to
provide
input
on
what
data
resources
are
of
value
to
them.
B
B
We'd
love
to
understand
how
to
prioritize
resourcing
effort
to
make
these
data
sets
easy
to
get
when
somebody
puts
in
a
request
for
120
000
files,
that's
pretty
disruptive
to
the
people
who
just
want
to
restore
10
and
download
them,
and
so
how
do
we
figure
out
who
to
prioritize?
When
do
we
draw
the
line
and
say
hey
we'd
like
to
understand
what
you're
doing
with
the
data
before
you
access
all
of
jgi's?
B
I
don't
know
historical
data
records,
and
so
this
is
going
to
become
a
growing
issue,
because
we're
only
seeing
these
types
of
requests
increase
and
I've
been
pretty
happy
that
we've
had
the
infrastructure
in
place
to
help
support
it.
So
it's
kind
of
a
balance
in
wanting
to
open
up
the
floodgates,
but
then
making
sure
you've
got
good
infrastructure
in
place
when
people
do
come
and
want
to
download
the
data.
B
So
in
summary,
we
are
moving
towards
this
resilient
and
stable
infrastructure
in
an
ideal
world.
I
it's
going
to
be
pretty
hard
to
get
there,
but
that's
our
goal,
and
so
we've
moved
to
this
distributed
computing
model,
which
presents
a
lot
of
challenges
for
staff
and
for
management
for
social
engineering
and
for
having
the
right
software
infrastructure
to
support
it.
B
B
Fair
is
only
going
to
increase
in
popularity.
I
think
over
the
next
couple
of
years,
there's
all
this
like
fair
data
for
artificial
intelligence
and
machine
learning
that
you're
going
to
hear
about
too,
and
I
do
believe
that
if
you
don't
have
kind
of
this
corpus
of
usable
scientific
data,
it's
going
to
be
really
hard
for
us
to
leverage
the
methods
that
are
being
developed
right
now,
and
I
think
there
are
people
here
that
are
much
better
versed
in
this
than
me.
All.