►
From YouTube: Evolution of Data Services for Science
Description
Part of the Data Day 2022 October 26-27, 2022
Please see https://www.nersc.gov/users/training/data-day/data-day-2022/ for the training agenda and presentation slides.
A
Thanks
for
coming
today,
today,
I'm
just
going
to
give
a
brief
introduction
to
desk
services
and
and
where
we
see
things
going
at
nurse,
so
nurse
has
a
huge
number
of
data
services.
Those
supports
these
are
across.
You
know
all
different
areas
from
data
transfer
to
data
management,
to
visualization
to
containers
and
data
analytics.
A
So
one
way
of
kind
of
representing
this
that
I
produced
for
a
Blog
a
few
years
back
is
you
know,
kind
of
in
this
way
and
I
know
it's
kind
of
more
complex
than
this.
But
there's
you
know
need
to
get
data
from
scientific
instruments
store
it
on
big
file
systems
interact
with
that
in
an
efficient
manner,
steer
workflows
either
interactively
or
via
workflow
managers,
and
this
interfaces
with
services
that
might
be
Standalone
databases
or
there
might
be
a
more
flexible
service
platform.
A
And
then
you
need
to
interact
with
HPC
system
and
there's
all
kinds
of
policy,
and
also
technology
needed
to
to
make
that
efficient.
So
including
things
like
containerization
technology
and
then
actually
producing
results
is
also
a
kind
of
data
problem.
So
there's
analytics
tools
and
increasing
amount
of
machine
learning
tools
and
visualization,
and
all
of
this
is
a
ever-changing
ecosystem.
A
So
if
you
actually
look
at
the
blog,
I've
updated
many
of
these
entries,
but
a
lot
of
the
picture
is
static,
but
some
of
it
very
quickly
changing
so
I
thought
it
was
interesting
to
look
back
at
one
previous
day
today,
so
in
in
those
days
we
had
less
technical
problems
because
most
people
are
in
building
50
Auditorium
in
person,
but
many
things
other
things
have
changed
as
well
in
this
time,
so
I
thought
it
was
interesting
to
look
at
this
machine
learning
talk
here
and
who
even
remembers
what
lasagna
is
well
well
named
framework,
but
not
it's
not
that
long-lived.
A
So
you
know
tensorflow
was
just
kind
of
starting
them
and
then
turned
out
to
dominate
and
even
high
touch
didn't
even
exist
at
that
times.
You'll
see
now
it's
kind
of
rapidly
growing,
so
things
have
certainly
changed
since
then,
so
this
plot
only
starts
from
2018,
so
2016
was
even
further
back,
so
it
was
probably
less
than
100
users
of
Jupiter.
Now
and
again.
So
now
there's
over
a
you
know
several
thousand
and
in
fact
we
can
see
from
daily
usage.
A
Now,
basically,
everybody
uses
python,
I
think
you
can
say
to
First
approximation
and
not
even
that,
but
it's
also
used
this
kind
of
numbers
about
what
batch
jobs
use
both
then
in
some
way,
and
that's
also
a
large
fraction
of
the
jobs
that
we're
running
with
deep
learning
that
we
saw
a
growth
in
just
three
years
of
6x
and
as
I
mentioned
before,
paytouch
wasn't
even
on
our
radar
in
2017,
for
example,
and
it's
now
pretty
much
overtaken
tensorflow
I
mean
this
is
last
year's
plot,
but
it's
now
overtaken
and
then
containers.
A
We
also
see
a
similar
growth,
hundreds
of
users,
but
perhaps
even
more
impressively.
The
top
500
result
that
permature
submitted
and
put
it
at
number
five
was
in
fact
run
inside
a
Shaker
container.
So
these
Technologies
really
are
not
no
rapidly
become
part
of
the
mainstream.
A
A
So
all
of
these
Services
run
on
our
big
super
computers.
You
know
that's
the
the
essential
for
or
interface
with,
our
supercomputers.
So
that's
the
core
of
what
we
do
at
nurse
and
you've
probably
seen
various
presentations
on
this.
A
But
just
in
case
you
haven't,
you
know,
permatures
most
of
the
compute
power
is
centered
in
these
GPU
nodes,
the
Nvidia
a100
accelerated,
and
this
is
a
great
resource
for
deep
learning,
for
example,
but
then
there's
also
a
large
number
of
CPU
nodes
which
can
be
used
further,
more
traditional
analytics
of
experiment,
workflows
or
things
that
are
difficult
or
impractical
to.
A
That's
important
to
data
and
analytics
such
as
the
all
flash
file
system
and
the
fast
connections
out
to
external
facilities
and
to
larger
file
systems
in
this
and
across
these
systems,
data
services
interact,
and
so
this
shows,
as
I
mentioned,
that
data
comes
in
from
outside,
runs
potentially
on
CPU
nodes
and
on
GPU
nodes
interacts
with
the
file
system,
and
this
can
all
be
driven
by
workflow
integration
and
I
just
wanted
to
comment
here
that
you
know
we're
really
at
the
start
of
parameter
and
we're
going
to
be
seeing
increasing
data
capabilities
integrated
into
a
parameter,
and
this
is
particularly
true
for
like
work
for
integration
or
expect
to
be
able
to
bring
containerized
Services
closer
into
the
system.
A
Okay,
so
this
was
already
planned
kind
of
in
the
nurse
nine
review
period,
but
for
nurse
10
we're
just
starting
this
planning
now
and
it's
going
to
be
even
more
workflow
and
data
capable,
and
so
this
is
kind
of
the
just
overview
slide
of
what
we're
talking
about
with
nurse
camera
shows,
firstly,
that
it
extends
out
into
the
system,
but
also
I
mean
out
into
esnet
and
out
to
instruments,
but
also
that
it
will
have
workflow
Services
built
into
that
okay.
A
So,
as
I
pointed
out,
things
have
moved
on,
so
we
now
have
all
of
this
great
data
transfer
tools.
We
have
I
o
Library.
We
have
performer
file
systems.
We
have
these
flexible,
python-based
Frameworks
that
allow
really
sophisticated
tools
to
be
kind
of
at
the
fingertips
and
containerized
services
that
enable
complex
Stacks
to
be
there,
and
also
for
this
to
be
portable
on
different
systems,
and
we
have
all
these
tools
for
Building
Services
that
sit,
for
example,
on
the
side
of
the
machine
and
drive
things.
A
But
you
know
there's
remaining
challenges
and
I
I
outlined
kind
of
some
of
this
direction.
In
a
talk,
that's
linked
here.
A
longer
seminar-
and
this
wasn't
necessarily
coordinated
but
I-
think
some
of
the
talks
that
we're
talking
about
in
this
meeting
actually
touch
on
many
of
these
aspects
of
these
challenges.
A
So
one
important
area
is
IO
where
data
volumes
are
still
increasing
larger
than
faster
than
IO
can
keep
up
with,
and
this
both
means
I
think
that
we
need
developments
in
kind
of
the
the
way
that
we
store
data
and
the
way
that
we
do
processing
on
storage,
so
that
those
are
research
aspects
that
I
know
Sean
nuclear
works
on
as
well,
but
another
important
area
is
actually
just
improving
the
I
o
that
we
do
and
I
O
profiling
is
an
important
piece
of
that
that
we'll
hear
about
soon
and
then
this
area
of
workflow
services
and
bringing
them
into
HPC
systems.
A
And
so
we've
got
a
bunch
of
talks
about
how
that
can
be
done
with
our
spin
service
and
with
workflow
managers
running
close
to
the
HPC
system
and
using
of
apis
to
do
that
and
then
about
using
these
productive
languages.
I
mentioned
python.
Libraries
are
pretty
capable,
but
also
using
them
with
large-scale
compute
is
not
a
solved
problem,
but
there's
various
directions
that
can
help
with
that
that
we'll
hear
about,
and
then
you
know,
maybe
python
isn't
the
right
language.
A
A
A
So
we're
going
to
hear
about
that
and
with
some
you
know,
demos
on
how
to
achieve
this
from
Steve
and
then
there's
also
this
kind
of
emerging
tool,
I,
guess
Jacks,
which
not
only
kind
of
helps
to
address
the
the
kind
of
question
of
scaling
python
onto
onto
gpus
but
also
I,
think
an
important
part
of
this
is:
it
brings
potentially
Auto
differentiation
to
software
written
in
Jax,
and
this
I
think
is
this
towards
this
last
point
about
adding
you
know
direct
inference
on
experimental
data
by
interfacing
incineration
with
differentiation
foreign
agenda.
A
Obviously,
we
had
a
somewhat
stunted
Stark
here
at
the
AV
problems,
but
definitely
there's
a
lot
to
a
lot
to
come
here
so
stay
tuned
for
it
all.
Okay,.