►
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title:
Tiled: A Service for Structured Data Access
Speaker:
Dan Allan, NSLS-II - Brookhaven National Laboratory
Abstract:
In the Data Science and Systems Integration Program at NSLS-II, we have explored various ways to separate I/O code from user science code. After seven years of developing in-house solutions and contributing to external ones (including Intake), we propose an abstraction that we think is a broadly useful building block, named Tiled. Tiled is a data access service for data-aware portals and data science tools. It has a Python client that feels much like h5py to use and integrates naturally with dask, but nothing about the service is Python-specific; it also works from curl. Tiled’s service sits atop databases, filesystems, and/or remote services to enable search and structured, chunk-wise access to data in an extensible variety of appropriate formats, providing data in a consistent structure regardless of the format the data happens to be stored in at rest. The natively-supported formats span slow but widespread interchange formats (e.g. CSV, JSON) and fast, efficient ones (e.g. C buffers, Apache Arrow Tables). Tiled enables slicing and sub-selection to read and transfer only the data of interest, and it enables parallelized download of many chunks at once. Users can access data with very light software dependencies and fast partial downloads. Tiled puts an emphasis on structures rather than formats, including N-dimensional strided arrays (i.e. numpy-like arrays), tabular data (i.e. pandas-like“dataframes”), and hierarchical structures thereof (e.g. xarrays, HDF5-compatible structures like NeXus). Tiled implements extensible access control enforcement based on web security standards, similar -to JupyterHub Authenticators. Like Jupyter, Tiled can be used by a single user or deployed as a shared resource. Tiled facilitates local client-side caching in a standard web browser or in Tiled’s Python client, making efficient use of bandwidth and enabling an offline “airplane mode.” Service-side caching of "hot" datasets and resources is also possible. Tiled is conceptually “complete” but still new enough that there is room for disruptive suggestions and feedback. We are interested in particular in exploring how Tiled could be made broadly available to NERSC users alongside traditional file-based access, and how that work might prompt us to rethink aspects of Tiled’s design.
Bio:
Dan Allan is scientific software developer and group lead in the Data Science and Systems Integration Program at NSLS-II. He joined Brookhaven National Lab as a post-doc in 2015 after studying soft condensed-matter experimental physics and getting involved in the open source scientific Python community. He works on data acquisition, management, and analysis within and around the "Bluesky" software ecosystem.
Host of Seminar:
Bjoern Enders
Data Science Engagement Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
B
A
Scientific
software
developer,
he
works
on
data
acquisition,
management
and
analysis
with
a
blue
sky
ecosystem.
That's
a
thing
in
the
light
source
community-
not
maybe
not
some
not
so
much
known
here
at
nurse
for
crd
yeah,
and
he
will
talk
about
tire
which
the
service
they
use.
They
are
going
to
use
to
make
data
available
for
the
users
at
the
same
time.
So
then
adam
just
take
it
away.
Sure.
B
B
So
it's
not
something
we're
new
to
thinking
about,
but
this
latest
iteration
had
its
first
commit
in
february
of
this
year,
so
we're
at
a
stage
of
looking
for
feedback
on
how
we're
framing
the
problem
any
opportunities
we
might
be
missing
in
the
way
that
we're
thinking
about
it
connections
to
other
work
that
we
might
not
know
about,
and,
and
especially
people
who
want
to
play
with
this
early
and
and
give
us
feedback
and
maybe
even
work
on
it
with
us.
So
that's
that's!
Why
we're
here?
B
Some
of
this
work
was
influenced
by
collaboration
with
a
project
called
intake
out
of
anaconda
run
by
martin
durant,
working
with
garrett
bischoff
at
nsls2,
so
to
zoom
way
out
the
problem
that
we
have
at
user
facilities
like
nsls2,
which
are
effectively
nsls2's,
28
small
research
groups,
sticky
taped
together
right
and
they
have
extremely
different
data
scales.
We
have
groups
that
are
producing
a
gigabyte
in
a
whole
year
and
groups
that
produce
terabytes
in
a
day.
So
there's
a
lot
of
volume
and
there's
a
lot
of
velocity
and
these
things
that
people
talk
about.
B
But
at
a
synchrotron
somebody
can
walk
up
with
a
random
new
instrument
and
and
expect
to
be
able
to
plug
it
in
and
importantly,
from
our
point
of
view,
incorporate
that
in
their
data
structures
and
in
their
data
processing
pipeline
downstream.
So
the
workloads
are
very
dynamic
and
there's
a
lot
of
heterogeneity
across
all
the
groups
that
we're
responsible
for
supporting.
A
A
C
B
So
we
have,
we
have
many
file
formats.
We
have
to
some
degree,
that's
driven
by
whatever
the
vendor
of
the
the
hardware
specifies
and
that's
not
always
the
most
convenient
format
to
work.
In
also,
we
have
a
large
number
of
scientists
that
we
work
with
and
support
both
the
people
that
that
work
in
our
facility
and
can
be
to
some
degree
wrangled
and
the
users
who
we
serve
as
as
more
or
less
customers
who
really
really
can't
be
your
ankle
that
we
they
they
want,
what
they
want.
B
They're,
also
largely
stuck
under
this
mode
of
parsing
data
out
of
file
names
and
not
really
thinking
about
searchability
or
databases
in
any
scalable
way.
This
is
just
where
our
field
is,
and
as
a
software
support
group
like
the
one
that
I'm
in
this
gives
us
a
lot
of
variety
that
we
have
to
wrangle
and
invest
in
so
we'd
like
to
become
unopinionated
about
file
formats.
We'd
like
to
have
this
idea
of
a
service
that
we're
calling
tiled
sitting
between
the
data.
B
However,
it
is
right,
so
some
of
our
data
is
in
nice,
searchable,
scalable
databases,
some
of
it's
just
a
directory
of
files,
and
some
of
it
might
be
in
services,
including
globus
or
one-off
use,
case-specific
data
services,
and
what?
If
we
could
transcode
the
data
into
the
format
that
the
users
want
on
the
fly
and,
more
importantly,
or
just,
as
importantly,
access
it
partially
so,
rather
than
knowing
that
we
just
have
an
opaque
file.
B
Knowing
that
we
have
a
specific
structure
like
an
array
or
a
table,
with
certain
columns
and
being
able
to
access
that,
both
through
purpose-built
clients
like
the
python
client
that
we
have
and
others
that
we
may
plan
and
something
as
basic
as
curl
or
a
web
browser
or
or
a
purpose-built
web
portal.
All
of
these
things
are
interested
in
accessing
the
parts
of
the
data
that
they
need
in
the
format
that
makes
it
most
convenient
for
their
downstream
work.
B
We
want
to
get
away
from
relying
only
on
metadata
in
file
names.
We
want
to
support
searches
both
that
rely
that
expose
the
full
complexity
of
some
underlying
database
like
where
you
could
filter
your
data,
sets
by
the
motors
that
you
used
or
a
temperature
range
or
the
sample
under
study
and
higher
level
queries
that
might
rely
on
less
programming
knowledge
and
might
also
be
more
portable
across
storage
mechanisms
with
different
backends.
B
Its
goal
is
to
improve
data
access
for
data
science
and
scientific
analysis
generally
leveraging
open
web
standards
and
scientific
scientific
computing
standards
trying
to
be
as
as
non-clever
as
possible
and
using
standards
that
people
can
already
easily
interact
with.
So,
as
I
said
at
the
very
top,
this
comes
on
top
of
years
of
thinking
about
similar
things.
I
personally
trace
it
back
to
a
project
for
abstracting
over
getting
image
sequences
into
python
tom
caswell,
and
I
both
happened
to
collaborate
on
at
bnl.
B
We
tried
to
generalize
this
to
not
just
image
sequences
but
all
kinds
of
data.
We
learned
about
a
project
out
of
anaconda
with
similar
goals
of
abstracting
over
I
o,
but
at
the
end
of
the
day,
very
python,
centric
right.
Getting
your
data
from,
however,
it's
stored
into
python,
but
intake
had
the
germ
of
this
idea
of
a
server
where
you
could
have
on
the
server
side
all
the
complexity
of
doing
the
actual
I
o
and
then
just
sending
a
rather
thin
python
client,
the
metadata
and
the
data
in
this
chunk
wise
fashion.
B
But
intake's
approach
was
not
as
performant
as
we
needed,
and
it
was
also
focused
very
much
on
python.
The
http
api
wasn't
something
you
could
easily
interact
with
from
from
curl
and
make
any
sense
of
it.
So
we
set
out
to
reimagine
this
in
a
service
first
approach
that
abstracts
out
the
I
o
in
the
same
way
as
these
other
attempts,
but
does
it
in
a
more
multi-language
and
web
oriented
way.
B
So,
at
the
end
of
the
day,
the
goal
is
to
separate
the
storage
of
data
from
the
access
of
data
where
the
service
in
the
middle
acts
like
a
transcoding
service
right.
These
things
exist
in
industry
for
video
transcoding,
where
youtube
is
transcoding
different.
You
know
versions
of
a
video.
This
is
kind
of
like
that,
but
for
scientific
data
is
one
way
to
think
about
it.
B
So,
as
we
run
into
particular
use
cases,
we
can
just
add
them
by
separating
how
we
store
the
data
and
how
we
access
the
data,
we
can
change
our
minds
about
how
we
want
to
store
the
data
over
time
without
breaking
the
analysis
code
and
a
use
case,
in
my
mind,
always
is
the
postdoc
who
writes
a
really
fancy
analysis
routine,
but
deeply
embedded
in.
That
is
the
assumption
that
the
data
is
stored
in
tiff
files,
with
a
certain
directory
structure
and
file
name.
B
And
then,
if
you
want
to
change
that,
to
blob
storage
and
czar
or
some
other
format,
you
have
to
go
back
and
understand
the
science
code.
You
can't
just
switch
it
out
if
we
can
have
the
science
code,
interacting
with
numpy
arrays
and
pandas
data
frames
or
in
other
languages
the
analogous
structures.
B
Then
we
can
separate
that
I
o
and
future
proof
our
code
better.
So
by
standardizing
on
structures
and
not
formats,
and
by
kind
of
surrendering
that
we'll
never
get.
Everyone
agree
on
one
one
format
we
can.
We
can
get
a
lot
of
value.
I
think
because
software
tools
might
disagree
on
the
ideal
storage
format
at
rest,
but
they
universally
understand
concepts
like
a
strided
array
or
a
table
of
data
and
hierarchical
structures
of
these
things.
B
These
are
agnostic
really
to
how
they
happen
to
be
stored
or
the
programming
language
that
they
happen
to
be
used
in.
So,
for
example,
if
I
have
tabular
data,
I
could
request
that
as
apache
arrow,
which
is
the
modern
type
of
format
that
seems
to
be
taking
over
and
it's
pandas
blessed
wire
format,
and
we
can.
B
B
B
And
we
have
all
the
standard,
useful
things
you
can
do
with
a
web
service,
modern
web
security
standards,
plugable
authentication,
and
to
here
we're
really
looking
to
jupiter,
which
has
had
such
success
being
deployed
across.
You
know
all
of
the
institutions
that
we
really
care
about
so
rather
than
having
a
user
signup
system
or
something
we're
doing
bring
your
own
user
management
system
says
jupiter
lab
that
should
say
jupiter
hub,
integrate
with
pam
or
openid
connect.
We
have
plans
to
integrate
with
others
in
the
future.
B
We
have
an
idea
of
programmable
access
enforcement,
so,
for
example,
at
nsls2
we
have
access
control
at
the
level
of
a
proposal.
Users
submit
a
proposal,
take
data
on
that
proposal
and
then
their
access
is
by
proposal
and
tiled
has
to
integrate
with
our
proposal
system
to
know
who
is
allowed
to
see
what
and
that's
about
50
lines
of
python.
I
might
have
a
chance
to
show
it
to
integrate
with
our
particular
access
control
and
you
could
do
as
well
with
others
because
we're
a
service.
B
We
can
do
clever
things
to
make
things
fast,
so
we
we
have
an
ambition.
I
don't
think
we've
achieved
this
in
any
case,
I've
measured,
but
we
have
an
ambition
to
make
this
faster
than
than
local
file
access
in
some
cases,
because
oftentimes
there's
a
significant
encoding
step
and
some
disk
that
you're
waiting
for
to
pull
data
up
into
ram
and
into
a
memory
layout
that
you
can
actually
work
with
it
and
we
can.
B
We
can
use
our
knowledge
of
what
data
sets
are
popular
or
likely
to
be
accessed
to
cache
in
advance,
or
you
know,
based
on
based
on
user
access
patterns,
the
data
directly
in
a
memory
layout
that
it's
useful
to
the
analysis
code
make
things
very
fast.
We
also
have
a
concept
of
client-side.
Caching
like
a
web
browser
has
so
once
you
access
the
data
set
a
first
time
it
might
get
saved
locally
and
then
be
faster.
B
The
second
time
with
standard
web
semantics
around
checking
that
the
data
hasn't
been
changed
since
you
looked
and
an
offline
mode
as
well
all
right.
So
I'm
going
to
get
into.
Let
me
actually
pause
here
for
a
breath,
I'm
going
to
get
into
an
interactive
demo
and
then
try
to
leave
plenty
of
time
for
discussion.
B
Each
dataset
has
metadata
just
like
an
attribute
in
hdf5
we
have
arrays,
including
numpy,
record
arrays,
structured
arrays.
We
can
do
data
frames
and
apache
arrow
tabular
structures
as
well
as
x-ray.
We
can
integrate
with
das
to
create
das
objects
that
wrap
http
requests,
so
you've
got
something
that
you
can
use
as
if
it's
an
array,
but
only
actually
fetches
the
data
it
needs
when
it
needs
it.
I'll
demo
that
we've
got
configuration
on
the
server
side
on
the
client
side.
B
That
has
nice
validation
and
auto
generated
documentation,
we're
doing
compression
in
all
the
popular
ways.
We
have
a
nice
cli
that
I'll
demo
for
running
the
server
and
inspecting
the
config
and
so
on.
We
can
serialize
in
a
bunch
of
formats,
some
of
which
I've
named
and
we
have
documentation
on
how
to
add
more.
B
We
have
some
nice
instrumentation
and
prometheus
metrics,
so
we
can
observe
the
performance
of
the
web
application
as
people
are
using
it
and
optimize,
and
we
have
pretty
decent
docs
what's
in
progress
right
now
or
prototype
and
working,
but
I
wouldn't
you
know-
wouldn't
stake
my
life
on.
It
is
a
basic
shared
secret
authentication
like
like
a
basic
single
user.
Jupyter
notebook
would
have.
We
have
oauth2
authentication
with
refresh
tokens
implemented.
B
B
We
have
standard
web
caching,
we
have
service
side
in
process.
Caching,
we
we
have
a
way
to
dispatch
to
custom
custom
objects
on
the
python
side,
to
give
users
exactly
the
the
structures
and
the
kind
of
overall
user
experience
that
they
want,
so
that
tiled
can
kind
of
be
buried
inside
domain
specific
applications
that
a
couple
of
our
users
have
have
built
for
themselves.
So
the
idea
is
that
tile
doesn't
have
to
be
the
thing
that
the
users
know
they're
using
right.
B
They
can
just
kind
of
be
hidden
in
the
stack
and
be
hopefully
giving
some
nice
features
for
free
behind,
whatever
user
experience
wants
to
be
at
the
front
end
of
that
interaction,
we
have.
We
have
dreamed
of
batching
these
transcoding
jobs
in
an
asynchronous
way,
having
some
sort
of
redis
based
caching,
so
that
multiple
workers
can
share
the
cache.
We've
been
talking
with
the
developers
of
awkward
array
which,
if
you
don't
know,
handles
ragged
ragged,
deeply
nested
structures
coming
out
of
the
nsf-funded
cern-affiliated
group,
iris
hep.
B
We
care
about
sparse
arrays
and
image
pyramids,
and,
and
and
and
so
on,
and
so
on.
I
guess
I'll
I'll
borrow
through
the
rest
here
a
little
bit
because
it's
a
lot
of
a
wall
of
text.
The
last
piece
is
title
is
currently
for
reading
the
data
underneath
it
can
can
change
over
time
and
it
can
update
to
that.
But
it
doesn't
let
you
write
through
the
through
the
web
client.
B
So
a
user
or
client
of
child
can't
initiate
a
change
to
the
data,
but
that
is
something
we're
very
interested
in
and
are
likely
to
prototype
early
next
year.
So,
in
terms
of
you
know
our
commitment
to
this
and
and
the
scale
of
deploying
it
I've
said
it's
very,
very
early
work.
I
would
call
it
alpha
code,
so
we're
really
here
for
feedback,
but
if
you're
interested
in
the
trajectory
of
the
project,
it
is
three
clause
psd
licensed
it
has
a
has
a
governance
model
in
progress.
B
It's
supported
by
the
program
that
I
work
in
led
by
stuart
wilkins
and
stuart
campbell
at
msls2,
and
our
our
deployment
is
internal
but
will
be
exposed
externally.
As
soon
as
we
finish
undergoing
a
penetration
test
in
coordination
with
bnl
cyber
and
we
are,
we
are
betting.
Our
data
access
on
this
we've
been
using
the
the
python
based
data
access
that
this
grew
out
of
for
years
and
we're
looking
to
build
on
that
with
this
service
and
and
support
it
for
the
long
haul,
so
some
final
thoughts
before
the
demo.
B
You
know
in
terms
of
what
this
really
is:
it's
not
file-based
access
like
globus,
it's
something
that
would
sit
well
alongside
a
service
like
globus,
because
it
has
to
actually
open
the
files
right.
It
has
to
know,
know
what
they
are
in
order
to
provide
this
partial
access.
It's
not
it's
not
a
serverless
framework.
It's
not
a
format.
B
I
think
that
tile
db
and
czar
are
in
the
space,
solving
a
similar
problem
of
partial
and
highly
paralyzable
access
to
data,
by
asserting
that
everyone
should
standardize
on
that
format,
and
I
think
there
are
fields
where
that
works,
but
I
don't
think
synchrotron.
Science
is
one
of
those
fields
where
you
can
assert
a
standard
storage
format
that
everyone
works
with.
We
need
this
ability
to
meet
scientists
where
they
are
and
let
them
you
know,
let
them
take
the
data
in
whatever
form
they
need
it.
B
B
B
B
Around
collecting,
good
ideas
and
some
nascent
interest
diamond
is
hiring
someone
or
is
placing
one
of
their
new
hires
on
this,
to
look
at
it
in
january
to
see
how
it
might
fit
into
their
architecture
and-
and
a
couple
other
folks
may
be
trying
it
in
the
near
future.
So
with
that
I'll
take
a
breath
and
set
up
a
demo,
I
can
take
a
question
or
two.
While
I
switch
contexts.
B
Okay,
let
me
move
around
some
things
to
get
this
aspect:
ratio
right.
How
does
that
look
instead,
great
okay,
so
this
is
what
it
can
look
like
to
serve
some
data
with
tiled
in
python.
This
is
like
the
most
minimal
version
of
this
that
you
can
make
where
you
say.
I
just
have
a
numpy
array
and
another
numpy
array
and
a
data
frame
with
three
columns
in
it.
That's
partitioned
up
and
some
metadata
with
a
date
in
it
or
something-
and
I
want
to
make
this
available
this
data
available
to
my
users
via
tile.
B
So
what
you
can
then
do.
This
is
sort
of
a
demo
route
entitled
is
tiled,
serve
pi
object,
pi
object
and
give
it
the
the
path
to
this.
So
this
happens
to
be
within
tile
itself,
wouldn't
have
to
be,
and
that
starts
a
web
server
and
if
you've
ever
typed
jupyter
notebook,
you
see
a
similar
output
from
them.
They
generate
a
one-time
or
for
this
for
this
process
they
generate
a
random
api
key
to
make
it
secure
so
that
other
processes
on
this
machine
can't
spy
on
that
data.
B
B
B
To
demonstrate
that
that
works
with
curl,
but
it's
nicer
to
use
this
program
http,
which
gives
you
a
nice
color-coded
output,
it's
a
little
more
modern
than
curl.
So
it's
going
to
describe
this
data.
It's
going
to
tell
me
that
it's
in
apparently
one
single
chunk,
that's
100
by
100
and
we
can
see
the
endingness
and
the
size
and
that
it's
float
and
that
it's
an
array
and
it
gives
us
various
links
that
we
could
use
to
get
at
the
data.
B
B
So
having
done
this
from
just
a
basic
web
client,
now,
let's
upgrade
to
a
python
client
and
we'll
just
connect
to
localhost
port
8000
from
the
python
client,
and
what
we
have
now
is
a
user
experience.
That
is
something
like
something
like
h5
pi.
If
you've
ever
used
that
it
basically
treats
this
tree
of
data
as
a
dictionary
oops
capital,
a
as
the
dictionary
that
I
can
get
into
to
get
various
things
that
hasn't
done
any
major
io
yet
because
it
doesn't
know.
B
If
I
want
the
whole
array
or
just
some
of
it,
I
could
get
the
whole
array
in
the
same
way.
I
would
with
h5pi
if
I
could
get
just
part
of
it,
and
it's
only
sending
what
I
actually
want
with
the
dataframe
client.
I
can
grab
just
two
of
the
columns.
For
example,
I
think
it's
called
x
y
yep
there's
a
data
frame
with
two
columns
in
it.
We
go
over
and
look
at
what
the
server's
up
to.
B
We
can
see
it's
asking
for
block
0
0
of
this
array,
it's
asking
for
a
partitions,
0,
1
and
2
of
this
data
frame,
and
so
on.
So
from
the
user
point
of
view
in
the
python
client.
This
feels
just
like
I'm
working
with
a
file
or
something,
but
under
the
hood
it's
making
potentially
authenticated
http
requests.
B
B
B
B
If
I
remember
right-
and
I
could-
or
maybe
it's
I'm
not-
I'm
not
going
to
swear
by
that
size
anyway,
large
enough
to
take
a
moment
so
that
we
can
demonstrate
this
next
bit,
where
I
can
say,
use
a
special
registry
of
clients
for
this-
that,
instead
of
greedily
downloading,
the
data
will
create
desk
structures
that
that
do
so
lazily
and
so
now
the
same
thing
gets
me
a
desk
array
with
the
given
shape,
and
now
I'm
just
in
desk.
So
nothing
about
this
object
is
particular
to
tiled.
B
Then
we'll
see
batches
of
requests
going
out
for
different
slices.
You
can
see
these
slice
parameters
here
changing
and
this
is
these
are
happening
in
batches
of
about
eight,
because
that's
the
number
of
desk
threaded
workers
that
are
running
in
parallel
here.
So
all
this
falls
out
pretty,
naturally
without
a
lot
of
with
no
like
special
knowledge
of
tiled.
In
my
analysis
code,
I
can
write
this
like
it's
like
basic
python
data
science
code
and
everything
kind
of
works.
The
way
that
I
expect
I'm
going
to
go
into
a
web
browser
now.
B
Okay,
so
open
up
a
new
tab,
localhost
8000.
If
I
just
go
to
the
route,
it
tells
me
some
things
like
what
version
is
this
and
what
formats
does
it
support
for
each
of
the
different
structures
and
so
on?
I
know
what
I'm
looking
for,
so
I'm
just
going
to
type
it
I'm
going
to
say
I
want
the
full
array
of
medium
image
and
browsers
when
they
make
requests,
give
a
list
of
formats
that
they
understand.
One
of
them
is
html,
so
tile
generates
an
html
page
with
an
image
in
it.
B
B
B
B
All
right
and
now
I've
grabbed
just
these
two,
these
two
columns,
so
it's
a
partial
partial
access
is
the
selling
point
here
all
right.
The
last
thing
I
want
to
show
is
an
example
where
we
have
data
that
just
comes
to
us
as
files,
and
so
it's
probably
maintained
by
a
research
group
that
does
not
know
or
care
about
structured
access
to
data.
They
don't
have
a
database,
yet
they
just
have
a
directory
of
files.
But
I'd
like
to
have
you
know
something
useful
for
them.
B
I
can
say:
tiled
serve,
tiled,
serve
directory
public
and
then
example
file.
So
let
me
let
me
show
you
what
the
example
files
are.
First,
actually
so.
Here's
the
example
files.
I've
got
some
nested
directories.
I've
got
some
tiff
a
csv,
an
excel
spreadsheet.
That's
what's
going
on
in
here.
So
now
I'll
tell
tiled
to
serve
this
directory,
and
this
is
just
a
sort
of
a
built-in
convenience.
You
know
one
use
case,
one
way
of
integrating
where
it's
going
to
walk
this
directory.
B
B
B
B
Something
else
I
might
want
to
do
is
cache
this
so
that
I'm
not
downloading
it
all
the
time.
At
the
moment,
we
don't
do
that
by
default,
maybe
someday
we
will
but
cast
on
disk
my
cache
stuff,
that's
going
to
generate
files
in
a
directory
that
are
treated
as
internal
you're,
not
supposed
to
poke
at
them
or
know
what
they
are,
but
tiled
will
manage
them
where
if
I
go
and
access
some
data
like
this-
and
maybe
what's
in
this
one,
maybe
this
and
then
I
kill
my
server
so
that's
offline.
B
B
B
I
see
their
questions
in
the
chat
but
I'll
circle
back
at
the
end
here
right,
so
here's
an
example
of
from
the
documentation
that
you
can
find
I'll
put
links
in
the
chat
at
the
end,
where
there's
a
public
demo
deployment
of
tiled
that
you
can
log
into
with
with
your
orc
id.
If
you
have
an
arc
id
and
so
what
happens
is
it
gives
you
a
url
to
paste
into
your
browser?
You
paste
that
into
your
browser.
You
get
prompted
to
log
into
orc
id.
B
C
B
This
is
an
optional
aspect
of
tile.
I
think
some
people
use
it
as
a
personal
utility
running
it
from
the
command
line,
like
I
showed.
Other
people
will
maybe
use
it
as
a
fully
public
data
service
with
no
authentication
needed.
But
if
you
need
to
integrate
with
system
user
accounts,
you
can
do
that
with
a
pam
authenticator,
just
like
jupyter
hub
does.
B
B
If
I've
got
data
sets
a
b
c
and
d,
I
can
generate
some
some
users
with
a
really
dumb
dictionary,
authenticator,
that's
just
for
for
toys
and
testing,
and
I
can
have
this
access
policy
that
says
who
can
see
which
data
sets
that
obviously
does
not
scale.
So
the
the
way
to
do
this
at
a
facility
scale
is
to
have
some
other
programmable
access
policy.
B
That's
making
you
know,
making
requests
against
some
some.
You
know
user
user
management
system
and
that
just
to
show
you
you
know
it's
not
really
that
much
code.
This
has
caching
and
logging
and
all
the
bells
and
whistles,
so
the
actual
functional
part
is
like
10
or
15
lines
to
get
something
that
integrates
right.
So
that's
at
a
10
000
foot
view
how
authentication
and
access
works
I'll
do
a
quick
demo
of
that.
I
know
the
font
size
is
terrible.
Give
me
a
moment.
D
B
Okay,
so
here
I
can
connect
to
nsls2
with
a
little
shortcut
that
has
the
url
and
other
configuration
that
I
want,
and
I
can
connect
to
this
tiled
demo
deployment
and
I
can
grab
different
different
data
sets
out
of
the
two
that
that
are
based
on
searching
for,
say
something
that
has
to
do
with
nickel.
So
in
this
public
deployment,
I've
got
some
reference.
B
Some
reference
data
set
that
is
on
nickel.
That
I
know
is
that
I
know,
is
good
and
interesting,
so
I'll
go
ahead
and
grab
some
data
out
of
that,
just
like
you
would
out
of
an
h5pi
dataset
or
something
like
that,
and
I
can
grab
some
data
out
of
the
nsls2
bmm
instrument,
also
based
on
nickel,
where
I'm
looking
for
the
same
fields-
and
I
can
you
know,
look
at
those
two
data
sets
and
presumably
pop
them
together
and
so
on.
A
B
Sure
so
the
question
is
what
standards
does
tiled
leverage,
so
I
would
say
the
authentication
standards
that
that
I
mentioned,
starting
with
openid
connect
and
oauth2.
The
way
that
authentication
the
way
that
we
we
we
get.
Your
identity
is
a
standard
and
the
way
that
we
refresh
your
access,
uses,
jwts
and
refresh
tokens
and
something
called
sliding
sessions,
which
is
basically
the
user
experience
where
it's
use
it
or
lose
it
so
up
to
some
very
high
limit.
B
So
that's
what
we
do,
then,
the
standards
of
the
way
that
the
metadata
is
laid
out,
the
formats
that
we
support,
we're
trying
to
invent
as
little
as
we
can,
and
so
a
lot
of
this
is
based
on
the
http
spec,
modern
authentication
standards
and
modern
data
science
formats.
That's
the
that's!
The
one
sentence
version.
A
D
A
D
I
missed
a
few
things.
There
are
a
lot
of
specs.
I
just
wanted
to
add
for
the
record
purpose
of
the
recording
http
cache
controls
for
is
supported
and
you
didn't
invent
a
whole
new
format
for
requests
and
responses.
You
follow.
It
follows
the
json
api
standard
for
for
for
the
actual
message,
bodies
right-
and
you
know
so
just
a
point.
Another
thing
I
pointed
out
in
the
chat
but
I'll
say
again
and
out
loud.
D
The
the
python
client
hides
a
lot
hides
the
fact
that
there's
http
going
on
underneath,
but
all
of
those
features
that
are
being
served
by
as
an
hp,
hdb
server
are
intended
to
be
available
to
hdb
clients
easily.
So
one
of
the
things
we
really
wanted
to
do
was
to
be
able
to
have
sort
of
simple
access
in
a
web
page,
in
addition
to
more
complicated
data,
science
use
cases
from
from
a
python
client
all.
B
B
That's
the
simplest
thing
we
can
do,
let's
look
at
what
that
did
so
that,
first
of
all,
as
for
what's
what's
even
in
this,
and
it
got
back
a
response
and
in
that
response
you
can
see
timing.
This
is
really
useful
for
optimizing
the
thing
so
we
can
see
it
spent
like
19
milliseconds
in
the
application,
basically
less
time
that
we
could
measure
compressing
it
and
a
little
bit
of
time
tokenizing
it.
B
Similarly,
down
here
we're
asking
for
a
block
of
an
array
box,
zero,
zero,
which
I
think
is
the
only
block
and
sending
in
all
these
things
we
can
see
it
was
boss,
encoded
and
the
time
that
it
was
spent
reading
it
tokenizing
it
compressing
it.
What
kind
of
compression
ratio
we
got?
This
is
an
array
of
ones,
so
it
compresses
very
well
and
43
milliseconds
all
told
spent
inside
the
app.
B
So
all
the
stuff
in
the
python
client
is
doing
http
requests
to
do
everything
it
does,
and
we
hope
that
we
have
not
baked
any
pythonisms
into
this.
We're
working
on
a
javascript,
typescript,
client
and
we've
played
around
with
the
c
bus
plus
client
just
to
make
sure
we're
not
baking
python.
Isn't
in
straight.
Is
this
question
next?
What
when
do
you
when
you
serve
up
something?
Without
do
you
need
to
tell
what
backend
to
use
that
should
be?
B
That
should
be
something
the
server
figures
out
and
not
something
the
client
has
to
say.
So,
if
you're
doing,
if
you're
accessing
nsls2
data,
we
have
a
proper
database
backing
all
of
this
right,
and
so
we
have
a
database
that
either
has
the
data
or
is
referencing
hdfi
files
somewhere
that
have
the
large
array
payloads.
So
that
never
has
to
guess
what
format
it's
supposed
to
read
and
everything
is
just
handled
for
the
for
the
directory
walker,
which
is
sort
of
a
you
know.
B
I
wouldn't
say
it's
a
toy
we're
trying
to
make
it
very
good
and
fast,
but
it's
the
scaled
down
use
case.
Where
you've
got
you
don't
have
database
infrastructure,
you
just
have
a
directory
of
nexus
files
or
a
directory
of
tiffs,
or
something
that
you
want
to
serve
for
that
it
is
using
heuristics
to
figure
out
what
files
it
has
and
there
are
ways
to
to
tell
it
specifically.
B
C
Me
rephrase
the
question,
so
I
I
guess
I
was
so
yeah
it
was.
I
think
it
makes
sense
that
the
client
is
agnostic
to
the
back
end
type,
but
so
on
the
server
side.
Let's
say
I
have
a
data.
I
have
some
data
in
a
database.
I
need
to
tell
tile
specifically
this
is
a
database
and
you
know
I
guess
how
do
I
tell
it
how
the
data
is
laid
out.
A
B
B
That
looks
kind
of
like
this,
where
you
can
basically
give
a
sub
path
and
say
this
is
a
directory,
and
here
it
is,
and
then
this
is
just
another
directory,
but
in
in
your
case
it
would
say
the
name
of
a
python
pathway,
python,
object
that
connects
to
your
database
and
presents
the
top
the
interface
expected
by
tiled.
We
have
an
example
of
that
in
public
for
ii,
but
it's
probably
you
know
not
reusable
by
you
right.
It's
an
example.
B
You
could
work
from,
but
it's
not
it's
specific
to
the
way
our
collections
are
organized
and
the
dot.
What
the
documents
within
them
mean.
I
think,
there's
probably
there's
probably
low
value
in
having
like
a
generic
postgres
adapter,
a
generic
adapter.
You
could
make
such
a
thing,
I
think,
but
it
might
be
too
literal.
I
think
there's
usually
meaning
in
in
the
internal
layout
of
the
database
that
you
want
to.
You
want
to
program
against
a
little
bit
to
give
the
users
the
experience
that
they
want.
B
E
Yeah,
this
is
really
exciting.
It's
really
nice
to
see
the
the
rate
of
progress
as
well.
I
think
you
mentioned
like
the
first
real
commit
was
like
less
than
a
year
ago.
I
want
to
ask
a
question
about
scale
yeah,
because
you're
shifting
you're
shifting
the
transport
of
the
data
over
to
http
and
at
nurse
when
we
buy
traditionally
when
we,
when
we
buy
a
super
computer,
we
buy
platform
storage
attached
to
it.
E
So,
for
instance,
if
I
wanted
to
today
read
in
20
terabytes
of
data
into
200
gpu
nodes
or
something
like
that
on
promoter.
The
obvious
way
to
do
that
is
to
put
the
data
on
promoter
scratch
and
if
I've
done
my
job
right,
I've
got
it
in
a
good
format
that
will
do
gpu
direct.
I
o,
or
something
like
that
right.
The
question.
C
E
What
scale
of
data
set
are
you
envisioning
being
a
reasonable
fit
for
this,
and
just
to
even
like
be
more
provocative,
maybe
like?
Should
we
stop
buying
platform,
storage
and
just
abstract
away
all
the
hardware
behind
these
types
of
interfaces
for
users
can
can
that
kind
of?
Do
you
think
that
is
the
way
to
go.
B
E
Definitely
see
the
the
value
behind
the
interface
that
you've
got
it's.
It
definitely
addresses
a
major.
What
I
think
is
actually
a
major
issue,
but
it's
often
like
described
as
like
people
being
kind
of
undereducated
or
whatever
it's
just
that
people
don't
even
understand
the
concept
of
files
and
directories
really
anymore
and
we've
had
at
least
one.
E
You
know
facility
tell
us
like
we
hate
your.
We
hate
the
depth
of
your
file
hierarchy
right,
so
just
like
make
that
go
away
right.
I
think
this.
This
goes
a
long
way
to
actually
providing
the
interface
that
people
want
to
think
about
in
terms
of
accessing
their
data,
but
I'm
kind
of
I'm
a
little
bit
dubious
that
if
we
were
to
you
know
you
mentioned
it
set
up
and
spin.
Today
we
would
basically
just
be
kind
of,
like
you
know,
extending
a
long
straw
around
just
simply
drinking
from
the
cup.
You
know.
B
B
I
don't
ever
want
to
get
to
the
point
where
someone
can't
get
a
file
if
they
want
it
right.
I
don't
want
to
be
itunes
necessarily,
but
it
is.
I,
like
the
tiled
serve
directory
mode
because
it
you
can
imagine
someone
using
getting
at
their
data
the
old
way
directly
on
the
file
system
when
they
want
to,
but
also
having
the
option
to
get
it
at
the
new
way,
and
I
think
for
for
nurse
deployment
specifically,
the
path
may
be
deployed.
B
D
I
was
going
to
make
that
point
that
there
are
a
lot
of
use
cases
that
have
different
usage
patterns.
Some
of
them
will
be
very
amenable
to
to
caching,
both
on
the
server
and
the
client
side,
and
some
of
them
won't.
You
know
that's
going
to
be
to
me.
That
seems
like
the
limit
of
http's
ability
to
to
serve
quickly
right.
F
Hey
this
is
heya.
I
wanted
to
kind
of
build
on.
Well,
you
mentioned
like
in
one
of
your
comments
about
the
user
experience,
and
so
I
was
wondering
like
in
terms
of
scale,
is
a
single
server
enough
to
meet
the
user
communities.
You
know
expectations
of
the
experience
that
they're
supposed
to
get
or
you
know
what
kind
of
back-end
requirements
are
needed
or
or
do
you
tell
people
who
are
interested
in
using
tiled
that
they're
going
to
need
to
set
up
in
order
to
meet
a
certain
user
expectation
of
responsiveness.
B
Right
so
it's
a
little
early
to
be
pitching
it.
As
you
know,
a
robust
scalable
tool.
I
would
say
that
we
are
designing
it
to
someday,
be
robust,
we're
trying
to
think
carefully
ahead
about
about
what
we'll
scale.
But
I
don't
you
know-
we
haven't
used
this
for
more
than
a
couple
power
users
at
a
time
so
far,
and
until
we
do
that,
we
can't
make
firm
recommendations.
B
What
we
know
so
far
is
with
a
single
node
deployment
and
a
heavy
tomography,
dataset
user,
pulling
of
order,
100
terabytes
down
with
das
workers,
and
so
on
that
we
can.
We
can
service
that
with
a
single
node
with
about,
I
think,
it's
64
gigabytes
of
ram,
and
I
think
I
8
8
cores
working
in
pro.
I
I'm
hoping
that
of
order.
F
B
So
at
the
small
scale
everything's
in
for
us,
so
it's
not
really
in
files
anymore.
We
only
are
using
file
based
access
for
the
large
data
sets,
which
you
know
necessarily
avoid
the
many
small
files
use
case.
So
I
think
we'll
be
limited
by
the
performance
of
the
I
o,
but
I'm
not
aware
of
anything
specific
and
how
tiled
interacts
with
that.
I
don't
know
dylan.
Do
you
have
a
better
answer?
There.
D
Not
specifically,
I
think
I
think
we
have
seen
some
use
cases
where
the
I
o,
if
with
without
caching,
without
warming
up
and
cache
the
I
o
on
small
files
on
slow
file
systems,
has
been
the
the
first
slow
point
that
we
have
to
deal
with.
So
that's
that's,
certainly,
what's
something
that
has
to
be
looked
at.
D
For
example,
this
the
spin
right
now
for
a
while
that,
on
our
spin
implementation,
we're
just
reading
off
of
the
community
file
system
and-
and
we
didn't
do
enough
tooling,
to
really
you
know-
instrumentation-
to
really
know
exactly
where
it
was.
But
it
was
not
as
fast
as
we
would
want
to
for
lar
for
10
000,
very
small
files
to
access
right.
B
C
B
Yeah
so
we've
we've
been
working
on.
You
know
we
look
at
this
as
sort
of
the
next
phase
of
data
broker
and
I
won't
get
into
the
long
answer
of
how
it
relates
to
data
broker,
but
maybe
it's
a
more
general,
a
general
foundation
that
that
works
alongside
it.
Data
broker
has
been
going
for
six
years
and
we've
bet
our
facilities
data
access
plan
on
it.
B
So
I
think
we're
firmly
attached
to
something
in
this
space
and
we're
at
a
point
now,
where
you
know
we
just
put
out
a
bnl
press
release
about
tile
last
week,
so
we're
we're
talking
about
it
in
public
we're
keeping
the
name-
and
I
see
stewart's
video
is
on
which
means
he's
here.
So
thank
goodness,
go
ahead.
Stew.