►
From YouTube: OpenActive W3C Community Call / 2020-06-17
Description
Improvements to RPDE
- The harvesting model has limitations
* Difficult to gauge progress
* Requirement to harvest from the beginning of time
*Cannot query selectively
- Pagination data
- Parameterise the RPDE querystring
- Movable first page
A
A
No
worries
no
worries
for
once
I
managed
to
start
recording,
for
once
I
managed
to
start
recording
before
everybody
joined
the
call,
so
we're
starting
on
time
for
once.
So,
as
I
said,
rpg
the
problem
is
that
it's
fairly
slow
in
the
sense
that
you
have
to
get
all
the
data
before
you
start
working
with
it
at
all
and
then
a
particular
annoyance
simply
on
my
part,
but
I
suspect,
for
data
consumers
as
a
whole
is
actually
that
it's
very
hard
to
gauge
progress.
A
A
A
A
First
of
all
that
this
doubles
the
query
load
in
the
sense
that
you
have
to
make
an
additional
query
on
the
publishing
side
to
support
this.
Some
kind
of
count
query
indicating
precisely
what
the
number
of
remaining
items
would
be
and
then.
Secondly,
this
ends
up
invalidating.
Caching,
of
course,
because
if
that
number
changes,
then
the
cache
needs
to
be
refreshed,
so
the
efficiencies
of
caching
with
rpd
would
be
lost.
Under
that
scenario,.
A
C
C
A
C
C
You
know
where
you've
got
like
a
gladstone,
where
you've
got
lots
of
different
types
of
customers
and
they've
all
got
their
own
data
set
sites,
and
so
because
it's
all
dynamically
rendered
right
now.
If
your
data
set
site
query
as
well
as
querying
the
database
for
the
organization
name
and
everything
else,
it's
growing
for
also
queried
to
the
total
number
of
records
which
may
be
cached,
then
that
would
that
would
do
that
and
then,
for
example,
in
gladstone
right
now,
the
dataset
side
is
rendered
from
the
database
and
is
cached.
C
D
A
A
A
A
B
The
relief
of
charlie
and
tom
it
sounds
like
I
was
gonna
say
my
tim,
my
only
question
because
I
I
don't
like
to
not
understand
things,
although
I
probably
will
regret
asking,
is
what
what's
the
impact
and
which
is
the
nice
general
pointless
question,
but
if
I'm,
if
we're
playways,
has
no
intention
and
doesn't
take
any
direct
feeds
from
any
booking
systems,
so
we
only
we
only
intend
to
have
it
really
use
the
I'm
in
feed
on
our
own.
What
does
that
concept
it
have
on
on
impact
in
that?
B
A
A
C
A
B
A
C
Thank
you.
Thank
you.
Well,
the
other
benefit
is
it
doesn't
break
anything
that's
existing
because
we
already
need
to
go
around
and
update
everyone's
data
set
sites
when
the
new
spec
comes
out
anyway.
So
this
isn't
going
to
add
any
additional
lobbying
effort
or
otherwise
to
changing
our
pde.
If
that
was
the
thing
that
we
need,
it
would
yeah
it
doesn't,
it
doesn't
add
any
more
to
than
already
there.
B
A
Okay-
and
I
suppose
also
in
terms
of
workflow-
it's
easy
that
the
dataset
site
specification
still
has
to
be
written.
So
it's
easy
enough
to
add
that
line
item
in
there.
Okay,
so
I'll
migrate
that
issue
over
to
the
dataset
site
specification
repo,
then
I
don't
know
if
I
described
the
next
proposal
in
the
best
possible
way.
I
don't
think
this
one
needs
very
much
discussion
either.
A
So
then,
the
only
point
of
debate
was
really
about
which
approach
to
use
the
difficulty
is
a
little
bit
technical
in
that
to
create
that
capacity
to
start
harvesting
from
the
moment
that
the
query
is
launched.
Actually
because
of
the
way
the
specification
is
written,
that
would
drop
the
first
item
in
the
feed.
A
But,
generally
speaking,
it
seems
like
looking
at
the
comments.
Everyone
who
could
be
bothered
to
comment
seems
to
be
keen
about
the
first
approach
there.
So,
even
though
it's
a
fairly
well
moderately
significant
change
to
the
query
itself
approach,
one
seems
like
it's
getting
all
of
the
votes
right
now.
C
But
I
suppose
it's
just
because
that's
the
that
query
is
the
bit.
That's
that's
most
often
done
wrong
and
because
we
haven't
got
because
of
the
way
that
we
test
our
pde,
which
is
just
checking
the
invariants.
It's
going
to
be
quite
difficult
to
also
test
that
this
is
done
right
without
kind
of
the
test
harness,
for
example,
of
the
booking
spec
being
enhanced
to
add.
Well,
I
suppose
it
already.
It
already
does
do
this,
so
so
using
the
testing
of
the
booking
spec.
C
You
could
do
something
like
add
an
opportunity
and
check
that
it
comes
through
and
and
then
check
that
so
the
check
that
this
all
works,
but
because
of
the
the
kind
of
weird
edge
case
about
that
first
item
to
actually
check
this
properly.
You'd
have
to
really
you'd
have
to
know
the
first
item
in
the
database
right
to
be
able
to
or
or
artificially
insert
the
oldest
item
in
a
test
suite
and
then
check
it
came
through.
C
A
A
A
C
Sorry,
yeah
no,
no
carry
on.
I
just
remembered
the
reason
that
we
didn't
do.
The
other
stuff
is
because
of
the
string
constraint.
That's
right.
So
it
has
to
work
for
every
every
use
case,
because
there
is
a
simpler
option
available
if
you've
got
an
id,
which
is
not
a
string
which
is
one
of
which
is
what
some
of
the
other
approaches
we're
talking
to.
But
it's,
I
think
it's
fair
to
say
a
lot
of
people
use
strings
as
ids,
because
they've
got
guides
involved.
So.
A
B
A
Okay
and
then
the
last
one
I
think,
we've
actually
already
covered
in
that.
I
think
one
of
one
of
the
frequently
voiced
difficulties
with
rpd
is
that
you
can't
query
it
basically
that
you
can't
say
I
just
want
everything
in
this
particular
geographical
area
or
something
like
that.
You
have
to
download
everything
and
then
slice
it
up
yourself.
C
Yeah,
the
yes,
that
this
is
exactly
it,
the
second
that
you
start
adding
any
types
of
parameters
outside
of
what
you've
already
got
in
there.
In
fact,
even
the
limit
parameter
in
the
in
the
rpd
specs
it
stands
doesn't
really
work
for
caching,
or
at
least
yeah
it.
It
creates
problems.
So
you
could,
I
mean
a
good
implementation.
A
C
Does
it
actually
protect
my
servers?
The
answer
is
yes,
it
absolutely
does
as
it
stands
because
of
the
constraints
we've
put
on
the
on
the
on
the
endpoints.
If
you
wanted
to
do
anything
else,
you
would,
I
think,
it's
fair
to
say
from
all
the
feedback
that
we've
we've
had.
We
would
need
to
start
adding
api
keys
and
the
challenge
as
soon
as
you
start
doing.
C
Was
you
know
that
you
have
a
form
where
you
fill
it
out
and
then
they
approve
it
and
you
get
an
api
key,
but
they
might
not
approve
it.
If
you
know
I
mean
I'm
not
saying
it's
a
sports
week,
but
the
implication
is
they
might
not
approve
it.
If
you're
a
competitive
organization
or
if
the
you
know
it
doesn't
quite
so
you
might
have
an
open
license
and
so
and
and
the
odi
is
you
know
the
view
on
this
has
always
been
that
you
know.
C
C
We
probably
want
to
make
sure
that
open
means
open
and
we're
not
relying
on
aggregators
and
others
to
you
know,
redistribute
the
data
with
an
open
license
and
have
kind
of
those
additional
gatekeepers
on
there.
Just
because
we
want
to
lower
the
barrier
to
entry
in
the
market
really,
and
so
the
the
solution
in
rpd
at
the
moment
is
there's
inline
filtering,
which
you
can
use,
which
means
that
you
can.
C
The
rpd
is
designed
so
that
if
you
want
to
slice
the
data
set,
you
can
do
that
arbitrarily
based
on
the
data
in
each
day
in
each
payload
of
of
each
item,
and
you
can
do
that
in
line,
so
you
can
only
load
into
your
database
exactly
what
you
need.
You
need
to
store
everything,
but
unfortunately
you
still
need
to.
You
still
do
need
to
page
through
everything.
C
But
of
course,
if
those
pages
are
cached,
then
the
paging
should
actually
be
fairly
quick
and
you're
benefiting
from
that
edge
caching
to
go
through
and
only
take
what
you
need
from
the
data
set
and
store
it,
so
that
that's
kind
of
I
guess
it's
a
philosophy
point
really
as
much
as
anything.
It's
the
reason,
this
constraints
are
there.
C
But
it
also
increases
the
amount
of
work
required
for
everyone
right,
because
everyone
needs
to
manage,
have
an
api
management
solution
in
place
where
they
can
manage
api
key
distribution
manage
signups
and
a
lot
of
these
smaller
organizations.
Like
our
parks,
you
know
this
is
a
they
do
the
tech
work
and
then,
as
we've
seen,
they've
only
touched
it.
You
know
they
touched
it
four
years
ago
and
now
they've.
C
D
C
You
get
an
average
uk
connection
in
a
second,
so
I
think
if
so
so,
at
which
point
you
know,
if
everything
is,
is
edge,
cached
and
everything
is
included.
Then
something
can
should,
in
theory
be
able
to.
You
know
if
it's
downloading,
just
as
frequently
as
it
can
do
be
able
to
get
the
entire
data
set
very
quickly,
and
then
they
can
decide
at
that
point
whether
to
keep
that
bit
of
data
or
just
drop
it
on
the
floor
based
on
their
own
filtering
if
they
want.
C
But
so
if,
if
we
were
to
add
query
parameters
that
do
geography
geography-based
filtering,
then
that
reduces
the
amount
of
data.
But
as
far
as
I
can
tell,
there's
not
there's
not
a
great
amount
of
data,
even
when
there
are
a
lot
of
locations,
but
you
don't
get
any
of
the
benefit
from
the
edge
caching.
So
the
data
gets
there
much
more
slowly.
D
C
C
So
if
you
wanted
to
get
a
static
version
of
you
know
what
what
what
the
sector
looked
like
in
a
certain
point
in
time,
you
could
probably
get
all
the
systems
to
pull
that
in
a
csv
and
then
shove
it
somewhere
and
then,
but
that's
yeah,
that
doesn't
that
doesn't
achieve
the
objective
of
creating
that
real-time
view
of
what
sessions
are
available
tomorrow,
and
so
that's.
I
guess
why
this
this
is
geared
towards
making
the
data
like.
C
I
guess
that's
why
our
pde
is
about
more
the
real
time
element
of
it
unless
the
kind
of
downloading
from
the
beginning,
because
the
value
you
don't
get
the
value
from
this
by
one
download
right,
the
download
in
itself
is
great
for
the
first
time
someone's
got
it.
So
I
totally
see
some
of
the
you
know
challenges
around
the
first
time.
C
Someone
harvests
the
feed
you've
got
lots
of
stuff
to
download
if
you
need
to
resync
that
feed,
but
the
ongoing
value
you're
getting
from
that
as
a
business
as
a
consumer
or
whatever
is
not
downloading
the
whole
thing
again.
It's
just
the
fact
that
you
get
to
with
a
minimal
amount
of
work
in
the
middle
of
of
bandwidth.
You
can
get
those
changes
and
there's
actually
not
very
much
going
on
there
in
terms
of
when
you're
just
calling
for
changes.
A
A
Is
the
problem
with
the
filtering
solution
is,
supposing
you
change
your
database
schema?
You
then
have
to
re-harvest,
meaning
you
do
it
all
from
the
top
again
meaning
you,
then
you
know
sync
the
time
again.
So
it's
it's
not
very
agile.
I
suppose
it's
good.
If
you've
already
got
an
established
business
methodology
and
logic
that
says,
okay,
we
can
just
keep
on
pulling
in
deltas.
Basically.
C
So
if
you
change
the
definition
of
one
you've
got
to
go
through
and
make
sure
you've
got
some
homogeneous.
You've
got
you've,
got
to
figure
out
a
mechanism
so
that
you
get
the
same
kind
of
data
back
from
everyone.
So
I
think
the
well
you
could
it
could.
It
could
be
viewed
that
the
current
situations
as
it
stands
with
the
harvesting
is
actually
the
most
agile,
because,
as
a
data
consumer,
you
can
really
radically
change
the
kind
of
filtering
you're
doing
you.
Can
you
can
merge
munch
the
data.
C
C
So
if
this
is,
if
what
this
is
doing
is
giving
you
basically
a
fire
hose
of
all
the
data
and
you
can
choose
how
you
cut
and
do
what
you
want
with
it,
depending
on
your
use
case,
then
that's
optimizing
for
the
thing
that
you
know
it
makes
it.
It
might
have
a
bit
of
slowness
in
terms
of
downloading
it,
but
you're
literally
saving
years,
compared
to
trying
to
get
that
data
from
whatever
apis
are
in
the
in
the
you
know,
or
lack
of
apis
or
in
the
systems
as
it
stands.
A
C
Yeah,
absolutely
absolutely
no,
and
also
you
you're,
more
agile,
because
you
because
you've
removed
the
complexity,
you've
got
access
to
all
the
data
and
the
the
approach
you
you
can
use,
which
you
can
apply
to
every
data
set
is
you
know,
filtering
based
on
the
standard,
whatever
parameters
you're
interested
in
and
you
can
decide
to
apply
new
and
interesting
filtering
to
that.
You
know
you
don't
have
to
wait
for
everyone
to
implement
the
more
than
date
field
for
the
end
date.
C
Right,
like
you'd,
have
to
go
through
and
wait
for
every
data
provider
to
do
that
and
the
time
that
would
take
and
the
cost
for
all
of
those
implementations
you.
You
just
add
that
single
line
of
code
to
your
harvesting
tool
press
go,
go,
have
a
cup
of
coffee
come
back
and
then
see.
If
it's
worked
yeah
I
mean.
D
A
C
Absolutely-
and
I
guess
that
was
the
previous-
the
previous
issue
right
around-
making
sure
that
we
can,
we
can
limit
the
amount
of
data
in
the
feed
to
that
useful
set.
Yeah
significantly
reduces
that
yeah
you're
right,
because,
if
you're,
if
you're
pulling
from
the
beginning
of
time
from
gll
you're,
looking
at
literally
millions
of
records,
yeah
yeah.
A
But
yeah,
I
think
that
it
shouldn't.
We
shouldn't,
underestimate
the
the
burden
that
gets
put
on
data
consumers
as
a
result
of
removing
the
burden
from
data
publishers.
That's
all
but
yeah
yeah.
I
feel
like
the
the
caching
is
invaluable
for
a
publisher,
but
it
creates
headaches
for
for
a
consumer
but
within
within
the
domain
of
the
possible
yeah.
This
does
do
a
nice
job
of
making
sure
that
there's
actually
some
data
there's
some
data
published
that
consumers
can
use
okay.
A
So
I
think
this
might
just
be
a
question
that
we
revisit
based
on
the
ability
to
start
harvesting
from
now,
rather
than
from
the
beginning
of
time,
because,
as
you
say,
gll
has
got
millions
of
records,
but
of
course
most
of
those
are
historical.
So
we
become
much
more
agile
if
we
can,
if
we
can
throw
away
obsolete
data.
A
A
C
Yeah,
well,
I
guess
I
guess
the
the:
what
do
we,
what
was
the
name
of
it,
the
retention
period?
So
that's
that's
on
the
open,
active
docs
already,
and
I
know
that
we've
been
pushing
that
with
bigger
feeds
to
try
to
get
those
feed
sizes
down
anyway,
just
as
part
of
just
general
discussion
as
they're
kind
of
moving
to
the
next
version
of
whatever
they're
doing
so.
C
C
It's
probably
not
easy
to
add
to
the
rpd
tests
in
the
validator
just
to
harvest
tests
see.
We
probably
would
need
to
do
this
in
more
of
the
way
that
the
booking
is
sweet.
Yeah
insert
a
record
check
the
records
come
through.
You
know
that
that
type
of
level,
which
which
is
definitely
yeah,
definitely
a
feasible,
as
I
mentioned,
there's
already
similar
tests
in
there.
C
So
maybe
it's
just
adding
a
couple
tests
in
there
I
mean
you
could
even
add
it
as
a
feature
in
the
way
that
that
framework's
currently
built
the
future
being
stateless
retention
period.
And
if
someone
chooses
to
support
that
feature,
then
it
goes
green
and
then
we
just
promote
that
feature
along
with
the
other
features.
Actually,
I
was
thinking
this
the
other
day.
You
know
that
that
open
active
test
suite
is
actually
pretty
good.
C
If
you
don't
implement
booking
because
it
does
do
data
set
site
validation,
it
does
full
harvesting
of
the
data
feed.
I
know
that
josh
in
playways
found
some
bugs
in
the
open
data
feed
just
by
using
the
test
screen.
So
it
might
be
that-
and
this
is
just
an
idea
I
had
yesterday-
but
it
might
be
worth
looking
at
adding
validation
of
the
open
data
pages
into
the
test,
suite
just
doing
that
as
it
goes
or
having
an
option
set.
C
So
you
can
turn
that
on
because
that
what
that
then
does
is
means
that
you've
got
this
kind
of
full
feed
download
test
test
harness
you
can
use,
and
if
you
combine
that
with
this
feature,
then
you
could
almost
imagine
like
a
profile
of
you
know.
You
can
configure
that
test
suite
with
a
profile
of
features
which
is
just
open
data,
actually
nothing
to
do
with
booking.
So
it
just
covers
the
open
data.
A
A
Getting
a
specification
up
for
the
data
set
site
published
and
testing
against
that
as
part
of
getting
these
two
improvements
to
rpde,
actually
practical
and
verifiable,
and
then
the
third
point
about
panoramatization
becomes
much
less
much
less
critical,
yeah,
okay,
yeah!
I
know
that's
a
that's
a
that's
kind
of
a
nice
way
forward.
I
think,
looking
at
the
test,
suite
is
more
than
just
booking
and
extending
it
to
cover
open
data
is
a
is
a
really
nice
way
of
tying
that
package
up.
A
C
I
suppose
there's
only
one
just
about
the
data
set
site
specification
is
obviously
that's
that's
something
that
is
being
presently.
I
mean
the
whatever
iteration
of
that
that's
there
in
in
the
github
issues
is
being
used
in
the
test
suite
because
there
has
to
be
something
in
there,
and
so
I
guess
it's
just
a
comment
that
there's,
obviously
a
present
implementation
is
happening
against
that,
by
necessity
with
the
current
implementers
and
the
playways
is,
is
already
underway
and
others.
C
So
I
guess
that
there
might
be
in
parallel
to
completing
that
test
suite
and
which
I
know
we're
all
is
an
urgent
thing
to
to
be
done.
It
might
also
be
that
there's
a
an
urgency
around
finishing
that
data
set
site
spec
so
that
what
things
are
done,
what
you
know,
the
definition
of
done
means
really
done
not
like
diana
will
have
to
revisit
it
when
that
other
spec
comes
out
in
like
a
few
months
time,
that's
just
a
thought.
A
C
Yeah,
like
I,
I
even
noticed
the
other
day
I
just
I
thought
it
was
a
bug
actually,
but
it
turned
out
to
be
a
well
a
feature
or
a
result
of
that.
The
the
case
of
content,
url
or
something
I
can't
remember
what
the
there's
a
content,
url
or
access
url,
whatever
the
url
is
that's
being
used
for
the
booking
spec
stuff,
which
has
come
from
dcat,
actually
has
a
different
case
of
url
than
the
scheme
of
the
org
stuff.
C
Oh
no,
all
right,
because
d-cap
uses
access,
url
or
whatever
it
is.
Actually,
you
know
I
check
actually
check
d-cat's
original
spec
and
yeah.
They
have
url
capitalized,
whereas
nothing
in
schema.org
has
url
capitalized,
and
so
even
I
mean
it
sounds
super
basic.
Doesn't
it
but
obviously
that's
a
conformance
thing.
We
need
to
make
and
that's
something
that
now
exists
by
default,
as
you
say,
because
this
is
kind
of
evolved
and
not
really
being
cross-checked
and
so
just
just
kind
of
yeah,
even
even
that
stuff
just
needs
to
be
well.
C
A
C
A
I
mean
there's
not
there's
not
actually
that
much
to
say
essentially
the
web
api.
Well,
the
last
time
I
looked,
which
was
a
couple
of
months
ago
now,
the
the
web
api
conversation
within
schema.org
had
spiraled
off
into
big
questions
of
scoping,
which
were
kind
of
outside
of
our
remit.
Basically,
so
there's
not
a
lot
to
update
specifically
on
us
because
it
wasn't
it
wasn't
like
it
was
not
a
dialogue
of
we've
got
this
proposal
to
make
and
then
schema.org
thought
that
there
were
some
problems
with
it.
C
A
C
A
A
C
C
A
C
C
So
I
don't
know
if
you
you
saw
that
there's
a
a
separate
w3c
community
group
like
open,
active
like
this
one,
that's
for
web
api,
and
so,
though,
and
a
couple
guys
on
there
seem
to
have
on
the
github
repo
put
together
an
initial
spec
and
that's
part
of
the
thread,
and
then
it
looks
like
that's
there
and
then
there's
a
bunch
of
other
thoughts,
but
no
one's
really
kind
of
bringing
it
together.
It's
just
kind
of
it's.
It's
like
opening
the
diamond
up,
rather
than
closing
it
down.
C
C
A
Mean
I
think
I
think
the
problem
is
even
just
the
name
I
think
indicates
this-
that
the
scope
actually
has
to
be
pretty
wide.
The
number
of
questions
that
have
to
be
answered
is
quite
high.
I
don't
think
open
active,
particularly
we
can
drive
the
conversation
there
if
we
want
to,
but
I
think
it's
going
to
be
a
long
conversation.
C
A
A
C
A
C
A
C
Oh
yeah,
okay,
that
makes
sense,
so
I
guess,
if
we
set
it
up
in
such
a
way
that
future
conformance
to
schema
would
be
a
it's
all.
So
I
guess
what
what
this
means
is,
because,
obviously
part
of
the
point
of
the
data
set
sites
is
that
google
can
index
them
and
others
can
index
them
that
there
will
be
a
necessary
step
at
some
point
to
con
to
align
with
what
they're
doing-
and
I
guess
sounds
like
what
we're
saying
is
we
set
this
up
such
that
everyone
who's
harvesting?
C
So
as
long
as
we
like,
you
know,
the
version
that
we
put
out
there
now
and
the
version
that
schema.org
eventually
decides
is
the
thing,
and
so
I
guess,
if
we
set
it
up,
such
that
it's
not
like
yeah,
like
we've
done
what
we
can
to
to
to
use
schema
terms
so
that
it
minimizes
the
difference.
I
guess
and
then
just
like
some
kind
of
obvious
switch
in
the
type,
maybe
like
open,
active
web
api.
That
means
that
someone
who's
consuming
this
can
write
like
a
simple
switch
statement
and
then
do
everything
yeah.
A
Yeah-
and
I
mean
I
think,
if
we
get
a
solid
kind
of
if
we,
if
we
create
a
solid
enough
standard
and
people
are
using
it,
I
think
we're
in
a
stronger
position
talking
to
schema.org
to
say
actually,
here's
a
syntax
that
that
should
be
supported
or
should
hold
weight
with
you,
but
we
are
simply
in
a
position
where
we
have
to
decide.
First,
I
think
okay.
A
From
me,
oh
thank
you
all
apologies.
If
some
of
this
conversation
seemed
a
little
technically
involved,
but
I
think
there
were
at
least
some
some
clear
actions
going
forward
and
I
think
quite
actionable
and
reasonably
urgent,
so
we'll
be
pushing
those
forward
in
the
very
near
future
and
thank
you
all
for
joining
I'll.
Give
you
back
10
minutes
of
your
day.