►
From YouTube: IPFS Weekly Call 2019-07-29 🙌🏽📞
Description
Presentation: Supporting Web Archiving via Web Packaging
Speaker: Sawood Alam @ibnesayeed
Notes
* From Old Dominion University
* Web Packaging … not currently suitable for archiving
* Web Packaging will “bundle” a bunch of web requests and deliver them to a web browser in a single signed exchange
* Signing is different from encryption
* Signing is more suitable for non-private communication
* WARC is a format for storing request/responses for archiving
* Largest crawling tool is Heritrix from Internet Archive - misses a lot of resources (doesn’t recognize JavaScript) - other tools archive more, but are slower
* Server side rewriting … archives need to be modified to be played back on different origins
* Proxy based rewriting is another approach where it appears transparent to the user (eg. oldweb.today)
* Archives are often displayed in ways in which they never existed
* It would be great is browsers natively supported “Memento Timegate”
* Proposals: TimeGate/Named Cache for Loading
* Fixity - no way to prove non-repudiation (eg. Internet Archive was accused of modifying an archive)
* Signed Exchanges only last for 7 days
* Temporal validations: verify that a signature was valid in the past
* Would also like native Memento support in web browsers
A
A
So
this
we
saw
a
big
opportunity
in
web
packaging
for
web
archiving
and
we
thought
we'll
write
a
position
paper
and
see.
However,
in
its
current
form,
it's
not
suitable
for
web
archiving,
so
we
suggested
some
changes
and
we'll
go
through
those
things
in
this
slide.
So
to
briefly
summarize,
what
web
packaging
is.
A
A
If
the
resource
is
not
found
in
the
bundle,
then
it
will
go
on
the
live
side
to
fish
that
now
the
cool
thing
about
bundling
is:
if
the
bundles
say
it,
this
resource
came
from
original
a
then
the
browser
will
believe
it
came
from
origin
a
and
then
that
doesn't
need
to
be
any
rewrite
or
anything.
The
way
it
is
done
in
amp
URLs,
where
you
know
and
side
from
cnn.com
appears
to
be
coming
from
google.com,
which
won't
be
the
case
in
this
case.
Now
how
trustworthy
will
that
be?
A
If
the
bundle
says
it
is
coming
from
origin?
A
do
we
believe
it
it's
same
as
if
you
were
using
a
proxy
and
you
were
using
HTTP,
your
proxy
can
respond
resources
on
behalf
of
any
origin,
anything
it
can
change
it
to
anything
and
you're
gonna
believe
it
now,
then
they
introduced
HTTPS
to
ensure
you
know
no
one
in
the
middle
have
seen
what's
coming
on
the
wire
or,
and
you
trust
that
it
is
coming
from
the
origin
you're
talking
to
so
to
bring
that
trust.
A
There
is
signing
that
was
included
now
signing
is
different
from
encryption,
because
encryption
is
more
like
one-to-one
where
the
client
and
server
meet
in
time,
and
they
ago
she
ate
on
its
shared
key,
which
is
then
used
for
encrypting
resources
in
signing.
We
trust
that
the
resource,
indeed
is
coming
from
the
same
origin,
but
it
is
in
plain
text
anyone
can
see
in
misrata,
so
it's
more
useful
for
publicly
accessible
resources,
not
something
where
you
exchange
your
credit
card
information,
for
example,
so
so
going
back
to
web
archiving.
A
Here
is
a
simple
web
page
that
I
created
last
year,
or
this
implies
of
summit,
and
this
simple
HTML
page
includes
two
external
resources:
a
stylesheet
and
a
logo
in
link
and
image
tag
respectively,
and
there
are
some
hyperlinks
as
well
on
the
stage
now
when
we
try
to
archive
this
capturing,
this
HTML
source
is
insufficient.
So
what
we
do
is
we
try
to
archive
HTTP
headers,
as
well
as
the
payload
now
combine.
Http
header
and
payload
give
more
information
at
replay
time
and
also
the
corresponding
HTTP
request
record
that
was
made
to
get
this.
A
This
response
from
the
server
that
is
also
preserved
and
all
these
things
are
preserved
in
a
file
called
war
file
and
what
files
have
different
record
types.
So
the
one
that
we
are
illustrating
here
on
the
right
hand,
side
is
a
response
which
I
hate
interrupted.
I
think
we
might
not
be
seeing
the
right.
A
A
Basically,
so,
on
the
left
hand
side
we
have
HTTP
headers
in
payload
that
was
responsible
for
that
HTML
page,
we
just
Illustrated,
and
this
whole
thing
is
put
together
in
a
work
record
and
since
it's
an
HTTP
response,
we
call
it
HTTP
a
response,
type
of
work
record
and
in
what
what
it
is
like
the
same
way
we
have
payload.
On
top
of
that,
we
have
HTTP
headers,
we
have
some
extra
headers,
we
call
it
work
headers,
and
this
whole
thing
combined
is
known
as
one
work
report.
A
Basically,
and
there
are
many
work
record
types,
there
is
response
type,
there
is
request
ID,
there
is
for
info
type
and
there
are
some
resource
side.
There
are
seven
or
eight
different
types
defined
in
what
specification.
So
a
work
file
is
basically
a
concatenation
of
arbitrary
number
of
these
records,
and
then
it
is
indexed
outside
using
the
original
URI
and
the
date/time
when
it
was
preserved,
and
this
is
how
web
archives
basically
serve.
So
they
have
these
work
tiles
collected
in.
A
It's
coming
back
to
the
crawling
process
of
the
bar
cotton.
Currently,
the
most
popular
and
de-facto
tool
that
is
being
used
for
any
large-scale
archiving
is
called
heretics
that
was
built
by
Internet
Archive
then.
So
there
was
a
research
done
in
2015
where
phantom
ga.js
used
to
be
the
the
well-known
headless
browser.
So
what
we
realized
is
like
if
we
use
W
gate
or
hare
traits,
we
miss
a
lot
of
resources,
things
that
are
of
the
default
representation,
which
they
only
appear
when
the
page
is
properly
rendered
and
some
JavaScript
is
executed.
A
So
if
you
were
to
use
something
like
a
headless
browser,
you
would
be
able
to
discover
more
resources
and
we'll
have
a
better,
a
more
complete
archival,
but
the
problem
was
using
phantom
JS.
We
were
able
to
discover
75%
more
resources,
but
it
was
an
order
of
magnitude
slower
than
heretics.
So
so,
what
we
believe
is
if
the
web
adapts
to
the
packaging
and
they
start
using
it
and
they
package
their
their
web
pages.
A
So
so,
however,
archives
replay
they
basically
they
take
the
originally
archive
resource
from
work
file
and
rewrite
all
the
links
in
hyper
references
and
everything
so
that
those
links
do
not
point
to
the
live
web
or
to
wrong
URL.
They
always
point
to
an
archived
version
of
those
resources.
So,
for
example,
the
stylish
dot
CSS
is
now
changed
to
slash
web
slash,
a
14
digit
number
followed
by
the
original
domain,
and
so
on.
The
whole
thing.
This
is
all
so
that
the
browser
next
time
when
it
is
looking
for
its
got
CSS
file.
A
It's
not
looking
for
a
random
style,
but
CSS
file,
words
getting
from
the
web
archive
itself,
and
this
is
called
URL
rewriting.
This
is
kind
of
necessary.
The
problem
is
when
the
resources
are
injected
using
javascript
for
a
server
to
identify
properly,
which
URLs
to
rewrite
is
become,
becomes
difficult,
and
for
that
there
are
two
tools
like
a
client-side
solutions.
A
Like
you
inject
a
JavaScript
here,
which
kind
of
traps
all
the
traffic
and
tries
to
rewrite
or
read
out
those
those
things
are
on
the
client-side
itself,
and
one
such
solution
was
something
that
we
published
last
year
or
the
year
before,
it's
called
reconstructive
and
it
is
used
in
IP
way
back
our
way
back
system
that
uses
IP,
IP
FS
and
illustrated
in
the
corner
below
it.
It
injects
a
banner
and
it
tries
to
trap
so
in
the
figure
above.
A
What
is
shown
is
like
there
was
an
image
that
was
supposed
to
come
from
external
example:
dot
net,
but
the
serviceworker
that
we
installed
kind
of
trapped.
That
request
and
said:
hey,
don't
go
to
external
that
will
be
live
leakage.
We
don't
want
life's
resources
in
in
an
archive
page.
So
then
it
will
strategically
reroute
it
to
archive
and
then
ask
the
browser
to
request
from
their
target
style.
So
in
this
case
rewriting
was
not
needed.
It
was
trapped
when
the
request
was
being
in.
A
Then
there
is
Ilias
proxy
based
solution
that
is
being
used
in
old
web
da
today,
where
a
web
browser
is
run
in
a
container
on
a
remote
server
which
is
configured
to
use
an
archival
proxy.
So
if
you
see
in
this
screenshot,
the
page
is
coming
from
the
original
domain
inside,
but
that
is
actually
talking
to
a
proxy
and
the
proxy
is
smart
enough
to
realize.
A
Are
you
are
looking
for
an
archival
copy,
so
I
will
read
out
it
to
corresponding
archived
copy,
so
internally
the
the
proxy
is
accessing
from
an
archive,
but
not
telling
the
browser
that
is
coming
from
an
archive
now.
This
is
the
kind
of
behavior
we'll
be
able
to
exhibit
if
we
were
to
use
their
packaging,
because
then
we
can
store
bundles
for
create
bundles
ourselves
in
web
archives
and
deliver
that
in
the
browser
will
believe
like.
This
is
how
the
web
is,
or
it
was
so.
A
A
So
such
a
page
does
not
exist
on
the
live
side,
but
this
is
how
our
kyv
serve
you,
because
there
were
some
cookies
in
play
and
they
were
mixing
resources
in
different
languages.
And
so
basically,
you
are
seeing
a
page
that
never
existed.
The
second
thing,
origin
violation-
and
this
is
the
situation
in
cnn.com.
A
If
you
go
to
a
recent
copy
luxe
last
two,
two
and
a
half
years,
or
something
like
that
in
Internet
Archive
and
try
to
access
those
cnn.com
landing
pages,
they'll
be
blank
and
the
reason
is
there
is
a
piece
of
JavaScript
that
says:
if
origin
equals
cnn.com,
then
do
these
things
and
render
the
page
and
since
archive
page,
is
served
from
Internet
Archive.
So
the
origin
changed
now
that
piece
of
JavaScript,
if
Clause,
will
come
into
where
it
will
not.
A
Let
anything
render
the
third
temporal
violation
in
that
case,
basically,
because
different
resources
were
captured
at
different
time.
The
the
weather
page
tells
a
different
story
in
the
text.
Form
tells
a
different
temperature
and
condition
of
the
weather,
but
the
satellite
imagery
does
not
agree
with
that,
because
it
was
like
weeks
or
days.
Apart
from
so
the
state
of
the
resource
was
changed
by
the
time.
A
Word
catcher
and
was
not
consistent,
the
fourth
his
life
leakage,
and
we
call
it
zombies
and
here,
depending
on
where
it
was
accessed
and
off
so
the
second
term,
when
the
Barack
Obama
was,
you
know
going
for
the
second
term
election.
What
happened
is
there
was
an
ad
which
was
live
lead
at
that
time
from
the
live
web,
and
is
it
appeared
to
be
prophesizing?
A
The
outcome
of
the
first
election
result
because
the
ad
was
recent
and
the
page
wasn't
passed
basically
so
as
if
somehow
this
page
knew
Barack
Obama
is
going
to
be
elected
as
in
the
first
term.
So
these
kind
of
situations
happen
in
web
archive
all
the
time
where
you
see
pages
that
never
existed,
and
we
want
to
avoid
these
things
and
we
think
the
packaging
has
potential
to
to
deal
with
these
problems.
Now.
The
problem
with
web
packaging
right
now
is
they
favor
recency.
A
Headers
so,
for
example,
server
tells
browser
that
head
unpacked,
this
bundle
in
a
cache
namespace
by
the
name,
I
provide-
and
in
this
case
it
is
a
string
to
2000
1907,
but
it
could
be
any
small
string
or
like
an
a4
back
string
that
that
browser
reads
that
and
says:
okay,
we'll
create
a
cache
with
that
name
and
unpack
all
the
resources.
And
then
the
server
also
tells
exchange
loading
policy
and
says
bundle
only,
which
means,
if
you
do
not
find
resources
in
this
bundle,
do
not
in
this
cache.
Do
not
look
anywhere
else.
A
Do
not
look
in
any
other
cache
do
not
look
live
just
returner,
you
know
network
error
or
ask
bad
form
or
bundled
resources,
or
something
like
that
now.
This
is
not
something
that
is
supported
right
now,
but
this
was
one
puzzle
that
we
kind
of
put
forward
that
will
solve
our
archive
it,
a
curricula
problem
now
so
moving
on
to
the
next
problem.
A
Right
now,
archives
do
not
have
a
way
technical
means
to
say
this
content
indeed
came
from
said
origin
and
no
one
modified
its
original,
and
here
is
a
story
that
we've
logged
about
recently
so
giant
read.
She
recently
claimed
that
copies
of
her
blog
post
in
the
Internet
Archive
were
modified
or
hacked,
or
something
like
that,
and
because
it
was
like
a
decade
old
captures
of
her
plot
post,
where
she
posted
some
homophobic
material,
and
so
she
was
denying
the
ownership
of
that
now.
A
Internet
Archive
denied
it
publicly
in
the
wrote,
a
blog
post
about
it
again.
They
did
not
have
a
technical
mean
to
prove
it,
but
they
basically
said
it
and
that's
how
they
do
it
in
when
they
are
summoned
or
something
there
is
a
someone
from
the
archive
goes
and
as
a
personnel
and
tells
like
yes
in
the
court
of
law.
Basically
they
say
no.
This
is
the
situation
now
we
wanted
him
technical
means
to
prove
it
so
that
we
don't
involve
personal
in
that
and
and
I
think
signed.
A
Exchanges
will
be
great
to
have
in
those
bundled
resources
and
that
way
we'll
be
able
to
prove
now
the
current
problem
of
scientists
changes
is.
It
does
not
last
for
too
long
like
hardly
for
seven
days
or
something
like
that.
So
we
have
some
solutions
to
to
make
it
last
longer,
and
that
is
basically
some
of
them
involved,
like
you
know,
utilizing
so
I
think
I
was
ahead.
A
Ok,
so
then,
so
here
we
we
think
we
can
fix
that
problem
using
temporal
valuations
of
sign
exchanges
and
temporal
validations
means
rather
than
saying
this
is
a
signature
is
valid
or
invalid.
You
say
in
a
historical
context,
you
say
this
signature
would
have
been
valid
if
it
was
checked
back
in
time,
as
reported
by
the
date
header
and
women
to
date,
time
headers.
A
Basically,
so
we
call
a
temporal
validation
and
basically
the
new
state
will
be
called
temporally
valid
now
if
it
is
not
temporary,
when
it
that's
fine
too,
and
in
order
to
you
know,
elongate
the
the
signatures,
we
have
some
position
for
that,
but
right
now
we
can
use
certificate,
transparency
logs
and
some
other
systems
that
that
now
exist.
They
were
not
there
in
the
past,
so
we
have
infrastructure
to
somehow
push
this
forward.
A
Now,
with
all
that,
we
would
also
like
browsers
to
have
native
support
for
momento,
or
at
least
they
acknowledge
that
resource.
That
is
being
seen
whether
it
is
coming
from
an
archive
that
is
coming
from
a
bundle
or
locally
captured
mhtml
file
or
something
in
all
those
cases.
You
would
like
browsers
to
acknowledge
that
this
is
an
archive
resource,
and
here
we
have
illustrated
basically
when
I
think
where
we
use
the
chip
of
the
address
bar
to
expose
more
information
about
the
archive
web
page.
A
So
to
wrap
up
the
packaging
has
a
unique
opportunity
to
devise
the
technology
that
supports
the
archiving
and
provide
some
much-needed
capability
to
verify
the
integrity
of
about
that
archive
resources,
and
we
are
looking
for
four
aspects
like
with
the
effect
Bundle.
Next
HTTP
exchanges
will
get
efficient
and
coherent
crawling.
This
is
not
something
that
we
are
fighting
anymore,
because
this
is
more.
If
bundle
resources
become
a
thing
this
will.
This
problem
is
solved
already.
Basically,
it's
just
a
matter
of
tools
catching
up
and
bundling
things
properly
now
timegate
support
or
named
caches
for
loading.
A
That
will
be
something
that
we
asked
for
and
when
that
is
done,
we'll
have
temporally
coherent
replay
and
if
we
also
get
temporally
temporal
validation
in
signed
exchanges
we'll
be
able
to
have
non
long-term
fixity
and
non-repudiation
support,
and
if
native
momento
support
in
browsers
lands
will
have
a
visual
way
to
communicate
that
the
resources
are
tired.
So
thank
you
so
much
and
a
couple
minutes
that
we
have
I
can
answer
any
questions.
B
A
What's
talking
about
this
thing,
I
think
so!
Yes,
we
do.
The
reason
is
because,
if
we
are
able
to
communicate
this
information
or
some
of
the
tools
and
controls
that
we
provide
in
the
browser
itself
like
like
in
the
banners
and
all,
they
will
be
surfaced
off
in
the
browser
itself
in
the
toolbar
and.
A
People
will
be
more
acknowledged
that
the
resource
that
they
are
seeing
is
not
live
and
we
do
not
want
live
in
an
archive
web
to
be
mixed
together.
That
causes
more
problems
than
not,
and
so
there
is
a
battle
going
on
browsers.
We
were
pushing
for
this
for,
like
decades
now,
and
there
is
excellent
support
of
momento
InBev
archives,
but
we
wanted
Wikipedia
to
adapt
to
that
because
they
are
using
their
history
using
their
own
mechanism.
A
We
wanted
it
to
be
used
more
standardized
way
and
we
wrote
add-ons
and
all
which
gone
through
thorough
review
process
and
all
and
then
finally,
they
decided
not
to
deploy
it
because
they
said
hey
browsers,
don't
support
it,
and
when
we
go
back
to
web
browsers
and
say
hey,
please
support
it.
They
say
nah
big
sites
like
Wikipedia,
don't
support
it.
So
there
is
chicken-and-egg
problem
here,
still
firing.
A
So
this
is.
This
will
be
fantastic,
because
if
we
get
this
bundling
support
and
all
these
archives
can
bundle
things
and
they
can
nicely
put
the
entire
bundle
in
say,
ipfs
and
it
becomes
independent
of
where
the
resource
is
being
served
from.
It
can
move
around
and
then
deliver
it
from
anywhere.
The
browser
will
would
treat
the
whole
page,
not
a
single
resource,
but
the
entire
page,
with
all
the
page
requisites
and
replay
nicely
as
if
it
is
coming
from
thee.