►
From YouTube: Intro to Decentralized Web Archiving w/ Webrecorder & IPFS, Intro to pin-tweet-to-ipfs: Webrecorder
Description
Brief introduction to pin-tweet-to-ipfs. The first IPFS-Enabled extension. Marrying one of Webrecorder's browser-based archiving tools (express.archiveweb.page) and web3.storage to display the power of IPFS to non technical users.
A
A
What
is
what
do
we
mean
by
web
archiving,
and
this
is
one
definition
from
the
international
internet
preservation,
Consortium
process
of
collecting
portions
of
the
web,
preserving
collections
in
archival
format
and
then
serving
the
archives
for
Access
and
use,
and
so
what
do
we
actually
do
to
make
this
happen?
We
record
the
HTTP
Network
traffic
we
stored
in
archival
format,
and
then
we
replay
the
archival
data
back
to
the
user
in
the
browser,
and
so
how
did
web
archiving
start
and
who
does
it?
A
Traditionally
it's
been
done
by
libraries
and
archives,
that's
kind
of
been
that
where
we're
at
this
concept
and
practice
originated,
of
course,
there's
an
internet
archive
who've
pioneered
the
field
in
the
late
90s.
They
have
the
the
interactive
Wayback
machine,
which
many
of
you
are
familiar
with,
but
I
also
wanted
to
mention
organization
here
in
in
Lisbon
called
archivo.pt,
who
are
also
doing
web
archiving
in
Portugal
and
they've
actually
built
their
own
web
archive
over
the
past
15
years,
and
they
offer
actually
some
of
the
same
functionality.
A
They
have
their
own
Wayback
machine.
They
even
have
a
save
page
now
that
accepts
User
submitted
websites,
including
specifically
focusing
on
Portuguese
websites,
so
you
can
check
out
their
archivo.pt
and
they
use
a
lot
of
our
open
source
tools
as
well.
So
a
quick
shout
out
to
our
keyword,
fpt
there's
also
many
other
member
institutions
around
the
world.
It's
the
iepc
is
the
international
Consortium
of
libraries
and
archives.
A
I
think
it
has
over
50
members
that
represents
organizations
that
do
web
archiving,
and
so
the
web
recorder
project
focuses
on
Building,
open
source
tools
and
making
it
possible
for
anyone
to
create
web
archives.
And
so
our
goal
is
to
build
open
source
tools
to
make
archiving
more
accessible
to
everyone,
not
just
large
institutions,
to
focus
on
Fidelity
of
archiving
and
replay
and
create
portable
formats
for
web
archives.
A
And
by
that
we
mean
archiving
the
web
as
accurately
as
possible,
so
preserving
all
of
the
dynamic
interactions
as
much
as
we
can
and
re-rendering
the
data
as
accurately
at
a
later
time.
So
archiving
all
the
interactive
things
that
happen
on
the
website,
including
all
the
JavaScript
and
then
playing
that
back
from
the
archived
storage
and
we
use
the
browser
itself
to
create
web
archives.
There's
not
a
separate
kind
of
crawling
tool.
That's
that's
separate
from
the
browser.
It's
all
done,
using
the
browser
to
create
and
access
the
web
archive.
A
So
it's
a
core
part
of
the
archiving
process
and
who
uses
web
recorder
tools.
Well,
obviously,
memory
institutions
we
work
with
libraries
and
archives,
but
as
also
one
to
support
and
currently
work
with
digital
humanists,
journalists,
artists
activists
and
really
anyone
who's
interested
in
archiving
and
preserving
slice
of
the
web.
Perhaps
you
and
it's
a
little
bit
more
about
the
web
recorder
tool
ecosystem,
so
we
have
a
bunch
of
different
tools
that
represents
sort
of
these
stages
of
the
archiving
process
and
later
we'll
demo.
A
Some
of
these
we
have
a
browser
extension
that
works
in
Chrome
and
brave
and
any
chromium
based
browser.
We
also
have
a
single
site
that
can
archive
a
single
page.
At
a
time
we
also
have
a
web
crawler.
Then
we
have
a
portable
format,
called
wax
Z
that
we've
been
building
that
can
be
stored
on
ipfs,
as
well
as
traditional,
centralized
storage,
and
then
we
have
a
web
component
based
viewer
that
renders
and
replace
archive
web
pages
and
the
way
that
this
process
works.
A
So
here's
kind
of
a
brief
again
these
three
stages.
We
use
the
one
tool
to
create
the
web
archive,
it's
serialized
to
a
format.
Then
we
use
the
replay
web
page
tool
to
to
replay
the
web
archive,
and
this
is
for
smaller
scale
kind
of
manual
archiving
in
the
browser
and
then
for
larger
scale.
Web
archives.
A
We
have
a
crawling
tool
that
is
used
for
the
archiving
process
that
runs
automatically
either
via
command
line
or
via
UI,
again
produces
the
same
format
and
then
can
be
accessed
via
a
the
embedded
viewer
and
so
how
much
data
convinced
to
be
archived
with
these
tools.
Well,
we
have
quite
a
big
range.
We
can
have
anything
from
certainly
that
we've
tested
thus
far
from
a
single
page,
that's
100K
to
a
an
archive
that
was
over
one
terabyte
and
I'll.
A
So
it's
sort
of
the
simplest
thing
that
works
we
put
in
the
we
use
a
UNIF,
Unix,
FS
directory
structure
with
this
waxy
file
and
the
JavaScript
for
the
embedded
Replay
by
Page
viewer,
and
so
this
is
sort
of
what
the
directory
looks
like
there's
some
sort
of
boil
plate
HTML
that
includes
the
web
component
and
the
actual
waxy
file,
and
so
that's
for
a
single
web
archive
we're
also
now
working
on
putting
multiple
web
barcodes.
At
the
same
time.
It's
a
similar
process.
A
We
have
the
viewer
and
then
we
just
add
more
of
these
waxy
files
and
we're
also
working
on
custom
chunking
of
this
format
to
optimize
for
deduplication
and
also
figuring
out
what
the
optimal
schema
is
for
this
layout,
and
also
so
I've
been
talking
about
this
waxy
format.
What
actually
goes
into
it
itself
is
actually
a
zip
file
and
it
contains
these
other.
It's
a
package
format
for
this
other
data.
A
The
whole
thing
extends
the
frictionless
data
package
format
and
we
conclude
additional
metadata,
including
cryptographic
signatures
of
the
data,
so
that
the
the
Integrity
of
the
format
itself
can
be
verified
through
the
data.
That's
in
the
format
and
there's
a
we
have
a
spec
for
it
and
yeah.
So
that's
that's
sort
of
the
core
portable
format
that
that
we're
working
with
and
now
how
does
this
work?
A
B
Yeah
we
basically,
this
is
an
example
of
one
thing
that
we've
built
out
with
using
web
recorder,
specifically
the
archive
web
Express
tool.
It's
basically,
we
wanted
to
get
ipfs
into
like
really
end.
Users
pans
people
that
are
not
familiar
with
ipfs
people,
they're,
not
familiar
with
like
web
3,
storage
or
web
recorder.
Any
of
this
stuff
there's
a
lot
of
value
that
we
can
start
to
bring
to
people
right
now,
even
before
we
have
like
these
Integrations
into
chromium
and
all
this
other
stuff,
it's
very
usable
now
right.
B
B
This
is
going
to
be
the
first
in
a
series
of
like
ipfs
enabled
extensions,
which
some
will
be
like
super
simple
like
this,
and
some
will
get
a
little
bit
more
and
more
complicated
with
peer-to-peer
stuff,
maybe
like
audio
video,
recording,
there's
a
lot
of
different
ways.
We
can
go
with
this,
but
for
this
one
specifically
we're
really
targeting
end
users
that
are
not
familiar
with
ipfs.
B
So
the
features
is
that
we
archive
a
tweet
and
a
completely
reproducible
context
which
he
just
described,
which
is
the
waxy
file,
and
then
users
can
input
their
web3.
storage
API
keys
to
track
all
their
archives.
In
one
place,
implementation
is
basically
we
don't
do.
Hardly
anything
in
this.
We're
really
really
relying
on
ilio
for
this,
but
basically
just
like
copy
the
permalink
out
of
the
page.
B
B
Web
2.
storage
should
probably
put
this
before.
But
if
you
don't
know
what
it
is,
it's
a
simple
file:
storage
service
for
ipfs
and
filecoin
and
web
recorder.
We
got
some
explanation
of
that
before
and
we're
gonna
have
some
demos
after
this
I
already
kind
of
mentioned,
like
the
future
of
ipfs
enabled
extensions,
but
we're
looking
at
audio
video
uploading.
B
Maybe
a
quick
one-page
website
publisher,
much
like
move,
has
but
maybe
not
having
to
use
aggregore.
Yet,
although
we
all
should
be
using
it's
so
sick,
maybe
peer-to-peer
file
transfer
with
Duran
I
have
like
a
big
list
of
ideas,
but
I
really
want
more.
So
any
crazy
ideas
for
any
of
this
stuff.
Please
like
send
to
me
I'll
try
and
build
it.
You
can
build
it.
You
can
build
it
together,
whatever
so
the
road
map
for
this
right.
B
Now,
it's
like
really
simple
right
extension,
most
of
it's
done,
but
we
do
want
to
improve
the
UI
I
want
to
explore,
possibly
like
embedding
the
web
archiver
itself
and
to
the
extension.
There's
been
some
examples
around
that
there
is
an
existing
extension
that
does
web
archiving.
There
is
some
like
restrictions
around
the
Manifest
V3
stuff,
which
is
a
common
theme
in
our
lives
these
days.
So
we'll
see
where
that's
going
to
go
like
if
we
can
find
some
cool
workarounds
in
manifest
V3
and
then
maybe
we
can
integrate
it
in.
A
B
And
then
adding
more
gateways
uploads
to
more
gateways,
possibly
local
nodes,
eventually,
so
we're
not
relying
on
that
one
provider
of
web
3
storage,
it's
now
available
in
the
Chrome
web
store
it's
on
the
way
to
being
in
the
Firefox
store,
I
have
to
just
separate
it
out
and
make
a
separate
build.
But
if
you
want
to
go
ahead
and
download
it
check
it
out,
go
for
it.
B
A
Thanks
Dave
for
that
demo
of
a
particularly
new
integration
of
one
of
our
tools,
which
is
archive,
webpage,
Express
and
I'm,
going
to
just
do
a
very
quick
demo
of
of
this
tool
and
and
some
of
the
other
tools
that
we
have.
So
what
the
extension
is
using
is
a
site
called
archive
webpage
Express,
which
is
basically
a
single
page
archiving
system
that
runs
directly
in
your
browser.
So,
for
example,
this
is
a
just
a
ipfs
cam
site
as
soon
as
it
loads
in
the
browser
is
actually
archiving
it
as
it's
loading
it.
A
It's
loading
it
through
a
proxy
in
order
to
be
able
to
load
a
remote
site,
and
so
we
could
see
here
that
it's
already
archived
5.8
megabytes,
and
so
you
could
go
here
and
and
yeah
and
essentially
archive
any
any
page
one
one
page
at
a
time,
and
then
we
have
an
integration
with
web
3
storage
and
you
could
click
that
button
and
and
it'll
upload
it
to
to
web3
storage
and
yeah.
So
that's
archive
webpage,
Express
designed
for
a
single
page
archiving.
A
Then
we
also
have
the
archive
webpage,
Chrome
and
brave
extension,
which
is
for
more
kind
of
multi-page
archiving
in
the
browser
and
basically
allows
you
to
Archive
exactly
what's
loaded,
including
stuff,
that's
paywalled,
and
so
you
get
your
you
could
archive
with
yeah.
Essentially,
if
it's,
if
it's
loading
in
the
browser,
it
will
get
archived
with
this
extension
and
I'll
do
a
quick
demo,
let's
see
if
it
if
it
works
here.
So
let's
say
we
want
to
Archive
the
ipfs
camp.
A
Hashtag
and
I
already
have
the
extension
installed
and
so
I
go
here,
call
it
a
hashtag
again
and
then
I
click
Start
and
it's
using
the
Chrome
debug
API
in
order
to
be
able
to
fully
archive
a
page,
and
it's
actually,
you
can
see
here-
that's
already
loaded
this
much
data.
We
also
have
this
thing
called
autopilot,
which
will
actually
do
some
repetitive
Behavior,
such
as
clicking
on
tweets.
So
if
you
wanted
to
go
through
a
an
entire
Twitter
timeline,
you
can
use
this.
A
So
this
is
a
custom
behavior
that
we
had
to
build
and
we
have
them
for
for
certain
social
media
sites
that
our
users
have
requested
and
the
idea
sort
of
emulates
a
user
going
through
going
through
a
Twitter
timeline
and
so
it'll
just
basically
keep
keep
running,
and
you
can
see
here.
I'm
logged
in
with
a
I
created
a
special
social
media
account
for
this.
A
That's
archived
23
megabytes
here
and
now
it's
using
the
the
replay
web
page
viewer,
which
I'll,
which
is
embedded
into
the
extension
and
from
here
I,
can
actually
I
can
go
and
I
can
download
this
as
the
waxy
file
and
I
can
also
share
an
ipfs.
This
works
particularly
well
in
Brave,
actually
because
of
the
embedded,
ipfs,
node
and
so
I
can
click
on
start
sharing
and
it
should
and
now
it's
it
tells
me
that
it's
sharing
this.
A
This
web
archive
and
I
copy
the
IPS
URL,
and
so
it
should
load
pretty
quickly
because
it's
just
in
my
local
node-
and
you
can
see
here
that
it's
loading
in
from
ipfs
using
the
the
replay
web
page
viewer
I,
can
also
let's
say
I
can
go
into
yeah,
and
so
that's
the
archive
web
page
extension.
So
it
allows
for-
and
you
can
see
here
that
I'm
still
logged
in
or
it
says
that
I'm
logged
in
so
it's
archived
as
as
the
this
Twitter
user
and
yeah.
A
So
moving
on
I'll
talk
about
replay
web
page,
that's
the
the
viewer
for
web
archives,
it's
available
as
a
web
component
and
a
single
page
app
it
loads.
Web
archives
directly
in
the
browser
can
load
from
https
or
ipfs,
and
hopefully
other
schemes
in
the
future
when
using
it
on
Brave
it'll
can
use
the
the
native
protocol
Handler,
otherwise
it'll
use
the
Gateway
and
you
could
load
it
from
replayweb.page
and
so
from
here.
I
can,
for
example.
This
is
just
now
in
in
Chrome.
A
A
A
That
shows
you,
the
verification,
the
provenance
of
how
this
was
created,
the
tools
that
were
used
and
also
signature,
information,
and-
and
we
have
a
whole
process
around
that,
especially
for
web
archives
that
are
created
using
using
our
Cloud
crawling
tools
and
and
we're
working
with
Starling
lab
on
this
in
this
particular
feature
as
well,
and
when
I
was
here's.
A
demo
of
I
mentioned
earlier,
a
very
small
web
archive
re-archiving
geocities
from
internet
archive.
A
A
So
that's
that's
sort
of
the
the
range
of
the
the
tools.
Then
we
also
have
wanted
to
variable
yeah.
We
have
a
cloud
crawler
that
can
be
used
to
essentially
create
a
crawl
in
the
cloud,
as
I
was
mentioned,
for
automated
archiving
and
it'll
basically
start
a
crawl
and
crawl
whole
entire
site,
and
you
could
watch
it
as
it
goes.
So
in
the
interest
of
time.
I'll
skip
that
now,
but
last
thing
I
want
to
show
is
a
brand
new
tool
that
we've
just
launched.
A
Just
this
week
is
the
multi
waxy
uploader,
so
download
archiving
a
single
single
waxy
file
to
ipfs,
so
I
wanted
to
show
how
you
could
archive
multiple
files
at
once
and
that's
using
this
this
uploader.
So
this
is
brand
new.
So
if
you've
already
created
archives
either
using
the
extension
or
the
cloud
crawling
tool
and
you
have
waxy
files,
you
can
basically
go
in
using
this
uploader
and
it
will
so,
let's
let
me
let's
see
if
this
works,
and
so
this
is
the
and
so
it'll.
A
It
will
try
to
upload
all
these
files
to
web3
storage,
and
then
we
have
a
a
static
site
generator
that
that
will
be
used
to
to
create
a
web
archive
or
essentially
a
static
viewer
for
for
all
these
web
archives.
So
I
I
did
prepare
an
example
of
what
it
should
look
like
when
it's
done
I
believe
Maybe,
not
maybe
like.
Actually,
no
it's,
it
is
so
okay.
So,
while
that's
good,
there
is
an
example
of
a
so
a
previous
upload.
A
A
The
the
replay
web,
page
viewer
and
so
let's
see
yeah,
and
so
that's
that's
kind
of
the
the
end
result
is
that
you,
it
kind
of
this
is
again
still
a
work
in
progress.
Allow
you
to
combine
multiple
web
archives
and
create
yeah.
So
once
that
finishes
that's
this
is
basically
what
what
it'll
look
like.
A
Oh
sorry,
I
keep
jumping
out
yeah,
so
you
could
add
a
description
and
yeah
and
then
each
and
so
this
the
idea
is
that
this
can
scale
to
very
large
archives,
because
each
one
of
these
files
is
loaded
on
demand
and,
let's
see
what
else
do
we
have
here
yeah.
So,
for
example,
in
the
in
the
case
of
the
lab
week,
there
should
be
a
a
YouTube
video
that
also
got
archived
and
so
that's
embedded
in
there
as
part
of
the
archive.