►
Description
Mike Goelzer, Leader of Product and Strategic Management at Protocol Labs, explains the details of the architecture for the browser-based Filecoin retrieval market they has been working on.
At The First Ever CSCON[0] Virtual Event!
A
Yeah
so
without
further
ado,
I'm
going
to
introduce
our
last
speaker
this
time.
The
title
of
this
talk
I
find
is
entirely
captivating.
It's
called
architecture
of
a
browser-based
file
coin
retrieval
market,
it's
by
mike
gelzer,
who
leads
product
and
strategic
management
at
protocol
labs.
So
I'm
going
to
bring
him
on
right
now.
A
B
B
Okay,
so
let
me
introduce
myself,
my
name
is
mike
gelser.
I
work
at
protocol
labs.
I've
been
working
a
lot
on
the
filecoin
project
this
summer
and
what
I
want
to
present
on
today
is
a
tool
that
I've
been
working
on
an
embarrassing
amount
of
time
this
summer
and
and
fall
and
winter.
It's
a
browser-based
retrieval
market
for
file,
coin
content,
and
what
I'm
going
to
talk
about
today,
in
particular,
is
the
architecture
of
the
application.
So
this
builds
on
a
previous
talk.
B
I've
given
I've
given
a
few
a
few
talks
about
this.
They
mostly
demo
the
application,
show
you
what
you
can
do
stuff
like
that.
One
of
them
is,
is
here
this
link.
I
know
you
can't
click
that,
but
if
you
can
remember
five,
six
hrk
and
l
or
I
p
I
d
s-
you
can
visit
that
for
a
more
that's,
a
more
gentle
introduction
to
what's
going
on
in
retrieval
markets
in
in
filecoin
generally.
B
What
I
want
to
do
in
this
talk
is
is
focus
more
on
the
architecture
of
what
we've
built
and
before
I
I
move
forward.
I
just
want
to
point
out
in
the
bottom
left
corner
there.
We
we
hang
out
in
on
file
coin
slack.
So
if
you
don't
have
an
account,
you
just
go
to
file
coin
dot,
io
slack,
and
then
we
hang
out
in
a
public
channel
called
phil
dash
browser
dash
retrieval.
B
We
welcome
anyone
who
wants
to
join
our
little
community,
that's
working
on
this
project
or
related
projects.
So
please
we
would
love.
We
would
love
to
have
you
join
that
channel
and
we,
you
know
we
can
point
you
to
other
other
stuff.
That's
going
on,
so
I'm
gonna
condense,
the
kind
of
background
information
to
just
hopefully
like
just
two
or
three
minutes,
and
then
I'm
going
to
talk
more
about
the
architecture.
As
I
said,
but
just
really
briefly,
what
what
is
the
retrieval
market
so
filecoin
is
a
market
for
data
storage.
B
You
negotiate
a
deal
with
a
minor.
You
store
data
on
that
storage,
miner
and
then
later
on.
You
can
retrieve
the
data
from
that
miner,
but
there
are
some
use
cases
where
you
may
want
to
publish
data
for
a
more
general
audience
than
just
retrieval
by
yourself,
and
in
that
case
we
want
to
create
a
competitive
market
for
retrieval
of
that
data,
and
the
idea
for
doing
that
is
to
have
a
second
market.
B
This
is
our
main
repo.
So
what
we're
trying
to
build
is
a
browser-based
implementation
of
what
I
just
described.
Everything
runs
in
a
browser.
You
don't
need
to
run
a
lotus
instance
on
your
laptop.
You
actually
don't
need
to
run
one
anywhere
and
I'll.
Tell
you
a
bit
about
how
that
it
works
in
a
minute
and
you
you
can
query
for
cids,
you
can
store
cads,
you
can
resell
cids,
so
you
can
come
check
out
our
repo
okay.
B
B
A
B
So
this
demonstrates
the
peer-to-peer
nature
of
the
browser
network,
but
on
its
own,
that's
not
very
interesting.
How
do
you
get
content
into
the
system
in
the
first
place?
So
to
do
that?
Let's
put
let's
take
a
look
at
a
cid
that
is
only
available
on
the
file
coin
storage
network
and
in
particular
on
this
miner.
B
Okay,
I
query
for
that
cid
I
click
buy
and
what's
going
to
happen,
is
this
browser
is
now
going
to
in
open
a
connection
to
a
proxy
server
that
runs
an
instance
of
lotus.
That
instance
of
lotus
will
do
a
lotus,
client
retrieve,
and
then
this
browser
will
send
funds
to
the
proxy
server
to
then
actually
receive
back.
The
bytes
of
this
cid.
B
B
It
can
do
two
things:
it's
it's
a
peer-to-peer
network
of
browsers,
so
it
can
retrieve
from
another
another
browser.
That's
the
green
part
that
you
see
on
the
left
here
and
it
then
it
can
also
retrieve
from
the
file
coin
storage
network
and
I'm
going
to
talk
in
a
little
bit
more
detail
about
how
both
of
those
both
of
those
functions
work.
So
this
diagram
is
basically
everything
I'm
going
to
walk
you
through
it
in
pieces.
But
fundamentally
this
is
everything.
That's
going
on.
B
You've
got
this
peer-to-peer
browser
network
on
the
right
in
green,
each
of
those
boxes.
Imagine
that's
a
browser
and
then,
when
those
browsers
need
to
publish
messages
to
the
chain,
they
do
so
via
a
public
lotus.
That
just
means
it's
a
lotus
that
anyone
can
access,
it
doesn't
store
any
wallets,
it
doesn't
store
any
private
information.
B
So,
let's,
let's
dive
into
a
little
more
detail
about
some
of
these
parts,
just
talking
about
the
browsers,
communicating
with
each
other
and
how
do
they
exchange
files
with
one
another?
Fundamentally,
it's
a
lib
p2p
network.
So
if
you're
not
familiar
with
lit
p2p,
this
is
a
this
is
a
great
open
source
project
which
I
I
know
and
love,
and
you
can
too,
if
you
go
to
github.com
lib
p2p,
lib
p2p
is
the
main
repo
and
then
there
are
various
language
implementations
linked
to
from
there.
B
We
we
have
many
different
we've
implemented
the
lib
p2p
protocol
in
many
different
languages.
It's
an
extensible
upgradable
protocol,
it's
great
for
peer-to-peer
applications
so
anyway,
so
this
is
basically
a
lib
p2p
network
of
browsers
they
communicate
using
gossip,
which
is
one
of
lib,
p2p's,
decentralized,
pub
sub
mechanisms
we
support
a
couple.
The
best
one
is:
is
gossip,
a
gossip
sub.
B
So
if
you
have
a
gigabyte
file
that
you
want
to
upload,
the
payload
cid
is
just
computed
as
a
hash
over
that
the
one
gigabyte
of
data
and
then
that
we
call
it
sometimes
called
that
the
data
cid
or
the
file
cid
or
the
payload
cid,
and
then
once
it
gets
into
filecoin
once
it
gets
stored
on
a
filecoin
miner.
There's
another
concept
called
pcid,
which
I'm
not
going
to
go
into
right
now,
because
this
particular
tool
only
thinks
in
terms
of
of
payload
cids.
B
It
thinks
in
terms
of
an
entire
file
or
it's
zero
or
nothing.
Sorry,
it's
everything
or
nothing.
So
we
we
deal
with
with
payload
cids
in
here
in
the
future.
I
would
love
to
to
do
to
work
with
ipl
deselectors
and
more
fine-grained
control
over
what
you're
retrieving
like
from
a
large
data
set.
You
might
only
want
a
certain
set
of
records,
but
this
right
now
this
tool
is
much
simpler
than
that.
B
B
It's
called:
go
fill
markets,
slash
retrieval
market,
like
that's
a
github
file.
Coin
project
slash:
go
fill
markets,
slash
retrieval
market,
that
that
is
a
set
of
of
that's
a
a
block
of
code,
very
large
block
of
code
in
that
in
lotus,
and
it
relies
on
some
lower
level.
Components
basically
relies
on
go
data
transfer
and
then
go
data
transfer
relies
on
something
called
go
graph
sync,
so
we
want
to
take
the
collection
of
those
three
things
compile
them
into
wasm.
We've
managed
to
do
that.
B
We've
got
that
working
outside
the
browser
extension.
Now
it's
just
a
matter
of
porting
it
into
the
browser
extension
and
then
longer
term.
What
we're
trying
to
do
is
rewrite
all
of
this
stuff
in
native
javascript.
So
so
we
have
one
project
underway
to
create
a
js
graph.
Sync,
that's
the
lowest
level
component.
B
Then
we
would
create
a
js
data
transfer
analogous
to
go
data
transfer
and
then
we
would
finally
create
a
js
side
of
the
retrieval
markets
protocol,
but
for
now
we're
just
relying
on
a
much
much
simpler
way
of
just
moving
the
bytes
back
and
forth
and
in
terms
of
storage
like,
like
you
know,
if
you
retrieve
a
one
gigabyte
file,
where
is
that
stored,
so
it's
stored
in
the
browser,
local
storage,
we
use
the
js
ipfs
block
store.
We
think
it's
a
better!
It's
it's!
B
Just
a
hash
address,
block
store
it
based
on
it.
Chunks
did
it!
Well,
you
give
it
chunks
of
data,
it
stores
them
hash
addressed,
and
then
they
can
reassemble
them
into
a
into
a
file
based
on
the
cid
you're
asking
for
so
that
that's
how
the
browser
part
of
it
works.
Now
I
want
to
talk
a
little
bit
about
the
payment
side
of
it.
B
So
when
browser
a
is
requesting
a
file
from
browser
b,
it
necessarily
implies
that
browser
a
is
going
to
pay
funds
fill
to
browser
b
and
the
way
that
works
is
a
back
and
forth
kind
of
mechanism
where
a
little
bit
of
data
is
sent
by
browser
a
and
then
browser
b
requests
a
payment
voucher,
which
is
a
commitment
to
pay
a
certain
amount
of
fill
in
the
future.
Then
browser
a
sends
back
that
payment
voucher.
B
Then
browser
b
sends
a
little
more
data,
and
this
cycle
just
continues
going
until
all
data
has
been
transferred
and
all
value
for
that
data
has
been
transferred.
The
other
way
in
the
form
of
vouchers
and
in
order
to
use
payment
vouchers,
you
have
to
first
create
a
payment
channel,
create
or
or
recycle
a
payment
channel,
which
is
a
an
on-chain
event.
So
we
have
to
publish
messages
onto
the
chain.
B
As
I
mentioned
earlier,
we
we
use
a
public
lotus
for
all
on-chain
interactions,
but
because
it's
public,
we
can't
put
anybody's
wallet
on
it.
We
can't
put
someone's
you
know,
wallet
private
keys
on
it.
It's
there's
too
much
risk
that
somebody
would
figure
out
how
to
hack
lotus
in
a
clever
way
that
exposes
other
people's
keys.
So
we
keep
all
the
private
keys
the
signing
keys
in
the
browser,
which
means
we
need
to
sign
all
the
messages
in
the
browser.
So
what
we
do
is
we
generate
message
all
of
our
messages.
B
We
don't
use
the
lotus
rpc
apis.
You
know
the
rich
lotus
is
a
very
rich
api.
There's
thousands
of
calls
you
can
make.
We
don't
use
basically
any
of
those.
What
we
do
is
generate
all
of
the
messages
that
we
want
to
publish
on
chain
inside
the
browser
we
sign
them
with
the
pr
with
the
user's
wallet
private
key,
which
is
also
never
leaves
the
browser,
and
then
we
use
this
low-level
interface.
Mpool
push
to
publish
that
message
onto
the
blockchain
and
then
there
are
some
other
apis.
B
You
can
use
to
to
see
the
result
and
did
it
execute
successfully
and
so
forth.
There
are
some
future
ideas
about
doing
delegated
signing
in
lotus,
so
you
would
have
a
lotus
where,
instead
of
having
wallets
on
that
lotus
it
would
it
would
reach
out
there.
It
would
be
given
some
kind
of
a
callback
when
ever
it
needs
a
message
signed.
B
B
B
Here's
the
dream!
The
dream
is
the
browser
all
of
the
browser
nodes
and
all
of
the
file
coin.
Storage
nodes
would
be
on
the
same
lib
p2p
network,
and
when
a
browser
wanted
to
retrieve
from
a
storage
node,
it
would
directly
connect
to
that
storage
node
in
inlet
p2p
it
would.
It
would
open
a
stream
to
that
storage
node
for
the
transfer
we
would
use.
B
It
wouldn't
even
know
that
it's
not
talking
to
another
storage,
miner
or
another,
another
lotus,
full
node.
Technically
it
wouldn't
know
that
it's
not
talking
to
another
lotus.
It
would
just
think.
Okay,
I'm
talking
to
a
lotus.
It's
requesting
this,
offering
these
payments
and-
and
that
would
be
it-
there's-
been
some
practical
challenges
in
realizing
this
vision.
First
in
in
a
web
browser,
obviously
you
can
only
make
outgoing
websocket
connections.
You
can't
make
raw
tcp
connections.
B
Lotus
actually
supports
websocket
connections,
but
it's
not
turned
on
in
the
config
file
by
default,
so
we
would
have
to
convince
you
know
a
large
percentage
of
miners
to
turn
on
websockets
and
advertise
a
websockets
multi-adder,
so
that
that's
one
that's
one
thing.
The
second
is
js
graph,
sync
and
the
go
wasn't.
B
Work
is
not
complete
yet
so
we
don't
have
we
don't
yet
have
we
have
some
of
these
working
pretty
well
js
graph
sync
can
communicate
with
gograph
sync
in
kind
of
an
isolated
environment,
and
we
can
do
again
in
a
in
a
separately,
isolated
environment.
We
can
do
data
exchange
between
a
lotus,
minor
and
our
own
code.
Using
this
wasm
bundle
of
go
a
subset
of
go,
fill
markets
plus
go
data,
transfer
plus
go
graphs,
and
so
we
think
we
can
combine
all
these
things
in
a
way.
B
That'll
work,
but
they're
not
there
yet.
So
those
are
some
of
the
main
challenges
and,
as
a
result,
we
put
into
place
a
stop
gap
solution
which
is
a
proxy
server.
So
that's
that
very
dark
blue
box
at
the
bottom.
We've
had
to
put
that
in
between
the
browser
browsers
and
the
any
file
coin,
storage
node,
so
it
basically
it
works
the
way
any
proxy.
Does
the
browser
sends
the
request
for
what
it
wants
to
the
to
this
proxy
machine.
B
The
proxy
machine
is
running
a
lotus
full
node,
so
it's
capable
of
doing
a
retrieval
of
that
of
whatever
cid
the
browser's
asking
for
from
the
file
coin
storage
network,
and
then
it
just
sends
those
bytes
back
to
the
browser
this
whole.
The
pricing,
the
economics
of
this
is
very
tricky.
I
don't
want
people,
I
don't
want
to
create
a
situation
where
people
can
abuse
the
proxy
server
to
make
money
or
something
like
that
to
get
extra
income
for
their
storage
miner.
B
So
what
we
do
is
we
make
browser
clients
pay
up
front.
They
pay
the
proxy
server
two
two
times
the
price
of
retrieving.
What
whatever?
What
we
estimate
to
be
the
price
of
retrieving
that
that
cid
they're
requesting
and
then
it's
a
best
efforts
thing
and
there
are
no
refunds,
so,
in
other
words,
we
do
the
best.
We
can
it's
open
source
code,
we're
not
trying
to
rip
people
off,
but
this
is
very
much
it's
not
an
ideal
solution.
B
A
B
B
B
We
don't
require
anybody
to
run
a
lotus
instance
anywhere
on
their
computer
or
have
one
in
the
cloud
or
anything
like
that.
So
the
I
know
the
interface
is
a
little
raw.
We
did
that
wasn't
a
priority,
but
I
think
this
could
be
turned
into
an
application
that
would
work
well
for
for
any
end
users
and
be
simple
to
use.
B
Yes,
yes,
I
mean
this
is
a
classic
light.
Client,
it
doesn't
have
a
full
node
it.
It
depends
on
it's
an
infura
style
system.
So
if
you
know,
if
you
know
infira
what
the
way
this
our
system
works
is
there's
just
this
public
cloud
lotus
and
all
the
messages
are
sent
to
that.
But
it
doesn't,
everyone
is
sending
their
messages
to
the
same
set
of
cloud
cloud.
Lotuses,
it's
just
on
the
subject
of
inferior
inferior,
actually
has
a
filecoin
api.
B
I
would
really
like
to
use
it
in
its
current
state.
It's
I
don't
know
if
it's
fair
to
call
it
read-only,
but
basically
it
can't
do
some
of
the
operations
like
transferring
funds,
for
example,
that
that
I
need
so.
I
can't
switch
to
it
quite
yet,
but
my
hope
is
it'll
keep
getting
built
out.
Certainly
the
the
inferior
gateway
for
for
ether
is
has
all
that
stuff,
so
they
they
know
how
to
do
it.
B
A
B
Yeah,
that's
a
good
question.
I
think
that
I
think
that
a
few
things
will
happen.
I
think
that
the
the
first
thing
that
happens
is
as
puja
was
saying
in
her
talk,
we're
in
the
early
stages
of
of
onboarding
clients
who
to
store
data
onto
onto
the
network
right.
So
the
first
big
thing
was:
okay
launch
the
main
net.
Have
all
these
storage
miners
ready
to
accept
data?
B
We
now
have
this
huge
amount
of
storage
capacity
and
now
we're
on
boarding
clients
who
want
to
store
data,
and
if
they
want
to
store
data,
they
obviously
want
to
retrieve
it
at
some
point
and
some
subset
of
those
people
are
going
to
want
not
just
to
it's
not
just
a
backup
of
my
hard
drive
where
I'm
the
only
person
who's
ever
going
to
retrieve
it.
It's
like
the
the
you
know
the
genocide
database,
for
example.
B
You
know
you
can
imagine
that
that
would
be
of
interest
to
educators
at
a
lot
of
different
institutions,
and
so
it's
maybe
one
organization,
uploading
that
data
to
filecoin.
But
it's
going
to
be
many
organizations
that
want
to
retrieve
it
and
I
think,
as
those
use
cases
come
up,
there's
going
to
be
a
need
for
something
like
a
google
file
coin,
some
kind
of
an
index
where
you
can
you
can
type
in
genocide,
educational
material
or
something
like
that,
and
you
can
find
this
data
set.
A
A
B
No,
I
haven't
so
we're
we're
using
wasm
in
two
places
right
now,
one
of
them
I
skipped
over.
We
use
rust,
wasm
rust
code
compiled
to
wasm
for
a
lot
of
the
payment
channel
message
generation
and
also
a
lot
of
the
signing
logic.
I
have
not
looked
into
assembly
script.
No,
probably
should
yeah
that's
a
good
tip.
Thanks.