►
From YouTube: Retrieval Market Builders Demo Day - Zhenyang Shi, W3DT, & KEN Labs - October 28, 2021
Description
Every other week, the Retrieval Market Builders get together to share progress in their projects in a demo format. We want to sincerely thank all of our collaborators for demoing their developments and helping to improve the FIL Retrieval Markets one demo at a time.
October 28, 2021, Demo Day: In this video, you can find quick demos regarding the following -
• Zhenyang Shi presenting their work on content retrieval with payment network through FCR
• W3DT demoing CAR file tooling
• KEN Labs showing their updates on simple datastore and API for GraphQL query with pubsub support
__________________________________________________________________________________________
If you would like more information about the Demo Days please follow the link below to the Demo Days planning document.
https://docs.google.com/document/d/1IrNbBm2_P79Xi2fXFj9YUfQ0PjWjreGdPYFjFntD1QM/edit#heading=h.akao8aspby4
A
Hello
and
welcome
to
today's
retrieval
market
demo
day.
Today's
date
is
wednesday,
the
27th
of
october
2021
on
the
schedule
for
today
we've
got
an
incredible
lineup.
We've
got
zhenyang
she's
speaking
about
the
fcr
he's
built,
which
is
content
retrievals
with
a
payment
network.
We've
got,
will
scott
from
protocol
labs
data
systems
team
talking
about
car
file
tooling,
and
then
we
have
william
from
ken
labs
speaking
about
a
data
store
and
api
for
graphql
query
with
pub
sub
support.
A
So
yeah
with
that
we've
got
the
intros
done
a
quick
message
to
say
that
if
anyone's
got
any
questions
throughout
this,
please
put
it
on
the
retrieval
market
slack
channel,
and
then
we
can
answer
them
asynchronously
or
after
after
the
meeting
and
also
others
can
can
get
involved
over
to
you.
Zhenyang.
B
Great
hello,
everyone,
maybe
before
the
bam
demo.
Let
me
just
quickly
introduce
myself
to
to
people
who
hadn't
met
me
before
so
my
name
is
jin
yang
I
live
in
brisbane
australia.
I
have
been
working
as
a
research
engineer
in
pegasus
consensus.
For
almost
four
years,
I've
been
mostly
involved
in
cross
chains,
stateless
ethereum,
a
little
bit
on
these
two
and
recently
falcon
retrieval
and
ipfs.
B
So
fcr
is
another
secondary,
retrieval
client
that
I
have
been
personally
building
during
the
past
two
months
for
the
presentation
today.
What
I
will
do
is.
I
will
first
go
through
only
the
core
concepts
of
the
system
without
covering
all
the
details,
because
it
would
take
at
least
an
hour
to
go
through
everything
anyway
after
the
system
works
through.
I
will
then
do
two
live
demos,
so,
first
of
all,
fci
is
written
in
go
and
it
is
able
to
participate
in
a
cid
network
for
content
delivery
and
a
payment
network
for
payment
delivery.
B
So
ci
network
is
very
similar
to
ipfs,
so
I
will
just
quickly
go
through
this
slide.
I
assume
everyone
is
familiar
with
ipfs.
So
routing
is
based
on
cad
dht
provider.
Record
is
used
to
indicate
an
interest
of
serving
content.
That
is,
if
you
use
fcr,
to
start
serving
content.
Fci
will
periodically
publish
the
provider
record
to
dht
and
to
stop
serving.
If
someone
else
wants
to
retrieve
a
content,
it
will
first
use
dht
to
find
the
provider
record
after
providing
a
record
is
fund.
B
Fcr
will
carry
the
provider
for
a
signed
cid
offer.
So
this
is
a
cid
offer.
It
contains
a
few
things:
the
root
c
id
of
the
graph,
the
price
per
byte,
the
size
of
the
graph.
The
wallet
address
for
receiving
the
payment
plus
a
little
bit
more.
One
thing
that
I
wish
to
mention
is
a
linked
miner
in
in
the
offer.
B
So
the
idea
is
that
if
you
are
running
a
proper
miner-
and
you
also
wish
to
participate
in
the
secondary
tool
market,
you
can
attach
your
mind
address
together
with
the
proof
to
your
city
offer,
which
means
this
can
combine
with
the
proper
storage
in
foulcoin,
to
increase
the
reliability
of
a
cd
offer.
Also,
potentially,
this
can
combine
with
the
reputation
system.
B
Let's
talk
about
the
payment
network,
so
after
you
create
a
payment
channel
to
someone
you
can
choose
to
serve
the
payment
channel
for
others
to
use
at
a
given
price,
the
serving
information
in
the
form
of
payment
routes,
which
I
will
talk
about
in
the
next
slide.
So
so
the
payment
routes
will
be
published
to
subscribers
every
certain
period
of
time
until
you
stop
serving.
If
someone
wants
to
do
a
proxy
payment,
it
will
first
find
a
payment
route.
Then
user
payment
to
query
for
a
payment
offer.
B
Now,
let's
introduce
the
concept
of
payment
route,
so
in
a
network
of
five
nodes
here
the
arrow
is
the
payment
channel
being
served
and
there
are
six
possible
payment
routes
from
node,
a
a
to
b,
a
to
b
to
d
a
to
b
to
d
to
e,
a
to
c
a
to
c
to
d
and
a
to
c
to
d
to
e,
so
node
a
stores
other
payment
roots
from
it
locally,
so
a
knows
which
nodes
are
accessible
from
it.
These
payment
routes
will
also
be
published
to
a
subscribers.
B
B
So
this
is
a
payment
offer
similar
to
crd
alpha.
It
contains
some
information,
for
example,
it
has
price
per
period
period
and
max
amount.
The
price
per
pair
and
the
period
are
used
to
calculate
the
surcharge,
for
example,
for
an
offer
with
price
per
pair
to
be
five
and
a
pair
to
be
a
hundred
in
order
to
proxy
100
units
payment.
B
You
need
to
pay
105.,
okay,
so
no
d
reply
to
b
with
a
payment
offer,
then
node
b
will
create
another
payment
offer
that
actually
depends
on
the
payment
offer
received
from
d.
So
the
surcharging
payoff
for
b
will
include
both
the
surcharge
set
by
node
bc
user
and
the
third
charge
of
payoff
for
d,
so
node
a
will
then
be
able
to
proxy
payment
to
all
the
way
to
node
e.
So
the
payment
is
done
again.
B
B
Okay,
so
the
so
the
first
demo
is
to
show
how
retrieve
a
content
with
proxima
a
proxy
payment
in
a
five
node
network.
Look
like
so
what
I
have
done
before
this
presentation
is,
I
spin
up
a
docker
compose
that
is
consisted
of
five
fcr
nodes
and
one
mocked
lotus
miner
so
and
I
also
run
a
setup
script.
So
the
network
looks
like
what
is
shown
in
the
graph
we
can.
B
Maybe
we
can
have
a
look
at
the
setup
script,
so
the
first
thing
I
did
here
is:
I
did
a
bootstrap
of
these
fcr
nodes,
so
they
are
connected
then,
for
every
node
I
generate
some
random
files
and
use
fcr
to
import
those
files
into
the
system.
I
then
asked
these
nodes
to
start
serving
those
files
with
price
per
byte.
B
B
So
node
two
imported
two
cids
node
three
imported
one,
cd,
node
four
important
one
study
and
on
five
or
the
one
setting.
So
this
can
be
also
shown.
You
know
in
the
graph
as
well.
Also
node
two
served
a
channel
to
note
three,
no
threes
of
the
channel,
two,
not
four
and
note
four
serve
the
change.
Note
five.
So
if
you
understand
this
graph,
let's
assume
we
are
note
one
here.
B
B
B
So
if
you
are
a
miner
and
you
enabled
faster
retrieval,
then
you
can
import
all
the
pieces
directly
into
secondary
retrieval
market
by
using
import
sector
and
by
by
default
when
you
import
the
sector,
fcr
only
stores
a
reference
to
the
sector
without
creating
another
physical
copy,
just
to
save
your
disk
space.
Okay.
Now
for
that,
let's
focus
on
retrieval.
B
B
Some
use
the
id
here,
let's
retrieve
it,
to
the
current
directory,
okay.
So
so
we
as
node
one
we
found
one
cd
offers
through
the
dht
from
node
two.
Normally,
if
we
don't
have
a
direct
payment
channel
to
the
recipient,
fcr
will
find
corresponding
payment
offers.
However,
as
you
see,
node
one
does
not
have
any
payment
channel
to
any
other
node,
so
we
will
have
to
create
a
payment
channel.
So,
let's,
let's
do
this.
B
B
Okay,
so
this
time
we
also
find
the
steady
offer
for
note2.
However,
how
the
retrieval
works
is
that,
first
of
all,
it
will
carry
all
directly
connected
peers
for
offers
without
going
into
ghd,
and
we
have
just
created
payment
channel
with
node
two,
so
fcr
current
node
two
for
the
offers
first
and
we
we
found
an
offer.
So
we
don't
need
to
go
into
dht
stage
and,
let's
retrieve
using
the
offer
from
node
2
and
the
retrieval
is
down.
B
Okay,
so
this
time
we
can't
find
the
number
from
o2,
so
we
discovered
an
offer
via
the
dht
network
and
we
found
an
offer
from
those
three.
However,
because
we
don't
have
a
payment
channel
directly
to
node
three
fcr
searches
for
the
corresponding
payment
offer
and
because
node
two
served
the
payment
channel
to
note
three,
we
fund
one
payment
offer
from
node
two
and
the
payment
period
is
five
with
a
pair
to
be
a
hundred.
B
B
Okay,
so
this
time
we
found
an
offer
via
cid
of
vidht
and
the
corresponding
payment
offer.
However,
this
time
the
payment
offer
has
a
different
price.
The
payment
per
period
rate
is
10
with
a
pair
to
be
100,
and
that
is
because
this
proxy
payment
goes
through
node,
2
and
node
3.,
so
we
need
to
to
pay
surcharge
to
both
of
them,
retrieve
that
retrieval
is
done
again.
B
B
So
how
fast
retreat
works
is
that
you
only
need
to
supply
the
max
amount.
We
you
wish
to
pay
faster
to
our
first
query,
directly
connected
peers
for
cid
offers.
If
fcr
can
find
a
city
offering
this
stage,
it
will
choose
to
retrieve
using
the
cheapest
offer.
As
long
as
the
you
know,
the
total
price
does
not
exceed
max
amount.
B
If
fcr
fails
to
find
the
cid
offerings
in
this
stage,
it
will
cure
the
dht
for
cid
offers
and
for
every
steady
offer
it
will
try
to
find
all
the
corresponding
payment
offers
as
well.
So
among
all
of
these
offers,
fcr
will
choose
the
cheapest
cid
offer
and
the
payoffer
pail
as
long
as
it
does
not
exceed
on
the
max
amount.
Okay,
let's!
Let's
do
this.
B
B
It
is
a
10
node
network,
so
similar
to
the
last
demo.
I
have
got
a
docker
compose
up
and
running.
I
have
also
run
a
setup
script,
so
the
overall
network
looks
like
this,
and
each
of
the
node
is
hosting
a
unique
cip,
so
node
0
is
hosting
cid
0
node
1
is
hosting
cid
1
blah
blah
blah
note.
9
is
hosting
cid
9
and
the
arrow
is
a
payment
channel
as
well.
B
This
has
already
been
configured
by
the
setup
script.
I
have
also
configured
the
max
hub
to
be
10,
which
means
every
node
is
able
to
access
every
other
node.
For
example,
it
is
possible
for
node
9
to
retrieve
a
content
from
node
6,
so
proxy
payment
will
go
through
node,
eight
node,
five
node,
two
node
three
zero
one
and
finally
six.
B
So
if
you
get
idea
of
this
network,
then
we
can
start
the
demo.
However,
this
time
I
won't
do
the
demo
manually
instead,
I
will
rely
on
a
script
to
do
that.
Demo.
That
is
wrong.sh,
and
you
may
wonder
why,
because
this
time
we're
going
to
do
something
different.
Maybe
let's
have
a
look
at
the
script.
First.
B
Okay,
so
this
time
what
we,
what
we
will
do
is
first
of
all,
we
get
the
cid
served
by
each
node.
Then
we
will
ask
every
node
to
do
a
fast
retrieval
of
every
other
cid
from
the
network.
So
there
is
a
node
zero
will
retrieve
cd1
cd2
s3,
all
the
way
to
c89
node
one
will
do
the
same
blah
blah
blah
every
node
will
do
the
same
so
and
we
make
this
retrieval
concurrent
that
is
90
total
retrievals.
B
B
Yeah,
somehow
it
is
taking
a
little
bit
longer
than
you
expected
and
you
can
see
some
retrieval
fails,
which
is
normal
and
I,
which
is
also
expected.
So
normally
you
can
see
one
or
two,
only
one
or
two
retrieval
failed,
but
this
time
you
see
a
little
bit
more
than
that
and
it's
fine.
If
we
have
a
look
at
the
output
file
and
we
search
for
the
keyword
retrieval
done,
we
can
see
that
72
out
of
90
retrieval
succeed
with
18
belt,
which
is
a
little
bit
high.
B
So
so,
if
this
happens,
that
is
okay,
you
can
run
the
retrieval
script,
one
more
time
so
for
previously
successful,
retrievals
fci
will
retrieve
the
content
from
retrieval
cache,
so
no
network
activity
and
no
cost,
but
for
failed
one
fcr
will
just
try
again.
But
if
previously
the
retrieval
fails
in
the
middle,
you
don't
need
to
pay
for
already
retrieved
blocks,
because
they
will
also
come
from
the
case
just
right.
B
It's
a
folder,
for
example,
in
node
nine.
We
can
see
that
it
has
nine
other
cid
retrieved.
If
you
take
other
nodes,
they
all
should
have
every
other
cid
retrieved.
We
can
also
check
how
much
was
spent
by
inspecting
the
outbound
channel,
which
I
won't
do
today
due
to
due
to
time
and
yeah
so
cool.
That's
that's
all
for
the
second
demo.
Thank
you
guys
for
for
listening,
feel
free
to
reach
my
slack.
If
you've
got
any
question
or
comment
or
sort
yeah,
that's
it.
Thank
you.
A
C
D
C
So
there's
a
binary,
the
binary
runs
as
car.
You
can,
if
you
do
not
have
this
installed,
install
it
by
running,
go,
get
github.com,
ipld,
slash,
go
car
command,
car
and
a
car
is
a
cli
utility
that
works
with
car
files.
So,
for
example,
if
I
have
just
some
random
data-
and
I
made
a
folder
here
called
example-
an
example
contains
a
file
called
hello.txt
right.
So
I've
got
you
know
some
arbitrary
folder
directory
tree.
C
I
can
do
a
couple
additional
things
that
are
sort
of
useful
here.
I
can
list
with
unix
fs
semantics,
and
now
I
can
see
that
in
there
is
the
directory
example
and
the
file
example
slash
hello.text,
as
you
might
expect,
we
also
have
a
verbose
mode
that
just
gives
you
a
bunch
of
additional
details
about
this.
C
So
things
like
this
block
is
using
the
raw
codec
and
then
these
other
two,
which
are
the
the
unix
fs
metadata,
are
dag
pb
encoded,
and
you
know
you
can
see
that
this
one
is
a
directory
that
contains
example-
and
this
one
is
has-
is
the
directory
entry
that
has
hello.txt
as
a
as
a
file.
So
you
get
some
sort
of
detailed
structure
about.
What's
in
there.
C
C
C
A
D
D
D
So
the
idea
is
that
we
want
to
aggregate
all
the
metrics
from
multiple
deal
bots,
so
that
we
can
provide
a
reliable
and
perform
a
query
service
for
different
consumers.
For
example,
if
you
want
to
collect
the
metrics
and
work
out
some
score
for
reputation
purpose,
that's
the
way
you
order
the
service
you
you
can,
you
can
try.
D
So
this
is
the
simply
there's
a
motivation
and
the
poc
in
terms
of
the
poc.
For
now
the
deal
bot
has
no
notification
enabled
so
that
if
we
want
to
receive
any
update
in
the
real
time,
we
need
to
do
some
work
in
the
deal
board.
At
the
same
time,
we
are
working
on
some
mock
so
that
we
will
mark
the
data
produced
by
the
obot
and
also
the
the
mock
server
will
also
serve
some
sample
data
so
that
the
service
can
collect
the
curator
data
and
get
it
cached.
D
D
D
D
C
D
D
D
D
We
will
also
aggregate
them
for
future
usage,
and
in
this
way,
all
the
data
collected
will
also
be
then
served
over
the
graph,
qr
or
ap
http
api
or
over
the
graph
sync
interface.
So
that's
part
of
the
next
step.
So
in
next
step
step
we
will
implement
the
dag
update
location
in
devote
so
that
the
debug
can
integrate
with
this
service,
and
for
now
we
haven't
done
the
aggregation.
D
So
all
the
data
received
are
simply
cached
locally
in
the
memory
under
the
the
data,
of
course,
in
the
data
store
from
the
provider
side,
and
that
this
is
what
we
need
to
do.
The
dag
aggregation
and
ipld
data
store
for
local
storage,
purpose
and
yeah.
I
think
that's
it.
For
today
we
will
bring
more
features
in
next
two
weeks
yeah.
I
believe
this
yeah,
the
q,
a
will
go
to
slack
thanks,
cheers.
A
Great,
thank
you
very
much
shisha.
So
at
this
point,
has
anyone
got
any
questions
which
they'll
think
would
help
those
watching
the
recording
to
understand
any
of
any
of
the
talks?
We'll
have
those
questions.
First,
we'll
stop
the
recording
and
then
we
can
go
to
a
more
informal
conversation
afterwards,
so
yeah
anything
which
we
think
could
help
clarify
or
explain
any
of
the
talks.