►
Description
Every other week, the Retrieval Market Builders get together to share progress in their projects in a demo format. We want to sincerely thank all of our collaborators for demoing their developments and helping to improve the FIL Retrieval Markets one demo at a time.
August 4, 2021, Demo Day: In this video, you can find quick demos regarding the following - • PegaSys team demonstrating their file retrieving, caching, and libp2p demo.
• Myel demonstrating lazy retrieval markets.
• KEN Labs showing their indexing tool and performance testing.
__________________________________________________________________________________________
If you would like more information about the Demo Days please follow the link below to the Demo Days planning document.
https://docs.google.com/document/d/1IrNbBm2_P79Xi2fXFj9YUfQ0PjWjreGdPYFjFntD1QM/edit#heading=h.akao8aspby4
A
A
If
anyone
has
any
questions
during
any
of
the
demos
to
please
post
those
in
slack
as
well,
they'll
be
answered
async
and
anyone
who
missed
this
live
session
will
understand
your
question
as
well
and
be
able
to
see
the
answer
that
way
too.
So
we
have
three
demos
going
on
today.
It
looks
like
we
have
pegasus
doing
a
file
retrieving
mile.
I
think
I'm
pronouncing
it
correctly,
I'm
looking
at
lazy
retrievals,
and
then
we
have
ken
labs
as
well
looking
at
index
tool
and
performance
testing.
B
So
last
time
we
were
able
to
demonstrate
you
that
when
a
provider
published
a
file
offer,
the
client
was
able
to
get
it
through
the
get
away,
but
we
we
use
the
file
to
do
it.
So
I
will
show
you
that
now
we
have
real
file,
so
I
will
go
to
the
one
of
the
provider
container,
just
to
show
you
that
we
will
do
the
demonstration
with
these
three
files.
B
B
B
But
at
any
point,
if
you
have
any
insight
on
a
geolocation
mechanism,
we'll
be
really
interested
to
have
a
discussion
with
you,
so
I
search
for
all
the
gateway
around
the
clients.
There
are
many.
I
will
choose
one
so
I
will
appear.
It
means
that
the
client
will
use
this
one
to
query
the
offer
and
I
will
search
for
the
first
file.
You
see
the
cid
here,
so
I
will
try
to
find
the
offer.
B
B
B
Okay,
so
it
was
for
the
standard
discovery
now
we'll
use
the
second
mechanism.
It
is
the
dht
discovery,
because
we
have
two
type
of
getaways.
We
have
full
getaways
and
many
minimal
getaways
light
gateways
a
full
getaway.
We
store
all
kinds
of
offers
minimal
getaways.
We
only
will
only
store
a
few
set
of
a
few
bunch
of
offers.
B
B
B
Like
the
previous
mechanism,
we
can
once
we
have
the
offer,
we
can
use
the
digest
with
a
sorry
with
a
pass
and
we
are
able
to
download
the
second
file
and
the
third
feature
you
see
at
every
time.
We
have
to
to
ask
the
client
has
to
specify
which
getaway
it
will
use.
So
we
have
a
really
really
fast
mechanism,
so
I
will
do
it
for
the
first.
The
third
file
virtual
fast,
is
just
an
internal
algorithm
for
the
client.
It
will
be
able
for
the
client
to
automatically
find
his
content
at
the
best
price.
B
B
B
There
is
a
good
incentive
to
do
that,
for
the
clients
also
is
between
client
and
gateway.
Payment
channels
are
already
set
up.
So
if
a
client
asks
a
getaway
to
retrieve
a
popular
file
and
the
gateway
have
it
and
they
used
to
discuss
payment
channels
are
already
set
up.
So
finally,
reveal
will
be
very
fast
faster
that
if
it,
if
it
has
to
go
to
an
unknown
gateway,
so
I
will
demonstrate
the
caching.
B
B
B
B
C
B
C
Yeah
then,
then,
in
the
provided,
admin
switch
to
the
provider
that
is
running
inside
the
gateway
container.
So
basically
we
achieve
patient
gateway
by
combining
gateway
and
provider
altogether,
so
basically
gateway
caged
the
content
to
a
pass.
Then
the
provider
is
able
to
pick
it
up.
Then
we
can
publish
offers
for
that.
Cached
content.
E
D
F
All
right,
I
think,
can
can
you
see
my
screen
right
now,
yeah,
you
can
see
yeah
all
right
awesome.
So
basically,
I'm
gonna
start
kind
of
with
the
use
case
that
we've
been
kind
of
working
on
and
really
what
interests
us
right
now
is
kind
of
looking
at
you
know
at
this
case
of
a
simple
blog
you
know.
So
what?
If
you
stored
and
cached
a
single,
simple
blog
on
on
the
flightcoin,
retrieval
market
and
so
kind
of
what
we
were
looking
at
is
just
a
way
to
really
remove.
F
So,
for
example,
what
I
could
do
is
if
I
were
setting
up
a
blog.
I
could
just
start.
You
know
by
writing
down
my
my
slug,
so
this
is
just
kind
of
a
sandbox.
It's
not
really
a
user-facing
interface,
it's
mostly
for
demo
purposes,
but,
for
example,
you
know
if
I
put
a
slug-
and
I
say
I
have
all
my
cover
images,
so
I
would
just
kind
of
load.
F
F
And
so
I'm
going
to
start
kind
of
a
pre-fetch,
so
here
I'm
just
setting
the
maximum
price
per
byte.
So
I'm
saying
you
know
I
can
tolerate
any
provider
who
will
charge
me
below
six
out
of
five
per
byte
and
then
I'll,
just
kind
of
retrieve
and
so
automatically
I'm
creating
the
payment
channel.
And
so
what
happened
here
is
this
retrieval
actually
only
retrieves
the
index
of
the
file.
F
Basically,
and
so
right
now,
I'm
loading
a
payment
channel,
but
for
paying
for
the
entire
content
of
this
dag,
but
I've
only
retrieved
the
index,
and
so
essentially,
once
I'm
kind
of
done
with
this,
and
I
have
that
loaded,
I
can
reuse
that
that
same
payment
channel
with
that
peer
and
then
progressively
load
all
the
content,
and
so
it's
actually
there's
still
kind
of
a
lot
of
you
know.
Questions
around
I
mean
there's
a
lot
of
complexity,
basically
around
deciding.
How
to
you
know,
establish
this.
F
This
link
between
the
client
and
the
provider
in
terms
of
when
to
when
can
a
payment
channel
be
settled
when
can
be
updated
and
you
know
making
sure
that,
for
example,
the
provider
is
in
its
interest
wants
to
settle
as
quickly
as
possible.
So
it
can,
you
know,
get
some
of
the
cash
back
and
the
client
wants
to
keep
that
that
payment
channel
up
and
open
as
long
as
possible,
as
long
as
there's
still
funds
in
it.
F
So
here
it
should
it's
just
tiny
bit
slow,
but
it
should
appear
here.
We
go
so
I'm
kind
of
like
retrieving
all
these
things.
So
now,
every
time
I'm
I'm
you
know
doing
block
by
block
and
sending
more
vouchers
every
time
I'm
doing
this
so
every
time
I'm
doing
this,
I'm
kind
of
extending
the
the
time
I
have
that
I
can
keep
this
payment
channel,
and
so
once
I,
if
I
don't,
give
sign
of
life,
you
know
after
a
little
bit,
then
the
provider
will
just
assume
that
they
can
redeem.
F
You
know
those
vouchers
and
so
they'll,
just
compact,
it
all
into
one
voucher
and
then
update
the
state
and
then
settle
it
and
there's
also
a
thing
where
it
won't.
It
makes
sure
that
there's
also
this
the
different
parameters
that
you
can
use
that
are
like
the
minimum
set
of
height
and
things
like
that,
so
it
val
it
really
validates
those
vouchers
to
make
sure
that
it's
it
will
work
with
that
as
well.
F
F
So
if
I'm
subscribed
to
a
blog
or
something
like
that,
you
could
push
those
updated,
cids
and
then
automatically
preload
the
channel
for
that
content,
and
then
next
time
I'm
going
and
I
browsing
the
content
it's
already
loaded
and
I
can
load
it
right
away
as
a
scroll
and
everything.
So
you
get
that
really
smooth
experience
and
you
don't.
A
Wonderful,
it
looks
like
there
are
a
few
comments
and
questions
for
you,
I'm
in
the
slack
channel
so
make
sure
to
take
a
look
at
those
after
the
rest
of
the
demos
are
done.
Thank
you.
So
much
thomas
finally
looks
like
today
we
have
ken
labs
demoing,
an
indexing
tool
and
also
some
performance
testing
as
well
so
I'll.
Let
you
take
it
from
here.
G
G
Yes,
we
can
cool
thanks.
So
this
is
the
agenda
for
our
demo,
so
we
have
three
parts.
So
first
of
all,
ben
is
going
to
introduce
the
our
ipfs
watch
enhancement
with
the
url
resource
support,
so
which
is
we
commented
in
previous
demo
after
previous
demo,
and
the
second
part
is
the
performance
testing.
So
we
did
the
round
of
the
testing
against
ipfs
file
ad
interface.
So
I'm
going
to
present
on
behalf
of
our
colleague
junction
at
last,
I'm
going
to
give
a
brief
introduction
about
our
indexing
service.
A
G
Okay,
so
sorry
for
a
delay.
This
is
the
ipfs
api
performance
test.
We
we
did
in
past
two
weeks.
Let's
see
our
objective.
In
general,
we
try
to
understand
how
fpfs
behaves
in
handling
different
file
size,
especially
the
large
files.
In
this
way,
it
helps
us
understand
what
kind
of
file
we
can
we
can
handle
and
in
our
future,
the
indexing
service.
What
we
need
to
what
we
need
to
do
and
what
kind
of
file
we
need
to
prepare
or
even
even
handle.
G
This
is
in
general,
the
test
plan,
so
we
just
test
the
the
file.
Add
interface,
the
api,
the
http
api.
The
steps
are
simple,
so
we
run
the
add
operation
with
different
file
size
and
the
calculate
illustrate
the
trend
of
the
different
indicator
or
the
the
measurements,
for
example,
the
response
time
and
the
transaction
per.
So
here
we
we
are
using
transaction
per
hour
and
the
cpu
memory
usage.
G
G
E
G
You,
okay,
so
it
might
be
too
small
for
you,
I'm
not
sure
if
you
really
can
configure
out
those
figures,
but
I'm
going
to
share
with
you
the
result
and
the
analysis,
so
in
general,
here
on
the
left
side,
so
it
will
be
in
the
same
pattern
so
left
side,
it's
a
test
against
different
file,
size
and
right
side.
That's
the
matrix
against
different
chunk
sizes.
So
we
can
see
the
larger.
G
The
file
is
the
more
time
it
takes
so,
which
is
apparent
and
on
the
right
hand,
side
you
can
see
by
increasing
the
chunk
size.
It
cannot
always
increase
the
processing
speed.
So
in
practice
it
converge
at
some
point.
So
what
we
learned
from
this
case
is
that
we
should
run
testing
with
different
chunk
sizes
in
practice
to
find
the
size
that
suits
suits
our
use
case
or
test
case.
G
G
So
in
this
way
we
can
say
that
we
we
can
try
to.
In
general,
we
can
try
to
increase
a
bit
the
chunk
size
so
that
we
can
get
a
better
performance.
I'll
show
you
more
summaries
later.
So
this
is
the
data
we
collect,
and
this
is
the
cpu
usage
under
the
memory
cpu
and
memory
usage
consumption
on
the
left
side.
It's
the
metrics
on
different
file
size
on
the
right
side.
It's
the
different
chunk
size
on
the
same
file.
So
it's
not
very
straightforward.
G
So,
if
I
go
back,
we
test
with
chunk
size,
0.5,
0.25,
megabytes
and
a
half
and
one
megabytes.
So
these
three
size
chunk
sizes,
so
these
are
reflected
on
the
chart.
You
see
here
in
the
right
hand,
on
the
right
hand,
side
so
in
general
I'll
say
that
the
larger
chunk
sizes,
the
higher
tph
on
the
ipfs
side
in
general,
so
we
can
increase
the
size
of
it.
So
this
is
the
second
result.
G
G
G
G
G
I
I
B
I
I
E
G
G
The
idea
that
we
want
to
give
a
solution
that
user
can
search
the
sit
also
do
some
meaningful
search,
for
example,
for
text
search
which
will
be
based
on
some
semantic
index.
So
this
is
the
way
or
the
the
goal
we
want
to
achieve.
So
we
define
some
very
generic
data
structure,
so
the
intrinsic
attribute
index.
This
is
to
to
give
you
some
insight
of
that
set.
G
So,
for
example,
if
you
put
some
file
with
a
file
store
to
to
the
to
ipfs,
so
that
the
data
is
not
copied
to
rpfs,
instead,
you
have
the
file
pass
and
the
offset,
as
well
as
the
the
size
of
the
block
or
the
accumulative
size.
So
in
general,
all
this
information,
the
intrinsic
attribute,
so
we
want
to
have
some
kind
of
index
on
that
information.
G
Well,
I
just
put
the
organization
attribute
here:
this
is
to
to
to
indicate
where
this
data
from,
for
example,
from
any
organization
from
any
inch
to
tuition
or
some
university
or
some
company.
So
this
is
a
very
generic
purpose
for
the
organization
it
can
be
expanded
or
extended,
and
the
ins.
Besides
the
the
ia
index,
we
also
define
the
ea
index.
So
this
is
a
extreme
intrinsic
attribute.
G
C
G
G
How
we
can
we
can
produce
or
set
the
this
kind
of
attribute
to
to
the
data
or
that's
it.
Some
way
can
be
that
we
we
can.
We
can
leverage
manual
labeling
of
the
data
so
that,
when
you
type
ipf
add
file,
so
we
can
have
some
option
to
to
allow
user
to
to
tag
or
label
the
the
file
so
say
this
is
some
very
useful
technology
article
that
you
can
share.
So
this
is
some
semantic
information.
Besides,
we
can
also
use
some
deep
learning,
algorithm
or
the
npl
so
that
we
we
can.
G
G
So
this
is
a
extrinsic
attribute.
So
we,
when
we
have
these
indexes,
so
we
can
set
up
the
index
entry,
so
this
is
an
entry
for
each
set
for
each
set.
You
have
these
two
attribute
set
types
so
for
each
you,
you,
you
can
by
some
means
populate
them
to
the
entry
and
then
we
archive
those
entries
into
index
archive
structure
in
this
type.
There
are
two
main
aspects
one
appears,
so
this
is
to
identify
which
provider
has
this
content.
G
The
rest
is
the
entrance
list,
so
this
let's
show
you
who
has
what
content,
including
or
actually
what,
when
we
say,
content
here,
that's
the
index
and
this
advertisement.
This
is
to
advertise
what
this
provider
has
to
the
public.
So
we
can
publish
a
message
to
you
to
to
some
pops
up
topic
and
any
subscriber
will
receive
and
store
or
process
those
index
data
for
any
customer
needs,
so
they
can
also
expose
interface
for
different
kind
of
purpose
in
very
generic
search
or
full
text
search.
G
G
Every
time
when
you
receive
a
title
advertisement
from
the
topic,
it's
not
guaranteed
that
you
receive
the
latest
and
that
you
have
synced
all
the
previous
advertisements.
So
in
this
way
the
data
provider
maintain
the
chain.
Every
time
when
we,
when
you
receive
the
data
subscriber
receiver
advertisement,
it
will
try
to
fetch
feature
from
the
data
provider.
G
So
this
is
the
idea
of
this
data
structure
and
we
define
the
interface
for
now.
We
just
defined
when
the
indexing
feature
will
be
enabled.
So
I
am
going
to
talk
about
the
indexing
feature
shortly,
so
in
general
I
we
want
to
introduce
another
option
in
the
ipfs
ad
command,
so
you
have
the
dash
dash
index
in
this
way.
The
ipfs
node
will
know
that
we
are
going
to
produce
the
index
for
the
new
content
and
there
is
just
some
open
issue
here.
So
what
if
the
other
file
is
deleted
from
the
local
file
system?
G
G
Is
about
a
file
store
when
we
use
file
store,
and
we
know
that
the
content
is
not
copied
to
the
block.
So
what
if
the
file
is
deleted,
so
the
block
becomes
invalid.
So
how
can
we
handle
it
if
onpane
and
the
garbage
collection
is
enough,
so
we
are
going
to
we
need
to
investigate
we.
We
need
to
find
the
solution
for
that
case
also,
if
the
file
store,
I'm
not
sure
if
a
file
store
will
be
a
permanent
feature
or
solution.
G
G
So
this
is
the
interface
of
the
ipvs
node,
so
indexer
node,
we
I
mark
them
as
red.
So
currently
we
are
investigating
the
use
case
or
the
feasibility
of
store
the
index
project.
So
in
general
it
has
already
exposed
a
list
of
interface,
as
at
least
here,
I'm
not
sure
we
still
need
the
import.
So,
probably
yes,
so
our
mark
has
read
so
we
are
still
investigating.
G
This
is
the
architecture.
So
there
are
a
couple
ways
to
to
feed
the
data
to
ipfs
so
end
user
by
command
line
or
the
ipf
watch.
What
we
are
working
on
through
the
http
interface
or
maybe
from
through
by
our
other
protocol
or
the
ipfs
crawler.
The
crawler
is
going
to
crawl
or
walk
through
the
whole
ipfs
network
and
collect
data
collected
blocks,
maybe
blocks
or
maybe
other
type
of
data.
G
So
maybe
a
collection
of
blocks,
it's
not
clear
yet
in
general,
the
data
will
be
coming
from
crawler
and
the
field
to
the
ipvs
node.
So
going
through.
This
stack
take
stack
when
we
have
the
the
dac,
the
dag
from
the
store.
We
can
start
building
the
index.
So
that's
a
raw
data
sent
to
the
indexing
feature.
So
when
I
say
indexing
here,
it's
a
feature
we
are
going
to
introduce
to
ipfs.
G
So
here
in
general,
we
are
going
to
publish
a
message
to
the
topic,
a
pop
soft
topic,
and
here
the
index
node,
which
is
we
are
also
working
on.
We
are
investigating,
as
I
said,
store
index
the
project.
We
may
want
to
extend
that
project
to
subscribe,
to
some
topic
to
collect
the
advertisements,
then
fetch
them
from
the
ipfs
node
or
some
or
its
data
provider.
So
the
index
node
will
build
more
featured
in
indexes
so
including
the
as
I
said
so,
the
symmetric
index
or
the
intrinsic
attribute,
so
they
they
are
very
straightforward.
G
I
just
say
the
the
plain
index
so
that
it
will
give
the
in
interface
to
external
users,
to
build
more
reliable
or
more
advanced
application
or
features.
So
this
and
this
indexing
node.
We
are
still
designing
so
here.
So
it's
not
finished,
but
now
we
have
done
the
index
feature
design
part.
So
this
is
the
the
feature
design
and
the
workflow,
and
this
is
sequence
diagram.
G
We
haven't
started
the
implementation
yet
so
we're
still
reviewing
reviewing
this
design
so
later,
probably
tao
is
also
going
to
post
this
design
to
to
the
channel.
We
can
talk
about
talk
about
this
design
more
and
any
any
question
or
advice
is
highly
appreciated,
so
we
will
go
through
and
give
a
update
in
this
design
if
needed.
A
Let's
do
that
then.
So
for
everyone,
just
a
quick
reminder:
these
are
every
two
weeks,
so
the
next
one
will
be
august
18th
same
time.
It's
a
wednesday
just
like
they
normally
are
make
sure
everyone
goes
in
to
the
slack
channel
to
take
a
look
at
some
of
the
questions
and
the
answers
that
were
provided
there
and
there's
also
looks
like
some
suggestions
as
well.
So
thank
you
again,
everyone
for
coming
today.
We
really
appreciate
it
reminder
again.
A
These
are
every
two
weeks
so
august
18th,
the
next
one,
like
I
said,
and
for
anyone
who
wants
to
see
the
recording
of
this
or
potentially
go
back
and
rewatch
any
of
it.
This
will
be
uploaded
shortly
and
I'll
be
putting
the
link
into
the
slack
channel
as
well
for
everyone
for
a
quick
and
easy
use
all
right.
Thank
you
again
for
coming
everybody.
We
appreciate
it.