►
From YouTube: RETMKT Builders - Retrieval Market Indexing
Description
Will talks about content indexers at Retrieval Market Builders Mini-Summit in April 2021.
A
Seeing
happening
to
lotus
as
a
process
and
a
lot
of
what
motivates
that
is
that
there
are
new
capabilities
that
we
want
to
add
around
making
the
value
of
a
file
coin.
Miner
better
in
terms
of
you
know,
enabling
this
retrieval
market
and
and
one
of
those
is
indexing,
and
so
what
we're
meaning
when
we
say
indexing,
is
that
it's
sort
of
important
for
a
minor
to
know
what
content
it
has
and
to
be
able
to
get
that
content
back
out.
A
So
this
this
problem
of
like
content,
routing
routing
from
a
sid
into
the
the
entity
that
has
that
is
sort
of
what
ipfs
is
already
solving
right
and
and
is
the
content
writing
problem.
But
one
of
the
things
that
we
think
is
sort
of
the
useful
next
step
to
enabling
this
in
in
the
context
of
of
the
file
coin
network
is
we
want
to
have
some
interaction
between
an
index
or
some
external
node?
A
That
is
going
to
help
us
with
content,
routing
and
the
miners,
the
the
nodes
that
have
content,
and
we
think
that
this
extension,
where,
where
we,
we
have
this
interaction
and
interface,
that
we
define
for
a
node
with
content
that
sort
of
accumulates
new
content
and
is
going
to
notify
that
and
then
some
external
party
that
can
learn
what
that
index
and
what
the
available
content
is,
is
one
that
both
isn't
and
doesn't
need
to
be
specific
to
filecoin.
So
we
can.
A
We
can
make
this
interface
more
general,
so
that
existing
services
that
have
a
lot
of
content
address
data,
maybe
also
fit
the
same
model
of
they.
You
know
in
some
sort
of
checkpointed
way
will
get
more
content
that
they
want
to
make
available
in
a
content,
address,
network
setting
and
and
that
interface
probably
looks
pretty
similar
to
what
a
miner's
lifecycle
looks
like
of
content
sort
of
coming
and
going
in
batches.
A
A
What
does
the
indexing
service
need
to
do?
Well,
it
needs
to
generate
an
index
one
of
the,
and
so
what
is
an
index?
Well
at
some
level?
It's
just
like
the
list
of
all
of
the
sids
that
are
in
that
deal,
and
so
we
have
tools
that
can
go
over
a
car
archive
as
a
deal
is
currently
stored.
We
could
imagine
with
a
different
proof
of
replication
that
potentially
that
storage
is
different,
but
it
is
still
you
know,
a
bunch
of
content,
addressable
pieces
that
are
referenced
by
sits.
A
A
Perhaps
that
needs
to
live
on
the
minor,
because
it's
unclear,
if
either
the
sid
or
the
dag
of
sids,
that
there's
a
good
balance
that
can
be
provided
at
this
interface
that
lets
an
indexer
node.
Do
useful
sub
selection
just
on
that
list
that
it
may
need
to
be
done.
Someplace,
that
has
the
data,
which
would
be
the
miner
okay,
so
so
you've
got
some
service
that
now
has
this
list
of
data.
How
do
you
get
that
data
to
an
indexing
node?
A
That's
then
going
to
do
the
the
work
of
generating
an
overall
index
and
being
able
to
say
oh
this
sid,
that's
on
this
minor
and
probably
what
that
looks.
Like
is
two
pieces.
There's
a
there's,
a
send
of
advisory
of
new
data
available.
So
whenever
a
new
deal
finishes
or
or
there's
a
new,
you
know
chunk
of
data
items
that
have
become
available
on
the
miner
it'll
publish,
probably
on,
and
this
can
work
just
sort
of
as
a
gossip
sub
topic
that
we
agree
upon
that.
A
You
know
an
ipfs
like
thing
where,
where
these
are
all
ipld
data,
and
so
there's
a
new
sid
that
gets
put
out
and
the
indexers
fetch
that
sid
with
a
connection
to
the
miner
and
get
back
or
or
a
connection
to
this
index
process
that
maybe
exports
itself
over
ipfs
and
gets
back,
then
the
list
of
of
new
sids
that
need
to
be
indexed
and-
and
so
we
started
writing
up
that.
That
interface
a
little
bit
in
terms
of
how
we
think
about
these.
So
what?
A
What
is
the
advertisement
that
happens
over
pub
sub?
And
so
you
have
your
your
a
sid
that
points
to
sort
of
a
manifest
of
this
new
list
of
data.
That's
that
exists,
and,
along
with
the
previous
one
and
along
with
sort
of
how
to
connect
to
you
and
then
you
sign
it,
and
this
means
that
that
miners
sort
of
get
one
global
view
of
what
content
they
have
are
making
available
for
indexing
and
that
helps
them
from
that.
A
That
helps
us
to
sort
of
have
this
consistent
view
that
we
are
able
to
enforce
and
make
sure
that
miners.
If
they
advertise
data
as
available,
they
have
to
advertise
it
as
available
to
everyone,
and
so
that
that's
sort
of
one
step
where
once
we
have
those
semantics,
we
can
make
sure
that
they're
sort
of
the
availability
of
a
open
marketplace
that
you
that
you
don't
have,
at
least
through
this.
A
Okay,
but
but
then
there's
there's
also
the
the
subsequent
question
of
the
second
site
of
this
interface,
which
is
you've,
got
these
indexer
nodes.
Now
that
have
these
big
indexes
of
a
lot
of
sids
that
are
available
within
file
coin
and
and
which
miners
and
which
pieces
they
reside
in,
and
we
still
now
have
the
problem
of
okay.
You've
got
a
bunch
of
clients
that
want
to
do
that.
A
Do
that
lookup
of
I've
got
a
sid
and
I
would
like
to
know
who
has-
and
this
is
a
problem
that
ipfs
already
solves
via
a
api
called
content
routing,
and
what
that
actually
looks
like
right
now
is:
is
the
method
find
provider
where
you
give
a
sid
and
you
get
back
a
peers
address
info?
So
you
learn
which
peer
has
that
content,
and
this
is
the
interface
that
currently
is
exported
by
the
dht
and
by
other
content,
routing
modules
within
ipfs.
A
There's
there's
a
bunch
of
interop
sort
of
path
of
least
resistance.
That's
still
getting
figured
out
in
terms
of
how
do
we
enable
that
actual
subsequent
fetch
of
okay?
So
an
ipfs
node
gets
this
record
from
an
index
or
node
potentially,
and
then
how
does
it
know
to
actually
get
the
data
out
of
a
minor
and
and
that
there
there's
sort
of
a
few
different
ways
and
and
we'll
see
how
hard
they
they
end
up
being
relative
to
each
other?
A
You,
with
with
the
current
graph
sync,
go
data
transfer
process
that
exists
in
the
market,
part
of
filecoin
miners.
The
process
is
happens
in
a
few
steps
where
first,
you
on
chain
do
a
retrieval
deal
and
ask
for
that
data
and
and
then
that
results
in
a
voucher-
and
you
use
that
voucher
to
then
initiate
the
go
data
transfer
session
over
which
you
transfer
the
data
for
some
subset
of
the
data
that's
stored
it.
A
A
The
other
thing
that
we're
going
to
have
to
figure
out
is
how
does
the
ipfs
node
know
that
it
is
getting
or
or
ask
an
index
or
node?
So
if
we,
if
we
start
with
a
limited
number
of
these
indexer
nodes,
initially,
what
what
does
this
initial
arrow
look
like,
where
we
find
the
right,
indexer
nodes
and
make
use
of
them,
and
that
also
likely
goes
through
a
few
iterations?
We
could
imagine
that
you
can
do
sort
of
an
initial
delegated
content
routing.
A
Are
you
going
to
prune
down
to
a
smaller
index
of
head
queries
of
popular
sits
that
are
getting
asked
for
a
lot
where
you
can
respond
to
80
of
the
queries
in
relatively
small
number
of
milliseconds?
Or
are
you
going
to
keep
the
much
larger
list
of
all
possible
sits
so
that
you
can
answer
the
longer
tail
of
sort
of
rarer
queries
which
will
be
a
more
expensive
lookup
and
higher
latency?
A
So
there's
a
couple
places
of
potential
differentiation
there
in
indexing.
I
think
we
we
also
still
need
to
figure
out.
What
you
know
is:
is
the
story
going
to
be
that
these
indexing
nodes
get
get
renumerated
directly
in
this
protocol.
There
there's
two
places
there
that
you
could
imagine
that
you
know
retrievable
mining
indexing
playing
into
the
overall
economic
game.
A
A
The
indexer
could
get
some
small
commission
on
those
so
that
it
goes
within
the
existing
sort
of
payment
channel
thing,
but
basically,
there's
like
a
finder
speed
type
thing
where
the
indexer
node
that
did
help
make
that
deal
happen
gets
somehow
tags
in
in
the
record
that
it
returns
so
that
when
the
miner
does
its
payment
channel
thing,
the
main
the
miner
sort
of
tips,
the
the
indexer
for
fun,
bringing
it
in
the
traffic
to
will
quicktime
check.
What
can
you
wrap
it
in
one
minute
yeah?
A
Sort
of
a
a
bit
more
future,
looking
we're
just,
I
think
you
know
it
gives
you
a
sense
of
what
the
hopefully
sort
of
first
concrete
steps
look
like
in
terms
of
getting
to
something
that
works
and
then
a
bunch
of
the
things
we're
imagining,
but
we
would
like
feedback.
Certainly
in
terms
of
other
constraints,
we
need
to
be
thinking
about
and
other
ways
that
you
can
you
all
in
particular,
are
imagining
plugging
in
or
would
have
to
interact
with
this
or
things
that
sound
unrealistic.