►
From YouTube: Decentralization Paths for Network Indexers - @jbenet - Content Routing 1: Performance
Description
Decentralization Paths Network Indexers - presented by @jbenet at IPFS þing 2022 - Content Routing 1: Performance - https://2022.ipfs-thing.io
A
I'm
going
to
talk
about
decentralizing
paths
for
network
indexers.
This
follows
some
talks
about
the
network
indexers
in
general,
and
the
current
architecture
is
a
centralized
system,
so
the
network
indexers
that
we're
describing
coming
in
come
in
as
one
additional
content
writing
system
for
ipfs
networks.
There
are
many
existing
content
writing
systems
out
there.
This
is
probably
not
even
a
full
mapping
of
these.
This
is
just
kind
of
the
the
the
widest
wettest
used
ones,
so
the
public
hd
is
by
far
the
most
common
content
routing
system.
A
There
are
also
automatic,
like
local
area
network
dhcs
that
form.
So,
if
you're,
if
your
regular
most
kubo
and
a
couple
other
implementations
do
this,
where
they
disconnect
from
the
rest
of
the
world,
they
will
form
automatic
dhts
in
the
lands
and
so
on.
So
those
already
exist.
Many
private
networks
out
there
exist
today
that
run
their
own
separate
dhts.
A
A
So
the
way
ways
to
kind
of
hack
that
and
there's
been
many
people
that
have
tried
things
like
that
and
then
there's
just
local
dna
mdns
discovery.
So
many
ipfs
implementations
have
ways
of
connecting
in
local
area
networks
using
standard
protocols
and
once
nodes
find
each
other
and
connect
to
each
other
they'll
automatically
bitswap
content
to
each
other.
A
So
there's
kind
of
this
implicit
content,
routing
discovery
loop,
where
nodes
that
happen
to
be
in
the
same
network
will
happen
to
connect
to
each
other
once
they're
connected
to
each
other
content
requests
will
just
work
but
there's
kind
of
an
implicit
content
routing
system
there.
It's
not
an
explicit
thing
that
you
you
enable
now
the
start
of
the
hash
indexer
you
just
heard
about,
and
just
one
additional
content
writing
system.
There's
gonna
be
many
more
so
over
time.
A
The
community
is
gonna
experiment
with
a
lot
of
tools
and
I'm
just
gonna
kind
of
go
through
a
few
and
by
the
way,
all
these
are
kind
of
all.
These
systems
are
optional
and
different
groups
connect
to
different
systems,
and
they
have
different
reasons
for
using
them.
I
think
we're
still
working
on
how
to
build
composability
into
this
system
like
how
could
you
compose
these
into
a
thing
that
just
kind
of
works
for
most
use
cases?
A
This
tends
to
be
it's
not
clear
that
there
is
a
just
work
solution,
because
there's
a
lot
of
security
considerations
and
authentication
considerations
in
terms
of
where
you
want
to
ship
your
reader,
your
queries
to
so
as
we'll
see
tomorrow,
there's
a
lot
of
privacy
constraints
here,
but
anyway
it
nevertheless,
when
shifting
a
lot
of
content
into
into
one
indexer
that
currently
centralized
it'd
be
good
to
kind
of
describe.
What
are
the
paths
that
we're
considering
for
turning
this
into
a
different,
centralized
network?
A
One
other
thing
worth
noting,
like
the
amount
of
data
that
store
the
hash,
is
storing
now
just
can't
go
into
the
dht.
It's
just
an
enormous
amount
of
data
so
like
it's
either
like
spam,
the
dnc
and
like
wreck
it
or,
or
do
this
the
so
we're.
Currently,
these
are
like
five
different
ideas
or
five
different
paths.
There
are
many
more
potential
things,
but
just
to
give
you
a
sense
of
different
directions.
A
One
very
simple
thing
that
we
can
do
is
to
follow
a
model
similar
to
the
gateway
model,
where
there
are
just
a
lot
of
different
indexers
and
there's
a
registry
of
indexers.
If
there's
something
like
less
than
10
indexers
they're
big,
this
can
work
fairly
well
and
10.
Different
organizations
in
networks
tend
to
work
reasonably.
Well,
it's
not
great,
it's
not
an
ideal
solution,
but
it
can
work
for
some
period
of
time.
I
wouldn't
take
any
one
of
these
solutions
as
permanent.
A
I
would
think
of
all
these
as
kind
of
transient
systems
and
so
federated
separate
systems
could
work
fairly
well.
The
gateways
are
doing
that
today
and
they're
working
quite
well,
and
so
on.
There
was
in
the
past,
people
were
considering
building
one
gateway
out
of
multiple
gateways,
but
that
would
have
been
you
know,
kind
of
like
integrating
them.
So
that's
kind
of
like
the
second
idea,
which
is:
could
you
take
multiple
different
organizations
that
together
work
on
maintaining
one
system
for
something
like
the
gateways?
A
I
probably
wouldn't
recommend
that,
because
the
gateways
problem
is
high,
like
performance
is
like
it's
a
huge
constraint
and
the
variance
of
applications
is
very
high,
and
so
you
end
up
with
different
gateways
that
tune
for
different
use
cases.
But
that's
not
the
case
with
indexers
linux
source
is
basically
the
same
problem,
and
here
we
can
follow
a.
A
A
A
Here
where
this
is
a
threshold
cryptography
protocol
and
as
long
as
you
trust
enough
participants-
and
you
expect
them
to
not
cheat
then
this
this
matches
your
secure
model,
and
this
can
work,
and
so
something
like
that
can
work
here
for
network
indexers
as
well,
where
you
could
have
an
integrated
system
that
works
kind
of
like
drand,
where
you
could
have
multiple
different
entities
working
together
and
maybe
assigning
portions
of
the
index.
So
this
might
be
sharding
across
parts
of
the
key
space
and
so
on.
A
A
We
could
also
explore
explore,
like
traditional
peer-to-peer,
technol
techniques
against
this
similar
to
the
dht
model.
We
could
go
into
trying
to
there's
a
lot
of
past
protocol
designs
for
things
like
this,
so
we
could
find
a
distributed
peer-to-peer
indexing
system
that
is
meant
to
run
in
this
kind
of
trustless,
but
but
working
well
enough
structure
and
arrive
at
a
good
solution.
A
Like
will
end
up
with
large
routers
in
in
a
small
number
of
locations,
it's
fairly
difficult
to
guarantee
that
this
will
work
really
well,
there
are
other
directions
dealing
with
with
blockchains,
so
once
you
have
smart
contracts
and
mechanism
design,
you
can
do
verifier
networks
where
you
can
use
blockchains
to
commit
resources
against
running
an
indexer.
You
can
do
things
like
check
that
participation
is,
is
continuing
to
work.
A
Well,
you
can
reward
participants
for
doing
this,
and
this
is
the
kind
of
thing
that
can
yield
the
hardware
build
out
that
I
was
talking
about
earlier.
You
can
get
to
content
routers
that
are
massive
scale,
localized
and
work
really
well.
If
we
have
a
proper
economic
model
for
how
you
run
these
things,
that's
one
of
the
things
that
that's
one
of
the
other
things
that
might
make
structure
peer
appear,
not
work.
A
So
you
could
do
this
as
kind
of
an
l2
l2
chain.
You
could
also
do
take
utico
the
hierarchical
consensus
tool
that
existed
a
and
just
create
like
an
entire
like
l1
chain
just
for
indexing
with
or
without
it
like.
You
could
start
it
today
without
economic
structures
and
see
kind
of
how
this
this
thing
might
work
and
then
over
time
evolve
it
to
to
have
some
incentive
structures.
So
these
are
like
five
possible
paths.
All
of
these
have
varying
degrees
of
complexity,
varying
degrees
of
of
utility
over
time.
A
It's
a
time
to
like
discuss
these
things,
because
we
could
take
many
different
paths
here
or
we
could
take
one
of
any
of
these
paths,
and
so,
if
people
have
opinions
about
the
direction,
they
should
go
definitely
come
talk
to
the
network,
indexer
team
cool.
That's
that's
it!
For
that.
Any
questions
on
this.
B
A
Yeah,
it
would
depend
on
on
the
structure
here
like
what
exactly
do
you
do
like?
What
topology
do
you
use
like?
Are
these
full
index
full
replicas
everywhere,
or
do
you
do
some
of
that?
Regioning
that
I
was
describing
earlier
today
of
like?
Do
you
split
things
up
such
that
indexers
in
particular,
regions
get
different
distribution
of
of
content?
It's
likely
that
you
want.
A
Because
also
if
you
have
a
massive
content
index
inside
of
a
data
center,
that's
just
content,
that's
relevant
in
that
data
center
and
not
really
outside
you.
Don't
necessarily
you
don't
want
those
applications
to
have
to
like
come
up
with
some
economic
model
to
move
all
of
that
data
out
into
the
rest
of
the
world
and
replicate
that
everywhere
else.
A
A
I
mean,
I
think,
I
think,
for
web
use
cases.
You
certainly
don't
need
like
the
intra
data
center
content,
routers
right
with
web
users.
You
definitely
want
content
to
be
accessible
at
least
some
fraction
of
it
globally
and
a
large
fraction
of
it
regionally
with
very,
very
high
with
very
low
latency.
A
I
don't
know
five
to
ten
petabytes
of
content,
and
so
you
would
need
five
to
ten
petabytes
of
index
everywhere
and
that's
fairly
big
like
if
it
was
a
hundred
terabytes
100
terabytes
for
petabyte.
That's
easy
to
replicate
around
the
world
so
that
one
order
magnitude
is
like
kind
of
annoying.
You
go
from.
You
know
a
box,
this
big
to
like
a
whole
rack,
and
so
that
difference
just
makes
it
makes
the
scale
go
from.
A
One
person
can
go
and
set
something
up
in
one
data
center
or
in
one
internet
exchange
to
no.
You
need
like
a
whole
team
to
operate
this
thing
long
term,
and
so
I
think,
like
the
more
we
can
end
up
with
one
box
replicated
everywhere.
That's
that
you
can
get
ten
thousand
copies
or
like
a
thousand
copies
of
and
like
no
problem
yeah,
a
thousand,
probably
maybe.
B
B
D
You
know
like
different
entities,
could
go
and
set
up
an
indexer
node
and
by
that
they
can
observe
requests
coming
in
and
at
the
same
time
they
act
as
a
retrieval
provider
where
they
have
big
enough
cash
to
catch
the
most
popular
and
then
that's
how
they
make
money.
Out
of
you
know
the
most
popular
in
their
region.
A
Yeah
you
could
a
problem.
There
is
that
you
might
get
into
this
conflict
of
interest
where
they're
supposed
to
run
this
like
indexer
and
provide
all
providers.
But
then,
if
they're
making
money
on
the
retrieval
side,
then
they
might
like
only
start
showing
that
one
you
get
into
these,
like
it's
kind
of
like
when,
when
when
us,
when
a
search
engine
also
runs
the
shopping
tool
or
whatever,
and
it's
like
oh
check
out
these
results,
click
here
and
don't
look
at
these
other
like
lower
price
items
right
and
so
like.
A
A
B
D
B
A
Well,
I
mean
you
could
have
a
model
where
you
can
use
cryptography
and
smart
contracts
in
like
a
blockchain
setting
without
needing
an
economic
model,
meaning
you
could
run
a
separate
blockchain
that
has
no
economic
incentives.
It's
run
by
this
federation
and
you
encode
in
it.
The
way
of
checking
that
behavior
is
correct,
like
you
figure
out
the
cryptographic
protocol,
part
of
it
and
you
just
use
it
as
a
way
as
a
mechanism
to
catch
misbehavior
and
slash
it.
Well,
sorry,
you
don't
necessarily
slash
economically,
but
you
can
like
have
a
reputation
system.
A
A
B
A
A
Where
there
are
these
network
indexer
cache
copies
and
they're
they're,
not
writable
at
all,
and
you
can
you
when
you
run
a
gateway
or
some
large
service,
you
can
run
one
of
these
entirely
and
stay
up
to
date
from
the
indexer.
So
content
still
has
to
go
through
the
indexer
and
then
propagate
it
down.
But
then
you
can
now
run
one
full
one
full
replica
here
t
what
would
that
be
interesting
to
you
guys.
C
So
another
moment,
especially
because
we've
mainly
focused
on
like,
but
the
content
crafting
especially,
is
like
like
we're,
trying
like
like
historically
regular
handshake,
is
like
certain
communities
clusters
and
now
like
we're,
trying
to
move
towards
operating
more
like
like
directly
into
the
pops.
Until
that
becomes
a
lot
more
prevalent
issue.