►
From YouTube: Decentralization Paths for Network Indexers - @jbenet - Content Routing 1: Performance
Description
Decentralization Paths Network Indexers - presented by @jbenet at IPFS þing 2022 - Content Routing 1: Performance - https://2022.ipfs-thing.io
A
I'm
going
to
talk
about
decentralizing
paths
for
network
indexers.
This
follows
some
talks
about
the
network
indexers
in
general,
and
the
current
architecture
is
a
centralized
system,
so
the
network
indexers
that
we're
describing
coming
in
come
in
as
one
additional
content
writing
system
for
ipfs
networks.
There
are
many
existing
content
writing
systems
out
there.
This
is
probably
not
even
a
full
mapping
of
these.
This
is
just
kind
of
the
the
the
widest
wettest
used
ones,
so
the
public
hd
is
by
far
the
most
common
content
routing
system.
A
There
are
also
automatic,
like
local
area
network
dhcs
that
form.
So,
if
you're,
if
your
regular
most
kubo
and
a
couple
other
implementations
do
this,
where
they
disconnect
from
the
rest
of
the
world,
they
will
form
automatic
dhts
in
the
lands
and
so
on.
So
those
already
exist.
Many
private
networks
out
there
exist
today
that
run
their
own
separate
dhts.
A
There's
the
hydro
boosters
that
you
heard
about
and
there's
been
other
talks
about
them
in
the
past
that
come
in
and
support
the
public
dht.
So
that's
a
it's,
not
a
separate
content,
routing
system,
but
it's
a
subsystem
that
ends
up
helping
the
public
thc.
A
There's
a
bit
swap
content,
routing
tool
which
there's
there's
a
frequent
the
ways
of
enhancing
bitswap,
but
in
sort
of
like
hack
content,
routing
in
a
sense
where
you
can
use
once
and
so
on,
within
bitswap
itself
to
just
optimistically
end
up
in
a
spot
where
you're
likely
to
to
already
be
connected
to
peers
that
have
the
content
or
they're
connected
to
somebody
who
has
a
content.
A
So
the
way
ways
to
kind
of
hack
that
and
there's
been
many
people
that
have
tried
things
like
that
and
then
there's
just
local
dna
mdns
discovery.
So
many
ipfs
implementations
have
ways
of
connecting
in
local
area
networks
using
standard
protocols
and
once
nodes
find
each
other
and
connect
to
each
other
they'll
automatically
bitswap
content
to
each
other.
A
So
there's
kind
of
this
implicit
content,
routing
discovery
loop,
where
nodes
that
happen
to
be
in
the
same
network
will
happen
to
connect
to
each
other
once
they're
connected
to
each
other
content
requests
will
just
work
but
there's
kind
of
an
implicit
content
routing
system
there.
It's
not
an
explicit
thing
that
you
you
enable
now
the
start
of
the
hash
indexer
you
just
heard
about,
and
just
one
additional
content
writing
system.
There's
gonna
be
many
more
so
over
time.
A
The
community
is
gonna
experiment
with
a
lot
of
tools
and
I'm
just
gonna
kind
of
go
through
a
few
and
by
the
way,
all
these
are
kind
of
all.
These
systems
are
optional
and
different
groups
connect
to
different
systems,
and
they
have
different
reasons
for
using
them.
I
think
we're
still
working
on
how
to
build
composability
into
this
system
like
how
could
you
compose
these
into
a
thing
that
just
kind
of
works
for
most
use
cases?
A
This
tends
to
be
it's
not
clear
that
there
is
a
just
work
solution,
because
there's
a
lot
of
security
considerations
and
authentication
considerations
in
terms
of
where
you
want
to
ship
your
reader,
your
queries
to
so
as
we'll
see
tomorrow,
there's
a
lot
of
privacy
constraints
here,
but
anyway
it
nevertheless,
when
shifting
a
lot
of
content
into
into
one
indexer
that
currently
centralized
it'd
be
good
to
kind
of
describe.
What
are
the
paths
that
we're
considering
for
turning
this
into
a
different,
centralized
network?
A
One
other
thing
worth
noting,
like
the
amount
of
data
that
store
the
hash,
is
storing
now
just
can't
go
into
the
dht.
It's
just
an
enormous
amount
of
data
so
like
it's
either
like
spam,
the
dnc
and
like
wreck
it
or,
or
do
this
the
so
we're.
Currently,
these
are
like
five
different
ideas
or
five
different
paths.
There
are
many
more
potential
things,
but
just
to
give
you
a
sense
of
different
directions.
A
One
very
simple
thing
that
we
can
do
is
to
follow
a
model
similar
to
the
gateway
model,
where
there
are
just
a
lot
of
different
indexers
and
there's
a
registry
of
indexers.
If
there's
something
like
less
than
10
indexers
they're
big,
this
can
work
fairly
well
and
10.
Different
organizations
in
networks
tend
to
work
reasonably.
Well,
it's
not
great,
it's
not
an
ideal
solution,
but
it
can
work
for
some
period
of
time.
I
wouldn't
take
any
one
of
these
solutions
as
permanent.
A
I
would
think
of
all
these
as
kind
of
transient
systems
and
so
federated
separate
systems
could
work
fairly
well.
The
gateways
are
doing
that
today
and
they're
working
quite
well,
and
so
on.
There
was
in
the
past,
people
were
considering
building
one
gateway
out
of
multiple
gateways,
but
that
would
have
been
you
know,
kind
of
like
integrating
them.
So
that's
kind
of
like
the
second
idea,
which
is:
could
you
take
multiple
different
organizations
that
together
work
on
maintaining
one
system
for
something
like
the
gateways?
A
I
probably
wouldn't
recommend
that,
because
the
gateways
problem
is
high,
like
performance
is
like
it's
a
huge
constraint
and
the
variance
of
applications
is
very
high,
and
so
you
end
up
with
different
gateways
that
tune
for
different
use
cases.
But
that's
not
the
case
with
indexers
linux
source
is
basically
the
same
problem,
and
here
we
can
follow
a.
A
We
could
take
a
path
similar
to
the
dram
network,
so
the
dron
network
is
a
a
a
blockchain
randomness
beacon
network
that
that
forms
a
that
emits
a
new
bit
of
randomness
every
30
seconds.
No
every
10
seconds
now,
every
second
now
three
seconds.
A
Cool
so
so
the
droid
network
has
a
here
I'll
share.
A
So
that
you
can
look
at
something,
so
the
newer
network
is
a
distributed.
Randomness
beacon,
it's
built
with
with
the
participation
of
many
organizations
there
are.
A
There
is
one
set
of
large
deployments
of
of
the
drain
system
called
the
league
of
entropy,
and
so
there's
a
number
of
organizations
that
together
run
the
league
of
entropy
and
the
league
of
entropy
is
one
that
deployment
has
multiple
different
networks
that
you
can
connect
to
that
emit
randomness
at
different
rates,
and
so
as
long
as
and
so
there's
some
security
properties
embedded
in
this
protocol
and
as
long
as
there's
some
sort
of
trust
threshold.
A
Here
where
this
is
a
threshold
cryptography
protocol
and
as
long
as
you
trust
enough
participants-
and
you
expect
them
to
not
cheat
then
this
this
matches
your
secure
model,
and
this
can
work,
and
so
something
like
that
can
work
here
for
network
indexers
as
well,
where
you
could
have
an
integrated
system
that
works
kind
of
like
drand,
where
you
could
have
multiple
different
entities
working
together
and
maybe
assigning
portions
of
the
index.
So
this
might
be
sharding
across
parts
of
the
key
space
and
so
on.
A
A
We
could
also
explore
explore,
like
traditional
peer-to-peer,
technol
techniques
against
this
similar
to
the
dht
model.
We
could
go
into
trying
to
there's
a
lot
of
past
protocol
designs
for
things
like
this,
so
we
could
find
a
distributed
peer-to-peer
indexing
system
that
is
meant
to
run
in
this
kind
of
trustless,
but
but
working
well
enough
structure
and
arrive
at
a
good
solution.
A
The
problem
is
that
we've
looked
at
a
lot
of
these
kinds
of
things
over
time
and
there
are
promising
directions,
but
it's
a
pretty
hard
problem
like
you,
you
in
order
to
meet
the
scale
requirements
and
the
performance
requirements
and
knowing
the
fact
that
these
indexers
will
become
both
very
large.
A
Like
will
end
up
with
large
routers
in
in
a
small
number
of
locations,
it's
fairly
difficult
to
guarantee
that
this
will
work
really
well,
there
are
other
directions
dealing
with
with
blockchains,
so
once
you
have
smart
contracts
and
mechanism
design,
you
can
do
verifier
networks
where
you
can
use
blockchains
to
commit
resources
against
running
an
indexer.
You
can
do
things
like
check
that
participation
is,
is
continuing
to
work.
A
Well,
you
can
reward
participants
for
doing
this,
and
this
is
the
kind
of
thing
that
can
yield
the
hardware
build
out
that
I
was
talking
about
earlier.
You
can
get
to
content
routers
that
are
massive
scale,
localized
and
work
really
well.
If
we
have
a
proper
economic
model
for
how
you
run
these
things,
that's
one
of
the
things
that
that's
one
of
the
other
things
that
might
make
structure
peer
appear,
not
work.
A
So
you
could
do
this
as
kind
of
an
l2
l2
chain.
You
could
also
do
take
utico
the
hierarchical
consensus
tool
that
existed
a
and
just
create
like
an
entire
like
l1
chain
just
for
indexing
with
or
without
it
like.
You
could
start
it
today
without
economic
structures
and
see
kind
of
how
this
this
thing
might
work
and
then
over
time
evolve
it
to
to
have
some
incentive
structures.
So
these
are
like
five
possible
paths.
All
of
these
have
varying
degrees
of
complexity,
varying
degrees
of
of
utility
over
time.
A
It's
a
time
to
like
discuss
these
things,
because
we
could
take
many
different
paths
here
or
we
could
take
one
of
any
of
these
paths,
and
so,
if
people
have
opinions
about
the
direction,
they
should
go
definitely
come
talk
to
the
network,
indexer
team
cool.
That's
that's
it!
For
that.
Any
questions
on
this.
B
B
Great
place
for
us
to
be
at
that
point
in
time
with
all
those
indexers.
A
Yeah,
it
would
depend
on
on
the
structure
here
like
what
exactly
do
you
do
like?
What
topology
do
you
use
like?
Are
these
full
index
full
replicas
everywhere,
or
do
you
do
some
of
that?
Regioning
that
I
was
describing
earlier
today
of
like?
Do
you
split
things
up
such
that
indexers
in
particular,
regions
get
different
distribution
of
of
content?
It's
likely
that
you
want.
A
You
have
one
content
set
that
you
want
to
be
available
everywhere
in
the
world
with
very
high
with
like
extremely
low
latency,
and
then
you
have
a
different
con
set,
which
is
you're,
okay,
with
like
a
that
content
being
available
in
a
set
of
regions
and
you're
willing
to
pay
for
that
being
like
super
low
latency,
but
then
you're,
okay
with
100
millisecond
or
you
know
one
second
latency
from
everywhere
else
in
the
world,
and
so
it
kind
of
depends
on
the
latency
requirements
that
you
need
for.
A
Because
also
if
you
have
a
massive
content
index
inside
of
a
data
center,
that's
just
content,
that's
relevant
in
that
data
center
and
not
really
outside
you.
Don't
necessarily
you
don't
want
those
applications
to
have
to
like
come
up
with
some
economic
model
to
move
all
of
that
data
out
into
the
rest
of
the
world
and
replicate
that
everywhere
else.
A
A
A
I
mean,
I
think,
I
think,
for
web
use
cases.
You
certainly
don't
need
like
the
intra
data
center
content,
routers
right
with
web
users.
You
definitely
want
content
to
be
accessible
at
least
some
fraction
of
it
globally
and
a
large
fraction
of
it
regionally
with
very,
very
high
with
very
low
latency.
A
A
I
don't
know
five
to
ten
petabytes
of
content,
and
so
you
would
need
five
to
ten
petabytes
of
index
everywhere
and
that's
fairly
big
like
if
it
was
a
hundred
terabytes
100
terabytes
for
petabyte.
That's
easy
to
replicate
around
the
world
so
that
one
order
magnitude
is
like
kind
of
annoying.
You
go
from.
You
know
a
box,
this
big
to
like
a
whole
rack,
and
so
that
difference
just
makes
it
makes
the
scale
go
from.
A
One
person
can
go
and
set
something
up
in
one
data
center
or
in
one
internet
exchange
to
no.
You
need
like
a
whole
team
to
operate
this
thing
long
term,
and
so
I
think,
like
the
more
we
can
end
up
with
one
box
replicated
everywhere.
That's
that
you
can
get
ten
thousand
copies
or
like
a
thousand
copies
of
and
like
no
problem
yeah,
a
thousand,
probably
maybe.
B
B
D
You
know
like
different
entities,
could
go
and
set
up
an
indexer
node
and
by
that
they
can
observe
requests
coming
in
and
at
the
same
time
they
act
as
a
retrieval
provider
where
they
have
big
enough
cash
to
catch
the
most
popular
and
then
that's
how
they
make
money.
Out
of
you
know
the
most
popular
in
their
region.
A
Yeah
you
could
a
problem.
There
is
that
you
might
get
into
this
conflict
of
interest
where
they're
supposed
to
run
this
like
indexer
and
provide
all
providers.
But
then,
if
they're
making
money
on
the
retrieval
side,
then
they
might
like
only
start
showing
that
one
you
get
into
these,
like
it's
kind
of
like
when,
when
when
us,
when
a
search
engine
also
runs
the
shopping
tool
or
whatever,
and
it's
like
oh
check
out
these
results,
click
here
and
don't
look
at
these
other
like
lower
price
items
right
and
so
like.
A
That's,
that's
the
issue.
I
think
you
can.
You
can
turn
it
into
a
thing
that
has
that
could
still
work.
Well,
if
you
have
a
good
guarantee
about
all
the
providers
being
being
returned
and
the
right
metrics
being
gathered,
then
I
think
you
can
arrive
and
like
the
internet
today
has
good.
A
There
are
many
systems
that
like
fall
into
this
and
and
we
have
found
good
ways
of
handling
it,
but
I
do
think
that,
given
that
we
have
mechanism,
design
and
blockchains
already
and
they're
well
deployed
just
use
those
just
use
those
to
verify
that
operation
is
correct.
I.
B
D
B
A
Well,
I
mean
you
could
have
a
model
where
you
can
use
cryptography
and
smart
contracts
in
like
a
blockchain
setting
without
needing
an
economic
model,
meaning
you
could
run
a
separate
blockchain
that
has
no
economic
incentives.
It's
run
by
this
federation
and
you
encode
in
it.
The
way
of
checking
that
behavior
is
correct,
like
you
figure
out
the
cryptographic
protocol,
part
of
it
and
you
just
use
it
as
a
way
as
a
mechanism
to
catch
misbehavior
and
slash
it.
Well,
sorry,
you
don't
necessarily
slash
economically,
but
you
can
like
have
a
reputation
system.
A
A
B
A
Yeah
we
can
follow
a
model
kind
of
like
ntps
or
drams,
so
we
we,
we
based
iran's
cache
layers
on
the
ntp
structure,
where
you
have
one
set
of
authoritative
nodes
that
is
figuring
out
reality
and
then
there's
a
set
of
nodes
that
are
just
caching
and
extending.
So
you
could
have
a
model
like
this.
A
Where
there
are
these
network
indexer
cache
copies
and
they're
they're,
not
writable
at
all,
and
you
can
you
when
you
run
a
gateway
or
some
large
service,
you
can
run
one
of
these
entirely
and
stay
up
to
date
from
the
indexer.
So
content
still
has
to
go
through
the
indexer
and
then
propagate
it
down.
But
then
you
can
now
run
one
full
one
full
replica
here
t
what
would
that
be
interesting
to
you
guys.
C
So
another
moment,
especially
because
we've
mainly
focused
on
like,
but
the
content
crafting
especially,
is
like
like
we're,
trying
like
like
historically
regular
handshake,
is
like
certain
communities
clusters
and
now
like
we're,
trying
to
move
towards
operating
more
like
like
directly
into
the
pops.
Until
that
becomes
a
lot
more
prevalent
issue.
A
Yeah
yeah
yeah
sounds
good
cool,
any
any
other
thoughts.
Questions.