►
Description
The Provide process in the IPFS network, or in other words, publishing content in the network is very slow (in Internet terms). Through measurements we’ve found that it can be improved by an order of magnitude. In this talk we’ll present our improvement proposals together with initial results and next steps.
https://www.p2p.paris/festival
(CC BY-NC-SA 4.0)
Pour soutenir l'indépendance du festival, et sa gratuité :
https://p2p.paris/fr/donate/
_______________________
L’objectif de ce festival gratuit est de proposer 5 jours de rencontres autour de toutes les thématiques liées au Peer-to-Peer. Si la notion technologique du P2P est évidemment importante, elle est également un ensemble de valeurs, qui compose une philosophie.
Les valeurs du Peer-to-Peer peuvent s’appliquer à un très grand nombre de thématiques, comme la sécurité des communications, la protection de la vie privée, le partage de savoirs, mais aussi dans les arts, ou encore les échanges monétaires.
Le Paris P2P Festival se veut une occasion de fédérer et permettre la rencontre des acteurs du Peer-2-Peer sous toutes ses formes.
Toutes organisations portant des projets, tech ou non tech, se reconnaissant de ces valeurs sont donc les bienvenues à se joindre à cet événement pour y organiser leur propre rencontre, y poser un stand, ou organiser un hackathon.
Cet événement est le “prolongement” des rendez-vous mensuels de Paris P2P, tous les premiers mercredi du mois à Ground Control. Il représente l’ouverture d’un réseau de personnes, collectifs, associations, entreprises (…) souhaitant profiter ensemble d’un événement par et pour sa communauté, sans but lucratif.
A
All
right
cool
thanks
everyone
for
joining
my
talk.
I
really
appreciate
that
you
have
interest
in
my
research,
so
the
talk
is
titled
optimistic,
provide
making
the
ipfs
dht
provide
process
in
order
of
magnitude
faster
and
I'm
very
excited
about
this
project,
because
I
don't
think
there
are
many
established
algorithms
that
have
so
much
improvement,
yeah
opportunities
as
this
one
right
by
now
and
yeah.
A
So
about
me,
I'm
yeah,
my
name
is
dennis,
as
marco
already
just
said,
I'm
I'm
a
research,
so
I'm
a
phd
student
at
a
student
at
the
university
of
gottingen
and
I'm
also
a
phd
fellowship
recipient.
You
can
reach
me
under
these
handles
on
github
or
twitter
and
I'm
yeah
feel
free
to
reach
out
after
the
talk
all
right
for
this.
A
For
this
talk,
I
think
we
need
to
get
on
the
same
page
of
how
the
provide
providing
content
in
ipfs
works,
especially
how
this
plays
together
with
the
dht
and
after
we've
done
that
I
will
just.
I
will
show
you
where
the
improvement
opportunities
lie
and
then
give
or
present
two
proposals
of
how
we
want
to
tackle
this
and
then
show
some
first
measurement
results
and
also
at
the
end,
show
you
some
next
steps
all
right.
A
So
when
we
start
at
the
beginning,
let's
imagine
you
have
a
file
called
my
file
and
you
head
to
the
command
line,
and
then
you
type
ipfs
add
my
file.
Ipfs
would
chunk
the
file
into
different
well
into
different
chunks,
construct
a
merkle,
dag
calculator,
cid
and
then
ipfs
starts
advertising
the
cid
to
the
dht,
and
this
is
so.
How
does
ipfs
actually
do
this?
I
think
to
understand
this.
We
need
a
very
brief
primer
on
distributed
hash
tables.
A
I
probably
won't
do
justice
on
this
topic,
but
I
will
give
my
best
for
every
anyone
curious.
I
highly
recommend
the
paper
as
much
also
did
yesterday.
It's,
I
think
it's
a
very
approachable,
read
but
anyways,
so
a
distributed
hash
table
is
basically
a
distributed
system,
that
maps
keys
to
values
and
in
the
case
of
ipfs
it
does
two
things
well
actually,
three,
but
two
relevant
things
for
us
right
now.
A
It
maps
c
ids
to
peer
ids
and
then
in
which
we
call
provider
records
and
the
second
one
is
it
maps
peer
ids
to
network
addresses,
so
looking
up,
content
in
ipfs
is
actually
a
two-step
process.
First,
we
look
up
the
peer
id
that
hosts
or
claims
to
host
the
files,
and
then
we
look
up
their
network
addresses
to
actually
be
able
to
reach
it
and
in
the
case
of
academia,
one.
So.
A
It
holds
peers
with
specific
distance
units
of
multiples
of
two
in
locally,
and
by
doing
this
we
can,
with
every
iteration
on
looking
up
content.
We
can
actually
go
and
get
closer
by
a
factor
of
two
and
half
the
half
the
distance
in
every
iteration
and
yeah.
We
don't
only
store
one
pier
for
each
distance
unit,
but
also
yeah
a
couple
of
peers
because
of
current-
and
I
will
get
to
this
in
a
second
as
well
and
with
these
system
parameters
and
also
network
parameters.
A
These
parameters
drive
the
routing
table
and
the
lookup
algorithm,
and
the
lookup
algorithm
in
ipfs
is
actually
in
our
case.
So
for
this
providing
content
use
case,
we
want
to
use.
We
want
to
find
the
closest
period
to
a
specific
cid
and
it
starts
with
we
load
the
20
closest
peers
from
our
routing
table.
To
I
would
call
it
a
query
queue.
A
A
We
ask
that
one
order
it
into
our
query
queue
and
we
continue
until
the
closest
known
three
peers
have
been
successfully
queried
and
then
we
terminate-
and
so
that's
the
gist
of
it.
Basically-
and
this
is
just
a
this-
is
how
such
a
process
could
look
like.
So
we
have
the
time
on
the
x-axis
in
seconds
and
on
the
y-axis,
the
different
peers
that
have
we
have
contacted
in
in
such
an
operation
and
well,
as
you
can
see,
it's
such
it's
quite
an
involved
process.
A
So
we
have
deployed
some
ipfs
nodes
all
around
the
world
in
different
aws
regions,
and
we
have
instructed
those
nodes
to
just
generate
random
content
and
provide
it
to
the
network
and
then
just
measured
the
time
that
it
took
well
to
yeah
to
provide
this
content
to
advertise.
This
content
to
or
the
cids
to
the
network.
A
I
think
more
approach
more
clear
is
the
table
on
the
right
hand,
side.
We
can
see
that
in
so
the
50th
percentile
is
at
around
30
seconds.
So
in
50
of
the
cases
it
took
around
30
seconds
from
from
all
from
all
regions
to
provide
content,
and
it
gets
even
worse,
obviously,
for
the
19th
and
95th
percentile.
A
Up
to.
I
don't
know
two
minutes
almost
well
even
over
two
minutes-
oops,
sorry
and
this.
So
there
are
a
couple
of
reasons.
Why
that's
the
case?
I
think
one
of
the
most
important
reason
is
no
churn
so
that
in
this
routing,
so
we
are
getting
results
from
remote
peers
from
their
routing
tables,
and
it
could
be
the
case
that
they
have
stale
entries
because
nodes
churn
in
the
network.
Here
three
graphs
that
show
the
node
churn:
let's
just
look
at
the
left
hand
yeah
of
the
at
the
leftmost
graph.
A
It
shows
the
uptime
and
hours
and
then
the
percentage
of
peers
that
actually
stay
that
long
in
the
in
the
network,
and
we
so
on
the
left
hand,
is
in
the
left
graph.
You
can
see
that
after
two
hours,
almost
50
of
the
peers
that
have
joined
the
network
have
actually
already
left
it,
and
I
mean
kademne
prefers
or
favors
nodes
that
are
quite
stable
in
the
network,
but
anyways,
and
we
will
definitely
get
stale
records
from
from
remote
peers
during
the
lockup
process
that
that
slows
down
the
whole
provide
operation.
A
A
But
if
we
just
have
a
look
at
how
long
it
actually
took
to
discover
those
peers,
so
we
went
through
the
whole
process.
Looking
up
the
closest
peers,
then
stored
the
provider
records,
and
if
we
just
take
a
look,
how
long
it
took
to
discover
those
peers,
we
can
see
that
it
it's
almost
so
so
in
over
90
of
the
cases
we
discovered
those
peers
in
under
one
second,
so
we
have
again
the
time
on
the
x-axis
and
then
the
cdf
on
the
on
the
y-axis
yeah.
A
Well,
we
see
that
we
actually
don't
need
to
go
through
the
whole
process,
which
takes
a
minute
or
two
and
to
really
be
sure,
to
have
found
the
closest
peers.
We
could
have
just
actually
finished
after
let's
say
a
second
or
half
a
second
in
over
90
of
the
cases,
and
so
so
this
these
were
measurements
in
the
past,
and
so
the
takeaways
here
are
well.
A
This
is
actually
some
idea
from
pittar
one
of
the
original
authors
from
the
kademya
paper
and
then
a
second
one
is
just
that.
We
estimate
how
suitable
appears
during
the
provide
process
and
well.
We
estimate
the
suitability
of
their
closeness
and
to
store
the
to
hold
the
provider
records
based
on
the
network,
size
and
yeah.
If
we
start
with
the
first
approach,
oh
well
with
the
first
improvement
proposal,
the
idea
is
that
we
start
multiple
lookup
procedures.
In
parallel,
they
are
independent,
so
they
don't
interfere
with
each
other.
A
We
are
running
them
concurrently
and
the
crucial
thing
is
that
they
start
from
a
different
set
of
seed
peers.
So
on
the
bottom
right
you
can
see
the
routing
table
and
the
idea
would
be
that
we
just
take
the
20
closest
peers
from
our
routing
table
and
the
20
cl
further
spears
and
run
the
lookup
process,
a
procedure
in
parallel
and
just
stop
as
soon
as
both
query
queues
have
an
intersection
of,
let's
say,
10
peers.
A
Then
we
abort
both
queries,
choose
those
10
peers
to
hold
the
provider
record
and
call
it
at
a
and
so
yeah.
So
that's
the
first
idea.
The
second
one
would
be
a
proximity
estimation
at
the
bottom.
The
horizontal
line
should
is
the
address
space
and,
let's
imagine
you
want
to
provide
a
cid
which
falls
somewhere
on
this
line.
A
A
How
likely
is
it.
Is
it
that
there's
another
pier,
that's
even
closer
than
the
one
that
we
have
just
discovered
and
well
in
this
case
it's
quite
unlikely.
It's
it's
quite
sparse,
but
if
the
network
is
a
little
bit
denser,
if
the
network
size
is
a
little
bit
higher,
we
would
probably
need
to
have
another
hop
to
to
be
sure
that
there
we
won't
probably
won't
find
anyone
closer,
and
this
improvement
proposal
will
include
two
estimators.
A
The
first
one
is
in
network
size,
so
we
cannot
so
right
now,
when
you're,
just
participant
of
the
network,
you
don't
know
how
large
the
network
is,
and
the
second
one
is
just
an
estimator
for
the
proximity
and
yeah
define
the
threshold.
What's
the
threshold
after
which
we
can
say
well,
we
probably
won't
find
anyone
closer
than
the
pier.
We
have
just
discovered
all
right.
Some
measurement
results
on
both
approaches.
A
So
imagine
we
have
finished
the
lookup
procedure
and
if
we
take
the
the
the
set
of
closest
the
20
closest
peers
and
take
the
average
distance
of
those
20
peers
to
the
cid
that
we
actually
want
to
to
provide
and
do
yeah-
and
do
I
don't
know
without
here
in
this
case
a
thousand
provide
operations
and
graph
them
in
this
particular
way.
So
we
have
the
distance
on
the
x-axis
and
again
the
percentage
or
cdf
on
the
y-axis.
A
We
can
see
that
in
the
status
quo,
so
in
the
current
implementation,
the
distances
are
away
lower
than
in
this
multi-query
approach.
So
it's
it
operates
in
the
order
of
0.06
norm,
xor
percent-
I
won't
go
into
that
right
now,
but
in
you
can
see
that
in
the
multi-query
approach,
it's
in
the
order
of
two
or
four
or
six
six
percent,
which
is
an
indication
that
these
queries
intersect
too
early
and
that
the
distances
of
the
peers
that
we
have
discovered
or
have
chosen
are
too
far
away
and
that
they
are
more.
A
A
So
in
the
literature
you
find
a
lot
of
approaches
where
you
gossip
around
you
ask
your
peers,
what
their
view
of
the
network
is
and
how
large
they
think
the
network
is,
and
this
would
incur
a
networking
cost
and
it
wanted
to
avoid
this
and
so
there's
an
approach
which
avoids
all
of
that,
and
I
highly
recommend
also
reading
this.
I
implemented
this,
and
so
this
is
this.
A
This
is
the
data
of
an
ipfs
node
that
is
just
running
on
my
local
machine
and
it's
just
listening
into
the
network
and
based
on
on
some
operations
that
the
ipfs
node
is
doing
anyways.
It
can
infer
kind
of
the
network
size
here,
so
the
blue
light
so
on
the
x
axis
is
just
the
time
at
the
time
of
day
and
on
the
y
axis
is
the
estimated
network
size.
A
The
blue
line
is
the
average,
and
then
we
have
the
first
one
standard
deviation,
two
standard
deviations
and
three
in
the
shaded
areas,
and
if
you
compare
this
with
the
numbers
that
the
that
the
network
crawl
gives
us,
it
kind
of
fits
nicely.
At
least
the
the
order
of
magnitude
is
right,
and
I
think
it's
a
pretty
pretty
good
estimate.
A
The
next
step
would
be.
Is
the
estimate
good
enough?
This
is
one
of
the
next
steps.
I
think
this
is
a
very
promising
result
you
can
at
the
bottom.
You
can
also
see
so.
The
the
green
line
is
the
total
number
of
nodes
we
have
found
in
the
network
during
the
times
of
the
days.
It's
also,
it
revolves
around
two
twenty
two
thousand
twenty
thousand
and
it's
in
in
the
same
ballpark
as
the
local
estimate
above,
which
is,
I
think,
a
very
promising
result
already.
A
A
First
result:
it's
it's
it's
preliminary
and
the
next
steps
would
be
just
to
experiment
with
all
the
parameters
that
we
that
we
have
all
the
all
the
switches
that
we
can
tweak
to
just
see
their
impact
on
the
on
the
closeness
and
so
on.
So,
for
instance,
for
the
multi-query
approach.
We
can
run
three
queries
instead
of
two
or
we
can
increase
the
size
of
the
intersection
and
for
the
proximity
proximity
estimator.
We
can
just.
A
We
can
improve
up
upon
the
network
size
estimate
with
with
some
other
parameters,
and
so
the
next
step
would
be
to
evaluate
the
network.
Size
estimate
across
a
fleet
of
ipfs
nodes
from
in
different
ph
spaces,
address
spaces
locations
for
in
the
address
space
and
in
general.
What
I've
not
shown
right
now
are
the
actual
publications
publication
durations,
which
is
actually
what
matters
in
the
end
and
yeah.
These
are
all
the
next
steps,
there's
a
repository
that
you
can
look
all
these
graphs
up
and
all
the
information
and
well.
C
C
A
A
One
of
the
steps
would
be
to
just
to
run
it
a
couple
of
thousand
times
and
see
what
the
lookup
looks
like
if,
if
there
are
actually
occasions
where
we
miss
missed
the
cid
and
also
when
you
look
up
content
well,
you
you
have
also
parallel
queries
and
you're
acquiring
a
lot
of
peers
for
for
these
provider
records.
So
it's
actually
quite
likely
that
you
will
will
find
it
anyways.
Although
we
haven't
stored
the
provider
records
with
the
closest
ones.
D
Hey
dennis
thanks
for
the
for
talk.
One
thing
that
I
was
wondering
is
when
you
are
doing
the
thing
where
you
start
two
queries
on
different
points
and
you
find
an
intersection
of
10
and
you
go
for
those
10
right.
Why
don't
go
for
the
30
exactly
yeah?
If
you
go
for
the
30
like
the
intersection,
should
be
oops
yeah.
A
E
F
E
A
G
A
So
I
suspect,
as
the
network
grows,
the
impact
gets
lower,
which
is
a
good
thing,
but
I'm
not
100
sure
about
this
yeah.
So
this
this
may
be
already
the
answer.
So
I
don't
I
don't
really.
I
cannot
really
tell
what
the
impact
of
the
network
likes
is
right.
Now,
it's
just
my
intuition
that
it
gets
that
the
estimator
gets
more
tolerant
to
network
size,
estimate
errors.
The
larger
the
network
is,
but
I
cannot
prove
it
right
now
with
any
any
graph.
It's
just
my
the
intuition.
F
Hi
thanks
for
the
talk,
I
know
you
can't
necessarily
prove
or
like
test
out
with
larger
nodes
in
the
network
or
like
a
larger
amount
of
nodes
in
the
network.
But
have
you
tried
this
estimator
algorithm
with
nodes
or
a
smaller
amount
of
nodes
in
that
work
and
see?
If
that
it
holds
for
true
for
smaller
networks,.
A
No,
I
haven't
done
this
yet
and
I
I'm
also
not
sure
how
to
do
it.
It
could
probably
work
if
I
simulate
the
network
and
test
ground,
maybe
but
yeah
and
like
the
so
how
these,
how
this
estimate
operates
in
the
limits
is
yeah
one
of
the
next
step.
As
I
also
just
said,
we
need
to
look
how
this
network
size
estimate
performs
when
the
network
grows,
but
also
when
the
network
shrinks,
so
at
which
point
it
gets.
A
Okay,
that's
a
good
point,
so
all
of
this
is
tested
in
production
and
one
of
the
the
cool
things
about
this.
The
both
of
these
approaches
is
that
they
don't
need
a
network
upgrade
by
any
means
known
system,
parameter
network
parameter
update,
but
it's
just
users
could
update
their
clients
and
benefit
from
new
functionality
and
new
performance,
and
so
so
this
is
already
tested
with
in
production.
If
that
was
your
question
yeah,
I.
H
A
Well,
okay,
so
then,
so
one
inaccuracy
that
we
buy
into
with
at
least
the
estimated
approach
that
could
be
that
we
are
storing
provider
records
with
more
than
the
desired
20
peers.
It
could
be
that
we
find
a
peer
store,
the
provider
records,
but
then
within
the
next
lookup
iteration
we
find
20
even
closer
ones,
and
then
we
use
store
with
store
the
parieta
records
with
them.
A
So
one
impact
could
be
that
we
have
a
slight
increase
of
provider
records
and
just
store
yeah
a
little
more
provider
records
than
is
actually
desirable,
but
it
could
also
be
the
other
way
around
that
if
we
tweak
the
threshold
in
the
other
direction,
that
we
could
yeah
just
store
the
product
records
with
18
peers-
and
maybe
this
would
in
the
in
the
mean,
would
even
out
so
this
yeah
could
be
considered.
Yeah
needs
to
be
considered
as
well.
Yeah.