►
From YouTube: DHT Routing Table Health - @guissou - Measuring IPFS
Description
DHT Routing Table Health - presented by @guissou at IPFS þing 2022 - Measuring IPFS - https://2022.ipfs-thing.io
Every DHT Server peer keeps a list of entries in its routing table in order to be able to communicate and redirect other peers to the peer that is most likely to have the required information. These entries need to be updated in order to keep up with network churn and other events. In this talk we'll describe the methodology we used to identify whether the DHT Routing Table is keeping healthy as well as results about the accuracy of entries.
The results of this study are part of RFM19, available on Github at https://github.com/protocol/network-measurements.
Slides CID: bafybeigdjyzepom74haqau7ojzay7zgwula4kss6nxoyqmrrrwwzjphf5q
A
All
right
so,
as
any
said,
we
have
a
lot
of
time
for
the
session.
We're
not
gonna
fill
the
the
whole
time,
so
there
will
be
some
some
time
for
discussion
after
so
if
you
get
lost
at
any
point,
just
shout
and
I'm
happy
to
take
it
more
slowly,
yeah
at
any
point
or
if
you
have
any
question
or
feel
free
to
interrupt
so
I'm
gonna
talk
about
the
the
state
of
the
drc,
routing
table
and
so
yeah
spoiler
alert.
A
The
the
the
health
of
the
dht
is
quite
good,
so
first
yeah,
what
is
the
academia
dht
routing
table?
So
academia
is
a
dht
which
is
just
an
overlay
network,
decentralized
and
in
an
overlay
network.
You
need
to
have
yes
some
some
peers,
so
you
need
to
know
the
contact
of
some
other
peers.
Otherwise
you
wouldn't
be
part
of
the
network
and
so
that's
the
peers.
A
So
these
peers
are
in
your
routing
table
and
so
that's
the
routing
table
in
the
dht
in
general,
and
then
academia
has
a
specific
yeah
implementation
of
the
routing
table.
So
it
keeps
appears
in
the
k,
barcad
and
sort
them
by
peer
id
sort
of
peer
id
by
short
distance,
and
each
backup
is
kept
at
20..
So
I'm
going
to
just
walk
through
an
example.
So
what
we
have
here
is
the
kalemniaki
space,
so
it's
so
in
calendar.
A
The
tree
is
not
balanced,
meaning
that
there
are
a
lot
of
nodes
that
don't
exist,
and
so
basically,
if
we
take
so
the
peer
ids
are
mapped
to
binary
identities,
which
would
be
like
this,
and
so
if
we
take
a
reference
node
and
try
to
fill
up
its
bucket
to
see
what
it's
like
so
we'll
take
the
nodes
with
the
common
prefix
of
length,
l
in
bucket
l.
So
what
do
I
mean
by
this?
A
So
eventually
we're
gonna
end
up
with
a
routing
table.
Looking
like
this,
so
we'll
have
the
the
low
id
buckets
that
are
full
or
capped
at
20
and
the
the
then
we
have
so
an
exponential
growth
from
the
high
id
bucket.
So
back
at
four
three
two
two
bucket
zero,
but
in
practice
is
going
to
be
capped
at
20
at
most
per
per
bucket.
A
So
that's
basically
how
the
routing
table
works
now,
so
in
calamia
we
have
a
replacement
policy
which
means
that
if
there
is
so
when
we
interact
with
the
network,
we
hear
about
other
peers
and
we
just
categorize
them.
Okay,
you
would
belong
to
bucket
x
and
if
the
bucket
is
not
full,
we
just
add
the
pier
to
the
bucket.
In
the
case,
the
bucket
is
full
so
as
described
in
academia
paper,
we
should
probe
so
ping.
A
All
of
the
piers
that
are
in
this
bucket
and
if
one
doesn't
answer,
we
just
replace
it
by
the
new
one
and
the
kubo
implementation.
So
it
means
that
for
the
buckets
so
as
we
have
a
lot
of
turning
ipfs
for
the
buckets
that
don't
have
a
lot
of
peers
like
bucket
four,
if
the
peers
goes
offline,
they
will
never
be
better
candidates
in
this
bucket,
because
yeah,
just
the
the
distributions,
is
very
unlikely
that
there
are
other
peers.
So
there's
going
to
be
a
still
entry
forever.
A
A
And
now
so
we
did
some
measurements
and
so
that's
kind
of
the
description
of
the
data.
So
we
use
the
nebula
crawler
built
by
dennis
to
crawl
all
of
the
appears
in
the
network.
So
the
the
nebula
crawler
gives
us
a
snapshot
in
time
of
all
of
the
peers
that
were
online
or
offline
and
the
state
of
the
routing
table.
A
So
all
of
the
peers
that
are
in
every
peers,
routing
table,
and
so
the
data
we
measure
from
is-
we
took
28
crores
over
one
week
and
it
was
in
april
this
year
so
and
the
the
methodology
for
most
of
the
measurement
is
so
we
get
the
global
snapshot
of
the
network
from
nebula.
Then
we
are
able,
with
the
list
of
peer,
that
is
the
routing
table
of
all
of
the
peers.
A
We
are
able
to
reconstruct
the
k
bucket
just
by
computing
xor
distance
between
one
peer
identity
and
the
identity
of
all
of
the
peers
in
those
buckets
and
from
the
global
snapshot.
So
we
know
all
of
the
peers
that
are
connected
in
the
ipfs
network
and
we
can
compute
the
x
closest
peers
to
any
point
to
see.
A
A
But
yeah
it's
a
tricky
topic
and
I'm
happy
to
talk
more
about
it.
If
you
wanna
chat
so
now
to
the
results,
so
we
just
so
yeah,
basically
in
each
of
the
k
bucket
of
all
of
the
peers.
We
just
wanted
to
measure
whether
the
the
records
there
were
still
records
or
if
the
node
was
actually
online,
and
what
we
could
observe
is
that
there
is
a
very
low
number.
So
in
the
ipfs
network.
A
Most
of
the
nodes
are,
if
yeah,
the
very
high
share
of
the
nodes
used
are
based
on
kubo,
which
means
that
they
have
the
replacement
policy
that
they're
not
going
to
keep
the
offline
peers
in
the
routing
table.
And
so
that's
why
we
have
such
a
low
ratio
of
unreachable
peers
in
the
in
the
k
bucket,
and
we
can
see
there's
a
slight
difference
between
the
low
id
buckets
so
zero
to
8
and
higher
id
buckets
so
starting
from
9,
and
I
think
that's
because
bucket
0
to
8
are
always
full.
A
A
So
on
the
on
the
x
axis,
we
have
the
bucket
id
and
on
the
y
y-axis
we
have
the
number
of
peers
that
are
in
this
bucket.
So,
as
we
can
see,
all
buckets
so
even
up
to
nine
are
full,
or
at
least
we'll
have
candidates.
So
the
orange
part,
the
missing
piece
per
bucket
is
in
the
case
I
don't
know,
a
pier
has
only
19
candidates
or
19
peers
in
its
bucket
number
one,
and
there
would
be
one
other
that
exists
in
the
network,
but
it
is
not
included.
A
A
A
A
So,
for
instance,
if
I
take
bucket
zero
and
all
of
the
network
has
the
ex
the
exact
same
peers
in
the
bucket
zero,
we
don't
have
a
high
diversity,
but
if
all
of
the
peers
are
present
in
one
another
another
bucket
zero,
we
will
have
a
high
diversity.
So
a
high
number
of
distinct
peers,
let's
say
per
bucket
globally.
A
And
so
yeah
in
canadia,
the
live
peers
will
never
get
replaced
and
it's
by
design
and
so
yeah.
Eventually,
the
buckets
that
have
a
lot
of
candidates
so
zero
and
one
the
unstable,
pier,
will
go
offline
and
get
replaced,
whereas
the
very
stable
peers
will
still
be
online
and
so
will
never
get
replaced.
A
And
so
it's
a
bit
of
a
concern
because
for
content,
far
away
you're
gonna
go
through
your
bucket
zero
and
if
it's
only
a
small
set
of
peers,
you
will
so
it's
bad
for
decentralization
and
you.
They
will
also
be
overloaded
because
they
will
be
in
the
very
stable
pr
are
gonna,
be
in
every
other
peers
bucket
and
so
yeah
just
going
through
a
quick
example.
So
we
can
see
the
colors
better
than
yesterday.
A
So
if
we
take
so
in
kind
of
yeah
bluish,
we
have
the
very
stable
nodes
in
yellow.
We
have
the
nodes
that
are
stable
enough
and
we
have
unstable
node
in
orange
and
so
that's
kind
of
the
state
of
bucket
zero.
So
we
have
four
sorry,
six
different
peers
and
at
the
beginning
it's
quite
balanced.
We
have
some
peers
that
are
very
stable.
A
So
they
will
fill
it
with
only
very
stable
piers,
and
so
all
of
the
very
stable
piers
will
be
connected
strongly
connected
to
each
other
and,
as
we
expect
those
very
stable
peers
to
be,
for
instance,
bootstrap,
peers
or
hydra
nodes.
They
are
very
important
for
the
network,
and
so
when
new
peers
that
could
be
stable
enough
or
unstable
join
the
network,
they
will
ask
the
bootstrap
peers
to
help
them
fill
the
buckets
and
so
eventually
we're
going
to
end
up
with
very
simple
peers
in
all
of
the
nodes
buckets
with
low
id
and
yeah.
A
So
that's
a
bit
concerning
because
of
load
balancing,
and
so
I
think
yeah.
So
we
measured
it
a
bit.
So
it
was
a
different
measurement,
so
it
was
so
yeah,
let's
say
one
column
per
week,
so
yeah
ten
points
of
data
one
for
each
week
and
at
the
the
measurements
were
based
on
two
crawls
per
day.
So
yeah.
Basically,
each
column
is
the
data
from
14
crores
and
we
measure
the
diversity
as
the
number
of
distinct
individuals
that
appear
in
each
k
bucket.
A
So
if
I'm
in,
for
instance,
bucket
zero,
it's
going
to
be
bucket
zero
of
all
of
the
peers
and
I'm
going
to
count
the
number
of
distinct
period
appear
there.
And
so,
while
we
measure
okay,
so
this
graph
has
a
lot
of
information.
But
just
if
we
take
the
blue
column
so
week
zero,
we
can
see
that
the
diversity
so
the
the
yeah,
the
number
of
distinct
peers
or
the
y-axis.
The
numbers
are
not
very
important
because
they
could
have
been
different
with
a
different
scale.
A
So
if
I
measured
over
two
weeks
or
over
five
weeks,
but
what
is
important
is
we
can
see
that
the
diversity
is
going
up
from
bucket
zero
to
bucket
nine
and
then
going
back
two
of
yeah
very
low
to
bucket
21
and
so
between
bucket
nine
and
21.
The
explanation
is
very
simple:
just
there
are
not
a
lot
of
candidates
that
fit
in
this
bucket,
and
so
that's
why
yeah?
We
don't
have
a
lot
of
diversity.
A
It's
because
we
don't
have
a
lot
of
individuals
and
the
rest
is
so
bucket
zero
and
one.
We
have
really
a
lot
of
candidates
to
choose
from
so
we're:
gonna
choose
the
most
stable
ones
and,
for
instance,
in
bucket
eight
we're
gonna
have
way
less
candidates
in
each
bucket,
so
we're
gonna
take
just
the
one
that
are
here.
A
A
So
what
what
I
think
should
be
an
easy
fix
would
be
just
to
change
the
replacement
policy.
There
are
many
things
that
that
can
be
changed
but
just
having,
let's
say,
a
limited
lifetime
for
each
entry,
and
so
you
gotta
refresh
each
entry
after
sometimes
would
help
for
the
diversity
or
yeah.
I
mean
I'll,
keep
the
rest,
maybe
for
for
later,
but
including
a
geo
locality
component
or
just
if
you
feel
the
the
key
space
of
the
bucket
uniformly
then
yeah.
It's
other
methods
to
that
could
improve
the
replacement
policy
so
yeah.
A
To
conclude
so
we,
the
dht,
is,
is
doing
very
well
overall
and
so
yeah.
The
we're
missing
only
the
quebecers
are
missing
only
a
few
number
of
peers.
We
don't
have
a
lot
of
sale
on
trees
and
the
peer
distribution
is,
as
expected,
the
the
nodes
know
their
training
closest
peers,
most
of
them
and
so
yeah.
We
observed
the
diversity
could
decrease
over
time,
but
so
it's
not
like
an
emergency
for
now.
A
It's
something
that
we
can
correct
and
anyway,
it's
not
critical
for
now,
but
yeah,
that's
a
path
of
improvement,
and
so
all
of
the
results
are
available
on
the
network
measurement
repo,
where
we
have
all
of
our
rfms
and
so
yeah.
That's
some
references
so
I'll
also
share
the
slide
with
the
clickable
links,
so
yeah.
B
A
I
think
it
shouldn't
change
much
or
so
yeah,
so
we
ex
I'm
not
very
familiar
with
the
hydra
implementation.
So
how
does
it
work
for
each
head
of
the
hydra?
So
there's
a
prit
and
what
about
the
routing
table?
Is
it
share
for
all
heads
or
does
each
head
have
its
own
routing
table
sure
sure,
okay
yeah,
so
I
think
the
hydra
node
would
would
help
the
other
nodes
to
at
least
yeah
populate
the
routing
table.
So
just
I
mean
the
writing.
A
C
One
question
I
have
is
if
we
apply
some
optimization
to
increase
your
diversity
yep.
Yes,
what
would
be
a
good
way
to?
Obviously
we
can
have
a
metric
that
would
show
the
diversity,
so
that
would
change
be
able
to
see
whether
it's
successful
or
not.
But
basically,
how
do
we
evaluate
the
performance
of
you
know
those
nodes
in
the
network
as
in
performances
and
finding
other
appears
retrieving
phone
content
and
so
on.
C
A
C
A
A
So
what
would
be
the
kind
of
like
goal?
You're
looking
for
to
move
like
for
the
health
of
the
dhc,
like
actually
being
more
diverse,
you'd,
be
like
more
performing
because,
like
my
understanding
is
like
the
stable
tears
and
I
get
one
to
look
for
stable
peers,
which
means
it's
likely
to
be
performing
than
reliable.
B
A
I
think
it's
a
trade-off
so
as
we
have
20
peers
per
bucket,
so
we
could
have,
I
don't
know
12
out
of
the
20
peers.
That
would
be
very
stable,
so
you
know
that
there
is
some
performance
that
is
guaranteed
and
so
the
way
you
select
very
stable
peers.
So
now
it's
just
if
appears
goes
offline,
it's
not
going
to
be
included
anymore,
and
so
maybe
you
just
set
a
threshold.
A
Okay,
this
period
is
stable
enough
for
performance
wise
and
set
a
threshold
and
have
like
I
don't
know
like
16
or
12
of
those
peers
matching
this
performance
and
the
rest
could
be
like
random
pierce
just
to
have
more
diversity
so
that
the
performance
wouldn't
be
impacted
so
much
and
we
have
yeah
more
decentralized,
so
more
load
balancing
and
I
think,
even
so
another
easy
improvement
or
yeah.
I
don't
know,
but
updating
the
the
replacement
policies.
A
A
Keys,
but
if
we
say
we
take
three
keys,
then
the
three
of
them
could
belong
to
the
budget
zero,
and
so
this
node
will
know
no
one
in
this
part
of
the
graph,
and
so
what
we
could
try,
also
to
having
a
replacement
policy
is.
If
there
is
a
gap,
then
we
could.
So
if
the
nodes
learn
about
this
one,
but
the
table
is
already
full
and
those
nodes
are
still
outlined.
So
it's
gonna
ignore
this
node.
But
what
we
can
do
is
say:
okay
here
we
know
already
a
lot
of
them.
A
We're
gonna
evict
this
one
because
they
are
not
closing
up
and
take
this
one,
because
it
will
help
us
to
reach
this
part
of
the
tree
so
yeah
that
could
be
one
problem
and
another
component
could
be
just
latins.
So
we
want
to
have
those
below
that
and
see
in
the
routing
table
so
combining
latency,
let's
say:
yeah
distribution
in
the
key
space
and
stability,
I
think,
can
reach
maybe
an
optimal
replacement
policy
or
yeah,
at
least
better
than
that.
A
B
And
also,
could
you
go
to
one
of
the
earlier
graphs
where
you
showed
the
missing
piece
yeah
that
one
yeah
did
you
consider
for
the
missing
piece?
Also
the
ones
or
did
you
exclude
the
ones
that
were
unreachable
during
the
crawls
I
mean
you
could
find
it
here
that
would
fit
in
the
bucket,
but
it's
not
reachable
in
this
scroll.
B
C
A
A
Yeah,
so
we
could
just
yeah
benchmark
the
replacement
policy
are
also
what
we
can
do
is
compute
the
perfect
routing
table.
So
as
we
have
a
snapshot
of
the
network
which
we
could
just
compute
at
each
step,
the
theoretical
best
writing
table
and
compute
the
difference
between
what
we
have
now
and
the
perfect
one
and
then
maybe
model
over
time.