►
Description
Merkle-DAGs are fun, but even more when they can be used to compute an ever-evolving state (i.e. a database) collaboratively maintained by a set of peers that may come and go as they please. Join us to learn about Merkle-CRDTs, what use-cases they fit and how they power things like IPFS Cluster pinset distribution.
A
And
we're
going
to
talk
about
American
crdts
as
a
way
to
replicate
data
between
peers
in
a
peer-to-peer
Network,
but,
more
importantly
than
understanding
how
they
exactly
work.
Is
that
you
leave
this
talk
with
some
ideas
of
things
you
can
potentially
do
on
top
of
the
ipfs
stack
first
things.
First,
my
name
is
Hector
San
Juan
I
work
at
protocol
after
the
software
engineer,
together
with
some
colleagues
samoli
pedroiannis,
we
put
together
this
America
crdpt
paper
a
couple
of
years
ago.
A
A
A
In
this
case,
all
participant
replicas
of
this
table
can
write
to
the
table
at
any
time
all
replicas
can
join,
can
live
at
any
time
and
in
any
number
anything
can
happen
to
the
messages
on
the
network.
They
can
be
dropped,
they
can
be
reordered,
they
can
be
corrupted,
despite
all
of
these
things,
eventually
in
time
converge.
A
If
you
build
something
with
a
solid
foundation,
this
happens.
You
see
that
Tower
this
was
a
tower
in
Saragosa
Spain,
the
Finish
was
pretty
nice,
it
was
awesome,
it
looked
beautiful.
The
foundations
were
not
so
good.
Have
you
seen
this
Tower
before?
Has
anyone
seen
this
Tower
before?
Why
not
because
it's
not
there
anymore,
they
had
to
they
had
to
take
it
out.
It
had
to
be
dismantled.
A
A
There
we
go
the
First
Foundation
in
Sleepy
to
be
gossips
up.
As
you
know,
lipid2b
allows
me
to
grade
a
peer-to-peer,
Network
and
gossip
sap
is
an
scalable
bus,
stop
implementation
that
allows
every
period
in
the
network
to
broadcast
messages
to
others.
This
is
using
Falcon
and
is
quite
solid.
These
days.
A
The
second
Foundation
is
ibld.
Ipld
is
a
number
of
interoperable
formats
to
create
content,
addressed
blocks
and
links
between
them
to
form
Miracle
Ducks.
An
America
duck
is
like
a
tree
in
which
every
node
is
content
addressed.
So
it
is
interesting
interested
not
only
because
all
the
properties
of
content
address
data,
but
because
once
I
Traverse,
one
of
these
Miracle
ducks
and
I'm
done
with
it,
I
will
never
have
to
Traverse
it
again
because
it
cannot
change
as
in
it.
A
The
third
Foundation
is
ipfs.
Ipfs
allows
us
to
make
our
ibld
nodes
available
to
appear
to
peer
Network,
providing
them
to
other
participants
and
retrieving
them
from
other
participants
when
needed,
using
content,
routing
and
P
routing.
So
that
means
ipfs
allow
us
to
replicate
the
medical,
dacs
and
I'm
building
with
on
IPL
deep.
A
A
A
We
can
Implement
our
key
Value
Store,
rather
simply,
instead
of
having
a
single
table
where
we
are
done,
we
delete
what
we're
going
to
do
is
to
have
two
tables
and
elements
table
for
added
keys
and
a
ton
Stone
table
for
deleted
Keys.
Additionally,
everything
is
going
to
have
a
unique
ID,
and
this
way
we're
going
to
avoid
conflicts
when
multiple
replicas
add
or
delete
the
same
key
at
the
same
time
since
they
will
have
what
are
seemingly
random
IDs.
A
What
is
the
process
of
checking
if
I
key
exists?
If
a
key
has
been
added
to
my
tables,
we
check
that
the
elements
table
for
all
the
elements
that
have
that
key
and
verify
that
at
least
one
of
them
is
not
in
the
Tom
storm
table.
That
means
that
they
have
not
been
deleted,
not
that
we
may
have
several
entries
in
the
elements
table
with
the
same
key
but
different
IDs.
A
In
crdd
jargon,
this
is
an
ad
wins
observed,
removed,
set,
but
additional
since
this
is
not
a
set
by
a
key
value
store,
we
are
going
to
need
two
more
tables
one
with
the
current
value
of
a
key
and
one
with
the
current
priority
of
a
key.
So
when
we
write
a
key,
we
can
use
priority
to
determine
if
the
new
value
of
that
key
should
overwrite
what
we
already
have
and
if
priority
is
the
same,
we
can
always
resolve
the
conflict,
for
example
by
alphabetically,
sorting
the
values
but
never
randomly.
A
A
The
priority
can
be
a
number
can
be
potential
timestamp,
anything
that
helps
you
decide
which
key
should
have
priority
over
the
others.
So,
as
you
can
see,
this
is
conflict.
Free
replicas
do
their
operations.
Anyone
receiving
those
operations
can
apply
them,
regardless
of
the
order,
and
eventually
they
all
come
to
the
same
state.
They
will
answer
the
get
questions.
A
The
least
questions
in
exactly
the
same
way,
but
note
that
for
all
replicas
to
converse
it
is
important
that
they
receive
all
the
operations
on
operation
based
crdds
have
the
problem
that
normally
they
require
reliable
message
delivery
so
because
otherwise
things
don't
converge.
If
one
message
is
lost,
you
will
never
come
to
the
place
where
you
want
to
get.
A
Therefore,
if
you
don't
have
reliable
message,
delivery-
which
really
you
never
have,
you
need
to
re-implement
strategies
to
ensure
that
your
message
are
delivered.
Therefore,
you
need
to
track
which
messages
you
are
delivering.
You
have
delivered,
which
you
have
not
delivered.
Etc
things
are
getting
very
complicated.
Very
fast,
in
the
second
part
of
the
CRT
work
that
we're
doing
with
our
witchery
help
us
deal
with
that
by
taking
advantage
of
our
foundations.
A
This
is
the
America
cidd
part,
rather
than
just
broadcasting
our
crdt
operations,
as
we
will
do
in
a
normal
implementation
of
operation
crdts.
What
we're
going
to
do
is
to
add
each
operation
to
America
DAC,
embedding
the
operations
in
iple
nodes
that
get
appended
as
new
roots
of
the
deck
instead
of
broadcasting
the
operation,
what
I
can
broadcast
it's,
just
the
CID
of
the
new
root
using
pops
up.
A
A
Second,
our
operations
are
all
linked
to
the
previous
operations
in
America
that
which
means
all
the
operations
that
you
know
of.
So
when
someone
receives
the
CID,
they
have
direct
pointers
to
everything
that
came
before
that,
if
they
need
them.
Third,
we
have
a
notion
of
order
because
I'm
adding
new
routes
to
my
Miracle
DAC.
We
know
that
this
parent
came
later
as
the
children
right
because
it
happened
later.
It
must.
A
It
must
be
an
event
that
has
happened
later,
because
I
added
it
on
top
of
the
children,
the
state,
the
whole
state
is
uniquely
identified
by
this
potential
root:
CID
I'm,
fully
rebuildable
by
traversing
the
DAC
and
applying
the
operations
that
are
contained
in
it.
But,
most
importantly,
the
state
is
now
a
set
of
ipld
nodes.
So
this
is
ipfs
favorite
food,
and
now
we
have
what
is
essentially
an
ipfs
powered
State.
We
can
move
it
using
ipfs.
A
The
replica
then
remembers
the
receive
CID
as
the
new
duck
head
or
the
new
root
of
the
deck,
and
whenever
it
wants
to
issue
a
new
operation,
it
would
just
add
it
on
the
top,
oh
that
in
a
system
with
many
replicas
that
are
writing
at
the
same
time,
to
this
miracle
DAC.
This
is
not
going
to
make
a
linear
blockchain
with
one
branch
that
is
growing
on
top
of
each
other.
A
A
Because,
if
you
have
a
duck
with
many
branches,
you
can
parallelize
traversal
of
the
attack.
If
you
had
a
duck
with
one
branch,
you
have
to
go
node
by
no.
If
you
have
several
brands
at
the
moment,
I
have
10
children.
I
can
have
10
workers
processing
those
children.
So
as
long
as
it
doesn't
get
too
bad,
and
this
isn't
a
bad
percent,
it's
actually
a
speeds
things
up,
and
so,
with
these
foundations
we
have
build
the
castle.
So
the
tower
is
very
straight:
it's
solid.
A
A
Is
that
all
the
things
that
I
didn't
have
to
do
that
matter
did
I
have
to
build
a
network
stack
or
Define
a
protocol
for
my
crdp
nodes
to
communicate
between
them,
no
likely
p2b
notes
and
ipfs?
No
do
that
did
I
have
to
implement
the
scalable
strategies
to
broadcast
the
information
to
the
network.
I
have
gossip
side,
it
was
already
done
for
me.
Did
I
have
to
implement
operations
tracking
logic
to
identify
replicas
that
are
missing
operations
so
that
I
can
re
resend
them
to
them?
No
I
can
just
if
there
is
no
activity.
A
I
can
just
wrap
broadcast.
The
current
miracle
that
root
every
minute
and
I'll
be
done.
Did
I
have
to
implement
logic,
about
which
replica
is
the
best
sweetie
to
send
operations
to
another
replica
that
has
to
receive
them
now
bit
swap
figures
that
out
on
content,
routing
did
I
have
to
add
any
logic
to
re-drive
fetching
on
a
different
replica
of
the
one
out
with
copy
operations
from
doesn't
provide
them
anymore.
A
No
I
can
like
Beat,
Swap
and,
and
the
dark
Sinker
will
take
care
of
that
did.
I
need
to
implement
a
linked
data
structure
that
is
content
address.
Now
everything
ipld
out
of
the
box
do
I
need
to
care
about
how
many
replicas
there
are
and
how
to
connect
them
now.
Leave
B2B
just
connects
things,
and
you
can
start
broadcasting
on
the
network.
A
A
A
But
laughs
so
there
are
also
a
number
of
downsides
or
even
I,
wouldn't
say
issues.
But
you
have
to
be
aware,
when
you're
working
with
these
sort
of
systems,
of
of
the
trade-offs
that
you
make
and
of
course,
I
told
you
like
a
very
nice
list
of
properties
that
I
have,
and
if
you
want
something
else
you
will
have
to
start
compromising
on
those
initial
properties.
So
we
may
be
acceptable
or
not.
A
First,
it
is
difficult
to
Traverse
Miracle
Ducks
efficiently,
it's
a
such
system,
so
it
is
a
bit
more
complex
than
I
showed
so
bronen
Traverse
branches,
I'm
picking
right
the
new
heads
when
you
get
a
fetch
error
as
you're
traversing
and
ensuring
that
exploration
of
all
branches
happens
and
that
you
leave
nothing
behind.
Even
when
you
face
errors,
It's
Tricky,
it
can
be
inefficient
and
you
certainly
if
you
want
to
operate
that
scale.
You're
gonna
go
with
a
very
damp
implementation
of
these
things.
A
The
problem
is
that
the
issues
that
happen
are
hard
to
see
because
an
issue
everything
in
a
third
it
is
everything
converges.
So
the
result
may
be
perfectly
fine,
but
maybe
to
get
that
result
you're
working
100
times
more
than
you
should,
because
you
made
a
bug
in
your
code
and
and
so
that
prevents
you
from
working
at
a
scale,
it's
very
hard
to
test,
etc,
etc.
The
underlying
the
underlying
storage,
so
the
underlying
key
value
store
that
is
holding
the
tables
that
I
showed
you
before
locally
in
every
node.
A
Why
did
I
put
it
there
because
well,
there's
a
trade-off
in
in
how
he
was
working
and
where
he
went.
Obviously,
in
order
to
get
into
abstraction,
abstract
geometry,
he
had
to
drop
a
lot
of
the
things
that
he
had
before
in
his
painting.
So
I
found
that
interesting
and
I
built
a
comparison
with
that
for
the
sake
of
the
talk
anyway,
Evergreen
deck
size,
I
told
you
whereby
the
American
duck
and
we're
always
adding
new
roots
on
top
of
it.
That
means
it's
perpetually
growing.
A
A
So
they
don't
know
when
when
they
should
get
rid
of
the
underlying
Miracle
dog
and
compact,
or
anything
like
that.
Having
that
requires
building
agreements
and
building
agreement
requires
consensus
and
consensus
usually
requires
blockchain
or
knowing
the
number
of
participants
knowing
the
number
of
replicas
so
that
they
can.
You
know
you
can
set
up
proposes
and
they
can
accept
it
and
so
on.
A
So
every
application
must
decide
if
it's
worth
to
trade
those
assumptions
for
additional
functionality,
but
what
makes
America
crdt
so
special
is
precisely
having
those
assumptions.
So,
if
you
start
start
stepping
away
from
those
assumptions,
the
American
CR
diseases
start
being
less
special
and
more
like
any
other
distributed.
Key
value
store
that
you
can
build.
A
Another
issue
is
how
to
protect
against
these
honest
actors
to
illustrate
Crusaders
entering
and
sucking
Conference
standing
opal,
which
was
Christian
2
at
that
time.
So
I
will
consider
that
a
bit
dishonest.
You
can
put
a
lot
of
rules
on
what
valid
crdb
operations
that
this
is
on
on
the
crdd
operations
that
the
system
accepts.
So
you
can
say
you
can
put
a
lot
of
rules
on
what
would
be
a
valid
operation.
A
I
have
and
half
notes
not
accept
invalid's
operations,
but
there
is
no
agreement
again,
so
you
cannot
you're
gonna
have
update
rules
to
the
state
that
are
based
on
the
state
itself,
so
you
cannot
Implement
a
cryptocurrency
with
wallets
and
balances
just
using
medical
crits,
because
the
system
cannot
possibly
agree
on
what
is
the
current
balance
of
a
wallet?
Maybe
the
balance
hasn't
converged.
Therefore,
you
cannot
put
a
rule
that
an
update
to
that
balance
is.
A
A
Finally,
there's
a
question
of
latency
batching
updates
can
give
you
a
huge
performance
boost
It's,
Gonna
Save
space,
because
you
have
to
issue
less
ipld
notes.
Instead
of
one
operation
per
node,
you
can
put
nine
thousand
you're
gonna
reduce
Network
congestion
because
you
don't
have
to
broadcast
so
many
updates.
A
But
of
course
it
introduces
latency
by
the
the
time
that
you
spend
building
the
batch
and
receiving
the
operations
that
make
the
batch
will
be
a
time
that
the
other
replicas
are
not
receiving
the
keys
that
are
being
written,
etc,
etc.
Applications
that
want
really
low
latency,
so
it's
really
important
that
they
get
a
new
key
very
soon.
Perhaps
they
don't
have
time
to
wait
for
for
one
minute
for
a
batch
to
be
completed,
but
others
can
certainly
can
certainly
think
take
advantage
of
this.
A
A
This
is
actually
what
happened
when,
when
the
Turks
conquered
on
Dusty
Noble,
there
was
a
water
blockade
and
Mehmet
II
decided
to
essentially
do
that
with
his
boats
to
attack
from
the
other
side,
and
it
surprisingly
worked.
If
you
have
a
chance
to
read
about
about
the
fall
of
Constantinople,
do
because
this
is
nuts.
A
So
what
is
this
all
for
it's
a
complex
word?
There
is
good
and
evil
and
there
are
different
applications
and
there's
number
one
that
one
one
fit
all
solution.
Some
of
them
are
weirder
than
others,
but
in
the
end
there
might
be
an
application
to
which
this
is
relevant
or
partially
relevant.
The
paper
that
we
wrote
happened
in
the
first
place.
We
were
looking
how
to
painlessly.
Sync
on
the
pinset
database
in
ipfs
clusters,
using
the
goibfs
stack
so
orbit.
Db
was
already
there.
A
A
It
works
very
well,
but
rough
clusters
kind
of
operate
if
half
of
the
beers
are
down.
So
that
was
bad,
because
if
you
have
a
two
node
cluster,
it
is
going
to
operate
if
one
beer
is
down.
So
that
was
like
a
huge
pain
in
the
ass
and
I
use,
ability,
I,
usually
usability
problem,
and
every
note
that
you
wanted
to
add
had
to
be
acknowledged
by
all
all
the
other
nodes.
Every
departure
had
to
be
acknowledged
by
all
the
other
nodes.
A
If
one
node
left
it
was
errors
everywhere,
and
every
update
that
you
wanted
to
send
to
the
cluster
need
to
travel
through
the
leader.
So
not
everyone
could
ride
on
the
the
leader
could
write
at
every
time,
which
means
redirecting
everything
to
the
leader,
etc,
etc.
It
was
not.
It
was
not
really
suited
to
the
kind
of
flexible
system
that
we
wanted
to
to
do
with
cluster.
Therefore,
we
implemented
our
pin
set
sinking
layer
with
Mercury
and
it
is.
A
A
Second,
you
can
do
more,
you
can
hit
several
nodes
or
you
can
parallelize
your
requests
to
the
same
API
and
so
on
and
get
more
more
items
per
batch
if
you
want,
but
this
is
like
the
most
stupid
approach
and
the
base
number
that
I
can
give
you.
The
setup
is
highly
scalable
and
the
system
can
absorb
right
Peaks,
very
naturally,
because
you
just
right,
you
can
build
on
top
of
America
that
and
you
brought
cost
as
you
as
you
want,
and
the
system
will
sync
as
the
conditions
in
the
system
allow.