IPFS Paris P2P Festival - IPFS Track, 13 May 2022

Previous Meeting

⏯

youtube image

►

From YouTube: Storetheindex, A distributed and consistent database By Marco Munizaga @ Paris P2P Festival #1

Description

Present storetheindex, a distributed and eventually consistent database. Explain what that means, what properties it holds, and how it enables IPFS ↔ Filecoin interop.

https://www.p2p.paris/festival
(CC BY-NC-SA 4.0)

Pour soutenir l'indépendance du festival, et sa gratuité :
https://p2p.paris/fr/donate/

_______________________
L’objectif de ce festival gratuit est de proposer 5 jours de rencontres autour de toutes les thématiques liées au Peer-to-Peer. Si la notion technologique du P2P est évidemment importante, elle est également un ensemble de valeurs, qui compose une philosophie.

Les valeurs du Peer-to-Peer peuvent s’appliquer à un très grand nombre de thématiques, comme la sécurité des communications, la protection de la vie privée, le partage de savoirs, mais aussi dans les arts, ou encore les échanges monétaires.

Le Paris P2P Festival se veut une occasion de fédérer et permettre la rencontre des acteurs du Peer-2-Peer sous toutes ses formes.

Toutes organisations portant des projets, tech ou non tech, se reconnaissant de ces valeurs sont donc les bienvenues à se joindre à cet événement pour y organiser leur propre rencontre, y poser un stand, ou organiser un hackathon.

Cet événement est le “prolongement” des rendez-vous mensuels de Paris P2P, tous les premiers mercredi du mois à Ground Control. Il représente l’ouverture d’un réseau de personnes, collectifs, associations, entreprises (…) souhaitant profiter ensemble d’un événement par et pour sa communauté, sans but lucratif.

A

Hello: everyone, I'm marco, I work at protocol labs and today I'm going to present something that we've been working on for the last couple months. That is a very practical tool for us to let anyone find where some content is stored on the file coin network.

A

So I'll get there's a lot of jargon there I'll break it down, but the thing I'm presenting is this distributed database for finding content. You can think of content as having an id. You hash it and you get this string and I just want to answer given this string. Who has this data that hashes this string?

A

So, let's start with the actual problem that we're solving. Maybe this can like give some context of like why this is even desired, so we have filecoin.

A

This is a blockchain using proof of storage and you can incentivize storage uh on the network, so you can store data on the network and pay someone to store that for you. We have a lot of data there right now. It's growing every day. I actually had to update this slide today by like almost 10, petabytes um and yeah. I mean we're going to have even more data in the future, so we're going to have a lot of these like cids, these content identifiers, and so how do we keep track of who's?

A

Storing what well the first thing we think it's like well ipfs does this right? Can we just put it all in the dht right now? The dht isn't quite ready for this number of cids to store.

A

It would be a big ask of each each dht node to just store like hundreds of gigs of data, um but I think there's definitely some interesting work that can be done here to improve this and still an open question what it could look like, because maybe you could have like heterogeneous nodes on the dht, like some bigger machines right and you have many small machines that are just kind of in people's closets and a couple like really beefy machines in data centers, but you end up with a lot of trade-offs here, like how do you know these machines are good?

A

They could be creating a civil attack, um open open question. So what? If we create our own federated network of nodes right? We know these nodes, we can trust them, they can be really beefy machines and they can do this mapping. Well, that's that's where what sort of the index is it's just a federated set of nodes that keep track of this mapping of cid to data or to the content that is referenced by that cid.

A

So I think, an illustration kind of helps and let's see how big that is. Maybe I can make that a little bigger.

A

So this is some user, they ask ipfs like hey, I want some data. That's referenced by this cid ipfs asks store the index or this indexer network and the indexer network keeps this mapping uh within the network, but it gets the data for this mapping from storage providers who publish something called an ad chain, and we pull that in and we index that data and I'll go a little bit deeper into each one of these things.

A

But this is like the high level overview and like what it looks like from both the storage provider, side and the user side.

A

So this is a actual example you, you can run this. In fact, I ran this to create this uh of how you use the indexer service, so in this case we're contacting a specific indexer and we're just asking like hey, who has the cid?

A

uh Who, where can I find the content referenced by the cid, and these are all the different source writers that have it? This is their peer ids.

A

So the next step is like how does uh where? What is the data that we're indexing and we get this data from the storage providers themselves? They publish you can think of it as this blockchain of advertisements and each advertisement references the previous advertisement.

A

It looks like this, and each advertisement contains the list of entries a list of things that this search writer now provides. So, like my cat photo, your cat photo. My dog photo right like uh and every time they get some more information. They publish a new advertisement and link it to the previous thing, and this is kind of what the type signature looks like it's pretty straightforward.

A

uh What this gives us is like sequential history, right like a huge problem in databases is like how do I know what came first. Well, we kind of like pushed the problem off to the to the writer side. So, like the writers here are the storage providers and they have to give us this canonical ordering of the history right. So in this example, a precedes b which precedes c. We know that the advertisement a happens before b, and this is true, no matter what order we get the advertisements themselves in.

A

Another property of this is, we have the indexers are eventually consistent, so if they they'll never give you wrong data like flat out wrong, but they'll just give you old data when you, when you query them- and this is a property of getting these sequential histories, because.

A

When they're looking at this advertisement chain, you know they might just look at a and then they only have the information from the advertisement a and they might later get c and then realize. Oh wait, hang on, I haven't processed b. Yet so I need to find b first and then I can do the rest. uh So you'll only be outdated, but you won't be flat out wrong.

A

We have like a defined order of how to process these advertisements.

A

And, what's cool is like you can imagine we have? We want to spin up a new index or node. We can just do that and then eventually, it'll just walk through the whole chains of, like all the storage providers and eventually it'll be back in the same state as one that has always been up, and so what does this look like from the storage provider?

A

Well, when it has some new information, it creates this advertisement uh block and then it puts it in like it puts it in the like chain of locks, and then it tells anyone who's. Listening over gossip sub that hey here's, a new head of my chain like here's, the new you know, entry node that you should look into and then indexers can lazily pull this list, and this pull model is really important. So, like another big problem with databases is how do you scale?

A

uh How do you scale rights, but you can actually kind of also work around this by being very lazy about it right. So if you're like, if the database itself is like well, I'm only going to handle rights when, when I have capacity, I'm only going to pull data in from sources. When I can do this, uh you you get these nice scaling properties where, like oh, if all of a sudden, for whatever reason, 5 pm on a friday, is really really popular for queries.

A

Well, you don't do any reads and you just handle writes this also allows the new indexer to come up pretty seamlessly like uh it. Just slowly takes its time when it finishes walking the chain it'll be up to date.

A

You do lose this property of reading your rights, but you know it's expensive to read your rights, uh it's hard to implement and it's hard to scale if, in this specific problem domain, like out-of-date, answers are okay, especially because we can cache things pretty aggressively once we do have an answer.

A

So everything I've talked about is really just like with one node, but this scales well across many nodes, um and you can imagine we're actually pretty lucky in this problem space, because we're just dealing with hashes hashes are pretty uniform.

A

We can spin up another indexer and just give it like a subsection of the hash and their read load should be the same. They should be uniform.

A

But so the index is just one implementation, and so, if another node does something similar here, uh it like reads processes advertisement chain the same way and it provides this query. Interface. There's no reason the federation only has to be stored. The index someone could go ahead and write like a russ implementation or or some other implementation, or something that just uses like you know, cloud building blocks uh that would that would work fine and could be part of the part of the federation and yeah.

A

I kind of went over this like we get a lot of properties here with hashes that make scaling much easier. uh The previous role scaling gets really hard because, like your data isn't uniform and you have to think and be very clever with how you shard your data.

A

So what does this do? How it gives us like? Ipfs and filecoin interop, so ipfs defines like a system of addressing and fetching data and filecoin incentivizes storage.

A

uh Here the ipfs client can ask the indexer like oh who has this.

A

It knows a storage provider has it fetches it from the storage provider and it can get that content directly, and this works today with with uh these special nodes called hydro boosters, which live on the dht and there's these special nodes that can query from other sources to return data faster to anyone, who's, probing them and it currently queries store the index, and this is kind of like a stopgap measure you can imagine, but because no change is required for the clients, but in the future, clients can just directly ask the indexer- and I I kind of made this section here, like what are some useful takeaways that you can take out of this and apply to your own projects.

A

I think there's like some good patterns here that are that are helpful to think about when building other systems, so first replication of sequential linked histories is easy. If I can give you like this linked list- and you know for sure that, like this event happens before this event right, uh it's very easy to replicate this. You don't have to worry about, like oh actually, there's like two parallel events. Oh, what does it mean to be like serializable versus linearizable?

A

Well, this is just like the strictest definition you can get. This is a sequential history.

A

One cool thing is in sort of the index. These blocks don't have to be provided by the storage provider themselves, and this is pretty interesting because you can imagine that many indexers all need to get this block of data of the advertisement, and you don't want all these indexers to just always be asking the sort rider so because everything is content addressed.

A

You also have this nice property, where the storage provider can put data into a cdn and then that cdn can give it to storage providers and then that cdn also asks or sorry the block the storage provider gives the block to a cdn, and that and indexers can get the block from the cdn rather than the storage provider, and a cdn can also use the indexers to figure out which storage provider it needs to get the data from. So you end up with this like fun, like uh feedback loop.

A

Another pattern that is definitely helpful to take away is, if you can partition things by hashes you should. It really simplifies the partitioning strategy, but it doesn't always work. So you know you might have like a case where you have clients and they have invoices. Well, it doesn't really shard very well, because some clients will have many more invoices than others.

A

One thing you don't want to. One thing you do want to avoid. Is this: like scatter gather pattern of read, queries it just doesn't doesn't scale well because you don't know what the answer is until you've asked everyone and then you're really just waiting on the the slowest link.

A

Okay, so sorry the index and you, how can you be a part of this? So if you want to make or propose changes, uh there's a sort of the index repo. We can start a discussion there. You can reach out to us on follow coin slack and store the index channel, and we have a fip right now which defines what the spec is of this indexer system.

A

If you want to join the index or federation, that's great. Let's chat, it is pretty storage intensive, it's going to be banned with intensive 2., although this is cached pretty readily you're still going to have to store pay for cash misses right now. You should be able to provide some decent uptime, I think in the future, with some cleverness we can reduce this requirement but yeah and then let's say you store content addressed out and you want to tell other people about it. Well, if you're a search, writer running lotus, you can already do this.

A

Otherwise you might have another system and you can take a look at this reference index, providing implementation and let us know, happy to unblock you with. However, we can and yeah so here's some some useful links. Sid.Contact is the website of the currently running indexer, and these slides are also on github and I'll. Take questions.

A

I'll repeat, your question.

A

How anchored to file coin directly? Is this versus being able to use it on ipfs? It's not anchored to filecoin, so the advertisements uh themselves, I I kind of like yeah yeah yeah, so the entries themselves have like information of how to fetch the data. It's it's not necessarily just like which storage provider has it it's instructions on how to fetch it, so the instructions could be like connect uh over bitswap to this other node, and it's like not even related to filecoin at all great. So.

B

Anything emitting content address or the advertisement structure.

A

Exactly yeah, it's a.

B

Topic is that a single pub sub topic, or can you is it configurable, like runtime or application time, yeah.

A

It's configurable right now we have like a main net and a test net, and but I mean anyone can just run the whole set of indexers on a separate uh pub sub topic. Thank you.

C

So um two questions and the first one is: what do you do if you had get two competing advertisements for the same previous hash and the second one would be? What is the incentive structure for running such a node.

A

Yeah, so the first question is: what do you do if you have two different heads right, yeah yeah, so that's undefined behavior, it's really right! Now, it's implemented as like, whoever, whatever the index or last sees, is the is the truth, but, like last, is a time based concept right, so it's basically undefined. uh If you want your data to be retrievable, then you don't do this.

A

uh The second bit is: what's the incentive yeah. This is. This is a tricky question. So there's a couple things one I mean, I think the simplest kind of answer here is like well, what's the incentive to like crawl websites and be google or be a search engine, and you get you can glean a lot of metadata from this. You also get like a lot of um you're like directly connected to the fire hose of what everyone's providing. So you could even add like a secondary index on content itself and then do something.

A

That's more like google, where it's like. Now you can actually index text or whatever.

D

Hello, um so what happens? If a storage provider stops providing a content if the deal runs out like how does this affect this chain of adverts and it like? It probably also relates to what you said about cdn.

D

So what if I'm a storage provider, I say: okay I'll store the data at that they see the end, but then, after I don't know a year, I realized I want to go to another cdn, so I have to change something there. um Is that going to mess up the whole history or how do updates happen so.

A

I guess there's two parts: um one is how do updates happen? Well, these, like entries, aren't just addition. They could be subtraction or they could be like a modification event. So, like you, could the search writer could say like. Oh actually that thing I published previously on the chain like remove it, I don't, I don't provide it anymore.

D

And then what was the second part? Sorry about the cdn. I guess it's the same. If I go to a different cdn I'll have to provide different information into the advertisement.

A

Yeah, so this doesn't specify like who's. uh I guess the cdn is kind of like separate in that the cdns aren't publishing to the indexer. No.

D

The storage provider does yeah.

A

Yeah yeah, okay, so the storage provider.

D

Will just have to update that and it's all good? Yes, yes,.

A

Yes, yeah yeah, okay, I understand your question now: yeah yeah.

D

A

Just like a modification like.

D

Attribute or okay field and and the service provider will have to sign that, so that um we know that this update is not someone messing about with the data and providing it's just. You know me updating what happened to the data.

A

Correct yeah, so the whole ad chain is signed by the storage rider, so it's not like another. Another person can like mess with that chain. Thank you.

C

C

I think it's good cool.

A

Well, thank you all for listening.

A