IPFS IPFS Camp 2022 - Data Transfer, 31 Oct 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Data Transfer Tutorial - Hannah Howard, Will Scott

Description

Let's dig into fetching data from IPFS together!

A

Okay, in this first half hour, we're gonna originally uh for this first half hour, I'm gonna ask you to use your imagination and imagine that you're, in a room of only 10 people and you're, all working together on your computers, and you all have good internet and you're all working through actually using our protocols um as it turns out. None of that is possible um because we don't have Internet and also it's pretty tricky, to do this on a setup like this.

A

So instead I'm going to be talking to you about the protocols, um it's very exciting cool. So let's talk about bit swap there's gonna, be a lot of talk about bit, swap in this hour feel free to tweet about it. Is that a thing yeah um give your thoughts. So what's bit swap well, here's bit swap the protocol very obvious. Of course, um bit swap is a block request protocol.

A

It means that blitzwap is a protocol for asking for blocks based on SIDS, very straightforward sort of again that minimal basis for working with content, address data and the way bit swap works. You can see these like little okay. This is a protobuf but and I know that's a little complex and Cody, and you maybe can't read it, but basically um what it has two sections. One is what list, which is basically the blocks. You want. It's essentially a list of Sids. There's some complicated things around.

A

What you want to know about those SIDS is the protocols developed and then um you have a series of blocks, uh so basically you can either want or you can get stuff or you can get it back um and uh interesting thing about this. If you look at this message format, it doesn't really look like a protocol.

A

That's a request response protocol, it looks like you can be both sending wants and getting blocks in the same message, and that is in fact the case with bit swap bit swap is more of a pub sub protocol, um which is interesting. It creates some interesting problems and challenges. So um one key thing I want to say about bit, swap um it's so key that it doesn't appear on the slide and that's sorry.

A

This is a mistake um anyway, that that is that a protocol is not the same as an implementation, so I showed you that little protocol slide, um and uh that is the bit swap protocol.

A

But the implementations of bit swap do a whole whole lot more, and this is an ongoing source of confusion- is that people will look at go bit swap or now rust bit swap, maybe even JS bit Swap and they'll say: oh well, that's just bit swap, but actually they're doing way more than bit swap they're in fact doing a number of different things that are key to the entire process of getting content on ipfs like content routing, which is what we talked about this morning anyway. So I just want to make that clarification.

A

If somebody says bit swap you should you can ask, do you mean the protocol or the implementation um and you'll sound? Hopefully you will not stump them. If you stump them, you'll know they don't know what they're talking about um cool. So uh we talked a little bit about ipld and we have and how it's made up of these blocks, that link to each other um and I I. Don't know if mauve mentioned this word merkeldag, but I wanted to bring it up.

A

Merkeldag means a it means, essentially a tree of blocks that are linked together. um It's a bunch of tree, a bunch of blocks that are hashed and inside of the block content is links to the hashes of the other blocks right, um and so let's talk about how um you would synchronize a marble, dag right. If you are trying to do incremental verifiability with bit swap again, remember you don't want to download everything and then find out. You have the wrong thing much later.

A

So the interesting thing about this is, you can imagine um if I started with the root block right inside the root block, are the hashes of the blocks immediately below it. So that means that I can download the next layer blocks.

A

Let's say: I Traverse over the left side and I get the next, the the first block at Layer, Two right and when for when that block comes back, it comes back as a bunch of bytes and I can rehash it and check that the rehashed that the hash of the data I actually got matches the hash of the data I I requested and that's how again you do this verifiability right and then once I had that I could look at the data in that that layer, 2 block and I could say what are the hashes that are inside this block, and that would tell me about layer three right.

A

That would tell me where to go to get layer. Three and now I can request. Maybe I go down another way to the left and my second request is for the far left block at layer three and then I get the next block at you know to the right and now I'm all the way back up at the top and I start going down to the right and I ask for the second block at Layer, Two I get the hash back um and uh I get the block back. I verify it.

A

I look at its content. I get the hashes from that content to the third layer on the right and then I Traverse those right. So that's actually a relatively complicated process um you can and- and uh you might does anyone know what the problem with this process is from bit swap standpoint like why it might be slow?

A

Yes, what round trip? Yes, so I don't know the hashes of the stuff at the bottom, when I start right, so I have to get the first block in order to go to the next level, and that means I'm looking at just for this simple, dag, maybe three layers of round trips just to get the content and if you've worked on the internet, you know that a round trip is like a huge penalty. It's like 100 milliseconds of pause, which is terrible um in good cases.

A

You can imagine, though, that the one nice thing about being a block protocol is that if my dag happens to be super wide right, I can request both of the second layer in parallel, because I know both of them and then once I have them.

A

I could go right to requesting the entire bottom layer at once, because again, bit swap can request lots of blocks at once and it can request and download them from many peers right where bit swap excels is the multi-pier, and so this is wonderful, because I can basically really only have three requests: three rounds of requests to get everything unless I'm sinking something like a blockchain and there's no way to avoid the round trip problem here right. You essentially have to step down each layer with a round trip and that leads to a slow transfer.

A

So let's talk about our other protocol, which attempts to remedy some of that.

A

um Let's talk about grassling graphs, so bit swap, is the original ipfs data transfer protocol seen a lot of work, a lot of evolution, um you'll you'll hear from someone who literally just re-implemented the whole thing in a moment and grass Inc is now I, don't know it's like the I feel like it's it's if bit swap is like the senior senior citizen now, like grass, think, is like the middle aged person. It's not quite a young upstart anymore.

A

um Basically, graph sync is an attempt to write a data transfer protocol that doesn't understand blocks, but rather understands dags understands ipld, so that you can ask other peers not to send you a single block, but to send you a whole dad back um and this this essentially addresses the round-trip problem, the dot. What it does mean is that grass Sync has to understand data at a much more complex level than bit swap um it has to.

A

In order for two peers to communicate over graph sync, they have to understand almost all the aspects of ipld. This includes codecs blocks the data model, even schemas and ADLs. In some cases, right and I- don't know if you all remember in the early. If we go back to the the opening presentation, one of the big challenges is making sure that both sides understand the same complex data model and we get into a lot of problems with this. um If, if the sides don't have quite the same implementation, it makes it complex right.

A

Grass Inc allows you to express queries against ipld um against ipld data using something called selectors. Selectors are essentially a query language for ipld I, don't know if Mo covered it in their talk, but you did yes, sorry, no, maybe not. Okay, oh cool, so this is new. You can imagine selectors as a way of expressing a Traver, a full or partial traversal of an ipld graph. You can imagine if you look at you.

A

If you imagine this sort of like dag image here and then you look over here on the the right you can see like the green one, you can imagine that as a path selector essentially saying I want you to Traverse a series of nested nodes or blocks or whatever that are on certain named paths, and this makes sense a lot for like directory traversal.

A

You might also say something like I: want you to Traverse a series of no codes via a path and then uh once you get to that path, I want you to give me everything underneath it. It's quite a complex query: language probably too complex, but it does a lot of stuff. I know there was a request for maybe there's a better replacement for bit. Swap and I would say possibly under certain conditions.

A

One is that you do need to do complex queries and have them be pretty fast end to end over a single peer grass snakes Achilles heel is. It is primarily a single, pure transfer protocol, it's very hard to think about splitting up, dag requests without knowing anything about the dag ahead of time.

A

um It is at this point pretty tested and I would incur. You know um one reason you might want to do it rather than rolling your own is. There might be some complexities that you don't yet know if you haven't tried to re-implement something like this, um uh so it's there and finally, if you happen to be using filecoin uh well, you used to have to use graph sync, but not anymore.

A

uh There was a talk here yesterday about how you can now use bit, swap it's very exciting cool all right, but these are amazing protocols. Grass ink bit swap, but we've got one more thing. That's right! That's right! This is amazing, stuff right here now, folks we are introducing a revolutionary new data transfer protocol.

A

This protocol is set to disrupt the ipfs world and auto magically upgrade our transfer speeds and we're releasing it here today on stage, because everyone loves an on stage: release: okay, here's what it's we call it HTTP, something or other um it probably doesn't work at all, um but it might, uh and it's new and new- is cool. So to talk to you about how to introduce a new H new protocol to ipfs we've got will Scott uh my co-worker.

B

um I think this is really more mad science than trying to uh talk about something practical or something that we're actually going to use. But really what we wanted to um talk about is that we have extensibility now uh and we're moving to a world where we should be able to think about a lot more extensible data transfer protocols than what we've had and so I think. There's a number of things.

B

Probably most excitingly is the ipvm and thinking about shipping code to another node rather than just data, because then I can ship code that causes it to understand some new protocol and and continue to evolve uh protocols in the way that make the most sense.

B

So what what would we need if we wanted to add another protocol and and we'll use uh for this example HTTP as a thing that we might want to be able to support the first step, as we covered uh for most of the morning, is content routing. uh We need to learn that there's a provider that has content address data that would be compatible, that we could actually try and get um I'm not going to try and do internet things uh on the stage.

B

But you can follow along at github.com willscott HTTP Dash Sid Dash data, um which is a little GitHub uh repo that has some blocks in it, um and so you can, you can see it as a GitHub Pages hosted repo uh where there's a directory- and you can say you know thing.github.io uh Slash and then a Sid and you'll get the block that hashes to that sit back and okay. So so this is a scheme that we can come up with. This is very simplistic, um but right, like that.

B

That is sort of like bit swap in some sense, except over HTTP I'm, asking for a block. I get back the block if the. If the server has it, so, if you, if you take that as your provider, how would you then learn that this provider exists in for what blocks um Network indexers allow an index provider someone who has content to announce that content in one of two ways either over lip P2P, which is how many of our providers are announcing it, but also over HTTP?

B

And so we have some providers like nft storage, for example, who maintain an S3 directory where they keep their list of all of the SIDS that they have and the indexer slurp that and so in this repo you'll also see that and you'll see a readme for how you could generate that yourself. But you make an advertisement as another file. That's named as its Sid. That says the set of Sids that exist in that directory, and then you can tell the indexer hey.

B

This exists as a provider and it's able to parse that and index the set of Sids that it should return. A record saying those SIDS are available in this directory that that may I I think I described that very simply, uh and the the reality uh involves a lot more uh fiddling with things. But it's conceptually not that hard I think. The thing that you should focus on is not those mechanics, but rather what? What sort of additional protocols would you want? So asking for a single block can work.

B

It looks a lot like bit swap in terms of round trips. I make a request: I get a block, but what about car files? We could potentially make a request for a car file, like you already do on the ipfs Gateway, where you ask for a dag export and get a whole streaming set of blocks back.

B

This is also incrementally verifiable, so there's a whole class of additional data protocols that we can imagine that are not going to be that hard to add in and the real trick is going to be figuring out the ipvm layer and what the interfaces are to allow another ipfs node somewhere else to start trying and and being able to uh iterate and and have different logic in how it does. Data transfer.

B

How many of these slides are there cool? So okay, so you define your transport format. uh I misused the HTTP multicodec as the number of the metadata that I say that it is so. This looks like HTTP, um which causes many people to be unhappy at me. um The multi-adder I added a new multi-adder component called HTTP path. Where I can then say it's actually in a subdirectory. We don't have that in multi-adder. That means JavaScript multi-adder crashes, because it can't handle new types of codecs or new multi-adder sub components and doesn't have an extensibility.

B

So we'll we'll need to work on that. We told the indexer and then we have a client that can fetch it. There's an experimental client in ipfs Shipyard called the web3, retrieval, client, w3rc and there's a branch there that is able to understand this additional record that it gets back and make an HTTP request.

B

So I will turn this over, but I will say again before I. Do the URL github.com will Scott slash HTTP Sid Dash data, and you can see the different steps that you go through too experiment with adding additional transport protocols.