IPFS 0.5 Meetup, 22 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Content Routing Changes & Improvements - Adin Schmahmann

Description

Our performance work centered around content routing, the process of finding and fetching data you need to load a website, watch a video, or use an app—crucial work that we focused our 2020 project roadmap around. Let Adin Schmahmann walk you through the details of our DHT work, and content routing improvements.

For more information
+ read the go-ipfs 0.5 release announcement: https://blog.ipfs.io/2020-04-28-go-ipfs-0-5-0/
+ visit the project website: https://ipfs.io
+ or follow IPFS on Twitter: https://twitter.com/IPFS

Sign up to get IPFS news, including releases, ecosystem updates, and community announcements in your inbox, each Tuesday: https://ipfs.us4.list-manage.com/subscribe?u=25473244c7d18b897f5a1ff6b&id=cad54b2230

A

All right, hi I'm, a Danish Bowman and we'll talk a little bit about some of the DHT improvements and got a good pass 0.5.

A

So overall, we've got four things are gonna. Look at one is the way in which the auto not service. That Steven mentioned briefly is utilized by default in the DHT.

A

Second, is how we've cleaned up our routing tables, but third, is the lookup algorithm how you find stuff and how that's been improved and then, finally, a little bit of the testing and how we sort of made sure that things are gonna. Look good on the network, so Auto not cleaning up the DHT. We have this Kadhim, lea network, which is the algorithm that I can just uses for its distributed hash table.

A

Where you basically just ask questions like do you, you know: do you have the data X or do you know someone who has it? But what happens when you cannot ask some of the people who are told given to you as responses? You know you're you're told about ten peers. You go talk to them, but six of them are unavailable. The the network structure starts to degrade and the like mathematical properties that allow for sort of logarithmic lookup times start to fall apart.

A

So the plan, is you ask other peers if they think your dialable and you sort of collect that information? And if you decide that you after you know interpreting all the results from your peers are not dialable, then you stop telling people that you are part of the DHT routing layer that you're a server. You are available to respond to queries because you're not you're unavailable.

A

We also have sort of a back-up plan, which is there where you will defensively. Look at connections that come in and see. Does this look like a does? This look like a pure that other people can talk to. They have a publicly diable address or all of their IP addresses. Look like localhost, or you know one 92168 types of addresses all right, so we've cleaned out a lot of our routing tables, but now we've gotten rid of the bad guys.

A

How do we make sure to keep some of the good guys in there before we were trying to be a little defensive with our routing tables, because there were so many inviolable nodes? We said, let's try and shave off some of the unviable if we only ever keep people around who were connected to then we're guaranteed that when we hear about appear from someone that they're not offline because they're in the middle of being connected to someone else, this is nice.

A

But what happens if your router blips and you lose all your connections now you're gonna, have to start like you're gonna have to like reset up you're DHT from scratch from the bootstrappers again, which is very unfortunate.

A

What if your machine is just overloaded, you have you know a thousand really important and valuable bits for connections or other application layer things, and so there, your DHT connections keep getting killed because they're just less valuable to your application. Overall, you still know where these peers are. You can still ask them questions the fact that you can't maintain a connection to them, because you're out of resources doesn't mean you should forget them, and so now we only remove peers from our routing table.

A

If we've they failed us, we've either ask them a question and they've failed or they've been around a while and every time we ask them a question. They just don't give us the responses. We need and we find someone better who might give a response in time.

A

So, look about your than the thing that sort of you tend to look at is the culprit of. Is this faster slow? Even though frequently it's the other two layers, this the routing layer and then the whole substrate appears underneath that's problematic. All right who do we ask.

A

So again, cadet Malia, you just keep asking, do you know anyone closer than X and that's like the question you just keep on asking.

A

Previously, with these routing tables full of bad peers, because the lack of auto net integration and the fact that we were you know losing peers because of you know, you know, intermittent disconnects or dropping connections will make things take a long time, both because of the failed dials, but also because the lookup algorithm needed to compensate for a lack of confidence in the like mathematical structure of the routing tables and as a result, we had to search more of the network.

A

We couldn't say: oh, we think we're close, because there might have been some one that was much closer.

A

But now because the the structure looks not like, looks cleaner and is more stable, we can give up much earlier.

A

We can ask we can find the three closest peers in the network to the piece of data that we're looking for the three people, three of the twenty people that are supposed to have copies of what we're looking for and that's enough because we should know there should not be any one any closer that we didn't know about, because the structure of Kedah Malia is such that everyone should know the peers that are closest to them, and we can now trust that all right so testing, let's, let's see how this thing works so another before we go into this- is that the tests were done on this on test ground right, which is very helpful, but also an isolated Network, not gonna, have all the same conditions: the real Network I'm not going to simulate all the conditions of a real Network.

A

You know a couple of the big things at play where we set all the latencies to be about 100 milliseconds for round-trip times, and and we could also control you know, but we could control the network to say like what percentage in nose look dial a bowl or undoubtable, and this is what it looks like this is how long it takes to Aizen tells someone that you have a piece of data, and you can see that the differences are enormous.

A

The one side is what it would look like if the entire network was 0 for 23 and the right side is how long would take it. The entire network was 0.5 again. This has the 100 millisecond, latency, z' and other sorts of assumptions built in, but the discrepancy is really large. In particular, the 95th percentile are very big, because when the routing structure is a little bit off, you can sort of get lost in the network, and now you won't.

A

This is effectively how finding the first provider this is, how long it takes for you to start downloading data with bits law and that's gotten faster, but not quite as fast as that and then finally, I pianist get x, which were also quite slow, as you can see much slower than the IP FS x, and those have gotten a lot faster too.

A

All right, that's been a crash course to what we've been up to with the HT improvements, and anyone has any questions. They've got a little bit of time.