IPFS Pinning Summit 2020, 14 May 2020

Previous Meeting

⏯

youtube image

►

From YouTube: go-IPFS 0.5 Updates - Steven Allen

Description

IPFS technical lead Steven Allen gives a presentation on recently released IPFS 0.5, and the features and metrics relevant for pinning services and other infrastructure and service providers.

Read more about the inaugural IPFS Pinning Summit on the website: https://ipfspinningsummit.com/

For more information on IPFS
- visit the project website: https://ipfs.io
- or follow IPFS on Twitter: https://twitter.com/IPFS

Sign up to get IPFS news, including releases, ecosystem updates, and community announcements in your inbox, each Tuesday: http://eepurl.com/gL2Pi5

For more information on Filecoin
- visit the project website: https://filecoin.io/
- or follow Filecoin on Twitter: https://twitter.com/Filecoin

Get Filecoin community news and announcements in your inbox, monthly: http://eepurl.com/gbfn1n

A

Hello, everyone, uh I'm gonna, be talking about the uh improvements we've made to go ipfs in the zero to five release. This is probably one of the largest releases. We've cut, uh not the most breaking that was probably 0.4, uh actually not very breaking at all, but it definitely has the most like large scale protocol improvements, or at least the most man hours put into it. um Okay, first up, we've made a bunch of change improvements to content, routing uh content.

A

Routing is uh the logic has around like announcing content to the network and then finding content in the network. uh The main thing we use for this is something called the dht distributed hash table. um We've made a bunch of improvements here to massively increase performance and reduce some overhead uh connection forming and stuff like that.

A

uh So one big improvement here was cleaning up the dhc in this release. We're using a tool we call autonet to determine their nodes will use this tool to determine if they are publicly reachable and if they are not publicly reachable, they will like drop off the dhc they'll still be able to use the dhc. They just won't like ask other peers to give them or to send queries their way, because no one can actually reach them in the previous dc.

A

What happened is peers would join the network um and then other people start routing queries their way. But these these peers were not reachable so like 95 to 99 percent of queries would just fail or time out, because you couldn't actually reach the period you're trying to reach. um So, uh in addition to this, we've also uh improved the routing table management, quite a bit um in previous releases. If you disconnected from here, you would sort of forget they existed, or at least you're moving around a table uh for a cottage here.

A

The branding table is kind of like, instead of fingers into the network for routing around the network and finding the content you're looking for um really, it allows you like jump across the network closer and closer and closer to content.

A

But if you don't properly maintain these tables, it means that, like a query that would normally like sort of like uh take uh a logarithmic size, the network time to complete, could take like linear and size that we're kind of completely walk around that we're going to rank um so yeah we've massively uh improved round table maintenance such that like prairie time, should actually be logarithmic in the size of the network.

A

uh These two, unfortunately, these two features here will only really benefit users once the network upgrades. So please, please, please upgrade your nodes and tell your friends to upgrade your nodes as people upgrade, the entire network should be faster.

A

Finally, uh we could look up algorithm. uh This will actually uh help nodes who are upgrading now, even when the rest of the network hasn't upgraded um where basically the previous kindle project we had. um We keep on searching and searching and searching and searching and searching um mostly because, like we had these like poorly maintained wrapping tables and we had a bunch of unbelievable peers, so you kind of get stuck in corners, networking and like backtrack really far and then go back and search more and like keep on searching.

A

uh We've made a bunch of movements like how we handle parallelism in the writing, table or sorry in the query logic, and because we also improved the uh the writing table logic. We've been able to like be less pessimistic. In our query logic, um this has led to a three to four x reduction in uh dials and a massive improvement in uh provide times.

A

uh So how do we test this? Well, uh this is actually a big problem for us. We're like we want to make a bunch of changes to the dht, but we can easily test like query side. Changes like we can say like well like. Are we making, like our queries to the network, uh completing in less time? um The problem?

A

Is it's really hard to test, like, um I guess, server side changes on the live network uh because, like we can't just tell everyone hey, please upgrade immediately or please like test this new thing, um so we built a testing system called test ground. It allows us to make basically arbitrary distributed system tests. uh We'd use this for a bit swap now use it for gossip sub, use it for uh or use it for the dht.

A

uh This allows to spin up like a thousand to two thousand nodes uh in a virtualized environment uh with realistic network latencies um and actually say hey. Does this dht work? um So these results are preliminary. um These are results from testing, so they're not going to match the real world. However, at the scales we are testing, uh we are seeing the access provided times in the ddhd, we're 20 to 30 times faster than the old dhc. Again, this is in a testing environment. You're, not gonna, see this in the real network.

A

uh Definitely not now and you're. Probably not gonna see it quite as good when it's a full network upgrades but it'll start approximately a lot closer to this, but the the multiplier should be about accurate, not the absolute five times, though um the fine times got to be around like two to six times faster.

A

And the ips get times got around five times faster in the new logic. So yeah made a lot of improvements there. um So does this actually show up in the real network? Yes, uh so even though most network is upgraded, we're already seeing massive improvements again we're seeing uh like in the upper left-hand corner here, you can see uh they provide uh like dials plus queries have gone down a lot. Well, the number of queries hasn't, but like the number of extraneous dials making the network have massively produced.

A

This should reduce the chattiness of the network, the amount of bandwidth overhead, just a matter of like load on the network in general. uh In the upper right hand corner um you can see uh the uh a graph of like queries that complete you can see that basically, you've just gone from like 60 uh percent, provide queries to like area and um uh 95 or 90 uh find queries to 100 both sides uh in the bottom left-hand corner.

A

You can see the provide times how they basically gone from effect like on the left-hand side, you're, saying they're, actually, timing out there, those red lines or timeouts um on the right they're, actually completing they take a while. uh This will get better as the network upgrades and as we make more improvements, but they're actually finishing, which is nice and the content writing fine times have been massively reduced in terms of the actual like reduction in fine times. um The 50th percentile uh has reduced from like eight seconds to around one.

A

Second, at this point um or one and a half seconds, sometimes uh the 90th percentile has roots from like 40 something seconds to 10 to 14 seconds. uh These numbers are highly variable they're, like uh I'm, trying to average this out over like many many days. So in this case, I guess a week of date at this point um and these numbers I'm seeing uh these numbers also get a lot better as the network upgrades um and we're making more improvements, uh but we're already like very happy with these numbers. They're.

A

Sorry we're not happy with these numbers. We are happy with the improvements we've made. We want to get much much better, um but yeah. This is a start. Okay, uh the next big change we've made in this really is content is in the context change.

A

This will have a large impact for uh pinning services um because, uh like basically like any wasted bandwidth um in our context, change particles like bitswap uh costs, money um uh and like if anything, if bitswap is slow, people think oh well, this is slow and they don't want to use fps. They don't use bing services, uh so we made a couple of very large improvements here.

A

um One is in terms of like uh time, defense, the actual performance um we basically you've massively improved performance for fetching for multiple peers um uh or, like basically, bitswap, is better at like sharing bandwidth and pulling from multiple sources at once and like actually getting the proper speed ups uh in the past, like it would get some speed up and pulling from multiple peers, uh but like not nearly as much. uh You can look at this graph later. uh Second of all, in terms of um wasted or duplicate blocks, we've basically gone to zero.

A

That red line at the bottom is the new bit swap the blue line. So the old bit swap, uh you could say basically like as we're ready squares, we're just not seeing duplicate blocks. Now, um when you start increasing the latency, uh we do start to see some people's blocks. We've hit some timeouts internally in the next release.

A

We've added a feature to better estimate these timeouts, so uh you should see even fewer duplicate blocks, um but yeah we've made a bunch of improvements here, uh graph, sync, okay, so uh graphsync is this new fancy protocol that lets us request an entire dag all at once, instead of requesting each individual block. So if it's swapped, like you say, I want this block with this block above this block.

A

um But this means you know a lot of round trips and if you want to like stream a bunch of data, all at once graphic looks like you like describe the subset of the data you want and pull it all in, um unfortunately like uh we haven't like because basically we have inside of graphs. It just does that where it pulls all the data in and like spends spits the data out, but we have not completed the um like the rest of the smarts around graph.

A

Sync, so they can actually pick which peers wants to download from split requests across peers resume requests all that kind of stuff. So, in this release we don't have photographs and support. Instead, we have graphic server side support. What this means is that an ips node can serve data over graph sync to some other node, not necessarily ipfs, but some liberty, p node- um that has graph sync client-side support. So what does this mean for pinning services?

A

um Well, uh if you want to speed up like fetching data from people trying to pin data- and you know their peer id, you can quite easily build a tool they just like get exactly put up here and they use graph sync to suck up all the data. This is assuming the user has enabled this experiment. This is not enabled by default. You know it might be defaulted in a future release, but assuming they've enabled it or you've told them to enable it. You just suck up all the dates and graph sync.

A

If you want an example of this tool, there's an example in the codefest repo in the tests. You can just read the code. There uh so yeah we're hoping that eventually we'll have like full graphs and support, which should uh improve performance of data exchange and ipfs in general, not quite the area. We've made a pretty big step here: okay, uh next up, import and export. uh This is how you get data in and out of ipas and made a couple improvements again very important.

A

Finding services um uh first up, uh we've improved ad performance uh on batcher, specifically uh so uh badger is like the next gen data store, we're trying to switch to um we've been slowly sort of digging through uh uh bugs and like regressions in it and like trying to sum down a list of issues we have we're almost there. We have a couple remaining issues: we're hoping to get this out in 0.6 the next release as the like the stable. uh By default. um Sorry, it's like the the by default data store.

A

um However, in this release, we've we're saying: okay, it's stable, at least uh so we're recommending that people try it out. um If you look at the bottom right hand corner here, you can see the performance of badger versus flight fs versus battery in this release.

A

Badger uh in this release is now on par, with, um at least on disk is on par with, like reading the file off the disk and then adding or uh or it's on par with copying a file from one location just to another location on the desk.

A

Basically, so it's not quite as fast as like fetching the data from the network and then writing to the disk, because you um we're using some of the disk io to read the data from the same disk, uh but for most users, if you're, just trying to add data to ipfs, it should be just as fast you're at databases is to copy a directory uh yeah.

A

uh Second up uh we have uh improv or sorry we've added some commands for importing exporting tags or sorry uh ipd dags, that is like entire directory trees or like arbitrary idealty data structures. um Previously to do this, like you, have to like fetch the data out of bfs into a set of directories and then re-import the data or like pull out individual blocks and then react them or like move around to data sources like this.

A

uh Now you can just give my cid and just pull out this archive of like this entire, like graph of data and then re-import it into ipfs. The cool part here for pinging services is like this is a great way like. uh If you need to get a bunch of data from a user, you can literally have them like run ifis tag, export uh export to a hard drive mail. You the hard drive, then you just import that ipfs.

A

That's probably the fastest way to move data around to this point in ibms uh or just in general on the internet, because, like shipping hard drives, is faster than the internet, um uh so uh yeah, if you, if you have users, they just need to like give you a ton of data. um This is an option.

A

uh Okay. Finally, um I have a bunch of uh heads up for, like people running activists and infrastructure, first up, um we've enabled a service called autonet by default. Autonet is again that service we're using to improve the dhd to allow nodes to determine if they're, dialed or not the heads up here is that nodes will now help other nodes determine if they're available by default. um If it's an issue, you can turn it off in the config, the documentation, godfs repo, but this is a very limited service.

A

We don't expect it'll be problematic but like if you don't want your node to like perform any extraneous styles, uh then you could turn this off. Please don't because you hope you're helping the network by leaving it on, uh but just heads up. uh Second of all, we switched to tls as the default transport. um uh This, like will basically give us like a better secure transport with better or uh better understood security properties. uh It's generally faster, actually a lot faster uh than our previous uh transport psycho. uh However, there's no open still support.

A

We have an open issue for this. um uh This is open bounty on it. um uh Yeah so like this does mean that, like as people upgrade, you may see an increase in uh cpu usage because open ssl's, um uh uh open the cells are. Simplification is much much better than goes um and secure has open ssl support so like if you've been building open secondary with open ssl support, uh then, like yeah, you may see that some more cpu usage from tls connections- yeah just just a hands-up uh subpoena gateways.

A

uh You can read more about this. um uh I believe in the blog at this point, um I'm not sure if we focus on blog posts yet uh but yeah. This lets us like put the uh the cid or the domain name or the ips key in the subdomain, so it gives every um ips site a a separate origin.

A

um uh This is uh like yeah. This is just generally important for like running writing dapps in ibfs. This affects um uh your infrastructure um because, like you may change how you run your gateway uh to probably support some different origins.

A

uh Finally, we made some improvements to the daemon uh one embedded system d support um uh uh so like uh we now support the system, notification, api and socket activation. This is useful if you're, trying to like start knife automatically or like order dependencies um yeah and we've made it so that you can serve the api over domain socket for improved security.

A

uh We've also there's also two migrations in this release. They are very minor um they're, just moving some things around, like fixing some config values, but just heads up. If you upgrade um and you automatically migrate, your node will have to go and download the migrations on upgrade uh to like make these small changes to the repo so just plan for that um and there's a ton more. But I am definitely out of time. So you can read over the slides later.

B

Do I have time for questions you do if anyone has questions for stephen stick them in the q, a or in the chat. Your your one question is people want the link to your slides later. uh I can send that out.

A

I also recommend that you read the release notes. They include everything here and in more.

A

Detail uh the answer to that is encrypted on top of private forms is yes, uh yeah, the underlying so um with a private swarm, uh we'll encrypt the connection. First, with the the swarm key, um uh then tls will be like on top of that encryption or sorry. I guess underneath that encryption, we'll encrypt the cls and then we'll encrypt this.

A

The question is: uh can we disable tls to point cpu usage? uh We do not have a flag to disable tls in this release, because we were planning on switching to the new tls version. If you have not been manually compiling your nodes with uh like dash tags equals openssl, you will actually should actually notice a decrease in cpu usage. It's only if you've been explicitly compiling your nodes with open sl support. If this might happen- and this is only as node's upgrade.

B

The question from cody is what is some of the history behind stuccio versus tls.

A

Oh um that's complicated. I don't actually know a lot of it. um uh I know that now we initially looked at using tls, but tls was like so tied up with um what one like the original theos particles took many round trips. Second, I think, to fewer, I believe, I'm not trying to train you don't check that. um Second of all, like tls, was so tied up with the ci system. You wanted a system that was simpler, easy to understand uh and not tied up with the cia system. Tls.3 uh mixed, fair number improvements here.

A

uh It even technically supports like uh like dropping certifications entirely. We don't do that toward like um better interoperability with other tls stacks um yeah. Sorry, I don't know the full history there.

B

I think a good summary.

A

Then I think we can move.

B