IPFS IPFS Implementations - May 2022, 23 May 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: A call for IPFS implementations - Adin Schmahmann

Description

In this May 2022 IPFS implementations workshop, we heard a number of first-hand accounts from builders & thinkers working on and with IPFS, taking stock of the current implementation story, and imagining what’s possible through shared effort on a protocol with such broad applicability.

A

Hi everybody. uh This may be a little a little late in the day for this, but this is a call for new ipfs implementations. My name is adin. I am a fan of ipfs. I've been working on it for about three and a half years, and I am one of the ipfs maintainers a little bit about ipfs. uh It's the earliest implementation of of ipfs. I think the first commit was in about 2014..

A

It is the most used implementation of ipfs and importantly, it is not exactly equal to what ipfs the protocol is other things that are not ipfs include ipfs.io or at siblingdweb.link things that support bitswap and the ipfs public dht things that only support bitswap things that support tcp. These are things that are not exactly equal to ipfs.

A

So what is ipfs? uh I'm not saying, um I think the boundaries of what it means to to be an ipfs implementation are are a little fuzzy and something that, like as a community with with many stakeholders and what ipfs is, is an area that we can sort of define boundaries with together. um It's not like, oh go, ipfs does x. Therefore, dean is a maintainer of golipfs and says this is what ipfs is right. That feels like we're back to go. Ipfs is not ipfs, uh it is content addressing there.

A

That's certainly part of it. uh Some areas you may want to think about of like is this pushing. It is uh bittorrent. Clients can use info hashes to download data, so are they ipfs implementations?

A

uh If not, why not? If I put the magic incantation below uh which, for those not not fluent in multi-formats means, I am a bit torrent file that is cidv1 with a sha-1 that is 160 bits long. uh Is that enough?

A

If not, what else do we need like there's some area to explore there, there's already a bunch of ipfs implementations? So so this isn't this? Isn't news uh there's about 15 on this list that uh some folks compiled in the lead up to this to this meeting? You can click there. uh The the link for this is heck at a dean, more ipfs with an a because that's what we're looking for here.

A

uh So, what's wrong with just having a single primary ipfs implementation, there's just a lot of use cases we've been through this sort of already and we've seen this through a number of the presentations so far today, um putting them all in in one place hurts uh it's hard to keep everybody.

A

You know sort of everybody's thing, happy you're like oh, I want to add in. Can I have you know ipvas daemon dash dash upside down. That only works for me, but I swear it's fine. It's fine! It's just a config flag. It's all! I need, but it's gonna be great, and it's hard to do that, because it interferes with the various conflict facts that everybody else wants um and basically it's just it's sort of having this this very wide.

A

You end up having this very wide interface of like this. One implementation needs to solve all problems, so you have effectively the empty interface for what does go. Ipfs do which is not, which is not great. You don't get to move as fast as you could like, as, as uh you know, draupa mentioned earlier today.

A

uh Also. um I wish it came this way when, when I got repo permissions but becoming a go, ipfs maintainer does not mean that uh that you're, perfect, uh you make mistakes. uh You make bets that, maybe you shouldn't, and you don't make bets that maybe you should people should be able to make these to make these sorts of bets right uh for what is what is the right way to do content routing? What is the right way to do? Data transfer?

A

What's the right way to do mutability, um we can try and make things pluggable some of the work that was done uh around dns link uh and how we can sort of enable that to uh as an as an api to cover many different utability systems is helpful, but sometimes you want to build things in and deciding what to build in requires making a bet on on some technology.

A

um And if we look around right, there are many hdp clients and servers, and even you know, bits aren't, which is which is uh not not too dissimilar from us uh in a number of ways also has many clients.

A

um So this is, you know, even within the peer-to-peer world, if you're going to sort of grow. Your adoption, like this having lots of clients seem seems like a reasonable thing to do yeah. So this is just more just more ipfs implementations and- and some of these are, when I say, small and big here, these aren't- these aren't meant is like.

A

Oh one of these is a better thing to do than another, but just the scope of like how much can change right writing with a different language or for a different platform is, is one way to do this right? um How you want to change the structure of where how the data lives, how you interact with the data?

A

um What sort of assumptions you can make about the data right these? These are all part of it. um You know elastic. You know the thing that linux to ipfs right made made assumptions about how you got to interact with the data that could make things faster and better for that use case right. This is this is part of it. This is these. Things are important, um but there's ways to change this that go further that are like. I want new protocols. I want new protocols for content routing. I want new protocols for data transfer.

A

um I want you know. I want new protocols for for mutability. How do I do this? um I want. I may want networks that are some are disjoint. I may want ways in which we can combine them.

A

How do we want to do this? What are the right ways in which to combine to to make networks that are are slightly different compatible right? I have a group that is offline in the woods and then they come back to everybody else. How does how do we want that bridging to occur, so they can interact with each other in one way and then come back and interact with everybody else separately, so examples um elastic iqs. This is one one example.

A

uh The goal of this implementation was, I would like to serve data to other ipfs nodes and make it easy to work running it in like a commercial cloud environment. That's like the basic pitch and they they are doing this right. This goes back. This is like yeah we've reworking things to do, you know, have have the databases and apis that we need to make this work for us, but you could you could go a little broader. uh You could have a you know an ipfs and bittorrent server. So ipld is pretty cool uh means.

A

We can basically represent any of the existing hash linked data formats, uh which is almost everything in the peer to peer space, has tried this and something's, not in the peer-to-peer space as well.

A

um So couldn't I just take a bittorrent server and then plug in some pieces to like serve the data over bitswap and advertise it to some content.

A

Routing system, dht indexer effect a tracker, a bittorrent tracker, wouldn't wouldn't that also work, and that way I could, I could sort of serve data to both networks without duplicating the data or anything that and it sort of works in similar ways lets people utilize both sets of clients to fetch the data you can flip this around and do it on the client side, which is people, could just use info hashes as cids with the the magic incantation from earlier uh and build a network and build a client uh that can fetch from both right.

A

So this is plugability, I think, as carson mentioned, not just at the and as dietrich mentioned, not just uh lib p2p as equal equals network. You know plugability, but yeah sure why not just swap in the bittorrent protocol. That seems like another way. You could fetch data, that's based on a content-addressed identifier.

A

It only works for that one type of data and that one type of network, but but why not?

A

And more broadly, if we look at some of these content addressed systems or if we look at systems that try and do like decentralized storage or sharing in some way, they're all using hashes uh they're, all using hashes underneath and they tend to do my system, they want like urls that look like my system.

A

They want, like you know, for for many, like blockchain storage, things like blockchain storage, call in slash my transaction id you're like why why? Why would you do this? Couldn't? Wouldn't you be happy to get that data from anywhere, not specifically like a transaction, but it's a content address? I could just make it a hash.

A

I have a hash. I can put a put a codec in front of it and now I know how to sort of work with the data here and we can allow ipfs to be like a little broader in terms of the sets of data in which it which encompasses right, make it easier to work with things that are not not unix fs uh and we've already.

A

You know there's already work going on there in terms of how this goes on with gateways and working with car files, and the great workbooks have done on on other codecs, whether it's like dag or dicosa, and work with ucans, so we're getting there, but there's room to to keep moving forward and experimenting with with new protocols.

A

You can show what is what is different show why yours is better um data transfer protocols alone. I I have like a list of like four or so things that I want to exist that are not there right now on protocols that just transfer over manifests of cids. I want protocols that can handle transferring very large like shot 2 and blake 3 objects and verifiable ways.

A

I want protocols that are optimized for things that are merkle trees, but like not well suited to bit swap or graph or graph sync, because they're the blocks at the bottom are too small right trying to move around things like file coin pieces like there's so many options here. Why are we restricted just these two um we're content routing systems really just many like there's so many possibilities. People have ideas, uh people should be able to feel free to experiment with some of these.

A

um I think ipfs were trying to to help with some like plugable apis here, but that's just go ipfs like take the libraries rip them apart, use what you need um and then of course, like authentication and mutabil and and mutability and ace encryption is related in here too, where, like there are very opinionated ways to do authent anything related to like protecting users in some way, authentication mutability, encryption uh mutability is a little different because it's about maybe sometimes it's about identities, but there's opinionated ways to do this.

A

There's not only there's not only going to be. You know one one best way. You should be able to try them out and they shouldn't all be like it's not like ipfs. It's like something similar. It's like no. These are ipfs implementations, they're, just you've just chosen to use a different protocol that if your thing works out well, other people start adopting, because you had a good idea- and I guess maybe the correlator to that is there are no perfect implementations.

A

It's it's not that you're not gonna. Do it people of different use cases is what this is all about. It's not oh, if only go ipfs did abc. That would that would be the dream. uh There's there's many different ways to do this.

A

um Try it out, maybe it'll, be maybe the architecture or your design will be good, maybe not, and then it's okay I'll try again, um you know. A call out of this is like webassembly is really cool. I like it. I have some experiments, I've done with it. It doesn't solve all your problems, saying, oh, if I wrote it in web assembly that would that would solve all my problems. It could load other web assembly and that would solve even more of my problems, but but it's it is not.

A

It is not the the answer to to everything um which sort of makes sense. Almost nothing is some things. Maybe we want to watch out for uh is buildup works for you. This is good being able to explain how it works and how it fits together with other implementations is part of this, as I sort of mentioned in the last session being able to describe to users and developers how it is that these different ipfs, implementations and pieces fit together um is important to not lead to to user confusion right help.

A

Them understand why something has worked, why something doesn't work. Sometimes this can be confusing like the model where we've switched to self-certified data, where you can pull data from anywhere.

A

It changes things uh a story I kind of like to to go to here is: I run, go ipfs on my laptop and I fetch some data, and you know speed was okay, but uh when I did it on my desktop, it was really fast. I think you guys should really figure out like your cpu usage and something because it's just not it's just much better on desktops than it is on laptops and you're like well. Well, that's not really. What happened? What happened?

A

Is you fetched it on your laptop from wherever it came from and then your desktop fetched it from your laptop at like land speed which was like really fast uh and and that's that's, what happened there, and so some of these intuitions that people come from like the http world of like yep, I asked for the data. Then you sent it back to me and I can like keep reproducing that right. um It changes and so making.

A

You know, tooling documentation, communication to make it easier for people to understand how the pieces fit together um is, I think, part of this.

A

So somehow we need more implementations. We need people to experiment, build more things join the crew. uh We have an implementer sink uh every every other week and come say: hi tell us what you're building see. What's going? What's going right? What's going wrong see if you can find other people who want to work with you on solving those things that are going wrong and and say, hi whether it's github matrix discord slack, we have a lot of places where you can find us.

A