IPFS IPFS Camp 2022, 2 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: IPFS Service Panel - Thibault Meunier, Matt Ober & Mario Camou

Description

Panel discussion of gateway operators, pinning services, and others who operate IPFS services.

A

uh So let's go ahead and get started and uh maybe give your name and uh what company you're with and the sort of thing that you're uh that your company is good at.

B

Good, um so hey everyone, I'm Tebow working at the cloudflare operating cloudflare public ipfs Gateway, as well as like dedicated gateways uh for people uh providing a59 certificate to all of them. As part of that, we're also leveraging cloudflare. Existing infrastructure should be for, like security, DDOS, analytics workers or zero trust.

C

uh My name is Matt Ober I'm, one of the co-founders and the CTO of pinata pinata is one of the earliest pinning Services that's been existing for a while, and we basically just do yeah web3 media storage built on ipfs. So we store your data. We serve it through gateways a lot of other stuff at play there, but those are kind of the two big two big factors.

D

And I'm Mario kamu I work for protocol Labs I, take care of our gateways and we're basically providing these as a public as public gateways as a public service for the internet.

D

So who wants to start.

C

Is this a question? Do we just talk between each other? Are you asking us questions.

E

So this is a question for cloudflare. um I was interested in what you guys are doing with ipfs like how are you running it and um what are your plans for the Gateway.

B

So for ipfs uh we started in 2018, I, think and kind of started, like just research project like we're interested in like these new Imogen binding for like distributed networks and like how people would use them, and so, as part of the research team would like this were tasks to start an ipfs Gateway and have it as a public good in a similar way that we run like our DNS. So we're in like various Services certificate, transparency or Etc, and just see what happens.

B

um It turns out that there was a lot more usage than what we expected initially um and so like recently like earlier this year. We transitioned like these Gateway from like a public good and like research project to keep the research project, but at the same time also offer like gateways for people that would like to like have actual slas. Have a support be able to like customize um the headers or I. Don't know various settings of their Gateway um in terms of going forward and I. Don't want to monopolize the conversation.

B

So I think all these guys are also doing super interesting things um and um yeah in terms of going forward.

B

um The HTTP gateways are definitely a good way to interact with the ipfs ecosystem at the same time, there's a lot of difficulty in terms of like carrying the trust and cryptography that, like ipfs benefit from into more verifiability to an HTTP world, and so definitely working on standards and Etc to like improve. This is definitely important for us and I. Think ipfs IO is also doing a lot to that extent with like car files, so um yeah I tied the mic.

D

Yeah well, uh like I said: I mean we ipfs IO yeah we've been running the Gateway since forever. I guess I mean I've been with I've, been with PL only for a little bit less than a year, so I don't I, haven't no. No. The whole history of the gateways uh but yeah like I said I mean we're providing that as a public service.

D

The idea is that there needs to be some way of bridging between the web 2 and the web 3 worlds, and we want to ensure that there's at least one way that is completely free and open.

D

Modulo takedown requests modulo malware blocking module all that sort of thing, uh and basically that's why we are keeping the gateways up and running. What's the thing with pinata.

C

uh Yeah I guess, on the subject of gateways, um we do operate a dedicated, Gateway service. I talked about this a little bit in an earlier talk today, but, uh similarly to you guys, you know operating these public free Goods. We do that as well through our public Gateway, but it does get crazy. So if you do want something, that's more in line with performance speed, but also protections against things like fishing malware, DDOS right, we kind of offer that up all packaged together and um yeah.

C

Think of us, as kind of like a fully integrated stack, so you pin your content to us and you can also serve it through a Gateway and so I'll kind of just one unified experience. So um that's really what we focus on is just user experience. It's it's a big part of what makes us a pinata.

C

I think we diverged a little bit from your original question, but um any other questions we have out here in the audience right now.

C

F

This is Jesse, um I, think I, look at a lot of people running the public Gateway. It's really free, um I figure out like anything, you want particle to help like maybe incentive program or something like that, can help more people on the public Gateway any situation or anything. You feel like super important for the people running the Gateway for free. Can you like share with us.

B

Foreign, okay, maybe.

E

B

It um say I mean definitely these gateways are free in a way that, like I, mean should be PL. uh Peanuts are like cloudflare, offering some Gateway for free um and at least for cloudflare, it's also as part of like good of the internet, like you would like to be able to access these services for free. Of course, it's not because the service is free. That, like you, should be able to.

B

Like you know, page someone with an issue or like uh you should be able to like just expect, um have the same expectation as it like paid service at the same time, similar to and I would really take like a normal traditional web infrastructure approach, but like similar to like cloudflare like who's operating like a free DNS, Server, Like, 111 or who's operating certificate. Transparency like all these are free services, and just because, like like love, those things, it's good for the internet uh to have these kind of like free services and public good.

B

At the same time, it also gives you kind of a good view and, like diver's view about like what's happening, and so that helps you improve the paying services. So, like the issues you would encounter um through the public service, you won't have them through the paid service later on.

C

Yeah um you touched, and I really got a lot of good points here, um in addition to public service right these, these gateways they kind of act as like, a uh called a free trial to the the distributed ecosystem. Here, um a lot of times these gateways are going to have to be built into whatever company is running them just built into their business model. So uh whatever that looks like is kind of up to the company. Doing that I know.

C

Cloudflare pinata ipfsio we've all kind of taken a little bit of a different approach here, and that just depends on kind of like what the company's goals are, how it fits into their business model and then kind of you know what services there they want to give a little bit of a free taste to to the General Public yeah.

D

Yeah well uh PL. One thing talking about I mean I, think that right now I mean the the heart of for us right now, I mean the hardest thing is actually uh scaling up.

D

Just basically that's what kind of you know could keep me up at night in the sense that uh we are basically dealing with an unbounded number of unbounded amount of traffic with a limited amount of service of servers, uh and that's what I think what I think would be the is the the toughest thing or the I don't know: um I I'm really waiting for Saturn to come along. To be honest, because that will take off uh that will take up a lot of load, but I mean it's on the one.

D

On the one hand, I think: do you think it's a bit ridiculous that we had to scale up to uh 120 Kubo nodes just to keep our gateways running I'm, not sure how many you, how many guys, how many notes you guys have uh uh I, don't know where am I going I, don't even know where I'm going with this uh with this rant, I guess I'm, just I'm just venting a little bit, uh but actually that's I. Think that might be a good discussion Point. What keeps you guys up at night.

C

uh Yeah I mean again I'll go back to the the previous talk I I did today. um You know the the big thing that keeps us up at night is not necessarily the load of the gateways. um Like I said you know, we've kind of adapted that to fit into pinata's growth trajectory as a whole.

C

We want to make sure it's a service we provide to the world, um but we want to make sure we're kind of doing that in a responsible way um and then speaking of responsible right, A lot of people don't use gateways responsibly, so I think again, malware phishing detection, uh illegal content.

C

That covers a wide breadth of things. But um you know that people use these gateways as kind of like a free CDN to serve whatever the content they want. And then, when the isps get mad, they look at your domain, not the person who initially uploaded the content. So this is this is kind of an ongoing fight we have at pinata and something that we're developing a lot of like resources uh towards.

C

um So it doesn't necessarily keep us up at night. But it's it's something that we're like putting a lot of thought and effort into as to how do we make sure that these Bad actors don't take down just us but like Poison, the Well of the entire ipfs ecosystem as a whole. um So.

B

Yeah I fully react to like an earlier point that you made about like the number of Kubota nodes and Etc, and it's kind of like I, think there's been like a different partying with some of like running a kubernet which, like really was designed as something you could easily run on your computer, to serve needs of like a Gateway and like like just web scale, and so the like, when you mentioned Saturn for instance.

B

It's definitely like one implementation that, like has been so differently, not something that, like you, would run locally but like something that has been made to scale and made different trade-off, and that's one of the things we've seen. Also internally is like from the beginning.

B

We relied on some of like the work that, like the go, ipfs and now Kubo team did but like the nodes were designing differently, because we had different trade-offs and it's in a very similar to like a lot of like the in the web, like across, like the web 3 space, where you have like I know, go ethereum that, like also had like a similar design.

B

You have like a know that you run locally, that you're able to suit the need of users, but that likely doesn't suit and scale as you want to run a public service at web scale. So I think as time will like pass. It would be more hopefully like there are more implementation in details that are just slightly Beyond, just like Kubo and one node size, but more in terms of like how do you scale all these.

D

I've actually been quite excited this week to see things like the elastic ipfs provider. Iro uh we're basically doing completely different things from what Google is doing, I mean Kubo. Yes, it's one implementation, it's getting better I mean I, don't know if you guys have tried zero 16, but it's uh uh it's a lot better than it was than the previous versions, at least in resource management, uh but I mean seeing that yeah. You know well, we'll just use the use uh Cloud providers to provide cool.

D

Why not to provide it to provide ipfest content uh like Alaska provider is doing use a micro services like approach like like iro, is doing it's really to me. It's really refreshing to see this sort of movement in the ipfs space, so I, don't know any more questions from the audience or comments.

A

What are some of the changes that that you have to make uh on top of Base Kubo to make the system work in your own system?.

B

um So the first thing like first first when we started like like when the project of like running an ipfs Gateway, was pitched just at clafler. The first thing the legal team said you need content moderation um so like that was the first thing and yeah I. Think like like Matt, like mentioned that, um like later on that, like, yes, like that's, a need wanting to operate a service just to be able to like answer like legal requirements that, like you may have um through this through the like, through time um and I.

B

Think like another different approach to like uh like to the native Kubo, is like. How do you make sure when you have like multiple nodes or Etc, that, like you're, not requesting content to do DHT like an infinite number of time, but like you can share like the resources that, like your content, would be like more cacheable than just like, improve time to retrieval, which might be slightly different for like pinning providers which have the content inherently. But.

C

Yeah speaking of that, so we we kind of do have a benefit on our end of being like a full stack implementation. Some of the things we've done on our end, specifically with how it relates to our nodes and our gateways, anything that's penned on pinata, when it's requested through one of our gateways. We we know that it's pinned to us. We actually don't even request it via like the DHT. We just go directly to the node. That has it and get it back, and we call that smart routing.

C

So this is kind of like a big benefit that alleviates a lot of resource load and also improves speed for the customer, which is you know, a huge win for for them, um specifically like fine-tuning Kubo, uh there's like there's a setting that a Dean one of the developers on Kubo added a while back. It's like the internal configurations.

C

You can change the amount of workers, the amount of tasks uh that are running in parallel, uh that that helped us scale up a lot. So if you haven't checked that out for your own Kubo nodes, that's uh it's a big win there um there's! So many little like fine-tuned things you can do in Kubo. That will make your life easier, but it kind of just trial and error based on what you're, seeing.

D

Plus a lot of that isn't uh very discoverable in the documentation. Let's just say that yeah um I mean even things like the accelerated vht client in the case that you are actually looking at the DHT like in our case right um new things like that. But one thing that we have introduced in protocol Labs internally is what we call the Thunderdome, which is basically multiple Kubo implementations, going only one will go out, or at least only one will outperform all the others. You might have seen some of the talks about I.

D

Think Ian talked about uh about Thunderdome at some point and we are actually starting to Benchmark Cuba versions against each other and also Google configurations, and that should help a lot with that sort of thing with finding out Optimum configurations for something that is not the standard way of using Kubo, as in start it up in your laptop right. But that's going to take some time to actually be able to get some insights from it.

E

Is this thing yeah? So when it comes to like I guess, I I mean it's part of content moderation, but like with ipfs being a decentralized service and um like running a Gateway kind of be encountered to that, because you can filter out a lot of the stuff like. What are you guys doing in terms of um like governance, to make sure that, like this stuff, that you're filtering out is just like illegal.

D

Well, like I mentioned earlier, in my talk, we have I mean in our case at least we have the uh any any takedown requests that arrive at abuse at ipfs.io uh they. Well, yes, we generally, we automatically generate a PR, a pull request for uh for any cids that are mentioned there, but then it goes to review through our legal department and they are the ones that actually do the merge. So yeah, the legal department is the one that says yeah. This is illegal.

D

Is this a legal takedown request or not uh uh that might be there? Might there might be more? That should be done at some point, but right now, at this point, that's the way that that is what uh what we're doing the legal department is doing due diligence. I, don't know about you guys.

C

uh Yeah, on our end, it's it's kind of a mix of a lot of things. um One thing I'll say, though kind of on our end right is like we don't want to block your content right, uh but we are the company that resides in you know a plot of land that is governed by a government right. If we don't comply with certain laws, we get taken offline, which means that everybody's content gets taken offline, which is what we consider a far worse thing.

C

The beautiful thing about ipfs here right is: you can use the protocol under the hood to get whatever content you want, and that is a very powerful primitive, but at the end of the day, right when things are associated with a company's domain name, you have to be very protective of that or else it can impact everybody, and that is a trade-off and it may not be for everybody and that is kind of what you, as a user, choose on your scale of decentralization like kind of what trade-offs are you wanting to make what's more valuable to you and make the right choice for you and your users um on our end kind of how we determine you know what we're blocking uh it is all legal on our end right.

C

We don't take anything down if it's like I'm offended by that content right, we don't take stances there. uh It's dmca content right, it's phishing content and a lot of the phishing content. We don't even get takedown requests for it's if we, if our systems automatically detect it and we notice hey you're, let's go fake, Microsoft page right, that's pretty obvious!

C

um So that's more of a preventative measure, but it is all content that is going to get us legally in trouble. uh We take no stances on politics. You know anything like that. It's it's purely what we are legally obligated to take down so.

B

Yeah I, don't have anything to add. Like traffic has a legal team, they have the tools to like content and like filter the content that like that has to be moderated for various reasons and from a technical perspective. We give them the tool, then that deal with it.

D

Actually, you you bring up a pretty good point right, I mean it's not that we want to take the content down. The content is there and if you run your own ipfs node, you can still get it. It's just that for legal reasons, you have to do it. Otherwise, it's not just a government that takes you down. It's isps that take you down. I mean uh take an example: I I live in Spain with the Catalonia process, uh with the catalya protests some years ago, uh the iisps actually blocked ipfest.io.

D

Why? Because the Catalunya, uh the people who weren't doing the Catalonia processes protests sorry were actually using ipfs to communicate with each other. So we had to do the take PL had to take down that content, otherwise they wouldn't have been. The gateways wouldn't have been reachable from Spain none of the gateways, and we are still having problems with some isps that are still blocking us. Using uh SSL injection they're, basically changing the SSL certificate and they do not for some isps.

D

You cannot get to the Gateway just because at some point there was some content that was deemed illegal by the Spanish government or by the ISP, and they blocked it. So and the problem is that that it's not just that they are blocking that content. They are blocking everything under ipfs.io, so we, you have to thread a very thin line there. You have to be very careful, I hope that answers your question.

G

Is it third sorry, uh I have a question related to what you guys mentioned about Saturn and it's about content delivery. uh There's problems about retrieving our data on ipfs. So do you think that the problem is maybe adding up a compute I, don't know computer layer, so it's faster to retrieve that data or to kind of cache, mostly any possible I, don't know part of the ipfs data.

D

So you're saying: okay, let me see if I'm understanding your question correctly, so you're you're talking about the problems with accessing data on the ipfs network in general, yeah.

G

Okay and content delivery, yeah yeah, I.

D

Mean accessing the data through the DHT uh I mean sometimes it's simply slow, because it's because of the way that DHT works or because I mean it could also be because it's slow, because the data that you want is on somebody's laptop. That is, on the slow end of a 33.6 k. Modem right.

G

Yeah, that's the point: I want to I, don't know. If maybe you can uh explain or something because uh then we need a compute power or kind of a computer layer to just not simply use ipfs as a storage to kind of serve that data faster.

D

Well, the thing is the bits have to get from point A to point: B yeah the bits themselves, and it doesn't matter how much compute you have. If your bandwidth is slow and you have to transfer x amount of bits, then it will take y amount of seconds minimum, just because I mean, if you're, if you're, on a one in if you're, in a one megabit line or say, let's just to make it easier another one megabyte percent. One say your line, gives you one one megabyte per second.

D

If you want to transfer uh 10 megabytes, you will take at the very least 10 seconds plus protocol overhead. So if you're in a slow line, it doesn't matter. If you have computer or not, if this line is slow, then the line is slow.

D

On top of that protocols themselves are slow or slower faster bit swap is not necessarily the most efficient protocol out there. I don't know. If you guys will agree with me that bitstop is not precisely the most efficient.

D

Well, there was a. There was a talk in one of the other tracks. uh I mean one of the problems with bit swap is that it's with ipld?

D

Is that it's kind of layered in the sense that if you have a huge file, you have a huge number of blocks and they are kind of assembled in a tree structure and the problem is you have to get the first block once you get the first block, you can see that what other blocks you need at the next level once you get a block on the next level, then you go to the next level and those are round trip times. Those are round trips.

D

So, if you go to on the internet round, trips will kill you round. Trips are what take over what, because you cannot parallelize too much so the protocol itself uh has these. uh Has these disadvantages, which is the one of the reasons why they came up with graph sync for filecoin, it's kind of yeah. We learned all of this stuff from from bitshop. Let's do something completely different, uh uh but still, if you are on a slow line, then you're on the Slow line and compute will not help you. There.

G

Right yeah. Thank you. Thank you. Sorry.

D

I, don't know if you guys want to address any more of that question. I kind of monopolize the microphone.

C

I'll say one thing: um so one thing to kind of keep in mind here too, is uh we're still growing right, we're still improving the amount of improvement that ipfs has seen in three years since the last ipfs Camp I, don't know if you saw the slides in the like the very opening keynote um you know, I went from like 50 seconds to, like you know, under a second right. It's huge gains, um same kind of thing now as you're starting to see this with the transport protocols he mentioned.

C

uh You know graph, sync I was in conversations the other night with somebody that was like we can do a bit swap better I, have ideas for this and there's like three ideas that were brought up. So you know people like that, and people like you, you mean you have the opportunity to influence these protocols right now and truly change them. We're not it's not too late right, we're still very early and there's a lot of gains that can be made so Cambrian explosion, I think is the term we've been using a lot this week.

C

Okay, you have a question. Are you just touching your chin, okay, cool, uh any other questions we got here.

A

Well, I know uh uh that peonado was taken completely offline and Mario just said that uh ipfs.io has been taken offline by uh you know not blocking content. I wonder uh is that the most out? What other outrageous things that might have happened uh related to you know, content moderation.

C

I mean that's, that's it's pretty much as bad as you can get um it's kind of hard to compare from being wiped off the internet. um Yeah, um it's hard to say. um That's, probably I, don't really have anything! That's worse than that to be honest off the top of my head, um just um yeah I'm, just rambling.

A

So cloudflare basically took that uh straight off uh you, you seemed like you guys tackled that right when you're starting to implement ipfs uh gateways at cloudflare, yeah.

B

Like that's I mean because of like running a web infrastructure for like some years now, um there's definitely like issues like that have been an encountered in lessons. We could take from like the building the web infrastructure that we have and like taking it like as we're building like the ipfs Gateway. Having like the same lessons like, we knew that, like matching like the like ability for a government to like have their own law on their own territory is important like especially for these problems, and so as part of like building the ipfs Gateway.

B

That's one of the first thing that has been done um and yeah. We had requests a little lack of a time like if you send an email or like uh just contact clever for like a dcml like a legitimate um like abuse request like the legal team um go through that make sure like it's like legitimate and like acts. uh Consequently,.

A

Did you say each country has its own law applied because I think that's different than what uh at least that's different than what we do. um Ipfs.

B

Yes, so the internet is like a worldwide system, but, like each country, has its own border and like its own right in terms of like what should be like, like its own laws, in terms of like what content is allowed, what it's not allowed, maybe like there's some great line in the middle, and so definitely each country should be Sovereign in their way of of like deciding um like club, would have a different approach depending on the country.

A

I guess uh yeah last question for me uh and um what uh I guess what would make it easier? uh What's what's the biggest pain Point using Kubo uh that I, don't know you wish would would be different.

B

um So if I start um I, think Kubo as it is, is just working like when I use, Kubo locally, it works um so I think Kubo inherently should not try to like be optimized for, like Gateway and like web scale providers that all the implementations like could do that better and like would make different trade-offs. That doesn't mean that, like what is being learned like at the web, scale cannot be improved and like improve every user performance. At the same time, I really think it's like two different worlds.

B

If we take on like the indexes, for instance, that like should improve content routing, that's something that came because of like Gateway scale is just too much for like one Kubo node to handle and like every time, query the DHT. So like you have an indexer, you just make one HTTP request and you get content Discovery, just in one request.

B

Instead of like I, don't know thousands of bit swap not counting for the DHT and that's something that kubernet would benefit um and overall I think it's okay to have to like two or like more different implementations. That would do multiple trade-offs.

C

Yeah I I think you kind of said it really well. um Kubo is really old, uh I mean it's like 2014 I think is when it first came out and it's just it's just kind of hard to you know this is just software in general, it's Tech debt right. It's a lot easier to start fresh and um build Purpose Driven than it is to kind of like steer a cruise ship.

C

um So Kubota has a lot of things that are really good and it's been a very valuable kind of like jump start for the ecosystem, but there's a reason we're talking about it right now and calling it Kubo and not go ipfs. It's because we wanted well, not we, but like the general ipfs ecosystem, where I determined that it was beneficial to not have only one go ipfs implementation. If you want to make a different one, that's written and go. You can do that right. It's not that um every implementation needs to do everything.

C

It can be purpose built. So we've we've talked about things like elastic this week. We've talked about things like iro. These are Purpose Driven, ipfs implementations that are either written. Like in Rust for hyperscalability- or you know, Cloud native, such as elastic they're, solving different things and I think you know that's important to keep in mind so like Kubo, it works with your local file system. It. You know it does all sorts of crazy things right. You may not need that at Cloud scale.

C

um You may just want to do a pinning and service, um so yeah I'm, very thankful to Kubo, but um I don't want I, don't want Kubo to have to do everything.

D

I think that uh we sometimes lose track of uh I'm going to be I'm, going to wax a little bit philosophical here, uh but I think we sometimes lose track of the name of the company and the name of the network. It's protocol Labs. What does that mean? The important thing is not Kubo. The important thing is that there's a protocol meaning that there can be iro there can be uh JS ipfs there can there can be elastic ipfs and as long as they can all talk to each other everything's fine.

D

And yes, uh let's not try to make a kitchen sink. That does you know it's a it's a floor wax and it's also a dessert dessert syrup. Let's not try to do that. I think I. Think that uh yeah iro is a elastic ipfs uh mini Ip. Is it ipfs, tiny, I, think or ipfs mini is also another tiny implementation for embedded devices. Ipfs embed I mean it's wonderful, that there are different implementations for different uh for different purposes. I mean trying to we in my previous job.

D

We actually were using go ipfs as an embedded system until we actually went with ipfs embed inside a rust binary, and it was so much better so yeah I mean the important thing is the protocols that are that are there and that they will enable multiple implementations for different things, so I'm all I'm. All for having all the implementations out there.

C

um I want to you touched on the really good thing here: um kind of like jumping into the individual implementations themselves. uh One thing that you may notice, if you're going through like the open source code, this is all open source first say like Kubo, JS or even like iro or elastic right, like everything's, open source. Here these these implementations of ipfs you'll, find under the hood, are built on tons of little reusable modules that you can swap in and out.

C

So jsipfs right is tons of these little npm libraries are being used in elastic ipfs right now, because well they do a great thing. They work with cids. They translate right like same thing with go right, I mean you can break these little things out and plug them and play them into your own implementation. If you want to do something better, so there's a lot of opportunity there for improvements.

B

And I also had like one thing on that is like when I have like specs for ipfs, which was not the case at the start of the year, and definitely that really helps improve to like inform like how do we do the implementation? What do we test against? What's actually like a reference before like if you wanted to do an implementation, you had to go like through, like go ipfs codes or generous ipfs code to bug people on slack just to try to see what was actually the thing you needed to implement.

B

So the fact that you have specs is also really helpful through that.

A

A

Well, thanks guys and thanks thanks to the audience sorry uh yeah, this has been the ipfs service panel.