IPFS IPFS þing 2022, 9 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: State of IPFS and HTTP - @lidel - Browsers and the Web Platform

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

Hi I'm lindel I'm working at protocol Labs within rpfs stewards group, I, was sure I will come up with a better title and I left that on purpose, also to remind myself that the direction of this talk uh was in the flux during this week and I redid, a bunch of slides based on sessions related to writable gateways, the way we think about https to transport and so on, and also happenings in our why their ipfs ecosystem, where we rename go ipfs to Kubo and create created more space for people to create more implementations.

A

So it's work in progress, but I think it's good for for that to be aware of recent happenings. So it's a state of ipfs and HTTP as today, where we are why we are here um what we need to do uh short term and what exciting things may happen. So, um for a long time, the way people thought about ipfs over HTTP or the presence of ipfs within HTTP was shaped by a single implementation named go ipfs. It's now now called Kubo, but the way it worked, you had a the demon.

A

You started the demon, it was ipfs node and it opened to Ports um on one part: it had ipfs and ipns namespaces and on another boards it had API v0, um so the one part was the Gateway and the second part was the RPC specific to this implementation of ipfs to this specific demon um and the API.

A

It's still at our docs website even today, and it's called HTTP RPC API. So is it like ipfs API for http? Well, that's the only thing there with HTTP. So no wonder people use it as a way of and they use it for creating mental models for working with ipfis.

A

um It's the same. The problem is: this interface was never designed to be used, consumed by like external entities for sure to not be used in the web browser context. It was designed for a to be consumed by a command line client, so your command line tool detected. Oh, is the demon running. If so, I will talk to it using this RPC and that's why it looks weird people are easy to rest.

A

Of course it's not it's the proprietary RPC specific to Kubo uh implementation of ipfs, so um I think one lesson learned is that ipfs implementations should be free to have their own rpcs and we already see that it's not like. We are suddenly permitting people to do that. Jsip fast already went their own Direction with a subset of rpcs. Initially it could just copied what Cooper did I wrote. Ipfs implementation in Rust uses grpc.

A

They already went their own way, and the point here is that those rpcs are fine if you want to interact or orchestrate the specific implementation of ipfis, but it's not the stable foundation for you uh to create long plan for us to create a long-term HTTP story um for ipfs, and that's why we go back to that Gateway, which was present. It was always present, but it was undervalued and also under.

A

um We never like invested fully in the Gateway um as the implementation, agnostic, generic HTTP interface for retrieving and maybe ingesting content address data, but I would like switch our mindset away from Legacy rpcs specific to one implementation and focus wider, ipfs, Community or people who care about content. Address data, and not really about specifics of How It's implemented to use gateways as a mental model and I will try in the following slides to make a case that it may be enough for 80 percent of things that you want to do and for remaining 20.

A

You can use implementation, specific rpcs.

A

If you use Gateway, you can swap in implementations, URLs stays the same. The behavior stays the same, I can swap probably I could swap a Cooper for Ira and I would not even notice it companion may not companion May notice it right now, because it's still talking to that RPC. But after we write it for manifest V3, you should be able to just swap the gateways um Brave right now it embeds Kubo and orchestrates it using RPC, but the interface that is backing ipfs and ipns protocol handlers. It's not that RPC, it's the Gateway.

A

So as long as the implemented new implementation, better implementation, more suited for this, like used in a browser comes in swap swapping it up, will should not require Brave team to change stuff, um so implementation, agnostic apis, are what I'd like to talk going forward and implementation specific apis. They have the replaced, but I don't think that's the way of doing interrupt so.

A

Closing go ipfs era, what's new, what's recent, what new possibilities in the Gateway world happened that are good to know when the most important part is. We now have specs for HTTP gateways. Historically, we had implementation in go ipfs and jst ipfs, but website docs. They said, uh there's RPC. There was no Gateway presence there. The specifications for HTTP gateways are in ipfs, specs repo and the specification itself is split into multiple layers. The most of common behaviors are in path Gateway.

A

This is the oldest type of Gateway um and it's a low level specification around hdb protocol. uh We have trustless Gateway, which is the subset of that only for sending blogs and cars. uh It's for uh fetching content address data without trusting Gateway and we have a web. We have a Gateway types which are specifically designed for use in a web, sub-domain Gateway and the nestling Gateway provide, namely origin isolation and that's final web 2 Gateway, which is DNS link.

A

So I I mentioned I shouldn't like ourselves that our dogs still say our PC API uh and no HTTP Gateway, so that we are in the process of renaming, go ipfs to Kubo and when Kubo 0.14 ships, we will merge a pull request which really reorganizes uh documentation around our apis and command line interfaces and part and HTTP. Apis are part of that. So we will be making it very clear that the RPC, which was historically just named RPC, is Kubo specific, that the CLI is Cooper specific and we will have this new entry.

A

As the very first thing, people see HTTP Gateway as the the default interface that the 80 percent of use uh should be happy with at least for data. Retrieval.

A

So we talk about Gateway. Apis I mentioned that during the first day, there's no API. There are just content paths and you request content path from the Gateway and you get the data by default. The Gateway will do the work for you, it will industrial, even if it's Unique so fast. It will just realize it for you uh if you use ipns or other immutable pointer. It will also resolve that um if you can use it in a browser context, it will provide you with origin, isolation, pair content rules.

A

However, the interesting thing that has happened recently is that we finally solved the problem of verifying the response that the Gateway was sending to me. So historically, it was just files and directories like the regular web, uh the serialization all the content, verification happened on the server.

A

It was fine if you trusted the Gateway. The problem was when we started talking with a external projects interested in content addressing- and we mentioned that. Oh there's, this Gateway for the interrupt with http.

A

There was always this question like what is the value of Gateway. What is the value of content addressing if I'm not able to verify the response?

A

So in go ipfs 0.13, because it was still the old name? We shipped a blog and car response formats, and now you are able to opt out from the sterilization Gateway, which means you can verify the response. You don't need to trust the Gateway. It also means people who don't care about the delegating trust to Gateway, have a smaller surface for implementation, and they now can only Implement car and block response formats.

A

So, where we go from here when we finally have content addressing on gateways for the very first time after years, um this may be controversial, but I think it will be also entertaining if you like, squint your eyes. A little bit. I will show you something um so HTTP itself supports, gets for getting data, you are able to ask for a Blog and you are able to ask for a bag of blogs. Http also has head requests when you only ask for headers without actually getting the payload.

A

um So the question is: are we are those Primitives enough to do something like HTTP transport and what what existing transport do? We have there's the graphing and so on, but I looked at the beach one, because this is the the simplest one. So this is how bit swap Works um like one peer asked. Another peer: hey: do you have the CID?

A

Yes, I? Have it okay, I want the CID I I won't block for that CID and the block is 10. That's it that's the entire this one. The complexity is inside, like optimization characteristics of who do I ask I, like remembering who was sending data to me related to Doug and so on, beatbox sessions, but the low level, the lowest level. Primitives are literally just two request types like do. You have the data can I have the data please um so now squinting in HTTP, you can send the request with cache control on leave cached.

A

And if you send that to some HTTP endpoints, it will either and it hits like head request. You ask: hey: do you have this data? Yes, I have the this data? Oh can I have a block or a car. Yes, you can have that. So essentially uh you could talk to uh you could talk to a Gateway as a beer and we could.

A

We have like all The Primitives will be there in Cuba 0.14, because I believe this is already support, for only if cache header is already in master and it will check hey uh when Gateway processes, the request it will check local data store. Do I have this root Block in my local data store. So essentially now, oh, if there's no roadblock, that they don't have blog and so on, so it may be even better than B12, because you may choose do I ask for a blog or do I ask for an entire sub branch.

A

And now, if we have this transport, HTTP transport exchange for exchanging blocks and dogs, what would be the main use cases well for sure, mobile browsers for sure iot would benefit from that, because they cannot run peer-to-peer at least not all the time. Maybe they also don't want to announce blogs they have. Maybe you don't want to broadcast your browsing history? Maybe you have limited storage and you only have ephemeral cache in the memory, um but then it's a useful transport like it.

A

May it's no worse than Beat Swap, probably um so, maybe just regular Brave could switch to this uh and users who have not imported any data could run it by default, um and maybe we could make it even better. If you are able to request car with selector, then you get even better optimization around that.

A

So just an idea, but Primitives are digesting. So what are the other low hanging, fruits around gateways and things that we have or may have with very little effort, retrieving directories that are stars uh if, if you still want to delegate trust or if you run Gateway as an interrupt glue in your infrastructure, being able to fetch entire directory tree pipe it to a tar, it's a small Gap that we probably need to close it's kind of like paying off technical debt.

A

It's possible to do this right now, using that RPC API, but it's missing from the Gateway another thing that's possible with the RPC from Kubo, but it's not possible at Gateway right now. It's requesting different ipld formats than using CFS.

A

Like all the code is there. We just need to expose it to the Gateway namespace in a way that it's not painful for implementers and, of course, register content types that they and I like we did for block and a car.

A

Then website hosting, we could see now that we have specs and we can like specialize the Gateway implementations uh for website hosting, could Implement things like supports for redirects, enabling people to move their publishing from the Legacy solutions to ipfs, with as little friction as possible. uh What uh more interesting, more interesting things that we could build uh if we start using Gateway as the the building block um I mentioned, that um the argument against gateways was that oh there's no way to retrieve data in a trustless fashion, we've added block and car responses.

A

But if you use DNS link, there's still a problem because you need to resolve DNS link foreign. The DNS link is the name which you can use as a mutable pointer, it's kind of like a web 2 Gateway. The problem is, the problem is uh to you: don't use dnsic directly. You use it to resolve the XD record at a specific domain name, and that has an sdid.

A

So from that point you have the verification, because you can ask for a car for that that dog, but the act of resolving DNS, the txt record was always a problem. You never had that end to end Integrity guarantees, um but maybe we could do better. Maybe we could uh now that we have car response format.

A

uh Maybe we could include DNS stack proofs along with responses for that domain name and something like Brave or other web browser vendor could, behind the scenes, request car with DNS Tech proofs, but in the user interface it would look the same way as it does right now. You see ipns, you see regular website. Just like you don't see. Is it going over HTTP 3, HTTP, 2, some cdns or load balancer or whatever some things could be abstracted away and some additional Integrity could be included without adding anything new uh the car.

A

It would be just additional block the format it would be: Dax dbor, so limiting the surface for implementers and then all this talk was about retrieving data from ipfs. What we, if we could push data to ipfest somehow we were calling this writable gateways, but I'm experimenting with different names and I I picked the most. The one I did not like the most just to force people to provide a better name.

A

So ideas welcome, but the idea is, there are different types of things that you would like to ingest into ipfs a file a directory more than one level or just random content address data that you created with external tools and that's just one time import. What if we could uh start patching existing data do I need to like fetch entire Wikipedia, to add one article, probably not. Luckily, we have uh now IPL depatch.

A

um The spec is in IPL Dio website in general. It's based on Json patch and where it gets interesting, is that IP LED data model can be represented as dock Json documents. Those are the value, Json documents.

A

So in theory, you could be patching arbitrary, ipld data using this and if we add support and gateways, you could be interacting with Gateway, with Json, without any additional software without any additional libraries and so on maybe exist like we there's some technical depth on the ipfs namespace and it's around file system. So it's still the class directories.

A

If you are living in a trust trusted context, maybe we could have IQD namespace with uh more advanced transformations and I will post here. I? Think I I don't know if I run off of time or whatever, but uh that's kind of like the landscape. I think the takeaway is that uh the gateways are here and it's probably the thing that we should be talking. The most are like investing the most, because that's the the common interface and the smaller surface.

A

Everything else will be painful for us and what was and is already so um specs are there. If you want to propose improvements, the read the first Improvement proposal for more details and I believe that's it. uh If I have time, I can take questions. If not I will remove myself.

A

Thank you yeah. We can do it. We can do a question or two.

B

um So you mentioned all of these new Gateway capabilities. Are there non-cubo gateways that are kind of like signaling internally that they want to support these new features? Or is this something where we have some ideas, but we still need to convince everyone else to adopt them so.

A

Interestingly, before we added like trustless responses to Kubo itself, there was like really no point in trying to fight that battle. Without uh doing the temperature check from people who are already operating gateways. Is this something they will be comfortable with? uh So we talk with uh existing Gateway operators, um end up like my layers and also that removes uh risk from them for some of them. So now, if you are a person, you can run only the trustless Gateway and you no longer host the serialized data, which removes some risks.

A

um I lost my train of thoughts. Could you remind me.

A

Oh yeah yeah, so I I wrote a rust implementation. They already like they prioritize Gateway. They use it for measuring their performance and so on. So for sure we also have uh Pub Gateway in jcpfs, so hopefully over time, we'll see more and more implementations.

C

Kind of to open that discussion, I think in general, um you need both in that. If we don't get major ipfs operators to run it, it doesn't actually matter even what we add to Kubo, because it'll just not run stuff, um so I think we're in a little chicken and egg of where we're switching to a new process, and that's good. So Vision has one person dedicated to the Kubo code base, even though we don't intend to do that, because we wanted to show a working implementation in cubao.

C

So I expect this to change over time, where a first reference implementation may not be in Kubo. Exactly uh and and that'll kind of, like switch over over time is, is what I expect to see. um This was really interesting on a very high focused on HTTP. Obviously, the default transport that our friend the browser communicates to uh I want to ask what you think about um the priority of other transports, like your friend and mine webrtc.

C

That also is a a browser friendly transport yeah.

A

uh I'd say it will depend on the runtime uh and by the runtime I mean both the execution environment, but also like the hardware itself. So, for example, webrtc sounds good, but then you try to run a lot connections to peers in chromium and you no longer have a good time or if you switch to a different tab. Existing connections get like hard throttle, so it depends. I, think that the power of HTTP is in that it's the lowest level, it's kind of like the lowest denominator of sorts. You know it always work.

A

It may not be as fast as bit swap, let's say, or it may not be as flexible. It was already truly peer-to-peer as webrtc, but then it always works and I think when it comes to implementations, it will depend. If you have a mobile browser, it will probably to save a battery default to http if it's like too slow. If non-gateway has data, maybe I'll spawn a small peer-to-peer, node ask local peers for data and then shut it down.

A

If I have some data that I paint and I want to like to keep providing that at least for 24 hours, maybe there are some code responsible for running the P2P stack with bit Swap and other things, or maybe we have. uh Maybe we just create a car and push that to external Gateway which, like ingests it and caches it, maybe like a Content address Alliance, could figure out a way for this, like a temporary ephemeral thing when I like closed my laptop, it's still around right.

B

You said you have somebody prototyping, something in Kubo is that the redirects yep.

A

B

Thanks Lionel.