IPFS IPFS þing 2022, 10 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: IPFS Gateways - Implementations in HTTP world - @lidel - IPFS Implementations

Description

Gateways in the past, present, and the future. IPFS Gateways - presented by @lidel at IPFS þing 2022 - IPFS Implementations - https://2022.ipfs-thing.io

A

Hi, I'm lydell, I'm parts of ipf stewards group working at protocol labs for a while, and I got two talks so for giving me. I speed, run I'll, try to say those things slowly, I usually say them too fast. um The first one is about iphone's gateways and the um the meme here is that ipfs gateways are implementation of ipfs for http world.

A

So what ipfs gateway is is a bridge between a traditional http clients and if infrastructure and the data content address data, that's on ipfs, either public or private and through the gateway existing http clients can download files directories if it's unix fs, but also now, since recently released of kubo any ipld data in block or car format.

A

More about that later, but in general you fetch it as you would fetch from a regular web server. um So the plan for this talk first, why http matters, uh how the api, how the a gateway api works? uh What are implementations architectures around implementing ipfs in hdp or using http, and quickly? What's new? What's next, what we hope to discuss this week?

A

So why hdb? I think everyone is on the same page, but just for the completeness, uh it's the universal blue. It's the most successful interrupt uh transport. I argue for at least retrieving data from ipfs from the first day. It maximized the way we can interrupt with outside world http clients tool services could run a gateway and integrate ipfs. That way, it's also maximized utility across all different use cases on the common line you could curl stuff using http gateway.

A

You could load content in the browser delegate, the content resolution and the serialization to the gateway mobile clients could fetch data without burning batteries. Iot devices could fetch stuff from gateways. Cdns could use content addressing for transferring and coordinating data so like with the fact that we have this universal glue out of the box.

A

We also can leverage existing caching solutions which are not designed for ipfs, and we could then move beyond that and create things that are designed with ipfs in mind, minimize time to the first byte for subsequent requests, maybe even for the first one, and when it comes to talent and developer experience, people know http. People know how to work with those services how to debug it. It makes it easier to onboard new developers into our ecosystem, using http as that gentle onboarding path, both developers and then users.

A

So let's talk about gateway api.

A

My personal belief is that the best api is no api, or by that I mean like the api, that you don't even see that it's an api and on the gateway.

A

We don't have a specific api because we have a content path and you just request that content path. You have immutable namespace for ipfs, you have a cid content identifier and you ask for that specific path on the gateway and that's it that's the api and just like, I didn't said, is ipns part of ipfs. Well for sure, it's part of some gateways you are able to resolve. Mutable pointers, be that ipns records which give you cryptographic, identifiers or even like human, readable, dns links.

A

It's a bit lucky to put dns name in the ns label that you could that.

A

But when you start using gateway on in a web browser, then suddenly you need to take in account that the web already had some solutions around security. Entire security model for the web is around origins. So in the browser context, we also support requesting content routes by putting namespace and the root identifier in the subdomain, and that gives every website every content should loaded this way, its own sandbox, all the web, apis all the cookies storage and so on.

A

Api permissions for camera microphone, whatever you do in your web, app are scoped to that content route and the same applies to mutable names and when it comes to gateways from the very early days, you are able to use it as a regular web server and the way web works.

A

The mental model is files and directories, so you were requesting a file or maybe you requested a directory, so the server returned to your directory listing, and it's perfectly fine, but there's a nuance here that if you do that, if you are limiting yourself to unix fs data content, that is data on ipfs and you allow gateway to do that.

A

The serialization to reassemble the dag, that means you delegate, trust to the gateway and for a long time I had this phase, because uh the gateways were the use of gateways was limited because there was always like you need to trust this gateway. It's fine! If you run it on localhost, it's fine. If you learn run it in your local network, your private cloud, maybe as your back-end service, but the moment you start using other people's gateways, you suddenly lose some of that content address guarantees.

A

So that's why I'm excited that. Recently, we've closed that gap and from last release of kubo uh it's possible to do trustless retrieval from gateways. uh You can opt out from this data serialization and just ask low-level content address data.

A

If you ask for a specific cid using let's say I want a route block or I want a bag of blocks in a car format, then the gateway will not do any work and give you data for that cad without any additional resolution.

A

So delegated retriever, like the retrieval with trust and the trustless retrieval, are two ways you can use apis today. um Let's talk about different implementations, so how that the implementation work well, the first implementation was in kubo, uh which I didn't already covered. It was like a long long time ago and took us a while to close the trustless retrieval gap.

A

um If you run kubo or if you run ipfs desktop or any other implementation that which follows the specs, you could run your node and that node is accessible on your localhost, but that's a single standalone, http server, essentially.

A

Today we have many implementations. A single http server is not enough. There are more advanced deployment architectures, because the gateways are used by millions of users um and to back that claim this. Those are, I believe, those are stats from our ipfsio and diva link, gateways alone, uh it's over 800 million weekly requests. We got millions of unique users and when you deal with those numbers, you no longer can run a single http server.

A

So we see http implementations of ipfs of sorts, so the kubo at the very bottom often does not even process the request when people make a request, for there are like multiple layers of load balancing. This is just an example. I don't even know if it's still exact architecture, we do, but it's one of our iterations. So here you can see two layers of load balancing. First one was at the at the routing level.

A

When you ask for a domain name, you would get routed to the closest cohort of load balancers um and then in this example, it's like 5050 and then that load balancer.

A

Let's call it let's say it's an nginx: it will route you to a specific cluster of ipfs nodes and on each layer you can have http caching, so most of the requests after the result for the first time gets resolved really really fast, and it's not just us with people realizing that, oh, I can implement custom thing, http things that use con help people to use content address data. We see um also like different architectures, depending on the use case.

A

So, for example, we have pinning services pinata and photo and others who provide dedicated gateways for their clients for a faster ahead of a queue retrieval of the cids data. They have. We see cloudflare, flick, streamlining the website, publishing experience to things like dns link, automatic pinning of assets of websites and so on, and we see very, very crazy things like uh nfd storage link gateway, which is a new beast, or it's also like show what happens when you push http implementations to their limits, given the existing technology. So it's a racing gateway.

A

When you ask this gateway for a content, it will try to fetch data as fast as possible. uh But what's interesting here and then once it's retrieved, it retrieves it, it will cache it. But what's interesting here, it's an example when the line between http caching and ipfs caching is getting blurred to the point when it's essentially like http cache becomes ipfs.

A

There's like no data, separate data store for ipfs at some abstraction layer. Here you simply like leverage http for ipfs caching, so that was me speedrunning, but I think the interesting part for folks here is what's new and what's next new blogs and cars trust us retrieval just landed in kubo 0.13.

A

I know folks already started using it because they can request their blogs and cars and do anything on on their end. You don't need to use ipld with this. You could invent your own. Ipld helps another things. Oh maybe if you want to use ipld, maybe we should add a built-in support for data models, so people could have a better experience or maybe people could request ipld data which follows data model in json or cbor formats.

A

If you trust the gateway- and you want to like ask it for like a delegated retrieval, you should be able to like fetch a directory in easier way than oh right. Now, you need to request a car maybe like requesting guitar is fine um and beyond retrieval. uh This is like a sneak peek like oh yeah. We still kind of solved, http solves the retrieval, but what happens if we want to put data on ipfs? Maybe pin maybe modify existing documents, so this is my shout out for the session writable gateways.

A

I believe it's happening wednesday join us another things improving website hosting. We have already spec proposal for supporting redirects file following new process, which I will describe on friday, and if you want uh to know more about the direction where we want to take ipfs in a in in http world, I believe there will be like follow-up session uh on friday after writable gateway discussions. So contents might change, but it will be interesting.

A

A summary, and one thing is like thank you and second thing is we have gateway specs now, uh so it's like an ester egg from me. I should mention that in the beginning, but this is the slide that stays for the longest, so we have gateway specs. You don't need to use kubo for gateways.

A

You could just go to specs repo, and if the specs are not clear enough open an issue, we want to have the specs which allow anyone to implement a compliance, server and or clients to follow the spec and be able to consume content address data uh in a way that it's independent of the specific seven years old implementations implement your own gateway experiment with caching, so yeah. Thank you.