Filecoin Retrieval Market Builders Mini-Summit, April 2021, 6 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RETMKT Builders - Filecoin Secondary Retrieval Market

Description

Peter walks us through the Secondary Retrieval Market from the Retrieval Market Builders Mini-Summit in April 2021.

A

If you've had a look at the slides online, then you wouldn't have seen this one, because I only put it in about- I don't know half an hour ago. So this is the team of people that we've got working on the um project, so you've got myself down the bottom left there and you've got raphael and um zhenyang igor samir, nelson and then frank and shu yao also are doing some support work as well. So um if you see names in slack channels, then these are the people involved.

A

So today we're going to really quickly race, you through um some stuff. So what are we actually working on and let's wait for it to turn up all right. So we all know what primary retrieval is where you are getting. The information directly back from a minor and you've got to really know where the content is secondary. Retrieval is trying to have that highly high availability, off-chain sources that can really try and get you that low latency.

A

um That one was talking about earlier, and so you also want to be able to discover where content is as well and so and we want to have a market mechanism for it. So having another look, so this is that primary retrieval where you need to know where the content is, and so what we're trying to do is come up with a system where we have discovery.

A

We have caching, the latency is really low, it's massively scalable and there's an incentivization scheme, and so we've got this idea of a cid author, and so the idea is that this is essentially a retrieval minor saying. Look. I will give you this content I'll supply, this content at a certain price, and this offer is going to expire on this date and so doing that sort of having an offer means that, rather than having to have um you know, have this sequence on the left.

A

You can have the sequence on the right, and so that takes out one whole round trip and taking out a round trip, reduces your latency.

A

And the other thing that it gives, though, is that you can have these cid offers in a node which is actually near the user so rather than the user having to ask all of the miners. Hey, do you have this content? How much can you supply it, for you might have a gateway close to you that can give you all of the offers all in one go and again that really reduces latency and the thought that we've conceptualized is also actually having these gateways cached local content, that's hot!

A

This is not implemented yet, but that again would massively reduce latency, because you would have content. That's near you.

A

So we've got the idea of a single hop dht, and so what this is all about is that, even if um you know a single node is not going to be able to keep two to the 60 cids, um all in one computer, it's just not possible, and so the nodes will have to be relatively selective about what content they they understand where, where it is but as well, you could.

A

We've got a dht which essentially stores cids with gateway gatewayers that have similar ids, and the reason for this is that, then all content can be discovered. So if you're a certain gateway, you know another gateway. That will be able to tell you where some content is, and so we've got a single hot dht.

A

So you can do direct lookup, so the roadmap is- um and so you know caveats all the normal caveats. These are plans and things change, but the road, the long term plan is this um where you've got a overall system. So I'm going to just walk you through um this and give you a few colors. So you've got um your client, your application off to the left there, which has a library that it integrates with, and it can talk to gateways and um say hey what content is available.

A

I've got this cid, um this pcid, where you know who's, storing this content and how much can I get it for and then they will be able to either have the information directly, because it will have been published by retrieval providers that they've subscribed to or they might have to use that single hop dht to talk to another gateway to find out where the content is so we've um to allow discovery between the gateways and also between these provider, components that are pro that are publishing when content has been stored and publishing those offers, you need a registration system, and so our thought is that we'll have an inbuilt actor that will be integrated into the lotus nodes um to administer the systems.

A

You also need um a admin app and a, um and that would integrate with a retrieval gateway, admin, library and the provider library, and so that's the overall system that we're working towards and so right. At the moment, we've got poc one which is available right at the moment, and that gives you essentially a system which is not connected at all to the file coin system we thought to um and to have as rapid development as possible.

A

We would just work on um stuff that was isolated from filecoin and so you've got um gateways that can that get published information from providers.

A

You have the client library which can interrogate gateways and so you're able to publish some data via the provider which publishes it to the gateways and then the gateways can be administered and then you've got the client.

A

So in at the end of june, um we're going to be having payment channel integration and um we're going to be aiming to have that client library, not just in golang but in javascript as well, and have that single, hop dht working.

A

And then, for uh as a stretch, goal it'd be great. If we can um have some wallet capability into that client library as well.

A

Another thing that we're looking to do for the end of september. So when we're finishing this initial mvp is have that registration service as an inbuilt actor in a lotus node as well. Try and deploy the system on a test net and have it production ready.

A

But there are more possibilities, and so um a really big possibility actually is something: that's not retrieval, based, um but storage based. So at the moment um you know how we you've got a lot of latency involved in a lot of round trips. To do these um storage um deals. So wouldn't it be good if similar to the retrieval cid offer so that offer where you're saying I will give you this content at this price.

A

um If you had a minor that says I will I've got space, I've got enough space and um I'm prepared to store, say a certain uh at this price per megabyte, and so then people can take them up on their offer. So you don't need to do as many round trips to do a deal. You essentially will have a openness like a a blank check kind of saying. Look. If you sign this, then we've got a deal and um yeah. The other thing is having those cases in network.

A

Another thing is payment brokers, and it's interesting to hear. Other teams are also thinking about this sort of thing as well, and so that would facilitate payments between entities that don't have payment channels set up yet and that will greatly improve the system there um because then you'll be able to have clients be using gateways that they don't have payment channels with, and you will allow the gateways to not have to worry about having payment channels with all the other gateways which is important for the system.

A

So what did we deliver in poc, one in to be precise, so I'm going to race through this quickly, so we had, um as I showed in that other diagram. We had the system, um so we've got an integration test system, which is our essentially application. We also had some documentation. We had the client libraries which will allow you to integrate with the system and um the actual node, so the retrieval provider gateway and register. So the register was a mock for those in-built actors, and so um the first thing the integration test system does.

A

Is it configures the system, so it generates key puts it onto the retrieval gateway, so the retrieval gateway key and then it registers that gateway into the contract. The register then um have um configure the provider, and so you set the provider up and maybe publish and put some content into the provider. So it's got some cid offers and then you put it into the register and then the gateway and provider discover each other, and then you add, so you tell the client look.

A

I trust this gateway, and so it's a it's something that we're going to use for um doing retrieval offers. You then um talk to the register to say all right. Well, this is this gateway that I mean that I want to use and you find out where it is in the in the you know. So what it's ip address, what port it's using, what key it's using and then you contact it. um Oh yeah and then you're putting you.

A

You publish that content into the provider which publishes it to the gateway, because the gateway is subscribing to content from that provider and then you from the integration test system, you use the client to request that offer and then you've got the cid offer other thoughts. So I think that these storage offers could really give um a lot of advantage to the um filecoin network.

A

I think that then you'd have you know pre-signed offers so that then you could just have this small client library, which you could so you could say, I want to store a file and then it can talk to a gateway and get a pre-signed offer sign. It store some data. um So, to my mind it feels like um something good, but maybe I'm missing something or uh yeah so anyway, that's all from me. So thank you.