Filecoin Retrieval Market Builders Demo Day, 13 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Retrieval Market Builders Demo Day - Myel & KEN Labs - October 13, 2021

Description

Every other week, the Retrieval Market Builders get together to share progress in their projects in a demo format. We want to sincerely thank all of our collaborators for demoing their developments and helping to improve the FIL Retrieval Markets one demo at a time.

October 13, 2021, Demo Day: In this video, you can find quick demos regarding the following -
• Myel demoing payed retrievals
• KEN Labs discussing their IPFS crawler and IPFS indexing end to end testing suite
__________________________________________________________________________________________
If you would like more information about the Demo Days please follow the link below to the Demo Days planning document.
https://docs.google.com/document/d/1IrNbBm2_P79Xi2fXFj9YUfQ0PjWjreGdPYFjFntD1QM/edit

A

Okay, we are now recording um welcome to this retrieval market demo day. um The today's date is wednesday, the 13th of october um and we've got an incredible lineup. Today um we have got firstly um mile speaking about paid retrievals and maybe cloudflare worker retrieval, and then ken labs, um speaking about the ipvs crawler and ipfs indexing work. They've been they've been uh working on, um as we always do.

A

Please put any questions into the retrieval market slack channel on the far coin slack and we can make comments and answer them asynchronously, um firstly, during the meeting and then after the meeting for anyone who isn't able to attend um so without delaying any longer um over to you, thomas and mile cool thanks. uh Let me.

B

Share my screen.

B

All right so um so for this one, I'm going to be demoing, uh some paid retrievals, uh so this is kind of our the javascript client that we've been working on over the last uh few months and um we're finally getting a few uh payments through the filecoin payment channels going through. So this client is basically connected to the falcon mainnet, so this demo will be on mainnet um and it's running into a service worker uh which allows us to run on a different background thread, uh so it doesn't slow down the web page.

B

uh So I'm kind of gonna load a few small files. These are all kind of um icons and so basically uh they're all individual svg files that have been added uh together, and so I just kind of uh this was just um basically retrieving them individually. uh So there's a few interesting things happening here. um One of them is, I'm actually loading from an existing payment channel. So my payment channel already has funds in it, so I don't actually need to wait for it to load which allows us to load this page immediately uh then.

B

So we have a few interesting concepts. So, for example, this payment channel has already been used for other retrievals, so it has kind of this concept of lanes. So I'm already kind of checking uh and verifying that the payment channel is, you know between the right parties, so my client and the provider, and then I can assign a new lane based on what lanes already exist and then, basically, initially, I make a first payment to um which is kind of a tiny payment, 300 data fields.

B

So it's basically just allowing me to retrieve the entries um and then figure out what files are in this entry. So then, from there I can target the specific cids and so there's actually five retrievals happening in parallel for this. Actually, the first one is first getting uh the entries and then the four, the four ones for all the these files are happening simultaneously and then, once it's completed.

B

uh Essentially, I've sent all my payment vouchers to the provider and then um the provider is just uh it's on the provider's end to kind of sell this, um and uh you know redeem the vouchers so we're still kind of working on on the process of you know, updating the payment channels once the the retrieval is done and when to basically settle them and collect the funds on the provider's end, because we want to keep the payment channel open for as long as possible so that we can use it for multiple transfers um and in order to also save gas.

B

So another thing that we're working on is, for example, merging the lanes together, so we can redeem the vouchers as a single transaction because, for example, all of these are individual vouchers on the separate lanes. So this could be combined into a one lane and so one message to retreat uh to redeem the entire funds. um So this is working pretty well at the moment.

B

The there's still a few things to basically adjust in terms of how many lines we want to use um and then basically how to when, at which point do we want to collect and um I mean, settle and collect the channels, uh so this is kind of all the demo for today. For us thanks.

A

Great thanks very much and yeah any questions. Please just drop them on the on the slack channel um and we can also go over to the end if you've got time um and yeah next over to ken labs.

C

Hey guys uh so I'm waiting from kin labs, so I'm also going to present you some items in this demo. So, first of all, it's the indexing performance testing. We rephrase- and this is the first one this one. So the other thing is a fs sniffer, also known as crawler, so with monitoring enabled will also show you the trend about how we, how much or how many cids and the providers we recapture and uh how the progress looks like okay. So, first of all, this is a the performance testing phase one.

C

We have implement a test suit for the intent testing, so it's based on rubber framework, so, ideally that we we can set up the different scenarios for different file, size, 10, megabytes, 100 and one gigabytes and ten okay. So this is how we we plan to to do it. For, for the future, we haven't run the the actual testing and get a report.

C

So probably we will have more detailed report and the output to share in the short future and uh yeah. This is the other thing I want to share, so this is ipfs sniffer, with monitoring enabled we have done a couple of pr code change in ipfs and we are using the most of the lower layer, modules or components of ipfs so for the poc and the the quick development.

C

So we didn't split them from the ipfs, so in general we would collect the following metrics, so total providers we captured or collected and total sits with provider found and indexed. So we don't have the index for the totals it's captured, including a provider found or not, and so this is the three metrics we have.

C

So if you see it, maybe two small I try to. I can show you the in the wrong time so here this is the the dashboard we set up. uh So you can see the trend here. It's just growing slowly here and you can see in the very beginning it's growing dramatically and uh because this is it's not linear, so we are using the log rhythmic, what's called scale type, so you you can see here it's kind of getting flat here.

C

So in general, it's slowly growing and we have three metrics here so total provider, which is a green curve here and the total record. It's a it's a yellow curve. You, you may not be able to see because exactly much the total record upload, which is a blue curve here, so you can see the matches totally match this means. So all the records all the sets found have been uploaded and the index in time rows. So there won't be any mismatch here.

C

So that's like a perfect curve- and this is a total provider, so see the the number here. It's less than 20 000 compared to the total provider providers here. So it's much less in the the provider's data, so it may be somehow confusing. That's because uh the total providers means the unique providers of cid, so each provider may have multiple loads of sets, so each is much less than the cid.

C

D

C

Why you see the huge gap here? uh If I try to change the type here you you can see it more clearly or make more sense to you. And yes, though,.

C

If I change to ling, so you can see it's how it grows here compared to the seats found, so the unique providers are much much less and the the providers we can also treat them as unique peers. So, like the peers we found in the network.

C

So all these data have been index in time, rails back end and how it's done we can see from the logs.

C

So we don't have the architecture diagram yet, but from the logs I can show you that we listened to the the request from the network, so we collected cne from want list and once we have the cid, we look for the provider, so this is in general, how it works. So here we find providers.

C

This is a peer id and this is a cid, so we any pair we find will be saved to the timeless back end. So this is how it works in the ipf demon and it's around constantly running and collecting data from the network. So this is the first step of the crawler. We have the set and provider.

C

The next step will be recognize. The the file type collect file metadata, for example, from the gateway, so that we can recognize if, if this is a file we can index or we can do the full text index if.

D

C

We will proceed. I retrieve the file content, push to the full text indexing service so that it can be full text index in our timeless backend. Eventually, users should be able to search from our front end the search engine. So this is how we collect data or crawl. The network on ipfs network yeah.

C

So this is the current status. The next step is that, for so this is a one of the the tasks we have planned, so we haven't made it. We haven't finished time to to finish so the pops up mechanism. The other thing- is a sniffer code refactor so for demonstration purpose. So we have our code, some of the configuration so to make it more flexible. So we have a plan for the code refractor under the configuration support.

C

So these are the items we want to introduce, including the basic functionality or the configuration items and our premise use setup so yeah that that's all, that's all the the effort from ben and the xiao quantum. So I really want to thank them to to the development and make it happen.

C

Okay, I think yeah as usual, the qa will go to the slack. Thank you guys.

A

Great thanks very much. um Let me just check slack see if anything's come through.

A

I don't believe anyone has added anything on slack um slack channel is the retrieval markets, one on the far coin slack. um But if I mean if anyone has got a question for zoom or um just to have have on the recording, uh is there anything that anyone like to bring up now.

A

I've got I've got one for ken labs, um so actually I've got. Two first question is just for my clarity. um Is there a difference between the sniffer and a crawler, and would we use those words interchangeably.

C

uh To be honest, it should be the same concept or the same to what we are working on or we have been working on so sniffer.

C

More, I think, for us, sniffer is more focusing on the listening on the network and the crawler means we act based on what we we collect. So we listen to the network and we collect content or data from the network for.

C

For this product, the feature, I probably will say, we should use a crawler in the future to make it more official and uh avoid any confusion.

A

Okay thanks and then my second question um is just wondering how and will maybe you could um you could pitch in on this? How does this relate to the story index work that and the indexing stuff which um the data system team has been working on.

A

D

We get a stream of a lot of the cids that are currently you know, either accessible through calling the dht or watching bitswap or we've got a lot of ways of sort of getting views into some various subsets of the cids in ipfs.

D

There's a plausible path for some of that also ending up in the index and it'll be yet another view and yet another source that you can then crawl by by using that index ingestion protocol, and you make something that looks like an index or node. But instead is using that stream of provided cids to ask for them and then do a full text index on them or other sorts of things.

D

um I think we're likely to see that large providers are the first that are most motivated to adopt um an index like um output, so as like, and the current one that you may have have noticed recently is the dht right now is getting a lot of new cids getting published to it by web3 storage um as it loads and pins a lot of data.

D

That means all that data is getting put into the dht, and so the dhc is starting to grow a little bit faster because of that and they're looking for ways to not add quite so much uh volume to the dht, and so they may um and they've got sort of one ipfs cluster. That has a lot of sids, so you may find that their content becomes available to the indexer, and so those sorts of you know partial views. It's another one. It's another strategy.

A

Great thanks um yeah anyone else got any questions they'd like to have um as part of the recording. It does help to clarify clarify things. If you know, there's a bit of discussion so feel free to raise them, and if not, we can stop the recording.

A

Great thanks guys I'll stop the reporting.