Filecoin Retrieval Market Builders Demo Day, 15 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Retrieval Market Builders Demo Day - KEN Labs, FILSwan, & PL Datasystems Team - September 15, 2021

Description

Every other week, the Retrieval Market Builders get together to share progress in their projects in a demo format. We want to sincerely thank all of our collaborators for demoing their developments and helping to improve the FIL Retrieval Markets one demo at a time.

September 15, 2021, Demo Day: In this video, you can find quick demos regarding the following - • ResNetLabs' Content Routing Team discussing smart records
• KEN Labs demoing IPFS indexing tools.
• FILSwan showing their updated reputation module.
• Protocol Labs Datasystems Team demonstrating indexing nodes.
__________________________________________________________________________________________
If you would like more information about the Demo Days please follow the link below to the Demo Days planning document.
https://docs.google.com/document/d/1IrNbBm2_P79Xi2fXFj9YUfQ0PjWjreGdPYFjFntD1QM/edit#

A

All right, hi everybody. Thank you so much for joining us for this version of the file coin. Retrieval markets demo days it is september, 15 2021.. We currently have three demos lined up for today's session before I announce them. Please remember to if you have any questions to, please put those in the slack channel that way anyone who's unable to view this live, will be able to go back in and take a look at the file coin, retrieval market slack channel for the questions and be able to see those answers.

A

Asynchronously also quick reminder: this will be uploaded to youtube and you can see all of our previous demo days there as well. Quick rundown, we're gonna, have three demos today looks like ken labs is going to be doing one about ipfs indexing file, swan we'll be doing one as well and then. Finally, the protocol labs data assistance team will be rounding out with our third demo for the day. So with that, I'm going to give it to ken labs, you can feel free to start sharing your screen and you can start your demo first.

B

Okay cool, so I'm going to do the demo on behalf of ken labs.

B

Let me share my screen.

B

Okay, so this is william far from ken labs. uh Let me share my slides here.

B

Okay also, I just leave it at this, so this is a demo for ipfs indexing. It's a follow-up with a previous development.

B

So if you see from the agenda today, I'm going to show you how we integrate our index node with the ipfs, so that the adjacent node will look for the provider from the index node if it's configured with priority, if it fails to find something they will move on on the existing logic in the epiphyse node and secondly, I'm going to show you the full text indexing on specific file types, including a pdf html and the plain text file uh in this case, so I can say that we are not going to index, for example, video audio files, those files.

B

So currently we are supporting these file types and at last I'm going to briefly talk about the next step. Okay, so this is a provider lookup and we have deployed our change to our host.

B

So if I go to the command line, interface, I'll show you a couple of out. Firstly, I'm going to add a text file, so this is a plain text file I just created before this demo.

B

So let me add it to the ipf node. Also. I can show you the content here.

B

Okay, this is just just before demo and if I go to.

B

My local console you can see, uh I did some experiment here and uh if I copy the set and apply it here, you can see it showed me that, just before demos, this this is from the the text file. I just added- and if you see here- and this is how it's trying to fetch the peer id from the index node, because we don't have a reliable way to present you how it's working internally. So I can also show you the log, alternatively, and not.

B

Alternatively, besides the the plain text file, uh the pdf and the html files are also following the same process.

B

Okay, so this is the.

B

How can I remove this bar here.

B

Okay. Secondly, so let's see the full text indexing, I prepare couple files, so firstly, it's a pdf, I'm going to add this pdf file to the.

B

And note: okay, it's not huge! So if I search here and correction okay, sorry added here, so it's indexed and I go to the time row.

C

B

Engine here you can see.

B

B

The handbook, so if I search on keyword like just it, should show me the pdf.

B

Here so if I try some keyword like young.

B

Hopefully it will show.

D

B

Yeah, so this is because the pdf content has the keyword young. Also besides this pdf file. These two files also have the same keyword, so it's listed here. So this is a pdf and.

B

The last thing is the the html, so actually this file, I have already added to the the node. So I plan to.

B

Just save this file, not not this one. I plan to do it here in the wrong time, so I can quickly.

B

B

Html file from the command line.

B

Html so I'm going to add it in the same way to the node.

B

Yeah, so this is how I added so if I search this new set from the interface.

B

Hopefully, I should be able to see it here. Yeah, it's indexed, and this is the the peer id of the whole that host I'm running so by file name. I can search.

B

I can do it here, hello.

B

Yeah, so this is the the file I just added, so these file types, plain text, html and pdf- can be indexed to our induction node. So these are the two things or the features we have done uh in the next step. We are going to continue the development once and we we failed to to finish in past two weeks is the the pops up mechanism.

B

So we have some uh bug fixing during the two weeks, so we we didn't, make it and that also in the photo buffer serialization, as well as the monitoring on dht and the index and node. At the same time, we are also looking for more data sources to to index.

B

So if anybody or any team has a resource or the the ipfs resources that can be indexed and interested in in our index node for free to reach out to us, so we can also integrate and try to index your data.

B

Alas, I want to use thanks ben and uh xiaohua zhang, who joined us very recently for the effort to to the development and make this demo happen.

B

Yeah, that's the the demo for today. I think, as usual, the qa will go to the slack thanks.

A

All right, thank you. So much it looks like there is um a question or two in the slack for you to take a look at um after we're done here. um Great presentation. Thank you. So much next step we have charles who's going to be doing a demo for file swan.

A

So if you could allow him to share his slides, that would be great.

A

There we go perfect and then charles, if you want to start sharing your slides.

E

Yeah right today, like we, don't really have slides. We just have a about a one minute, video to show the bit of web development recently so recently, like we have done, is after we build the bridge from the uh s3 to frye coin. So now we move it to one more step, as we get a lot of requests for people who wanted to download from the local fs3 store uh fs3 storage tools through the ipa bath to another location, basically like so they can doing like across different geolocation transfer.

E

Instead of using and gigs or something or building a proxy based on the fs3.

E

We are doing it, push it to the ipfs, then you can sending the backup files from ipfs and also you can directly sending those um files to the fragment network. Yeah.

E

B

First, like this is a local instance.

E

We're running on the uh modified mineral instance, so you can sending a task open ideas through the network, so understanding network. It will give you the success and uh all the tasks with all the information included, then the dressing into the frequent network. At this time, people will want to download the car fare from the icfs.

E

Then they can doing the offline import from ipfs instead of directly standing from other files like antiques, so we get as uh iprd uh hashcode here, so we can be able to download it from the terminal you can using wget or whatever, and to download this cover.

E

And at the time time you can get compared with the original kevar. So then you have two carafer matches, which means that it's consistency.

E

Then, like uh I've as long as it wishes for ipfs, so another company or institutions, you know different geo locations, you can get the cover from ipfs started in production and that's basically pretty much a very short video about what we have done. So this is one more step: moving from the fs uh s3 to the ipfs to park network. Okay, thank you. Everyone.

A

All right, thank you so much and then it looks like to wrap up. We have the protocol labs data systems team um really quickly um before we get started with that one. I do want to remind everybody that these are every two weeks, so the next follow coin, retrieval markets demo day will be september 29th. At the same time, you can take a look at your calendars. It should be there and I'll, be your host then as well.

A

So with that uh we're gonna have the protocol labs data systems, team demoing, their project indexing nodes right now, I'll? Let you take it from here.

C

All right, let me uh go ahead and share my screen here.

C

Let's just share the full screen.

C

So I have a couple rather boring looking windows here, so I'm just going to go ahead and talk through what I'm doing. I have a a indexer running and I have a simple client to that indexer. So what I'm going to do is I'm going to show how a provider can be registered, how we can see the registered providers, how we can index some basic content and then query for that basic content and then we'll see if we can do a demo of the advertisement and ingest protocol.

C

So uh first things. First, let's go ahead and see if it's the window, where I should be able to see content, let's just clear this out, so you can see new messages more clearly, all right, so not so interesting. Yet let me go ahead and run the client.

C

I have a few commands available: let's go ahead and register first off, so we can register ourselves as a provider.

C

If you look over here, we can see that our indexer did go ahead and receive the registration, and now it knows that this node here that I'm running client on is now a provider. And now it's allowed to go ahead and send contact uh content to be indexed and I'll uh talk more about what registration means and how they advertise in the context of advertisement protocol in just a minute. So let's go ahead and now we've registered ourselves. So let's go ahead and index some basic content, I'm just going to.

C

C

Sorry I want to index: um let's go ahead and just index this, so I've indexed some content for the cid. uh This um qm ends with lbc id and I've, given it some metadata metadata, just some scratch data that is useful to the uh provider um and that's useful for when the client goes and retrieves it from the provider. uh So I've indexed this this content, let's go ahead and find it. So if I.

C

Go ahead and find that, and here we go, and so we we get the response back so for the uh for this multi-hash um well, in this case, the multi-hash is the same as a cid, a v0.

C

I can see that I get some data back, um I've gotten it both for the provider I registered for myself, but I've also gotten it from some other providers who I've previously uh indexed content, the same cid. For so now we can see that this particular cid or other multihash indexes the indexes data that indexes the same data, that's provided by different providers, and in this case these two providers provide the same data but with different metadata, and in this case this provider here provides it um with the same metadata.

C

It happens to be as another provider, but it's showing it as a different provider. So we can index uh different pieces of information from the same provider or we can inform we can index uh information from different providers. That's what we're showing here and here's the different here's information for the different providers. So when the client gets this response, they can say.

C

Okay, I need to retrieve the actual data from the provider and here's the multi-address for the provider and for the other one they can choose by whatever means they want to go to one of those providers and see who can provide them that data.

C

I was going to show you. I believe I want to list the providers. We have. Let's go ahead and.

C

So we can see that we have these two providers that are registered. That's going to show you that previously, okay, so that's the the very very basic client interface. um We also have an advertisement protocol. We just did a lot of updates this morning. So uh while I can connect over liberty and pass advertisements, um I'm not sure that I can show yet that where the advertisement and ingest is is happening um so.

C

See I think I need to share this one okay, and we see that I don't think you can see the new window. I just brought in.

C

So I have the reference provider running in this window. I'm just going to import a car file and then I can go ahead and.

C

Connect to the provider and pass the advertisement, so I don't actually have debugging, so it does show that we connected to the pier, and if I had the logging setup for this demo properly, I would have been able to see that it received an advertisement and it was able to then go ahead and ask the reference provider for any of the content that it didn't have already, and I believe it also has the content already. So it's not going to go ahead and ask it again.

C

So um let me talk through what what I wanted to do, what what you, uh what this is deadwood that should have demonstrated. So what the advertisement protocol does is.

C

This is where a provider can advertise over the p2p that it has some new content to ingest the reference provider running here is what actually sends that advertisement over to the index or indexer receives that checks to see if it already has has uh ingested that content and if it has not, it calls back to the uh at some time later, depending on its own internal scheduling, falls back to the to the provider and says.

C

Okay, now give me the content and it walks back through receiving all of the content uh until it has received everything up to the latest advertisement, um either from the where it's last received it or from the beginning of of history.

C

So this advertisement protocol is able you're able to go and update index without having to fetch the entire thing. In other words, it's uh when they send here's an here's. A new advertisement here here are the new things to fetch and if an advertisement is missed, uh the next one that the indexer gets will will see that it doesn't have the previous one, we'll ask for the previous one last for the previous one until it uh receives it until it gets the content for an advertisement that it's already received.

C

So uh with these two types of things, we can either write simple clients or write the much more.

C

Fully functional uh ingest and advertise based clients, and then you can always get the index content back using a find query that any any clients can go in and search for either single uh multi-hashes or search for batches of multi-hashes and get all the answers uh back to what to who has what content that they're looking for so that? That concludes the demo. um Sorry, it wasn't too flashy on screen, but I would certainly like to answer any questions that you might have.

A

Just checking the slack channel, I don't see any questions in there for you at the moment andrew, but if anyone does have questions about this demo and they think about them later on feel free to drop them in the slack channel. That way they can be answered. Asynchronously um with that. I think that wraps up today's demo day.

D

Cool thanks for the demo andrew that was um yeah. It's good to see that we're making progress and it's good to start getting this out there, so that people can see the the flow that we're imagining the stuff going through. So.

A

All right, I want to thank everybody again for coming today. Quick reminder: these are every two weeks, so the next one will be september. 29Th. Please check the sign up um slack thread in the next couple of weeks for the next one I'll be posting that there and also, if anyone, has any questions about any of these, this video or any of our previous demos, you can find on youtube. Please direct that to the slack channel retrieval markets under the file coin, slack channel itself. Thank you again have a great day.