Filecoin Retrieval Market Builders Demo Day, 21 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Retrieval Market Builders Demo Day - FilSwan, KEN Labs, & PegaSys - July 21, 2021

Description

Every other week, the Retrieval Market Builders get together to share progress in their projects in a demo format. We want to sincerely thank all of our collaborators for demoing their developments and helping to improve the FIL Retrieval Markets one demo at a time.

July 21, 2021, Demo Day: In this video, you can find quick demos regarding the following -
• FilSwan showing their UI-based S3 compatible bucket storage.
• KEN Labs discussing their indexing tool which includes a directory watch similar to IPFSwatch, IPFS-sync to ingest data, and a web GUI for an index viewer.
• PegaSys team showing demos of DHT offer retrieving and improved user interface.
__________________________________________________________________________________________
If you would like more information about the Demo Days please follow the link below to the Demo Days planning document.
https://docs.google.com/document/d/1IrNbBm2_P79Xi2fXFj9YUfQ0PjWjreGdPYFjFntD1QM/edit#heading=h.akao8aspby4

A

See all right, hi everybody! Thank you. So much for coming to this week's retrieval market demo days looks like we have uh three presentations today. We have file swann ken labs and pegasus.

A

Please remember for anyone who has any questions to please ask them in the slack channels that way they can be answered async, so anyone who's not watching this recording live, can be able to see the questions and understand them as well, and then, after this recording I'll, be posting it to the google drive.

A

If anyone wants to go back and take a look at any of the key sections, so first off it looks like we have someone presenting about file swamp, um I'm not sure who it is because there's no name next to that title, so I'll, let whoever it is, take over and start sharing.

B

C

uh We cannot hear you yeah yeah.

D

Sorry yeah yeah yeah. I just share my screen. Hope everybody can see it yes like this is um the part two of the background. The part one is about how to send out a deal and the backup to the frequent network from the command line to us. We showed the last time this time we go to the even more complicated part. uh It is designed for the ui part. So you can click the button and share it here so because it's a little bit complicated and we just get it running on local machines.

D

So we would make a push request. So let's see how it works, yeah. So, basically, like uh with the current settings, you can easily set up the video with uh running a miner locally.

D

So after that you'll be able to connect with remote.

D

Remote and minimal servers, and uh after you log in.

D

Yeah, I will log in you'll be able to see you have file uploaded of that. We have a fancy button which is at the right corner, you'll, be able to upload it through the ui.

D

I can see the uploading progress.

D

Yeah as long as it'll show you up, you can click the button with edit icon share to five network. You can put the miner id, you want, set up the price you want to given per approach, and you set your issue how many days you want to save on the fracking award, then you click send. Then you are getting the dlc id, which means that you generated the transaction of that of the well.

D

If you look at through their list, you are explaining that at the back end the prg is sent out and then you should you you'll, be able to the minor side that be able to get as data from online transactions so to give people a really um aws compatible vetting, which is great for that and we're still working on some tiny issue.

D

You want to fix about the credentials the security side where they are doing the optimization about it, but uh this is uh first work we have done, and the interesting part is that the original illustrator push all the results we code. We already have to the repository about a few minutes ago and the interesting part in the browser part. We also do modification the previously. The browser was using the react. Yes, but we developed another version, view competitive version.

D

So basically we rewrite all the ui stuff using the view and using the new command to doing the build then yeah, but we still need to publishing all the results to make it much better. So I think next week we can get a full document for the team who want to running the mineral service as an independent library, so they can reuse it and integrate them to their platform yeah. This is just a short demo pass. Thank you. Everyone.

A

Awesome, thank you so much charles all. Right so remember. If you want any questions, feel free to ask them on the slack channel um david put it down there, but it is the uh retrieval market builder, slack channel specifically for anyone that has any questions. So now we have ken labs, they will be discussing a new indexing tool. It looks like.

C

Guys uh this is a william. Can you hear me yep cool, so this is the first time to to present here. So let me let me share my screen.

C

Sorry luke's: I need to unlock the security and the privacy.

E

If you're on a mac, you're gonna have to do that and then you have to restart zoom to make it work. It's it's pressure.

C

Oh yeah, I need to.

A

Gotta love having to deal with zoom security settings all the time right.

B

C

Back, thank you. Sorry for waiting uh now hope you can see my screen now we can cool. So I prepare a simple slice here.

C

Okay, let me start so I'm william from the kin labs. So today I'm going to present the ips ipfs indexing tool. So this is the agenda. So I'm going to talk about the objectives, architecture and the demo bit in our test server. Then I'll show you the future plan.

C

First of all, the objectives. So in this demo I'm going to show you the the simple tool or solution this is to allow ingestion or import files into the ipfs node. While there's no data duplication happens.

C

uh Some words on this duplication. So when you, when you add some file to the node, apparently when you create a block, it contains a device from the file content. So in some user cases, so with the the massive data collection, you may not expect the data duplication instead, so we we just save the the metadata of the file into the blocks, so it can save huge amount of storage space.

C

So this is especially useful, as I just mentioned, when the user has a large data set.

C

So this is a simple architecture overview, so you can see we have a ipfs node and uh sorry one second, how how can I remove this guy, this bar okay?

C

Okay, so we have our fs node and this is a general architecture. We have ipf sync process which will watch a directory or folder in the file system with any file change, including new file or file update or file deletion. So it will sync the change by adding the files to ipfs node.

C

Here we will use the ipfs file store, which is an experimental feature. So in this way we don't store the the raw data, the robots of the file content. Instead, we just have the metadata in each block generated by the ipfs, so it saves space.

C

So this is how we watch the directory and sync to rpfs node. At the same time, with any change synchronized. To that ipfs note, the ipfs sync process will receive a cid which is a root note of the dac.

C

So we push the cid and the local file pass the pair to the timer's backend, which is our own developed backend. So this pair is stored in our database. Also, this back end serves a front-end.

C

Sorry, I need to remove the port here, so we just switch to ad port, so this is the front-end ip address. So this is a generic architecture.

C

Okay, so, as I mentioned so we we are using the the go file store the feature in the solution and, more specifically, we use a no copy option that allows the ipf to just store the metadata of the block, including a file size offset and the pass the file pass. So in reading scenarios it will fetch the actual bytes from the original file or the the file in the local file system.

C

So it will also still work in the most of the ipfs interface, including the command line and http. So this module is we also so the ipfs sync module is also existing project or module yeah. We we did some enhancement, so in the watch process we we notified the the time rows backend, with the file change, as I just described in the architecture overview.

C

So all those uh cid to the file pass mapping is indexed by time. Rows backend. So in this way we know what files have been synchronized under what the crds so the root root node.

C

So this is a the architecture, so I'll show you some demo in a test server, so in general, I'll connect to the server and copy some files to to the watch directory. Before that, I'm going to launch an ipfsync process with a customized configuration in this way, I'll just.

C

Customize to some specific directory, so I'll show you that soon and then we refresh the file store page on time, rows front-end, we will see the status and I will show you more on the front page, uh including the lag overview.

C

Okay, this is from the slides. Okay, let me go to the server. I have already logged into the server, and this is the time rails front-end. So we conform to the web ui component. So so it's pretty similar ui, so you will be very familiar with with it now I'm going to.

C

First of all, let me let me show you here: this is the the ipvs sync binary we built and uh I can use this startup script to launch a process.

C

Something like this and uh specify say. I I create a test set folder here, and this is a boolean option for no copy. True means. We use no copy, okay, starting to watch this folder, and let me show you the the process.

C

So there are two activation process which one is running as no copy force, so it will copy the actual file content the device. So this is a the process we have been running for for a while. So this one is, I just launched with no copy true, and here I'm taking the uh the lock.

C

So, let's see and uh first step, let me add a so. I prepared two files so due to the limitation. I cannot prepare too large file for this demo, so I prefer two one is just somehow smaller. So let me move this file, so you can see. Let me move this file to root and.

C

C

And then, let's go to here yeah: this is the the file I just added. So it's java language specification, you remember pdf, so this is a pdf file. I just added, so you can see the file size and so you may you may wonder what are these two menus files is the one we built for the the actual files we we add to the ipfs node. So all the blocks have the device, the original content bytes. So we won't cover it today so far indexes.

C

This is what I'm talking about so the files I added with no copy option.

C

Okay, so let me click this file, so you can see this is the dark.

C

The build and the number is the the size and we will optimize improve the user experience here and if you click the the leaf node, you can see the order, the offset and the size so the size of the the block the offsite offset in the original file so by using these metadata.

C

So the the file store makes you to fetch the device from the file when user requests it and it didn't. It doesn't show the file paths in this graph, but here you can see the file information, so the cid, the file size and the pass there's no links on this node.

C

Oh so we didn't. We didn't add these links to to the other nodes that were the intermediate nodes. This is something we can improve. More importantly, here this is the crd information, so this is cid and we decode and show the the base version codec and the multi hash, including the the the short two two five six, the hash function and the including the digest.

C

So this is the dark and okay, let me add, a much bigger file. So this is a segment I split from the original wiki. We could pay a which a weekly archive. So it's about. Let me see a hundred megabytes.

C

Okay: let's go back to the page for indexes.

C

What's the name ci? Okay, so this is a weekly latest, abstract ci. So this is the file I just added. So if you click so don't surprise that you are now going to see all the details uh we have to improve on the the ui, the user experience, there are too many nodes, not nodes, yeah blocks. So if you expand, you should go to see, see something there and if you click the block, you can see the metadata under the file information and the cid all the details here.

C

Okay, so to be honest, so this is a size I just tested and the ipf is sync accept, so we are still improving the performance and the size above that may crash the process as there's some far-read uh implementation problems so that it seems like the file is loaded into the memory at once. So we are, we are still fixing the issue.

C

About that- and we also add some useful links to to this many other item- ipf explorer and the setting.

C

So this is the ipfs watch setting, but it's not implemented for the implement yet so, as I just show you so we we pass a watch directory in the command line interface and there are some other options we are going to support. This is also already supported in command line, so no copy option.

C

This is the front end and let me go back to the slides.

C

Okay, so the plan first of all is that we want to also watch http resources. In this case, we we can use the url store feature which also experimental, I think for now- and this is something we we also can take care of.

C

The other thing is the file store data storage, optimization, so you you know that the data will be chunked and it means that there will be massive files stored in the file system and especially in the big data collection.

C

So that's something will slow down or impact the performance, so we are planning to provide a solution with a distributed key value database. So in this way we can optimize it's similar in the pinner data storage. As we add files to to the node there's, the pin set will be expanding.

C

So in this way, if we look into the implementation, the painter is also loading the the data in the startup, so it will also impact pack the performance and the exhaust huge amount of computing resource. This also, we are trying to optimize and uh that's it next contributors.

C

So thanks to abel ben shopping, zhang, xiang.

C

For the contributions to this demo, so everybody has worked hard to to make it so we show some some of our product and our future plan thanks.

C

So I know that the qa will be moved to the slack.

C

Thanks see you.

A

Perfect all right, thank you. So much um so it looks like our final demo today will be from pegasus and also just a reminder for everyone to keep a lookout in the slack. There were a couple questions there that are going to be asked, hopefully asynchronously soon, so make sure you look there and just a quick reminder that these demo days are every two weeks, so the next one will be august 4th so keep that on your calendar to make sure you're able to come and take a look at that as well.

A

Okay, so, last but not least, we have uh pegasus with their demo. I believe I'm pronouncing it correctly.

F

Hello, so today we will demonstrate our book 2 demo. It was achieved 2 weeks ago, but we missed the demo, so we have recorded the video, but now we will be able to make the demo live.

F

It's about how to retrieve offer us on the secondary market in order in the next step to retrieve a file into this offer. At the end, we will have a little talk about what will be the next panel. So now I will just demonstrate you how we will be able to get offer from our services.

F

So, on the top left, we have um docker service launch with a lot of getaway who are running a provider and a lotus node on the bottom. We have a frontend that will demonstrate at the end and for other services we have a getaway, a provider and a client. All free service has started I've started. I just need to emit them, so we have a register mechanism which means that the getaways and the providers need to be registered in order to discover each other.

F

Also, like the clients, our client needs to get registered or get away to be allowed to communicate with it. So first I will register all the getaway and I will register also the provider.

F

Now we can see that each getaway is an identifier and also the provider as an a as an id notifier. I will first publish a dht offer so.

F

I will use his ids and for the sake of the example currently, there are no real files linked to that. It is part of the next step, so I will use a temp cid.

F

What happened is an offer has been published in the system and now we will try to retrieve it from the clients. So we will take this gate away. First, the clients need to add an active gateway to his list of getaways, and then he will be able to use this gateway to reach the provider to get the offer. So it is a dht offer so we'll use the dht command line to to find it.

F

So we specify the gateway and we specify.

F

The file we want to reach what we see is that it's a log, but we successfully, we will successfully get the offer in a few seconds. It takes a little bit of time.

F

Because it needs to make some transaction in order to enable the payment channel, so we get the offers back now it is from a dht mechanism. That is a kind of a search mechanism, but we have a more straightforward mechanism to publish offer which we are calling is published standard offer.

F

So we have a random mechanism to make the democracy.

F

F

We have published free random file in an offer that we will try to reach from the client. So I will go back to the client. I will just try to find the offer again from the get away I have subscribe and I will text this file. For example, no sorry.

F

I don't need it.

F

So we are about to get also the offer one last step, so everything is with a go, go clients and we have also built a javascript library to do it. So the interface is quite light, it's quite straightforward, but it was just to validate that the javascript part was okay.

F

So what I will do, I will do the same thing. This is the javascript ion. So first I will specify your gateway and I will specify one on the one of the file I was looking after. So if I try to find the offer, it is just the standard furnace, dht.

F

I'm able also to to get the offer.

F

That is for the demo. Now we will have a little uh ex little talk about what we'll do what what are what we have achieved in this demo for the past two weeks and what we will do for the next demo, so chen young, if you want to take oh.

G

Yeah sure um yeah sure. Well, I guess um there isn't like another update, like a big update from us, so which, unfortunately, we couldn't fit in today's demo is that um in past few weeks we have spent time sort of reviewing rebuilding some of the components of the system, and so our latest man is having quite a lot of big changes from what we demo today. Hey excuse me, so I guess the first thing is that we have already made the switch to lip2p in latest men.

G

um Second thing is that we have now um switch to user, go cid implementation, so it now works with real world content id rather than 32. Bytes placeholder and we've also got a working simple data retrieval from offers, so you've got offers and you can retrieve content based on the offers you found earlier and um publishing offers from real files and more complete ci together with many other interesting features and of which we have to demo next time, which is a pity but uh yeah, because the auto takes some time to prepare you.

G

That's I guess that's all. I wanted to say.

A

All right, wonderful! Well, that does it for today's speakers. It looks like remember. The next demo days will be on august 4th, so please sign up beforehand on the document that gets sent out a couple days beforehand. Again, if you don't want to take a look at any of these recordings, I will be uploading it to the google drive shortly, as well as be able to take a look, any questions that were asked on the slack channel. So thank you again for coming everybody.

A

We really appreciate it and I hope you have a good rest of your day.