Filecoin Retrieval Market Builders Demo Day, 8 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Retrieval Market Builders Demo Day - Glitter, FilSwan, & KEN Labs December 8, 2021

Description

Every other week, the Retrieval Market Builders get together to share progress in their projects in a demo format. We want to sincerely thank all of our collaborators for demoing their developments and helping to improve the FIL Retrieval Markets one demo at a time.

December 8, 2021, Demo Day: In this video, you can find quick demos regarding the following -
• Glitter shows their work building a search indexing protocol on Filecoin.
• FilSwan demos a S3 bridge to Filecoin network
• KEN Labs presents data backup on the Filecoin network.
__________________________________________________________________________________________
If you would like more information about the Demo Days please follow the link below to the Demo Days planning document.
https://docs.google.com/document/d/1IrNbBm2_P79Xi2fXFj9YUfQ0PjWjreGdPYFjFntD1QM/edit#heading=h.akao8aspby4

A

Hello and welcome to this week's retrieval market demo day. Today's date is wednesday. The 8th of december uh we've got an amazing lineup. Today, we've got ted leo speaking about glitter protocol, and this is a new team introduction. um We then have charles cause demoing the latest fs3 work and maybe some other stuff. I will see how we get on and then william from ken labs, demoing pando latest um so yeah as usual. uh Please put your comments in the thread on the far coin, slack and ted over to you to introduce uh glitter.

B

Yeah, thank you, patrick uh I'm sure, with my uh presentation here.

B

Hey guys, we are uh we're glitter, okay, it's a network. We are trying to do some indexing protocol on the on the final coin network and uh actually currently in the file coin network and facing some problems like uh there are quality like there are a lot of use, uh useless data on the farco ecosystem.

B

That's why we have the firecoin plus program and we want to incentivize people to uploading more and more useful data and uh storage usage grade is not like high and and most important, like most of the data on the barcode has never been retrieved ever since they uploaded and we're trying to solve this by by creating a pro indexing protocol and incentivize user to upload a lot of different kind of useful data, and in this way people we can use massive amount of storage on the firecoin and facilitate people to download data from the far coin and get make all the code data become hot.

B

Here is how glitter protocol works. It is like uh on this side. This is our miner network like uh this. Is these apps are just our ecosystem apps. We want the uh different kind of uh developers just developing their own apps, using our protocol and like uh when we're used to trying to uh when are users trying to access a piece of data from the internet or from different kind of broadcast if they try to uh like.

B

uh If they try to download some data from the internet or from the victorian network, they can directly uh use a linkedin directory track without my network and get a piece of data from the firecoin uh to see if that piece of piece of data is existing on the flat coin or not. If it is, if it's not there only far coin, then we now we can.

B

We just incentivize this user to upload that piece of data to another coin, then he can just directly uh spread the index uh broadcast that index information to our network.

B

This way we we can become a second of a decentralized version of the indexing uh protocol across the uh far coin network, and in this way we are building a bridge between the web 2 to new file, and uh I, like I think in in the future, we can have different kind of uh uh ecosystem yeah uh ecosystem apps, like uh we can be a decentralized version of the search engine and we can have the decentralized version of the netflix or spotify, and we can have a decentralized version of the download manager and also we are building this issng.

B

As I said, with isis reader for people like event, media original feed is now there any more people can still get get that isis feed from the firecoin and uh also on the science hub and the this is the one I like the most like uh we're trying to build. This uh firefox and chrome extension uh like uh if, if you even if the original website is not there anymore, people can still get the data like uh get the data from get a snapshot from the firecoin.

B

If, uh if we have no, we if we have a copy on the falcon- and this way like uh I think we can- we can get connect. We can build a bridge between web 2 to f3, and this is how our economics work, targets, works and uh uh like for a data retriever if they, if they want to retrieve the data from the uh to to do some query like if it's simple, quite likely to check this file exist or not, then that would be free.

B

But if they want to do something like a google like query, then they will pay a little bit and they will need to pay a little bit amount of the quality to the indexer, which is on mana network. Then.

C

B

Actually, here in the miner will get to decide like how much how much money or how much query feed they want to charge in the data retriever actually can directly set their layer. Query feed to zero like if they just don't if they just want like.

B

Actually we want all the miners to be competing with, with each other and uh like uh the data that the app developers like all of the users, they will need to interact with cell protocol and the firecoin file need the app developers so, and so the app developer will also get a card from that priority, and also on the data contributor we want. We want to incentivize people from uploading, different type of data to our core network, and actually we are indexing, a lot of data, the current uh the current file size.

B

We are indexing right now. It looks like we will keep indexing them. Just keep keep passing all this data, like the current uh totals data being stored on the file coin, is like uh 35 terabytes, and so so. Actually, we we already have our test network online right now and, uh like we'll start uploading, some news and web pages from the access reader.

B

So uh once we start uploading all of this data, I think we can become the single life uh storage, consumer of the entire final coil network uh and uh actually like. uh We have already bring like we will receive some investment from product lab and the one bad personally and zx is our advisor now, and uh I think we can help the farmer uh we can grow together within the farco ecosystem and uh and the mac on the data store about coin be alive.

B

B

Hopefully we can become the indexing hardware between the file coin and the only existing different kind of data protocols and, uh like uh I think, by the end of last year, we can will be able to index most of the internet web pages on the far coin and put put them all on the far coin and uh like uh currently, we have where we are testing our test net and the start uploading some data to the file coin and uh like uh this is our plan for the last year and uh yeah.

B

Let me check with the progress yeah.

A

Oh, I think it's breaking up a bit and we've lost the screen share.

B

Oh hold on hold on a little bit. Let me let me share with you another one uh lists our this is our. I just got connect list with our our access reader. uh I pre previously. This list list is not a decentralized search, but right now it is, it is like uh connect without test net and, like you can do some search. So this is a pretty. uh This is a result like I I I I just have all the two twitter from from from one bed.

B

So you, if you're search no file coin, you know test that. Then then you can get a lot of tweets from from filecoin and from juan bennett and, like you can tab, you can just.

B

It's kind of search, search, result and right now, like uh all these webpages, are waiting in a line to be uh to be mined on the final uh and um by the minus five coin: minus and we're just we're gonna release our uh python and the node.js sdk soon, so so every so every person can our users every developer can just build up their own search engine or get different kind of contents from the file coin.

B

Using one line of code and uh that's this is our go and uh we're gonna have a quick, uh we're. Gonna have uh our python and the node.js sdk out. I think within one month and uh like uh please feel free to try it now and I think we're gonna participate in this demo day uh for yeah for a serious different kind of demo days, and that's all for me right now. Yeah.

B

Let me check with your question here: awesome.

A

Thank you, yeah there's a few questions but yeah any clarifications that people would like to ask um in in the recording.

A

No okay, in which case I think, um yeah move on to charles and the latest fs3 stuff.

D

C

Hi, everyone and.

D

D

Yeah, can you see my screen? Yes, okay, great so after a very long time, development. Finally, we have the second version of ambassador ready. uh It's very easy to um it's quite easy to understand, so we don't provide any slides, so it might be live demo. We put it already online with a calibration f3 test night, which means it's connect, connect to the calibration network. So yeah we have a public demo account people can just access it and try it.

D

So it's pretty simple like when you log into the servers uh s0, you have the capability to create a package as the same as a three and you'll be able to upload the file which well, you know that I downloaded some rfg files from network, so you can choose any one and upload it.

D

Yeah so after you upload it, yeah you'll be able to make a online transfer. So, let's see like if I wanted to directly backup to frank network, so we can get. If you know the provider id you can use, it'll be like this. This is uh and I'll give you the price whatever id and give it a date.

D

Yeah then just send it so you're gonna see the dlcd created here and.

D

You'll get information, so those data study web is the one you'll be able to download after 30.. But, however, this is a v1 version. It's working perfectly looks nice clear. What we are bringing now is a v2.

D

The biggest difference between v1 and v2 is that we are be able to schedule a backup, the entire instance to fracture network. So what you can do is that you can create a backup plan here.

D

Then you can see that I want to give a plan for let's say demo day, and I want to pay this price for backup. My storage and I want to keep the copy on firecall network for 185 days. Then I can create it.

D

I've uh your backbone will be triggered here, which means that every day, at the same time it will be trigger. uh It will compress all your data volume become a cover upload into the firecall network and doing scheduled job. It will be backup every day after backup finished you can see here, you will have a schedule back up here. It will tell you finish: you can doing the rebuilding images, which means that if you love your data copy, your entire instant accessory instance locally, don't worry if you click the button.

D

Okay, there is that we are doing the rebuild for you, which means that you are fetching the backup from network review the images to recover your previous failure. Then you get all your files back online. I mean entire instance.

D

This is a successful body, backup jobs we have already done before and you can check in the history- and this is the new created the scaling of progressing, and this is the storage provider who is maintained there. I can see it's here. The test data storage provider opens the public. Anybody can send the dlc to it on a calibration.

D

It's a. We have one gigabyte network to accept deals for people testing on calibration because they don't really want to spend a fair coin um official midnight. So we also give you this id here, which means that if you want you'll be able to download the copy here from common mind, it's also working yeah. So, let's see, if we get back up to um this is same it has the id you'll, be able to fetch it myself as well yeah.

D

So this is the faster demo we have, and I think it will be a great product for enterprise usage and it will be a good use case for frycoin as a permanent storage contract based permanent storage. I don't want to miss missing the public, okay and yeah, so this is basically like what do we have for fs3 v2 on v3 we're going to have v3 launched soon? The v3 will be a cloud version, so people will be able to create their instance on the fly with us, yeah yeah here.

D

So I think I still have some minutes, so I wanted to introduce another great product multi-chain payment, so we want to provide a webstrate user interface and to make a payment from polygon network. So if you log out and just let me login, yeah, okay, I already connected to the network, so I cannot show you how you are connecting yeah. So basically you're gonna upload the files. Let's do another image, we have the image. Let's pick up a.

D

Three megapixel one: okay, then this is a price by usbc, not fred coin, but at the back end we're doing the estimation using the define to do in the replica token. So from sushi exchanges get the price feed, then you can choose low average and high. It's can I upload it. You will see that your file is uploading to the ipfs network.

D

Then you need to sign the contract to allow user.

D

To lock your funds, because before the deal is made, you are not supposed to pay. If you didn't support finish the job, you don't pay anything and your phone will be refund after six days.

D

You can see your context transaction and you can check online here. This is the amount you pay we create uh because there's no usdc token on polygon network now, that's network, so we create our own token.

D

So, after everything is done, you can see there. You have. Your images currently is uploaded it's in processing stage and it is written in the queue sent in to the storage server and you are paid, but the fund is locked for sure we go here.

D

You can see the information you'll be able to download the file from the ipfs network, we're also building a calibration fpfs gateway to enable people upload a file to the calibration apis network. Ipfs doesn't have calibration, we just name it.

D

Those are the three signatures you need to unlock the funds. This is the deal decentralize, the dow for match to universal key holders to sign the proof of on-chain from fraction network. So we get as a bounty join the bounty between frycoin and channing as storage proof provider, so we'll be the data provider to continuously prove the storage of the deal on chan to article training, the network. So all the networks people can use this proof to prove that this year is unpaid already, so it takes a while to get the to assign the order.

D

So, let's see like one order, which is already accepted for accepted order, you can see the data study proposed the id you can see the dr id as long as it is reached to the clients we are simply to unlock the fund.

D

If you have two thirds of the signature approved, you will be able to unlock the fund, so fund unlock. You can see here. Those are successful, deals found sending to the storage provider.

D

So as long as provider, you are asked to provide your wallet address, so you can use the funds, wrap the token uh wrap the file coin um from the client yeah. That's pretty much about our demo for the payment uh through mark chan, yeah thanks. Everyone.

A

Awesome thanks so much um I like the the ipfs loading wheel. um That's a nice touch. um Any clarifications. People like to ask at this point.

A

No all good all right, um william over to you.

E

Thanks, patrick.

E

Okay, let me share my screen okay, so this is a follow-up demo. That's about the the panda metadata service, so in previous demo, introduced what the metadata service is and the the high-level architecture. So in the last two weeks we have done a bit of features development, including data backup to file code network.

E

C

E

Consumers should also be able to find the data directly from the file coin network with the pando account, and this is down via s3. For now, I'm going to talk about the the situation in using the s3 after the demo shortly, so I'm also going to introduce the metrics to help us monitor the state system status and the the performance of the whole panda service.

E

Okay, so I'm going to give you a quick demo here so two things: one is I'm going to send some data update from the provider, which is a mocked application, so I run it from my local environment and then we can observe the data upload to s3. Then we we check the metrics.

E

So this is what we this is where we usually deploy our application, the service. So this is our rancher front end, so we can see the logs here.

E

Okay. So now I'm going to the console.

E

And this is the application I used in previous demo as well.

E

We were also going to tweak the the usage description a bit in next week, so for now I'll just reuse, this command, so the the parameters, the first one is uh the private key which is used to sign our request. Then we have the dhg number. So this is how many dark data we want to send to bundle service. Then we have the pando peer id the multi address and the last parameter is a bit tricky for for now.

E

So we, while we are sending the data, the idea that we want to also wait until the data is fully transferred, so we haven't moved all the functionality to a daemon process. So for now, if we don't have the timeout, uh it will directly terminate.

E

So let me run it so this is the output, so you can see here we have five root settings generated here which are for those uh duct data respectively.

E

So we can see all these steady from the pando service log. They will be received.

E

So I can check one in the log, so this is.

E

The ide, so you can see there are five. So all the dark data has been synchronized to the panel service.

E

So if we scroll down here- and we can see the successful backup, so the data is written to this car file. So this is a timestamp we can use it to to search later in in the estuary back-end.

E

This is the indication that the the backup is successful. So now, let's go to the actuary back-end.

E

So if we refresh so, we should be able to see the the file back top on actually, and these are all the files I think we we tested today in these two days, so the files will stay on s3.

E

It has the the process under the mexican to upload those files to to filecoin network.

E

So this is how we, as pando servers back up the data. The next thing is the metrics, so this is what we built.

E

So if I refresh so, if you notice just now, there is 64 dag persistence, count, which means there are 64 pieces of duct data stored by the panda service. At the time before I triggered the the the data update. So now there are five more, which is the same as a number of the dark pieces I just produced, and this is how many providers registered as we are testing. So we reached seven providers, and this is a subscriber account.

E

We have a interface to allow you the panel service, to subscribe to you to to some providers. So for now this is the http interface, so we allow multiple subscription to the same provider. So you should be able to see here. This is a big bigger number than the register count, so we register same provider multiple times, and this is a persistency latency. These are the the heat map, the idea to to show the latency on different aspects, and you can see here.

E

This is uh the latest, and I think this is what let me see.

E

Yeah, so you can see, this is a one persistence. This is a one operation with uh latency between 500 million seconds to one second and then one between 200 and 300, and three between 100 and 200. So this is how we we show and present the latency metrics. So this is a subscribe api, so this is pretty performant you can see here. So this is a simple http api, so the pando servers can subscribe to some provider. So nearly just one millisecond.

E

This is the register. Api is the same. So it's uh I did the registration before the demo earlier today, so it showed here, and it also takes nearly no time, and these two are the metadata related api latency.

E

uh Probably I can also show here. So this is a snapshot list, so these are the old snapshots generated and each call will be tracked by or monitored. So I just hit this meta list endpoint. Let me do another one to show the snapshot details. So this is the api. So in this demo, I'm not going to cover all the details up in the response uh we just trying I'm trying to show you the metrics.

E

There may be some delay to to see them on the dashboard. But now probably you can still see some blocks here. So these are the core made around a half past two this afternoon.

E

So this is the indication that it's pretty performant because of nearly no traffic to those hdb api, and these are the almost in memory operation. So these are all the metrics we have already implemented. uh So this is metrics. I also put a screenshot here and uh so for next step. uh One thing is a bug fixing. uh The other main thing we want to review is a backup solution.

E

For now, all our data backup will stay in at the actual site until it gets uploaded to filecom network.

E

So s3 has its own policy mechanism until the total size reached up to 3.75 gigabytes, so also they will still stay on s3. So we want to measure if this is what we expect.

E

Otherwise, we may need to do the data upload ourselves to to work to work around the the s3 service, so we we need to review and the last thing the performance test. uh We have done, the the future development we build up the the metrics, so the next step will be the performance test, uh probably end to end. So we want to see how much data we can.

E

We can store, we can manage and how many providers concurrent providers we can, we can support. We haven't covered the consumer side, so that's also part of the work in the short future.

E

Okay, that's, I think. That's it still, thanks to and ben who are the main contributors to all this feature, development and bug fixing thanks to them and uh yeah. I will use almost the same slice at the end. So thank you very much.

A

Awesome thanks um any clarifications that people would like to ask or bring up now before we finish the recording.

A

Cool all right.