Filecoin Retrieval Market Builders Mini-Summit, April 2021, 6 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RETMKT Builders - NBFS Filecoin Swan Platform

Description

Charles from NBFS delivers us a builder talk at Retrieval Market Builders Mini-Summit in April 2021.

A

Yeah so we're doing a data trans integration platform and so well, aiming at the transfer terabytes to paradigm data from one data center to another one. So then people can be able faster, retrieval it faster, moving the data from one place to another place so because it's better better buy data, so you need. Sometimes people want to monitor the process of the transfers. They want to make fun of the best, the price miner provider and we they wanted to save on different regions. So those and also they wanted to do something more like encryption description.

A

So all this is together more like and after the data was saved, they probably wanted to get the um calculation. So we wanted doing some machine learning or there are analytics so those kind of work. So we put everything together to building one platform.

A

So first we have a dashboard here. We uh doing the scan continuously from time to time it takes about uh every two hours, we'll be able to take a scan of the internet work and we take all the price of their status and put there and after we get all those informations we'll be able to give them a latest after time.

A

So we know that we also put a place for them to add their contact information, so people can be able to add the contacts, so they'll be able to communicate on the slide and uh which we also provide a way to manage. The different cards is accessory based, so after people send the deals, they'll be able to upload the csv to the system, then you can be able to download the accessory to see all the progress science. Currently, the frequent network is another trick in the tracking the offline offline activities.

A

So if it's in import interrupt or something like that it it, you won't be seeing that so you're able to provide this kind of information.

A

We are running a agent like a minor agent on the minor machines, the compliance scanning doing the import and the scanning everything happens, so it will get updated about some status like intel, import, interrupt or some logs so grab from the broken logs, so they'll be able to do some processing uh process much more faster, and so you can see different status on different minor ids yeah and we'll also provide apis to faster, retrieval all the different information you have. The system so you'll be able to download.

A

For example, if you have 500 tasks uploaded with each of them have three 30 deals. Each year has about 32 megabytes you'll, be able to aggregate it in a big csv file to download them and to find out which ones to success and not success. You can also do using the api to first retrieve all those informations and the business will we want to aggregate all those information together, so the platform label. We provided the task management, uh we're doing some id verification. So when miners upload their information say okay, this is my minor.

A

I want to give you some information about myself. We're doing kind of id verification using the signed messages. You need to sign using your wallet to prove who you are and we do a payment gateway. It's more like a payment gateway heart. So when people want to perform one-time payment, they don't need necessary to uh using the wallet to inside them themselves. They can just like exchanges, you make the deposit.

A

If you want to move into another account, you can use, withdraw or send, and so you can pay tips or something like that, and more often on the client side and the mining side will provide the sdk. So you can run it as a service and your local. We also provide the prp install. So when the new version comes, you just need to have your upgrade and you can get latest version and it provides a little bit of function of the encryption and for now we only imple implemented um ads at the moment.

A

So we are going to do it more, but we actually did a demo later. On the minor side. uh We we wanted to have kind of obsidian service uh in case if mine, if client, want to select on different regions, but we are not there yet I'm happy that mal is working on that. So probably we can integrate that service in the future. So we don't need to do those student services by ourselves yeah, but the interesting part is what is after we get the data.

A

We want to give people a minimum, integrated s3 services, so people can directly migrate their data from ipfs to minerals, um give them a feeling like aws sdk integration. Then they can upgrade directly and retrieve the data from the menu or ipfs and upload to the notebook template storage. So we have a big hdd instance um for sorry, sd ssd instance for each notebook all of them are container based, so they can run in the service inside the container to doing the real-time different learning or data analytics.

A

I also have a container native service, which you can.

A

Downl as a database instance, so in just in case, if you want to perform it on locally a data center management switch is usually for people have lots of data centers. Just like me. I have several miners in different data centers and I can use this service to management, but this is beyond the platform and some for some instance access as well yeah. So this is really a comparison to the aws snowball. So another snowball is a service for age computing.

A

What they did is that between several big data centers, you can migrate from one aws region, for example northeast to um to another region. I like um east coast, then using the hdd or using other services like um directly download, then moving around different regions that removing movement. You can also shift the storage from different s3 region.

A

Then, on the local data centers, they provide lots of gpu or cpu instance, so you can do real-time training or calculations there.

A

The fractal network currently have more than one thousand node globally and as a node, who is eligible to receiving deals about 400, so compared to aws only have 22000 over the world. Beyond that, adding those kind of calculation or computing capability, it's much making we're, making the fractional much more powerful than the aws yeah. Let's pretty much about the slides, so we're good to see the uh the apps.

A

So this is the first one network so basically like uh like just like terminal, you can see each of them if you click the miners, you'll be able to see the difference um about the many details.

A

So here we have a simple reputation system well, but this system is largely based on the deals they make on uh inside this platform for money for the more deals he made, they probably get more higher score because because those miners usually sign up as a platform, so they can provide their summaries to what kind of service they're providing and with those signed miners. If they doing the deals they get a higher score than the other online year that just uh cinder anonymously.

A

So this is a little bit different from other reputation system and we don't calculate the online vr at the moment, because we know that other team is currently working on the online vr transfer with a different reputation system. So we keep the place in the future if we can integrate those kind of graphs from them. Another way, we're doing that, we add a region weight for the different region, because if in one region there's only one minor, obviously they provide more values than the region has 10 miners, so building a complicated algorithm here.

A

So we put it clear and also a time-based, so we keep on tracking over their history and if his canadian improvement he may be getting high school also have some seven days or one month. Calculations.

A

Yeah, so here you can directly log in and to change your profile here and you can upload your avatar if you want the most important part about the deals. So here we give people the capability to place the bead. So while you place the bead, you can provide the information about your task and then different miners can provide bid. If I'm satisfied with him, I can assign him a little bit. Then they probably will get high school.

A

Then yeah after the different message is uploading from the system. uh You can see the entire cities and the import data you can see here. I have lots of import fails, but we have one success. The reason is, I delete some storage files while I'm doing the importing, so you can see that the client knows that you want you're not doing things well and with different settings markings when you send out storage, you have a copy on the local, so we provide a kind to us here.

A

So we use the dial to the deals, the kepler local copy. For this one csv, we call the meta csv. It contains a different information. More than just the cities, so if you keep this information and upload it to system, it generated the uid and this id for you yeah and the another function with currently still in development is the payment part. So we wanted to make a payment deposit to give people a feeling more like to support a market for currency payment.

A

If you have a fair coin, but you want to pay with usdt in some scenario for enterprises uh we embedded the um exchange behind the price, so we calculated based on the current usa td value, so you can send the tips based on this one. So this is a experimental feature, we'll probably release next week to the public, so yeah you can be doing the deposit and the general referral address and give you um exchange failings, yeah and yeah.

A

So after the data is retrievable that we consider doing experimental about the data retrieval, I make a small video here, yeah yeah. So basically, when you're doing this one, you can using. We created several python packages where probably open source later on. So you can directly download the save a key general key to the local inventory, so it helps you to do uncreate encryption with using the aes the interviewer generated a key file.

A

You can use it to doing the decryption in the future after you encrypted this file, you can using this command directly to upload the entire file to the frequent network, so the upload the zip file to a file framework. Unfortunately, we still not um getting completed currently in the uh problematic way, so you still need to using another tool, which is uh either the first one um packages or using some for like text, cells, propagate or other services to doing that.

A

But here, while we just showed away using the this one client, you can send us a file converted to a car file and upload it to the file.

A

Network yeah, so when you're doing that, it will show that the archive the entire archive file with the original data will be sent to the network. By the way the training is for how to recognize a monkey among different monkeys from private.

A

Network, yes, now it's generated the dataset id, that's that is generated and yeah. You can see the carfare here and then we start sending it in the network. We actually can also uh using this one to find who is eligible to receive minor. So we can choose the best minor one except udl.

A

This is we define the minor id, then yeah then just doesn't need. This is a minor located in north america region, which is actually our minor in canada and it print out all the details of the miner, the price and all the other details, and it will generate a final csv after this task generated. It will automatically upload to this one platform. Then you can start tracking the left cycle above the sealing process.

A

Then, after a few days, you'll be able to start a few hours, you'll be able to get the deal started downloading. The next part is retrieval.

A

Yeah. Here is the visual pack.

A

Then after retrieval is done, this is about 500 megabyte data, so it takes a couple minutes for downloading.

A

Maybe 10 seconds.

A

Yeah, I have a download success, we'll be able back to the jupiter network. Now we're.

A

Back we're back here.

A

I'm sorry a little lag. Oh sorry, we missed one step we needed using ipfs. Add is uh retrieval the data to the api, particularly then we will be able to get the data from ibms network. We also build all our own ipfs gateway for the swan gitaway, so our ipf's gateway, so we'll also be able to retrieve it from our swamp giveaway. So an important part I'll forget to mention that we already finished the firecoin's mineral part so actually was directly.

A

Send the backup to the menu then send it from the menu to the fragment network, which means s3 compatible now we'll be able to dunk dungaree and download the encrypted files from the ipfs using this one. We can find that here. This is our ipfs gateway on our dev environment. So we found the file and we're using the hash ids. We start downloading.

A

So after downloading we are using our package to decrypt the data yeah after decrypted, we start the machine learning process which parties for the image class simplification it's a bit longer process but uh close to the end.

A

Yeah the encryption is uh decryption is using the key representative and you can see on your local environment, local container instance, so you are very protected, because this is your local environment.

A

So after unzip everything we started, the machine learning finally get a.

A

A

So, just to speed up here, you're doing the machine learning, step-by-step you're, doing analyze from the local storage and you'll start processing the machining argument. This is actually the cargo test. This kind of context was happening last august, so we take the cargo test with images. The major task is that they give you some just test images and you will need to find which piece of those monkeys- some japanese monkeys, some of them. I don't really know so- yeah other different kind of training and takes a while we use it may take one hour.

A

But if you have a gpu on your machine, it will be five minutes. Then they will tell you which kind of speeds of those monkeys are. Then we are done yep. So um by the time we're doing those kind of training, our payment, channel and permanent components. We will charge you for the trigger instance. You are using. We are charging you for the different data volume you use from upload and download, but that's another component, so yeah, that's pretty much what we do.

A

Thank you guys.