Filecoin Retrieval Market Builders Demo Day, 11 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Retrieval Market Builders Demo Day - New Web Group & Leeway Hertz, 11 May 2022

Description

Every other week, the Retrieval Market Builders get together to share progress in their projects in a demo format. We want to sincerely thank all of our collaborators for demoing their developments and helping to improve the FIL Retrieval Markets one demo at a time.

May 11, 2022, Demo Day: In this video, you can find quick demos from -
• New Web Group
• Leeway Hertz

A

Sure hello welcome to this week's retrieval market demo day. Today's date is wednesday, the 11th of may 2022 on the schedule. Today we have new web group talking about titan ultra networks. The first milestone and leeway hurts speaking about the retrieval performance, dashboard so we'll hand over to edmund from new web group. To start us off.

B

Hello, everyone- this is new web group today, I'm going to talk about is about the titan outro networks, I'll share cdn for verb three.

B

In in this meeting, we have some senior software engineering manager, lean and project manager, yuan and me admin as a presenter for this. If you have any questions, you can ask lynn so I'll, do the translation for you and also you en. Basically, let's get started today's agenda. We're gonna talk to two things. The first we have. What we're gonna talk about is what is titan outro networks and also we're going to talk about the interpretation of tetanus networks, function, model.

B

So what is the titan ultra networks?

B

Titan outro networks is a high-speed cdn service for web3, based on ipfs protocols, which pushes the content close to user through certain rules, so that user can get the content they need nearby in order to resolve network congestion, problems, reduce network bandwidth, costs, improve business response, speed, improve user experience and ensure business data security.

B

And how do we do it? We can make full use of the shared network resources nodes, can cover pc clouds, mobile devices, etc, etc. To make full use of edge nodes storage capacity and idle bandwidth to provide a better user experience.

B

The role of titan outro networks. There are multiple parties involved in entire outro titan network systems, each with different needs in version 1.0. We divide them into user edge, nodes candidates, metadata schedulers reputation, system, data set and retrial portals.

B

The specific division of labor for each role will be explained in details on the titan websites and national documentations.

B

How to join the titan network, the service provider of edge shared nodes only need to install the titan outer networks, share resource clients, connect to the network, generate a file coin wallet and becoming a family edge ipf as lightweight nodes, participate in contributing resources and get few rewards idle. Bandwidths storage resources through the titan outer networks, transform into the titan atom and transform into a shared pool of resources which are cached in the ones by the intelligence scheduling center to achieve acceleration through resource usage, frequency condition for becoming an edge nodes.

B

The upload bandwidth, the initial bandwidth threshold is set at 10. Megabytes per second users are allowed to freely choose the amount of resources they wanted to contribute at the beginning of the installation process. The scheduler will not use more resources than the value field in by the user for the service provider notes.

B

The inspection cycle, the inspection is carried out in every 5 minutes. It will check every 12 times in every hour and the revenue will be generated after 900 times of inspection.

B

The note is examined for stability, resource quality security, etc, to determine whether it is a malicious node or not.

B

If the quality of service is consistent with the commitment, the nodes start to generate revenue after 900 heights, according to the volume of service, the requirement to become a candidate or verifier, the pledge coins candidates need to pledge the corresponding falcons election cycle.

B

According to our setting, each candidate is quanti qualified to be a verifier and each election cycle. That is seven days. A certain number of verifier will be randomly selected from the candidates to verify the validity of the storage and bandwidth of the node across network.

B

How do users get file accelerations in titan networks?

B

Users just need to enter the cid just like we use google search and then get the file to download based on the frequency of the content usage, the scheduler push the content blocks from verified nodes and conditionals to the appropriate edge nodes based on the source, ip of the user request and the ipfs block cids.

B

The user is provided with optimum delivery paths.

B

Accelerate principles, in other words, commanding scheduling. How does the scheduler know what blocks are stored in each candidate's nodes, verifier nodes and etch node, the scheduler directs the candidates, nodes or verifier nodes to pull blocks on car valves from ipfs felcoin. If the blocks are pulled, the cid are returned directly to the scheduler. It is pulled. Car file, the candidate nodes or verifier nodes need to return the cid of all the blocks inside the car file after pulling a whole car file.

B

Finally, based on the frequency of the content usage, the source, ip of the user's request and the ipfs block cids, the scheduler, the direct edge node to the candidate nodes verifier knows to pull the blocks for cache. This makes the transmission better.

B

How does a verifier node know what blocks are stored in the candidate nodes and edge nodes? Candidate node is kind of know that in titan networks, according to our setting, each candidate is qualified to be a verifier and elect a cycle. Alternatively, seven days, a certain number of verifier will be randomly selected from the candidates to verify the validity of the storage and bandwidth of the node in the whole network.

B

The scheduler directs the verifier nodes to the authentic nodes of the whole network and verify the results and uploads key information that is, the storage capacity, bandwidth and online hours and for each height.

B

Thirdly, later with the window fvm, the failcoin visual machine is online. We will develop the corresponding contract support after each election cycle. The scheduler will reward on punish the nodes based on the verification results.

B

How and what files are accelerated, our target for acceleration is ipfs blocks, so there's no restriction on the file, size or file type and the ipfs blocks can be accelerated for all file formats and file sizes.

B

When the file is uploaded to the ipfs network, ipfs cut the file and the number of blocks cut for large file is high. Well, the number of blocks cut for small file is low, so whether this is car file with your data etc, all can be accelerated.

B

What types of files can be accelerated, how large the file transfer can be accelerated? Okay, let's take a car file. For example, if a car file is pulled, then the candidate nodes or verified nodes will be split. The car file into several blocks after pulling the whole car valve and transfer the blocks to several edge nodes as a closer to the user compared with the direct acceleration of the carville. The effect f effect of accelerated blocks is much better and it requires less hardware.

B

Resources which is very competitive in terms of resource costs and consideration effect blocks transmitted by the titan networks. Are the combo combined into cloud file or data needed by user through the titan retrieve entrance provide the user with stable, high quality downloadable accelerations.

B

Thus, it supports falco file storage. Thus, the titan network's pcdn supports storage data. The titan network does not support user to store data freely in the early stage, however, but the candidates and the verifier knows of the proposed proposal are essentially ipfs nodes, so they can be supported in a later stage.

B

The consideration effect can be obtained by telling the pattern networks the cids and the target areas in the future. You can upload files to titan networks, choose your acceleration resources freely and pay according to the usage.

B

If you have any questions, please let not hesitate to ask me- and maybe I can translate to you with uh to dean or maybe you can ask uh yuan for.

C

Any further questions.

B

C

I can start okay uh thanks everyone uh for joining this call and giving us this opportunity to present. uh I am the book. Let me share my screen.

C

And uh let me know if you can see my screen.

A

C

Perfect, okay, so I'm deepak, I'm the cto with leeway. Heads and uh we've been working with uh protocol labs uh building on filecoin for almost uh uh three months now, and uh we've been uh working closely on uh basically uh capturing the the network performance, so we're working on retrieval performance uh dashboard.

C

So in this specific product, uh what we're doing is, uh as you all know, filecoin has been like successfully able to measure and analyze the storage performance of various files, although the retrieval performance is something that has not yet been monitored or has not been measured so far. So as part of this project, uh this is what we are planning to do. uh We have. We are basically comparing the file retrieval uh from various uh uh storage providers.

C

uh The value in this uh is to basically uh uh compare all these different uh storage providers and find out uh the the performance based on various matrices.

C

So the overall networks that we are planning to uh our storage providers that we are planning to cover in this project are cloud flare, uh estuary, meson, uh web3, storage, ipfs smile and and as we uh move forward, uh I will see uh if there are new storage providers, we'll include them also uh so based on uh different matrices.

C

What we are trying to do is we are trying to uh get the performance of the retrieval of a specific file based on the location, uh so the matrices are ttfb, which is time to first byte.

C

So, let's say, if I, as a user, tries to fetch the file from uh one of the networks, let's say nissan or mile how much time it takes us to basically fetch the first byte and then what is the overall time it takes us to download that file uh the c using the cid so ttlb, which is time to last byte.

C

uh We are also working like the extension to this whole project is to uh basically provide the reputation score also to these uh storage uh providers and retrieval providers.

C

So that's the overall uh project that we are working towards now.

C

This is uh the high level designs, I'll walk you through the whole design, what it contains and what all different components of this project are. So let me just jump to the.

C

Rear performance dashboard, so this is the uh the working prototype of the overall uh project so over here, uh if you see uh what we have is uh we have uh these different storage providers that we are trying to compare in this first version. So these are the uh the storage providers, then what we have done is we have launched like multiple bots in different locations, so these are the different locations where all these bots are available now corresponding to each storage provider.

C

What we have done is we have we have stored one file and corresponding to that file. We have the cid available, which is the content identity file, and uh we are trying to fetch that file using different bots that are running in all these different locations over here. What we are trying to show is, let's say if we have uh nft.storage as one of the storage provider and we are trying to see the performance uh in from the location north virginia.

C

uh This is the file, and this is the uh time when we last fetched this performance, so you can basically find this performance for last 7 days or 30 days, then, corresponding to this, we are showing. What is the best speed time? What is the average speed time and the worst speed time for this specific uh file for this specific location? For this specific storage provider?

C

Now, uh on the right hand, side, you can also see all these different nodes that are running different boards that are running from different locations and based on these color coding, these legends. We are basically showing what is the performance of that specific location to fetch that specific file? Now over here, we have, we are showing like all these uh like. This is like a real-time dashboard that we are showing here so for bahrain for estuary.

C

What is the current atfb for a given cid? Now all these files that we have uh stored on these different networks? uh They are pretty much of the same size. So I'll explain a little bit in more detail when I explain the source code structure as well, so this is like uh the current uh uh the retrieval time uh what it takes for uh any cid to to fetch uh like. uh If you see there are different color codes, so under 200 milliseconds, this is the color code.

C

Then we have divided pretty much it into like five different uh legends, excellent good average, bad and poor.

C

You can basically go ahead. Change the uh the dtlb performance here you can see the difference. uh So it's live. You can change the location also. So let's say if we change it from north virginia, let's go to southeast asia, so this is the time it is taking. For that specific content to fetch, uh you can basically try to see different storage providers. What are their performances, so this is the performance based on the file which is available the location.

C

So this is on a very high level how the dashboard looks like so now we can basically move towards the architecture of this application. Also, now before I move to that, there are a few new dashboards that we are working towards.

C

So let me just try to show it here so uh so this is uh the historical graph that we are trying to show with respect to these different uh storage providers here uh with respect to a specific location.

C

So uh how these networks are performing over a period of time and uh what we are also showing is like uh based on a certain event, uh if, uh uh if certain uh networks performance goes down or up so we are also trying to capture these different uh events. uh That has happened. Let's say there is a new version of the the storage provider, any upgrades to their network or anything, and what is the uh the performance after that specific event? That's what we're trying to show here.

C

You can basically change the the time period, also and over here, you'll see the dates and the the performance in terms of milliseconds. uh So this is something that we are still working towards now, uh if I just quickly move to the high level architecture for this application.

C

So what we have is uh what we have here is. uh uh We have created a config file uh in this system, which basically consists of uh the information around the content, url, the cid and the base url that we're using uh for that specific network and what is the destination from where we are trying to fetch that uh that specific file. So this is the config file, it is basic. It is uh available with the scheduler service. So in this you do your service uh it.

C

It picks up these uh uh this config file and keep pushing the events file, retrieve events in the amqp over here and it's happening like every 30 seconds, so that we can in real time we can monitor the performance of these storage providers.

C

So these are various boards that are running in different regions of aws. As of now I'll explain a little bit more around different uh infrastructure services that we are planning to use, uh but as of now, we are using aws and these are different boards, which are running in like almost 22 regions.

C

uh So these boards, uh they they receive the event and they have all the information in order to fetch the file. So they fetch the file and uh they have the logic built in wherein they create another file. Retrieval uh retrieved event uh to this next queue. uh In this uh we are, we are measuring the tplb for that specific uh file, content ttlb uh the the time period, the location at what time uh from which uh location, bot location.

C

We have uh retrieved that file and we are storing it in the database uh uh and then uh using the apis you're trying to show that on the front end. This is like on a very high level how the overall architecture looks like.

C

Now, if I just quickly jump to the source code, also, these are different microservices that are running in the system. As I explained earlier, we have a scheduler microservice, so in this uh scheduler microservice we have the config file defined under modules uh jobs. So this is the config.json wherein we have defined these content, urls network destination file, size and the cid for that uh file, so it it basically reads this config and pushes the content to the uh to the amqb, which is running again on aws. As of now.

C

This is the content retrieval bot, so these are the boards which are running in different aws locations.

C

Now what it does is, uh as I explained, this is what it does: uh fetches the file using the config and then capture ttfbtlb uh for a specific time uh time period and then accordingly construct a event and push it to the uh to the event uh to the amqp. Again uh again, there are a couple of micro services that we have created for uh storing and uh the for the api purpose, so this is on a very high level, uh the source code for this project.

C

Now, if I talk about like what's upcoming, what are the new things that we are planning to work towards, and then maybe uh after this, we can take up some questions.

C

uh So as of now, um as I mentioned, we are using aws as for launching these bots now, uh in order to get a real-time scenario from a end user perspective like there could be some network issues, there could be some latency from an end user perspective, so we are trying to utilize these infrastructure services that are available in different regions and so that we can have much better uh matrices.

C

uh Some of the uh networks that uh the storage networks that we are planning to bring in uh into the system like saturn is one of them. Then ipfs cloudflare is there titan and mile.

C

So this is from a matrices perspective. One more matrix that we would like to bring in uh into this system is uh so it's like an end-to-end uh end-to-end use case wherein a bot uh receives a file uh that bought basically store the file using one of these uh networks, uh one of these storage uh providers.

C

uh We store the file, and then we try to retrieve the file also so that from storing and then up to the retrieval. We can calculate the uh the overall time taken for that entire process, and in that case, what we are trying to achieve is one is the storage time. Another one is the retrieval time, and then there is the cloud distribution uh uh time as well. uh That is uh also one of the important aspect that we are trying to cover in this uh project.

C

uh Beyond that we have some thoughts around the the creation of these uh reputation system uh using the reliability, scalability, uh the performance of these uh storage uh provider networks and uh uh create some sort of uh score around that. So that's uh on a very high level. I know we can go a little bit in more detail, but we can always take in questions after the second presentation.

C

I think I'm done here, patrick any anything else. I would like to add. I can.

A

Great yeah, thank you very much. uh I'm just gonna have one clarification which is uh well at the beginning. You were talking about storage providers um yeah, we use the word storage provider to talk about the the the far coin. Nodes are actually storing the file and sending proofs to the to the uh the chain. uh So in this case it's retrieval providers or retrieval networks um that way that we're measuring the performance off but yeah all looks great. So thank you very much as anyone else.