Filecoin Filecoin Storage Providers, 22 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Miner Tooling Showcase and Dev Grants

Description

Explore some of the amazing mining tools being built by the community and learn about the new miner tooling grant program.

Join us for Filecoin Liftoff Week, an action-packed series of talks, workshops, and panels curated by the web3 community to celebrate the Filecoin mainnet launch and chart the network’s future.
https://liftoff.filecoin.io/

Events take place all week, October 19-23, 2020. #FilecoinLiftoff
For more information on Filecoin
- visit the project website: https://filecoin.io/
- or follow Filecoin on Twitter: https://twitter.com/Filecoin

Get Filecoin community news and announcements in your inbox, monthly: http://eepurl.com/gbfn1n

Chapters
00:30 Introduction to Hactar
05:50 Hactar Demo
08:23 Introduction to Hyperion, a lotus-exporter
20:18 Hyperion demo
25:00 Introduction to Miner Dev Grants

A

Haktar is a tool that we have developed as part of the grants program.

A

Our motivation for building haktar was a need that we recognize when mining ourselves other coins or generally, is that you need to have some data insights at your fingerprint at the tips of your fingerprints, because you don't want to connect to the remote server and check the data there run the cli commands, but you this is a tool where you can connect either from your mobile phone or from your pc to a web service and have all your stats in one place.

A

uh Our landing page is hacktard.app, so you can check it out and start using there and read more information about it there.

A

So currently, what haktar can do and which data it can read, is general status of a minor node. This includes on which lotus version your node is running. Does it need an update or not? So you have a check mark about that? Is it online or offline, of course, because that is a very important metric. uh What's the disc occupancy, is it the disc near full or not and general minor info which addresses use?

A

How about sectors and their sizes and minor power so a general overview, and then it contains some history overviews and graphs on minor performance, such as, when was a mining reward collected uh a history of uptime when the node was online or offline and list of deals and their status price and how long they have been active and, of course uh the important part for us was uh the offline monitoring. So you get a report on your in your email that your node is down.

A

uh This is the current version of haktar, in which information it has. If anyone uh has any other needs that has feedback. Depending on this, we will listen uh about it, so shoot us an email uh how hack that works is actually it. Com combines uh three parts uh on the left side. You can see that uh a miner has a few running nodes. Let's say on each along each node and lotus, node and miner.

A

uh You need to install a haka daemon app, it's a small app that reads from the lotus node and miner information and reports to our api, and then there is our application. That just reads reads our api and data from the database that we serve this data from so basically you don't need to expose your miner api or so on, but the hacked demon app reports, all the data I'll go through a short demo, how to set it up and how it looks like the demo was uh built on devnet just to note the numbers.

A

So this is our landing page with uh of the app where you can register it's uh free to register.

A

You can sign up using your email and sign in as well, so that would be the first step to go to analytics dot, hack, dot and register there. There you'll receive information. What do you need to do on your miners server and that is install hakata, daemon app.

A

So you connect your remote remote server where you run the miner. You download acta, give it execution permission sorry for the bad copy pasting skills here.

A

But you basically, you just need three commands to to run it and it works out of the box. You use the same login, of course, because we do have authentication there. Sorry, I misclicked here.

A

Sorry about that uh yeah and start it and enter your information, the same credentials that you have used to register on our website. This was the message that my miner is down.

A

So, just to set up everything I have run lotus node.

A

This is a familiar process to a minor, I believe and then running a one minor note, and now I can start factor because.

A

It needs a running node, so basically that's it. You just start it, and if everything is all right, it should be reporting. And if you refresh your dashboard on our app, you would be able to see the stats.

A

So here is the message that update is available, and this is the general miner info that I was talking about the sector sizes and the miner power. How all this information you can turn on and off the email notifications.

A

You can manage your notes and it supports multiple notes. So, if you're running multiple minors, you can all do from one dashboard.

A

The remaining data is balanced. How much your balance has changed depending on the previous day. How long has it been online? The disk space through time as well and mining rewards through time. This wasn't devnet, so not much of data and deals and their status, size and price.

A

So it's a basic data about your uh miner performance, which can be easily installed and easily viewed from from the app from wherever you are.

A

This is an example of email notification. If you, if your miner gets turned off by any chance, it's not working anymore.

A

If you want to learn more about haktar, you can check out our main repository on github or you can go to our application as well analytics app and just start using, and that's it. Basically, it is ready for use and would love to receive your feedback. Regarding that.

B

Hi everyone, so my name is julian: I'm a miner and today I'm going to present what we develop for visualization and analytics so quick introduction, um so uh I'm using um ipfs for, like I think, three or four years uh and I jump on the icon uh via the space race course months ago, and uh so uh today, what I'm going to present is that how we can use and leverage graphene, which is an analytic tool uh that we want to use.

B

So it's it's open source, um and here we go so the objective uh for us was like during the space race we identified that we were just a lack of reporting tool. uh We were not able to do any uh proactive monitoring. We don't have any push notification to mobile, so we want to to focus on that first, uh but part of this project. We also expect to uh um improve all the operation uh on managing a minor.

B

So actually we want to move from what from product from reactive to more proactive and don't spend all our time just watching uh our screen. um So this is the how it looks like today and how it uh was during the space phrase for me. So it's like a lot of terminals to watch all the different logs and and command lines with all these useful information.

B

uh So the good point for us is that um protocol labs uh with lotus. They provide an api to get almost all this information through this apis. um So so the idea is to move from this current monitoring to uh to grafana.

B

So graphana is open source. uh It has a very, very huge community and um we already have a lot of dashboards out of the box that we can leverage, but they were missing um lotus, minor information.

B

So um what we we did is like we leveraged a graphene with approach uh pro parameters, so parameters is a database that collects all this information and graphene is presenting this, and so we develop what we call a lotus exporter, which is a small component that you you deploy on your on your miner and you retrieve all the data that are then analyzed by the system.

B

So we are two to work on that there is the sender and myself sonic, so uh you can join and reach us when you want something on slack so lotus exporter exports, the the lotus information. So to promote this and graphene to be nice graph and notification, so here it's how we work and why we decided to to go for graphana. So um one of the main idea is that so super promoters can collect information from a lot of different uh protocols and and and targets.

B

um So the fight coin here uh icon is to present the lotus miner. So we deploy a node exporter which is a component from uh uh produced by uh prometheus. um It actually just bind the port and you can just retrieve a lot of data.

B

There is a very small footprint on the server, um so it was quite useful and uh one of the big difference uh with the other competitors or the other solutions is that this one, you are um putting uh the data and so because, actually our prominent server is uh on our private network.

B

We don't want to expose our private network to the the cycle nodes which is on our public network, um so what we did is actually we would prefer that just putting that having a software here that is pushing the data and once the data are in promoters, you just have a web ui, which is very, very nice, I'm going to do a demo for you um that is creating all the different graphs based on the data we have informatives.

B

So um why grafana? So there is a huge, huge community. You have a portal, you can download a lot of existing dashboards that are impressive. You can easily export and import this dashboard, so it's very convenient. The the ui is is very, very very nice. You can customize almost everything it's easy to deploy and configured um and the the component that is deployed is a very, very small footprint on the machine.

B

So I will explain that later on and out of the box, you already have dashboards for the all the system, information and the gpu, the nvidia gpu that we are leveraging. They already have a dashboard. So it's very easy and you will have a lot of uh reporting information uh here.

B

So what we developed is there is a parameters exporter, so that um it's it's. They have to define on their website how an exporter should work and we we just follow the guidelines. So um it's a very small program. It collects lotus nodes and and minor data everything is done via api. So the idea is is that when there is new release, we don't have to to to re-adapt the code based on the different command lines.

B

It's completely configuration-less, so you have nothing to do except running the script on the machine. It's pulling the data every minute um and the there is a small footprint. um The execution every minute is just one sec. So in this one sec, we we gather like between 1000 and and 2000, depending on the size of the miner information from this miner.

B

Data are pulled from parameters such as explained before, and you just deploy it on the lotus miner and you don't have to deploy it on all the different focus.

B

We have two different versions today of this exporter, so sender is working on a node.js version, which is a daemon running on the machine binding the port, that's connecting um in parallel as I developed another one, which is a python script that is run by the chrome tab and it's done every minute. So so the the result is exactly the same, just two different approach.

B

I will. I will not go through all this now. So, let's go through a quick um demo. um Okay.

B

So now you should see my screen.

B

Okay, so what I'm presenting here is uh yeah. So, let's start by this one, this one is the output of the the the exporter that is running on the miner machine. So this is all the lines that are produced by the exporter. So it's very easy to to troubleshoot, and you can I mean it's. The script. Is that big, so anyone can just review it and modify it. It's very, very quick! um So that's what is produced by the every minute, then you have the prometa server.

B

So the database that is collecting this data and presenting this data so same thing, you can connect, it's very easy, very convenient. You can run troubleshoot, I mean you can see, realize recline what are all the different values and then this is how it's presented. So here this is the standard dashboard which is coming. uh I mean out of the box in grafana. So as an example, this one, you already have all the useful information you expect to monitor the system. You have the cpus you have.

B

I mean there is hundreds of dashboards in this one and you can select the one you can easily take it. Create your own, like this modify it just in just in two clicks. Everything is is for the the you know, type script, almost anything, so you know that the script uh it's just configuration you can change the colors. You can change the threshold. You can change the notification.

B

Everything is very easy and very convenient, so this is the standalone one, which is, of course, missing all the latest information. So what we did is this one, so so the idea. So what is the target for me is to get rid of all the terminal, except when I have a problem and I need to connect to the machine, so I want to get all the different screens on the same one. So this is work in progress. This is just the beginning, we're starting this like a week ago.

B

So, okay, this is what we have today. So the first thing is that we created it to be able to manage easily different miners, so we can leverage this we today we have. I just I have a few uh miners, um so let's go through this one. um So what we presented here is that we retrieve information from the network like the chain head network. We are on mainnet.

B

You can see your your power, so this is um with time. So that's my power today different balance with different colors. So here like this, is the the available balance, the one that is the most important, because, when you're running out the phone on this one, you go into trouble.

B

So here you are monitoring the miner, the owner, the worker, the control address so and you get colors and you get notification when actually something is going wrong here you have some phones that are locked like the pre-commit testing pledge and uh also feedback um and other wallets. So I've got a few other wallets that you can. You can see here and I can easily monitor.

B

We can also retrieve useful information like the daemon version, so when you are managing different cluster, it's easier to identify if something is going wrong, so we are going to have a full inventory on the server that is available on this dashboard below. On the page, but you can also have some we can check what are the net pairs. So if it's going too low, we know that we are going to issue the same thing. The green is changing to orange and red.

B

Now the useful part that I was missing during the space race, um so we can follow the number of stealing process in real time. We have here, so what you usually have in the comment lines uh uh yeah. So as you can see so here, it's manual refresh, but you can set it up to like every five minutes or every every every minute, if you like.

B

So everything is live it's connected to machines, so you can see what are the ceiling jobs different stages here we created one that is showing all the the history of the ceiling jobs uh and you can see in the red line as an example. What is waiting so it gives you useful information to to troubleshoot and see that you have a lot of jobs waiting. So maybe your settings are not optimal or you have a problem in your in your scheduler uh yeah number of sectors, so sectors reporting.

B

uh We have this is the sectors I've got today: the probing sectors, the ampoule which is empty but like during sr2. It was a different story, so here we are monitoring, vm pool. So if I send a message you will show the message appearing here. I can try just to show you how it looks like here we go expecting it takes more than one minute.

B

B

Here we go so it's showing the message and we have some filter here, so it's showing the owner address instead of the existing address from the network. So it's more visual, it's more easy to troubleshoot. So when you have your your proof of space time coming, you will see become progress. So you can see there is a control address in the in the queue um yeah. That's all for this uh other graph uh that we are doing just to same thing, to troubleshoot the different team, jobs and and the queue the balance uh different balance.

B

So we graph over time. um What do we have here? Oh yeah, so this is some useful information that usually we don't really have a look, but that can be useful for troubleshooting from from the the daemon.

B

So you have some bandwidth and usage of the different components that are essentially data on the network. So that are same thing with time here and yes, that's!

B

The last one is the time that the the the the exporter is taking to retrieve the data. So, as you can see here, uh the last one was like two minutes, uh but most of two seconds, sorry, most of the time, it's more like it's more like two seconds or one sec.

B

What is taking time is uh to retrieve the data from the different sectors. So this is something you can disable. If you, you don't want to get reporting on the sector. So I personally at that time like to get that that type of information um yeah. So let's go back to the presentation.

B

So what's coming next, uh so we can easily integrate all the storage information. All the deals, information we can create more analytic, graphs, same thing if you've got ideas, uh feel free to to give me these ideas. I mean presenting the data. Is you have it's really a green field here, so we have plenty of possibilities and most of the time, I just don't really know how how to present these data. So if you get ideas, that's fine deadlines are coming next, so they're already available in the exporter.

B

So it's just a question how I present this data and useful information about the network, like the gas price, also the worker resources. We are not showing here that I used to leverage through the different command lines, and I would like to get them available here and the number of blocks. Of course we won't.

B

So um if you want to help you're more than welcome I mean creating graph is not very difficult, uh jumping into grafana it's just taking a day um so you're more than welcome. If you want to create and and propose, and and and try this, this software, um the python one, the one I'm leveraging for this graph, we just show um is actually not available right now on github, but I will release something soon, probably by the end of this week. So I will share the link.

B

In the meantime, you you can reach us through slack, so uh our two different addresses are just there.

C

Hi, I'm going to give a quick overview of the dev grants program. Then talk about the minor, tooling rfps we have, and the git coin grants quadratic funding program they're, currently a part of falcon dev grants. We accept proposals at all levels of the technical stack. This includes at the protocol layer all the way up to end user applications and user research. We also accept proposals for mining tools. Velma's hacked our app was actually funded by a wave one follow coin dev grant file coin dev grant wave submissions are normally due quarterly every few months.

C

We will accept your proposals on a rolling basis at any time, but at the beginning of every quarter we get a refresh budget and start reviewing and approving new grants after the deadline so for priority consideration in a wave. It's best to submit your proposals at the beginning of each wave currently we're in wave five, and that was due october 1st and our next wave 6 will be due january. 15Th and wave 5 recipients will be announced soon.

C

We currently have a special round for may net liftoff of grants for tools to support falkline miners. uh The deadline for proposals for this is november 21st, um and we have a 50k budget in grants for this. um So this is going to be part of the bitcoin clr quadratic funding program.

C

How it works is that for every one die, anyone from the community contributes to any of these grant projects. Protocol labs will match that contribution using a quadratic formula, specifically the sum of the square roots of each contribution.

C

So this allows many small contributions to matter more than a few big ones and as a way for the community to vote on these ideas. So even if you just contribute one die, it lets us know that you voted for this project and believe in it and the more people vote. The more protocol labs will fund it.

C

So quadratic voting has been a really cool idea. That's been around for a while in the blockchain community, and quadratic funding has been very successful for the theorem community. uh Vitalik buterin has a blog post, analyzing bitcoin clr rounds and wrote a paper on it with glenn whale and zoe hitsig about applying it to public goods. For the commons and as an example, there is a bitcoin community fund for implementing eip1559 by currently worth over 95 000 uh in in dye.

C

So we are trying this out for fog coin. For the first time uh also, anyone can submit their ideas for projects on git coin, so you can suggest an idea and submit it there, and uh you will see a few already submitted ones from this page uh on on that page, get coin, dot, co, slash grand slash liftoff and you can learn if the community supports your idea and protocol labs may also fund your project, even if it gets no other contributions.

C

I will review them so visit that url to look at the list of projects and contribute or suggest your own project. There are currently nine minor tools rfps to start, and we may post some more in in the next week or so and so to recap: we have nine starter rfps.

C

Anyone can propose new projects, anyone can contribute, die to or vote for a project, and we will match that with quadratic formula um and contributions have to be in die, but bitcoin is adding fall coins soon and we hope to help get all your ideas and tools built.

C

uh The current round ends in one month on november 21st and if you don't want to use bitcoin, um uh if you and you can submit any proposal, the regular way at the dev grants repo on github, and if you have any questions, please dm me at each on e-s-h-o-n at fall coin slack. um Next, I'm going to talk about the nine minor tool.

C

Rfps we have posted uh pl will be seed funding these tomorrow with a few thousand in dye to start to research, these rfp ideas, we interviewed various minors and got some terrific suggestions from community members. There are some tools to support smaller minors, specifically like the monitoring tool, which is just about helping a minor monitor their system. Resources is actually julian and sandra's idea.

C

uh So this could also just be a script or tutorial on how to set up monitoring using prometheus and or telegraph and potentially grafana, that a small miner could easily run we're interested in helping a small miner run it on the same machine as they're running running their their miner. um But we also welcome more advanced monitoring tools uh yeah, so julian sanders tool may be eligible for this grant. um Next up, we have the minor sector checker, which is about adding more cli for finding failed sectors and removing them.

C

Miners currently need to keep monitoring their sectors and respond to faults or failures, making this easier to check and then pull up relevant logs and messages along with the cili um and would be really great. So the third one we have in this list is the cid indexing tool for miners, just about making a minor storage, more transparent and accessible to storage, clients and retrieval clients such as developers built building storage apps.

C

So this idea came from juan bennet himself and is about indexing all the cits that a miner has, and it really would help the retrieval market for people to be able to look at like which miner is storing, which id um the next few we have is a one is a minor transaction history and earnings predictor rfp.

C

So it's currently not easy for a miner to figure out all the fees and gas and specific transaction history. They have even on a block spur like like field fox. So nor is it easy to predict earnings.

C

So uh we have we've had a lot of requests for an roi calculator, but it's kind of complicated to build one, so we're starting out with more basic components, and this is about just interpreting a minor transaction history in detail and then using that data to predict future earnings, and another basic goal of this is to help miners figure out how much they need to keep in their wallets for fees over time.

C

Next is the auto add new sectors based on capacity capacity. Adding new sectors is currently managed. Manually it'd be great to have a tool that can monitor systems resources and, if there's capacity available, automatically accept new deals and add new sectors.

C

Next is the mining task pool for smaller miners. This one is a little controversial because it's a trustless mining pool is not possible in fault coin. Right now, um we may have to like propose an fip to the protocol or wait for user to find smart contracts coming next year, but this was proposed by a community member, and some miners may want to try it out anyway, and we're interested in funding any updates to the fsm that could be used to farm out tasks to smaller remote miners.

C

Next few we have the gas tool and guide um any tools to help. Explain current gas, including base fees and feed cap for each tip set and average gas premiums could help community members also understand how gas works in fall coin and uh the next one minor, docs and community guides is kind of a fun one. There are two parts to this. We currently welcome really good prs to the file coin docs.

C

It will be at the discretion of the docs team, uh what qualifies as a good pr worthy of a grant and then the second part is if you write a blog post or make a tutorial or a video with tips and analysis about your fault coin, miner or mining and file coin in general. We might fund that. So here are some topics of interest.

C

For example like how did you set up your rig, set up tricks for lotus your redundancy strategies for storage, redundancy um monitoring tools that you use and your disaster recovery plans, so all contributions will be judged based on quality and award amounts determined afterwards at the discretion of the grants and docs teams. So a doc's pr could be worth anywhere between uh five to 150 dollars.

C

um An entire topic uh that you write about could be worth between one to five hundred and a minor video tutorial. It could also be worth one to five hundred. Finally, we have the minor hardware: profitability calculator, it's a general calculator for hardware, setup costs and how that impacts ceiling rates it doesn't have to include collateral. Someone else is building a model for that, so this is just about um sort of the estimation of hardware and ceiling rates.

C

So so those are the mining tooling grant rfps we have so far, um and if you have any questions reach out to me on file coin slack.