Filecoin Filecoin Storage Provider Bootcamp, by ESPA, 5 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ESPA Module 4C - System Administration

Description

Benjamin Højsbo (VP of R&D) with PiKNiK discusses system administration at the Enterprise Storage Provider Accelerator (ESPA) bootcamp week that took place in February 2022. This discussion includes operating system, storage architecture and hardware, power and renewable energy considerations, and data center considerations.

Data is growing at an incredible speed and much of this data is archived and/or simply lost by enterprises. Our program will accelerate net-new Enterprise Storage Providers into the ecosystem, using a Web3 protocol with an impressive incentive model called Filecoin.

Learn more at:
Sign-up: https://web3espa.io
Landing Page: https://m.fil.org/espa-bootcamp

Follow ESPA:
Twitter: https://twitter.com/web3ESPA
LinkedIn: https://www.linkedin.com/company/web3espa/

A

Thanks guys um yeah, so um let's look a little bit at system administration, so we're just gonna touch them. So there are some other topics of doing the system administration for launches.

A

So um the first thing here is yesterday: I talked a bit about the os using ubuntu and and what there is around that um so here we are basically going a little bit more into so. What does this mean when you're running low tooth on on something like like linux and yeah? So basically the thing that we need to watch out for when doing a system, administration tasks is basically that we want to make sure that we have as little downtime as possible.

A

Now for the first slide here, that's around the minor and the market node, so that would be like the central nodes and the ones side is like um client-facing internet facing. So you know for the client, it will be very easy to see if these are down because they won't have any service.

A

um So one thing at least that we want to make sure is like all the maintenance that we do. We never want to do that when we are window posting so for the smaller systems we would normally try to not make that happen. If there are, you know, windows, window post happening on the larger scale systems. We might end up going into having to prove 24 7 right and if we have 30 minutes between each deadline and we might have one or more partitions in there.

A

We need to use fair a fairly a bit of time on these deadlines to actually calculate, and that means that yeah we might be able to do some updates, but if we need a full uh os reboot, that's that takes some time.

A

So um so I'm not going to tell you exactly how you do this, but obviously for now most would be running the miner on on on bare metal you're not allowed to have multiple miners. We talked about this consensus uh slashing happening if you are mining, two blocks uh out of one minor, so that's probably that's the main reason why you can't just have like two miners just doing the same thing uh next to each other, but obviously this is probably where the where we will see it.

A

It will go into some kind of high availability on these services, so you actually be able to take some of this down for maintenance while still keeping the services up, but for now it's more a game of making sure that if we have these proving deadlines coming up that, that is not where we do like the really big update and have to reboot and uh yeah and do all that stuff.

A

um The other thing is like in general, we do not want any downtime, and we also touched on this before, but the main problem is I: if the miner lotus miner is down, if the daemon is not synced, then basically nothing will happen if we get uh the chance to to to make a winning post. I don't know if we said this, but actually what happens to start with is like we get what we call like a winning ticket and that's basically based on the randomness that is uh pushed out.

A

We figure out if our miner got like the lucky ticket um and if we do that, we are kind of allowed to do a winning post, um so winning post uh has to uh to to touch the storage as well and we'll get back to that, but it basically means that we need our miner to be online to actually make this happen.

A

So you just have to remember that if you take it offline for updating or whatever you're, potentially losing money and that's uh well, you would also be doing that with other applications in businesses in general. But it's like here it's a little bit more um direct.

A

It's pretty certain yeah where you could say like if the like, if your users in the company says like hey, my applications doesn't work. I can't do my work. It's like yeah, whatever we fix it tomorrow, he is like okay, you are actually losing money, so I don't know if your experience is that, but but yeah it's like that's just how falcon is it's a little bit more direct?

A

So you have your incentives in the right place: um yeah, okay, so another thing is like on the market note: uh it is kind of the same now. You could argue that uh you don't need to care as much for the market. Note, because, obviously you know you will still be proving your sectors and you will still be mining your blocks, although the market node is down, but if you are in the enterprise gray game and you do have clients well, they might not like that. You know they can't access your miner.

A

They can't talk to it. It will just be silent that doesn't look good, a bit like having your own web page down web server down. So obviously we also want the same for that to have it as as highly available as possible um yeah, but we're not uh we're not directly getting slashed on the market. uh Note yeah. One last thing is like when we do updates on the on the miner we have to remember. The miner is also the the scheduler, so it it's orchestrating all the ceiling, jobs, meaning that we have.

A

If we have our ceiling pipeline fully uh filled up and we restart the lotus miner, it will have to restart all these processes, and that might be a bad thing, because I don't know how much you have touched on this, but if you're running your ceiling pipeline really hard, you would probably you know have do some offset on when you start the jobs, so you keep on feeding jobs every every some period and they pop out in the other end, so it's kind of like a sausage factory.

A

What happens is like if you do a restart and you have it all filled up, they will all start at the same time, and that might not be funny. uh You know if you want to do a social factory that is trying to make it everything at the same time. um So that's just one thing to consider that probably you want to ramp down before you do some of these things: okay, uh so that was on the minor and the market. Note so on the workers um updates on the workers.

A

Well, uh the thing is like workers actually does not need any internet connection. um So obviously, if we want to update our linux and and keep it up to date, the easiest way will probably be to have some kind of internet connection. But if you are running like a really controlled uh environment- and you know exactly what packages, what versions so on, there's actually nothing keeping you from keeping all your workers totally on a separate network. All they need to do is to connect to the bluetooth miner process and that's an api call.

A

They need to be on the same subnet, but that's something that does not to need to be the subnet where the miner has is, or the market node has this internet connection. So that's just uh to know that it really depends on. You know what you're used to doing here, but obviously this could be an approach.

A

So the other thing is like this still applies that if you're shutting down your um your workers well make sure it's not like too congested, because it could be trouble. Then, when you need to start out, if it does happen normally what you would do is like you would do. A lot of uh you know upboarding a lot of the jobs.

A

That's not a problem. If you're doing cc, you just lose some time. If you're doing client deals it's a bigger problem, because what happens is like if you're, avoiding the jobs, you're, basically losing the deal, so you would have to go and ask the client to redo a deal with you um and that's again. You know from from a service perspective, it might seem a little bit unprofessional. If you have to go to the client and say hey you made hundred deals, I lost 10 of them.

A

So can you just remake those 10 because uh yeah it didn't work? um At least you know it's worth thinking about, but but that's probably what would happen what you would want to do if it gets really congested right and as it says here, you can totally keep your lotus miner running while so take the workers. Offline fix them, take them online again and they will reconnect to the to the miner and they will get the tasks assigned again so that that works.

A

So in general you know we probably will see that we move more and more into that. We can. You know, just keep the vital services up at all times and and don't have to care too much about what's going on around them, um yeah cool, so that was for the the updates um we're going to get a little bit more into detail in a bit. But this was just to say: you need to have processes in place, because otherwise uh this normally happens updates just piling up uh yeah. So stu was saying like seriously.

A

Is that your computer and I said like yeah, but it's only like eight of them. That is like security.

A

So it's like it's not that bad, I'm all good, but it's like it's just a reality that you need processes in place to actually handle this, because otherwise you know this will happen and- and it is a security concern, if you don't have a good process for how are you actually going to update it, because I just need to see the whole time and it just needs to win blocks, and it's like yeah, okay and then you get hiked.

A

No, so it's just it's really important to have the process.

A

uh So, what's going to happen, is like this worker right here that we talked about doing the sealing process that can do pc1 pc2 c2 window post looks a bit like a c2 process, so obviously it could be pretty fun if you could just put your pac2 on there.

A

What's going to happen is the worker process here is also going to have a little flag for doing the window post, probably also one for the winning post, meaning that now you can have as many uh window post workers as you like and what's going to happen is like if you have a deadline- and you have 10 partitions in there, because you build a really big miner now uh well, it will just schedule, like all the other jobs.

A

So probably you will see that all your you know, all your partitions will start proving at the same time just goes to different machines, so this will be solved but yeah. This is what we're working with. We started with a like a one box, uh lotus miner, where we did everything in the process, then we moved the market and the market note out of it.

A

What happened at that time was if we got a ton of deals coming in, then they started to do the import, taking up all cpu cores and totally flunking your uh window post, and then we thought like hey, that's not that smart! So now it's a different process and we can like put it on a machine or we can do like restrictions on how much capacity it's taking up so for lotus in itself. um Well, we would normally get it from uh from github where it's uh where it's published.

A

uh The new version is coming there. Currently, there is no real, like containerized version of it. I think it's in the roadmap as well to you know, move over to doing that.

A

So what you have in there is you have like a docker compose file um which you could use, but I would say it's not the same as having just a you know, a containerized version that is tested and working, and you basically just put it on your cluster and and it runs from there, but I, but that's probably going to happen um then um so for now the process is when we get a new version of low tooth. We basically have to recompile it on our hardware and you know having hundreds of servers.

A

That's a lot of resources that you use on compiling. What you can do is like. If you have identical servers, you can compile it on one of the machines and then you can distribute it to the others. You don't have to compile it on everywhere, but the thing is the reason why we are recompiling. Every time is because there is always some updates to how the proofs are.

A

There is also some little tweaks, um but also because um the I guess, the reason why is because lotus is uh getting a lot of updates in a lot of areas means that you can't really compare it to just like uh good old software, where everything is re done for you, meaning that when we are recompiling we're, actually you know it's compiling against. You have an amd cpu, it's the send two. It has these.

A

uh These instruction sets that I can use um it's looking at the the gpu so we're using a cuda toolkit to compile it and do all the code optimizations.

A

So it's like, um you basically need to do that, every time that we have a new version for now, and you could argue that you know when this was like more finalized software, then obviously this would have been done already and basically you just installed it. But for now it's like this process. We we need to go through so um yeah. So, as I said reason why we need to recompile and not just use the pre-compiled is because it basically needs to fit our hardware.

A

So if you do not recompile on your pc one worker, it does not know that your cpu has the shower extensions and can actually use those to uh to run pc1, and it will take you forever to do those computations. That's why we do it uh also. The same goes for it's also used for the daemon for the like the chain updates and so on. So it's it's pretty important to to do that.

A

um Other packages that we're using, I just put something up here, which is like right now, uh some of the modules that is being used- it's not that much! uh So so it's so I guess it's generally. It's! It's not that no, it's not a really big landscape that we're looking into one thing that you might want to keep in mind is like because a lot of this development is done on ubuntu.

A

It means that the developers might not take into account if what it looks like on red hat and that again it comes down to that. If you run ubuntu, there's a high probability that it will probably just compile, and if it doesn't, then it says I need this package and you upgrade that package and everything is good. If you do it on some of the other distributions, it might be that you need to go in and figure out okay.

A

So what does that look like for me and it's not to say that you can't work on the other distributions, because obviously you can do that and you have many miners that uh study dossaf. It's just a matter of that. You know you might be needing to go a little bit more into what packages is available for my distributions, uh the right versioning and the content, and so on. So it's a it's totally doable.

A

It's just something to be aware of that when you are doing something else and running something else and how it was developed, it might not take into consideration that they just added one package and that is actually not available somewhere. So it's like just to keep in mind um the other thing that I would say that you need to keep in mind with this when running the other distributions is like once you start going into having custom packages, and so on. You might end up in like a dependency mismatch, something and you know yeah.

A

I don't know how many of you have experienced that, but it's just like that's not really funny when you are in that situation, where you always start to look like something really custom so so for now, it seems like this is the easy way around it, and if your other preferences is totally fine, you just need to be aware of. You know there are these dependencies and basically you need to be able to provide those in some kind of way.

A

I guess right um so, let's jump so this is a release kind of how the releases are set up. So we have. um You could say that we actually have four different, because um stuart also touched on the release candidates which comes before this. But if we just look at what is like officially released, then you have major releases, so that could be like version one happened when we went to mainnet. I think um then we're gonna have a version two when we get the fvm yeah.

A

So that's probably you know when big things like that happens um when we change the whole access system and so on, you will probably have one of these major releases. But what is really important here is the one in the middle, the mandatory releases, because that is what we could also call the consensus breaking releases.

A

That is something where you need to have that in place before the network changes consensus.

A

So that could be that you have a fip we talked about that before, but it could say like yeah now we want to calculate something on the network in a different way and we need to set some date for when that is going to happen.

A

So we have a specific epoch where we say at that moment the whole network changes into the new consensus model, the new way of calculating meaning that basically we need that low choose version to be in place before that happens, if it, if you don't do that, so if the just keep on rolling on your own one, uh the old one that does not have this uh update, what happens is like at that epoch?

A

You start a fork right, and that means, if you try to do winner post, that's very well, but you are not really yeah you're in the wrong chain. So it's like. Nobody really cares. So your miner will be very happy and send all these messages and so on and you can send money around and uh there is like yeah. So only if you have some friends that want that chain that would care, but on them like on the main net, which is also ones where the exchanges are like coupled into er yeah.

A

You will see that you lose money, uh so this is pretty important and the reason well there's a couple of reasons why so one thing is like. Obviously, you need to look at the community channels like a sac, when it's announced that now we have a mandatory release, so you can kind of start to figure out okay. So when am I actually going to do these updates and so on? That's one thing.

A

The other thing is like: if you go down the road of starting to modify your low tooth, you can do it on some areas, which is like maybe into how your your sealer ceiling operation is working. You can maybe do it a little bit on the scheduler um if you start really changing how you're proving all your data, uh which is critical, that you can prove well, it's just a matter of that.

A

Every time you have a mandatory release, the clock starts ticking because you need your stuff in place before that happens, and the reason why there is this risk of doing your own coding is because, when this clock starts ticking, you might have built a version of low choose that is proving on a really big storage system, but it can only prove because you made your custom system.

A

So if you just put the vanilla version of bluetooth on there, it does not have the kind of the optimizations that you did and it can improve, and if you have a like one, exabyte miner, that can prove you're gonna lose at a lot of money. So it's a calculated risk. You can totally have your own uh developer team that basically maintains their own version of lotus. That has these tweaks that make sense for your system.

A

You just have to be aware that consensus breaking releases like this means that there will be a deadline where you need to fix your software to actually implement this, and a lot of the big miners is currently doing this. So it's just a matter of if you want to take on that risk, or you want to keep it vanilla, meaning that it's like the whole um community that will make sure that it's it's working.

A

It also means that say that there was an update and you're trashed your miner. uh Well, nobody will care about that if there was a consensus upgrade and the whole networks breaks, we'll probably get consensus about fixing it and we'll fixing it for all the miners. But if you did something yourself, it's like uh yeah. We can have a chat around it in in the community and we do that. Sometimes, when people spend like hundreds of thousands of fill on gas, it's like uh yeah, that's uh that's too bad, but you're, not gonna change.

A

Anything and you're not gonna, do a fork. You're, not gonna roll it back to you and say like yeah, that's also I'm! We are so sorry for you. um So the thing is like it's just it's a kind of a choice, but you can do it in a like if you start doing it into the proving that's probably where you see the highest risk.

A

If you're doing on your ceiling, performance and so on, I would say it's probably more sla risk. So if you're, you know, you can provide the double ceiling, speeds and double throughput, and now there is an update and you can make it work. So you now you have to run the normal code.

A

You will have half performance well, it might be that your clients wouldn't like that, but you know that may be a more manageable risk than not being able to prove your whole, uh like all your storage, okay, so uh that was a releases uh feature releases. Normally, that would be like new uh new cool stuff that you might want or not. um If you are not sitting around and waiting for it to come on, then probably you'll just wait until there is a new uh like a mandatory release.

A

If there are some features in those uh feature, releases that apply to you- and you know you have been waiting for it, then obviously yeah. You would start to just test it out and see it can do what you want and then you can implement that.

A

So we normally, you know, update with the feature releases, but if you have a really big setup and you don't really need any of the feature releases like on the short term, probably doesn't make sense to do the update before you have the mandatory release, which will obviously also include all these feature, updates.

A

Okay, so again it's it's! It's just against like what? What up times are you trying to accomplish so um yeah? I hope that makes sense. um Then we just have a couple of other uh little things, so video drivers- we actually already talked about that yesterday, um but yeah, it's pretty important that they work, and today we have this um kind of uh right now, if you want to do the fast ceiling times with the cuda optimizations, you also have to have the cuda toolkit installed.

A

Not only the the the video driver and again it's a little bit annoying to work with that, because now you can have multiple different versions. So one thing is your video driver.

A

We already talked about running it headless, making sure that it's not auto updating, but you should also be very aware of kernel updates, because when you do a kernel update, basically it recompils your video driver and now you have a probably a system that needs a reboot.

A

But the other thing is when we are running with the cuda toolkit. We actually have a guide on how to do that, because it's not a straightforward process. What you would see happen normally is like if you have an ubuntu server and you just go and download the cuda toolkit and install it with the driver.

A

Doesn't work works on some systems, uh but it doesn't work on all systems. So what you might want to do is install your video driver and then find a um a cuda toolkit which has like the at least a major and yeah a decent major version on the same so same cuda version, and then you just compile the toolkit and nothing else, um and the toolkit in itself should only be used when it's actually like compiling lotus.

A

But still you know there is some versioning to keep uh to keep track of to make this work. um But obviously the most important thing is about making sure that the that the video driver is online and working and you have the nvidia smi over there, which is showing you a gpus. That's a good sign, it's not exactly the same as it will work. You have the other one which is showing what the the cuda toolkit, but it's actually only showing the path that there is something there.

A

I actually had some issues when I did an update last time that it was uh still having some old paths and so on. So there can be some issues there, um but um I have a couple of other topics here that we have touched a little bit upon, but this we're just getting a little bit more concrete here on what what we're actually talking about.

A

So one thing is chain maintenance stu talked about this yesterday, but what actually is happening here is um that uh yeah the chain is growing quite rapidly we're doing new blocks every 30 seconds. So obviously you know we have a pretty big load of data that is uh is going on and what we want to do is we want to keep our local copy of the chain to a certain size and not go too big.

A

So I'm not going to the technical parts of why it is so. But there is something is about. You know the bigger snapshot you have of the chain, the harder it gets to. Actually you know calculate the states, meaning that the the bigger basically just the bigger chain file. You have the lower performance you get out of it and that's why we have this uh set up where we kind of prune our chain, and we have a couple of different ways of doing that.

A

So one way of doing that is basically relying on protocol labs, putting up a snapshot which can be downloaded and it holds like the last. I think it's 2000 but x 1000 states and you basically use that file. You download it. You uh delete all your own chain, uh your old chain data, and then you import this snapshot instead and that you know helps you keep the chain. Small, if you don't want this kind of um way, you need to trust a provider of a snapshot.

A

Well, so in reality, I think when we do the snapshots, we don't see that much issues with it because yeah you have to trust the provider of that that there's no malicious states behind. But then again I think you know, probably our the calculation of our hit like what is the hash of that would probably not match.

A

The other thing is like you can probably see right away when you're running it. If you know, are you on the same chain? Do you see the matches, as uh is the same and so on?

A

So I I think for most storage providers, it's not a big issue, but if you want to keep it really clean, what you can do is you can basically tell your demon to create a snapshot, so you can tell it to create its own snapshot and save it to a file you prune the chain again and you import your own snapshot only reason why people are not doing. That is because that takes up a lot of resources.

A

So basically, you just need a beefy machine to do that, because it also needs to keep the chain synced.

A

But then again you know if you have multiple demons running at the same time, to make that high available, you could definitely use the one not in you know not running the messages you could use that to create a snapshot. um So it's just about setting up the processes. For that, then we have storage maintenance and I kind of put this into yeah. I kind of put this into two different uh areas where we have the. I call this long term, but we're actually, you know here we're more talking about the the physical uh storage.

A

So what are we maintaining on the physical storage? um So one thing is: we need to be aware that if we are doing any maintenance on the physical storage, as I said before, winning post also needs access to the data. So every time that you get a winning ticket and is allowed to do a winning post, it will just do a little a little query on your data just to make sure that yeah it's still there.

A

But that also means that if you took your storage cluster offline, because you need to do something with it, it's not go it's not going to do the winner post, so you could say that you know every time you have your storage cluster down, you are losing money, so same thing applies there.

A

um The other thing is uh with the window post that, obviously you you know your storage cluster needs to be in good shape. For that uh I think we mentioned this before, but one of the issues that we see on the smaller storage providers is like. If you are not in control of your scrubbing and your scrubbing, is taking a lot of the resources, then you can.

A

You can end up with a situation where, basically you know that if your system starts scrubbing, then it won't do window post, which is pretty annoying um but yeah that it's worth to keep in mind on the smaller story providers. Obviously you have more. uh You know you have more time between the winner post. So, basically, you could have a cycle where, after I'm done with my like a stuart like having four or five deadlines after I've done my deadlines.

A

Now I have time to take everything apart and assemble it again, and hopefully it works when I have to do the winner post again, but if you are into the enterprise class having a really big setup. Well, obviously, it needs to prove 24 7. So it's like you, you don't really have that kind of time. So you need to think so. The other thing that we need to talk about is so what do we manage in lotus?

A

And this is actually what stu was touching on that you see over here that you have some visuals on? How is it filling up the different storage paths, so one thing that you can do when low tools is running is you can actually attach new storage paths, so it's made to it's made sure that you can always expand that you can keep on expanding while it's running and what happens? Is uh you see that we have? You know the different colors here saying, like I'm almost fill filled up.

A

uh You also see that we have the number of different. uh You know data in there unsealed data. How much is sealed? How much is cached um right underneath that you have a weight? How you can use the weight? Is you can still one of the storage paths to always have like weight to 20 instead of weight 10? And it basically just multiplies by that. So you can make your own like distribution, if you like, on how it's, how it's handling, otherwise, if they are all the same, so all your paths is all the same.

A

All your weights, all the same. It will just keep on you know putting it and filling it up at the same rate, but you can control these rates and also um yeah, I think stuart. You may touch on the max storage parameter that you can basically set a limit on how much it fills it up. So it doesn't like go to 100 percent, um so there is pretty good handling of that in in lotus.

A

But obviously you know you still need to handle the physical data underneath. So um there's not much to do here. uh We talked about that. You can move data around, so you can actually move the sectors around and lotus will go and look up where the sector is on the scratch space. So when we are actually sealing the maintenance needs to be done.

A

There is like, um if we like, if, if our low tooth worker process crashes- and we have some stuck uh data there- uh it might be that we have situations where it kind of leaves the the cache data on disk and obviously that's like this is 500 gigs per sector.

A

We don't want that, so we probably also have a little job of going in there and seeing okay did it leave something from when I I crashed the miner and everything did I leave something behind that is now not like automatically being deleted, uh so there's a little maintenance job there. The other thing is like over time.

A

We are rebalancing, sometimes uh how much nvme space we have, because we basically if, if speeding yeah so so, basically, if ceiling times goes down, uh we have less time where we need the sector to reside on the storage. So now we free up storage. We have also seen that lotus gets a little bit more clever and says like if I have actually moved the data over. Let's just delete it, and you know if things goes wrong too bad, but then again it's not. You know we aren't going to die from that.

A

So it's like not spam, you're, not using too much excess nvme space, because it's very expensive so migrate was just about moving them around and if we want to migrate a sector from one place to another, it could be that I bought a small in a like uh yeah, a small nas, where I have my first sectors on and now I'm building a new cluster. Now I want to move my data over there and you can totally do that.

A

uh But then again you know you might have these processes running to free up the old stuff, to reuse it to something else, and- and there is also this thing that we talked about- that you might want to consider moving it around. So you have a cluster which is not doing something for a period of time, so you're creating like maintenance, windows, yeah, okay um sector management, um so this was just to say that basically sectors, that's where we put the data in to has like three phases that it goes through.

A

So the first one is where we are creating it. It goes through the whole sealing process and the thing is like: if we lose the data there, if we lose a cc, we don't care that much. We are only spending a little bit of energy on the sector. That was lost, we're spending a little bit of time, but it's not a big issue, because what happens is like the majority of the collateral that goes into a sector that goes on chain. That collateral is moved when we do the the proof, the last message.

A

So if we screw up the the sealing process before the last message, we probably don't lose that much collateral right. So it's more, you know lost time on the deals we already talked about, that the issue is that basically, we probably would need to ask a client to redo the deal with us, which is a little bit unprofessional, but we didn't lose anything and we didn't lose the collateral as well, when we are on the on the the second one when we are actually storing the data.

A

So when we're storing the data in long term, we need to prove it, and uh we already talked about backups if it's needed or not when we go to hyperscale, it might be that it's not really feasible to do like uh one to one backups, um but at least for the ones that is doing it. The data that needs uh to be there to prove is the ones that is like the sealed sector itself, but it's also the the cache folder. The cache folder for each of these sectors is not that big.

A

But if you don't have it, you can't prove, and then you could argue, why didn't you just put it all in one file, but that's how it is right. So this is really important. So if you want to have a one-to-one backup and basically have a separate backup miner that could take over if everything just went down, it needs both of these folders to be able to prove so that's happening for a long period. Hopefully, and then we have the expiration and that's when we just talked about that there is a process for that.

A

There's also a link right here. For you know what does that look like when we are removing expired sectors? So that's basically the life cycle, um I'm getting a little bit over time here, so I just had a slide here on on the life cycle management.

A

This is no different from running all the other hardware operations. It's like falcon is not different in any kind of way. So the good thing about filecoin is it looks very much like a high performance cluster or like a machine learning ai clusters, meaning that there is a really good secondhand market for most of this. So the thing is, we have to think about. So how do we source this hardware? Do we maybe do uh do we do like leasing rent?

A

Do we just purchase it if we purchase it, we might think about so how much? um How much says do we have service contracts on it? Do we want to keep it only until we have the service contract and then get rid of it? You know these are like the normal considerations um on the more technical side. It could also be that we might want to.

A

You know, get some new hardware before the before everyone else sell out the the gpu servers and so on, while it's still in like high value uh versus you're, totally running it down. uh The thing is what we've seen over the last couple of years is like when we started out. My first miner had a 2080 ti right. It was desktop computer, so it has a 2080 ti.

A

It used 260 watts when doing a window post and also when doing ceiling operations. Now I have a little uh it's the krato, a 4000. It's it's a can only use like 130 watts. It's it's a really tiny uh gpu, it's faster right, so it's like it's a whole different footprint and if you have a ceiling cluster like that, you might be better off.

A

You know getting the old hardware sold to machine learning, others which you know this would work perfect for them, but it might be not as efficient anymore for valcoin, so there could be a business case and actually getting rid of it.

A

While you still have really good resale values, get some of the new stuff which has like a much lower uh power uh to uh like efficiency uh measures, um at least we're really seeing there that it makes really good sense to be on the newest versions of gpus, because the the optimizations also work very well on these gpus and your power footprint is much lower. So that's that's. Probably all there was to that thanks guys.

A