Filecoin Hackathon Presentations, 5 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Programmable Storage Markets - Matt Hamilton - FVM Dataverse Hackathon 2023

Description

Programmable Storage Markets presented by Matt Hamilton at the FVM Dataverse Hackathon Kickoff 2023.

A

uh Yeah there we go okay right, so hopefully that's uh uh my screen up now, great okay. uh Folks, um thanks a lot- and hopefully all is still good here- and you can all still hear me uh so I'm gonna be talking about running a local network uh for filecoin. So the reason I'm talking about this is, if you are dealing with storage on a file coin.

A

The way in which storage works is you have a storage client and you have a storage provider or a miner, and so the storage client makes a deal proposal that then the storage provider or the storage Miner will then pick up right. So um what we? um So when you do that on a public network, it can sometimes be a bit tricky uh to you know see what is what is going on.

A

um Okay, Steph asks if this session will be recorded and saved yes, it will be it's all me recorded, so um you you will have it here as well. um If you come back to uh back to here, you should uh you should see it again. So um what we're going to do is run it by running a local network. We can play both the client side of things and the storage provider see side of things.

A

So what we're going to do is we're going to set up and run an entire instance of filecoin running locally on um your own laptop right. This uses Docker, so you'll need to go ahead and install Docker. If you haven't, got Docker already installed and you can get Docker if you well so just first of all to show here just the um uh the URL to the repository we're going to be working from is here.

A

Falcon project, github.com, Falcon, project, Falcon, FM, localnet and in here are some instructions here and there's a link there for getting Docker installed as well right, so I'll take you to Docker and how to get Docker installed. uh Docker is a system that allows you to run applications kind of containerized in a virtual environment. uh So it means that you can get it all set up nice and quickly.

A

So this uh this repository here has the instructions, and this is what we're going to go through to get this set up. So the first thing we need to do is clone this repository here. So I can copy this and I've got a terminal here and I can paste that in um already exist. uh Let me just delete this.

A

um The clean yeah I was giving this demo just recently at consensus um in Austin Texas in the US. uh Hence already had it locally here.

A

um So there we go I've created that and I can then go into that directory and once in that directory, I could start everything up and so I give the command Docker compose.

A

Oh and it will start it all up now. The first time oh, cannot connect to the docker demon. Oh I, don't have Docker running. It seems that's because I just restarted my computer just before this. So let me just start Docker make sure docker's running um and uh oh uh close see. Hopefully this will start up. Okay right, okay, good! Let's, uh let's try this again, then so Docker compose up right. There we go now the first time you run this. It will have to go and fetch the docker image.

A

The docker image is about three gigabytes in size. So just be aware of that in terms of network bandwidth, um when you, when you run that um what this will do is it will start up five different containers. It'll start up a Lotus container, that's the node software that runs the filecoin node it'll start up Lotus Miner, that's the an instance of a minor or a storage provider.

A

It'll start up an instance of some software called boost, that is, software that is used to manage the storage provider, kind of business side of things and then two ancillary, uh two tools: booster HTTP and uh booster bit, swap that allow boost to kind of connect out uh with the with the outside world.

A

So um this is just starting up here and it'll. Take like I said it takes a while to download the image once it's got the image it starts up and it'll take about three minutes, or so this is running on a MacBook M1, like I said it should run on Windows Linux, um uh older MacBooks, whatever you've got available uh to run it now, what we're actually running here is, what's called a two kilobyte Network, so that means each sector, which is the unit of which storage providers operate, is two kilobytes in size.

A

uh What that actually means is the maximum size of a storage uh deal that we can create is also only two kilobytes, so bear that in mind is very small. The reason it's so small is. It means it's very quick to be able to actually demo this and get it all up and running on your laptop.

A

There is a file that you can edit um there's a DOT EnV file uh in that repository, and that contains you can comment out to create, what's called an eight Meg Network, and if you comment that line those lines out um there, then it will run an eight Meg Network. So I'll give you a bit more storage space. You can store up to eight Meg uh per item there, but it still should be small enough to run on.

A

You know most reasonably recent laptops, but when you start that up it'll download uh about seven gigabytes to download to get that up and running because what it has to do is download the initial. uh What are called proofs that are used or parameters that are used to create the initial Genesis Book.

A

So, let's see here, this is running up. Boost is starting up. um You'll see a couple of errors here you can just ignore those. For the moment what's happening is Boost is funding a bunch of wallets uh for the storage provider and it's checking to see whether those wallets have been funded yet and for a little while the wallets won't exist. So it'll take a little while, but once that's done, you'll see booster HTTP and booster bit swap started up and that's when we know it's working and we can actually see boost.

A

If we go back to our browser, we should be able to go to localhost 8080, and you should have boost up and running. Like I said this is the software that is used for storage providers to manage um their kind of network their side of process. What I'm going to do is I'm going to go into settings and I'm going to just set the price down to zero, which just means that we're not going to charge for storage deals.

A

It just makes this uh demo just a little bit easier, um but otherwise we've got that going so and we've got a brand new network. So what are we going to do with that? Network? Well, the first thing we're going to do on that network is we're going to deploy a storage provider smart contract. Now there is again in the filecoin project.

A

There is a repository fevm, hard hat kit um created by my colleague Zach, and that is a um a starter kit that has a number of contracts in including What's called the deal client contract, and so this deal client contract will let you um uh act as a client, but from a smart contract right. So it'll allow a smart contract to actually initiate a storage deal.

A

So uh let's do that then I think I already have it um down here. Let me just see here: yeah I've got it here, fevm starter kit, so what I need to do is I need to deploy that and again I. Think I've got the command in my history here. Deploy yes, so yarn, hard hat, deploy and I tell it that the local net, the network we want is localnet, so they'll configure the local net. Now that has failed now.

A

The reason that has failed is because there is no um ah I haven't got my key set for my wallet and I haven't given my wallet any funds. Yet so I've got a metamask wallet set up here. If I go here and click on metamask you'll see I've got a wallet here. I haven't logged into this. Yet so let me just wait for that. I think I need to wait for it to time out before it. Let me uh connect again, although it should connect to local network. Since we have this up and running now,.

A

um Switch networks- let me just switch so I can log in and notice. We begin to Parkland local net right there. We go Falcon local net and we have zero funds. Zero, T Phil, now Sarah showed you about on the hyperspace test net, how to get the faucet uh to deliver some funds, but as we're running this locally, we've got our own local network, so we can't rely on a faucet, so we need some other way to deploy funds to our wallet.

A

So what I'm going to do is I'm going to copy this address here and I'm, going to tell um I'm going to tell it to deploy um some funds to our net to our wallet. So what I can do here is within the um where the docker image is I can say. Docker compose exec, just tell me to execute a command I want to execute it on the Lotus, node and I want to run the Quran Lotus evm stat and give it that address and that's going to give me the corresponding uh F4.

A

In this case, we want to test Network T4 address that is used by filecoin right. So Sarah mentioned that there's this address class F4 address class and that can directly map to an ethereum address, which is one that starts 0x, so this t410 address and this 0xcb address they're both the same address they're just represented in a different way, and you can convert backwards and forwards between them and using this command.

A

We we just did that now that I've got that I can then now tell Lotus to send some funds to that wallet um and I'm going to send 777.

A

um T-Fill to that wallet and let's come back with a transaction ID and what we'll see is if we go back to our metamask, we'll see that come up with a um with some funds there now the reason the deploy didn't work is because I haven't actually set. My uh private key, so I need to set that now. This is just a test address, so it doesn't matter too much that I display this here generally, you don't want to display this on a live stream, but I can get My. Account Details get my private key.

A

um I can copy that and I can go back here and I can say um where we were here: export private key equals, and it's in and now once we've got some funds actually able to work. So, let's just double check and see. Yep we've got our funds. We've got 777 t-fill that has just arrived in our wallet, uh so we've got some funds to deploy a smart contract now, so I can rerun that deploy command and it should go ahead and deploy. So it is deploying that to again to our local network.

A

So whilst uh whilst that is doing that, um let's see here, what else can we do here?

A

um Oh, yes, we're going to need something to upload to the network, so I'm just going to create a a simple uh file here um and I'm just going to call that hello, uh hackers txt um welcome to as well welcome to the date of verse.

A

Marathon now remember, we can only store up to two kilobytes, so just a small text file is good. Now we need some way to prepare that text file to upload. So when you upload something to Falcon, when you create a storage deal, you need what's called a car file car file. Think of it a bit like a zip file. It's an archive of that file. So we need to create that car file and then we need some um uh parameters from that file.

A

Now, there's an easy way to actually do that that has been created by our friends at Lighthouse and nandit from Lighthouse is going to be on uh shortly as well to give a demo some of the stuff that they do so I can go to data.lighthouse.storage and I can upload my file to there. So let me just wait for that to load up login with GitHub, so once that's logged in there we go. I can upload my new file um and it is called hellohackers.txt.

A

So I've uploaded that- and we can see here this hello, hackers, txt and we've got some parameters here- that we need so we've got What's called the pcid um that is the ID of um when you create when you upload something to filecoin. It has to be padded out to a power by two size. So in this case it's been padded out to 256 bytes.

A

Our original file is 233, it's padded out to 256., so we've got the PC ID, which is the CID of the file, including the padding and we've got the payload CID, which is the the CID of just the original file there. So um I can plug those in the next thing. What we want to do is execute a command to invoke our smart contract.

A

Now I actually have the details of that command up here in a file here, um and what we're going to do is we're going to plug some of those details into into that file here. So uh we deployed our smart contract. We need the smart contract um ID here, so that's it there. So I can copy that and paste that in.

A

I need the piece CID.

A

A

Hit the payload CID that goes in the field called label.

A

We need to start in the ND pocket. This is when we want our storage deal to start and end. So it's on Epoch 2000. Remember: we've just started our new network, so you know we're just in the few epochs um Epoch in in this case is 15 seconds. So the block time is 15 seconds on this local network. Normally it's uh 30 seconds on the production Network and we need the size. It's 231 bytes, the peace size is 256 and we need the URL. So let's copy that URL.

A

So this is just telling this URL is just telling is going to tell boost where to actually get the data we want so 233 233 256 we've got. Let me just we'll check, that's out right, 233 yeah. We got that there so I can now copy. All of that and I can run it. So I'm going to go back to here, just clear.

A

The screen paste this in so I'm, going to run this command here, and so what this is doing is this is creating a deal proposal on the network and giving it the parameters of the file that we just uploaded to that data warehouse kind of staging site.

A

So that's uploading now and it says it's making a proposal on the network. We can actually look at our log files here and we'll actually see um we'll actually see this come by. We can actually look at the logs directly for Boost just itself. If we want so I can um create a new tab here and I can say: Docker compose logs boost F, and this will give us just the the log files for Boost and when a storage deal comes in, we should see that it's probably already happened. Actually um ah there we go.

A

There's the storage deal come in, it's recognized the storage deal, it's got the information here, so that should mean now. If we go and look at Boost in our web browser, we should have a storage deal here. Yes, we've got one storage deal um so that storage yield is now going to fetch data and it is fetching data from uh that temporary site on the data warehouse. So it's uh gone and gone and got that data there and it's ready to publish if we click on here it gives us some information about everything. That's happened.

A

um What it's done here! It's checked the cids um so comp here, that's the piece CID there. So it's checked all the cids check the sizes made sure everything is as expected and I can go to publish the deal now boost will publish it automatically in an hour's time. I want to publish it now, so I hit publish now, and it is now publishing it.

A

So it's waiting confirmation, so okay, Steph asks I tried using Lighthouse with a screenshot and it did not output either the piece size or pcid for me generate car work, though right. Okay, yes, there are some there's, there's some tooling uh some client tooling, you can use as well. Instead, there's this generate car command with a URL there that can work. I've also found out from Nanda as well. They've actually added some of this stuff into their Lighthouse CLI, so you can actually use that as well.

A

So this is awaiting publication, and so this is uh waiting here. Whilst that's waiting. What we're going to do is we're going to actually try and fetch this data back now so uh to fetch the data I'm going to use a tool called Lassie. Lassie is a tool that can fetch um uh data from ipfs or filecoin and uh I have Lassie installed on here. What I need is I need some information from boost. So uh where is the log files we have here um boost? Okay?

A

Let me just stop that I need from boost bit swap I need if I grab.

A

uh I think it's Port 888 I'll boost boost a bit swap.

A

There we go, um this has given us the details of where we can inquire to get our data back, because if we try and get our data back so normally you can say Lassie Fetch and you can give it a CID. So our Cid in this case is this deal root. Data CID I can copy that and I should just be able to say Lassie fetch that and Lassie will go away and fetch it.

A

Lassie uses a thing called the interplanetary network indexer to find the data, and the problem being is that you won't um Lassie won't be able to find that because it's running on my laptop. So what we need to do is we need to give it some providers and so we'd, say dash dash providers, and we can use the information from here. So uh we can use that localhost address there and we're connecting via peer-to-peer and then we need the peer ID, which is also there, which is there.

A

So there we go so Lassie providers we've, given it the address, it's a local host, 888 we've, given it the peer ID. So hopefully, now that should run and fetch the data for us uh nope. um What have we done here? Fetch or maybe I've got the things the wrong way: ah I've, not given an actual command uh providers uh fetch? Oh, maybe it's the other way around one. Second here, maybe it's Fetch and then I give it. The providers.

A

Just got those the wrong way around I think: let's try that try that last effect we've given a provider we've given that CID no a flag provided, but not defined providers. What am I doing here ah we've got a missing space. Let's try that again.

A

uh ah We've got the wrong peer, ID. Sorry, it looks like I copied the wrong peer ID from the uh from the file there, but it's told me which one so it's not about one ending UK. It is from here I just copied the wrong one from there and run that there we go. It's fetch the data and we have now a car file back again and I can run this command.

A

Ipfs car unpack and I can run that on that command on that car file and it will unpack it, and now we have a directory there. If I go into that directory, we will see hello, hackers.txt and there's our file. So that's gone right.

A

The way around and actually, if we go back and look at Boost now we'll probably see yeah, it's sealing the uh sealing the the sector, so boosters actually are star Miner, okay to start sealing the sector and that'll take a little while um and then once that's done, it will go into a stakehold proving which means that the miner has to every 24 hours, tell the network that it still has the data. So there we go. We've kind of Gone full circle around the storage deal process.

A

um Hopefully that was clear. I'll also put some slides. I've got some diagrams as well. I meant to bring up here, um but I didn't get a chance to grab them before the the talk, uh but I'll put those as well, probably in the uh Discord um there to help.

A

So uh any questions on that we're running just a little late the moment, but um hopefully um that is uh explanatory for people um people see here I'm just looking to see if there's any more questions coming up and then we will hopefully move on to our next guest uh Dawn says I'm good on time, thanks Dawn, um so I can't remember, I. Think the next guest is the next guest coming up now. Well, oh we're 30 minutes until the next guest.

A

Oh okay, that's great sorry, I uh I I got things uh uh wrong here. I thought we were going straight into the next guest, so um keep going, keep going, keep going. I've got 30 minutes, my goodness so I've gone through this thinking that uh I'm gonna gonna run out of time and uh the one time I do that and I've miscalculated the time um and we've got it here. Okay, um well, like I, said: if there's any, if there's any questions, um let me know I'm not sure if there are any so far.

A

Yet um if not, what we'll do is we will stop the stream and we will then restart the stream uh in about 25 minutes or so um uh we'll get in there. Oh uh figure out her Nikki has asked, can I dive into FAQs, so uh yeah um I'm not sure what if anybody has any particular um questions but just I think off my head in terms of frequently asked questions so.

A

That I get them at the most hackathons um yeah, so okay, well, so, first of all, uh Daniel is asking: is the session going to be available somewhere? Yes, it will be available back here. If you come back to twitch.tv, fill Builders you'll see it here, we'll also publish it on probably the filecoin YouTube channel as well, but it'll be here as well, so you can always come back. Twitch.Tv fill Builders and see it so uh tend to ask what type of file other than dot txt can be stored.

A

Anything can be stored um on there. The one caveat being like I said is that we're running a uh two kilobyte Network here, which means the maximum size. If I go into settings, I think it shows it here.

A

um The uh minimum piece size is 256 bytes, the maximum is two kilobytes, so the maximum I can store is two kilobytes. That's why I just did a text file because even kind of the smallest of of jpegs or something is going to be bigger than two kilobytes. That's just on this local setup. So that's nice and fast to run on a laptop on the main Network. The sector sizes are 32 gigabytes um in size and to put that in context, the sealing process takes about four hours.

A

For you know a pretty big beefy server with um you know, gpus and everything takes about four hours to seal. So the reason we've got it much smaller. Is uh it's much quicker to run here on this network, but, like I mentioned there is the uh this dot EnV file.

A

And if you comment out this eight Meg these two lines here, it will allow you to create an 8 Meg network instead, which will give you you know a bit more capability. You can store PDF files, images whatever up to eight Meg in size. Now, uh when you start that up it'll take about I, think it's about seven gigabytes, seven and a half gigabytes of data, it has to download um so again just be aware of of your connectivity when you do that.

A

um You know um just yeah so in terms of other questions, I'm trying to think what we get asked as well in hackathons, um so really the about the deal flow and about the way in which um storage deals are actually kind of processed and sent around. Let me just see if I can bring that diagram up.

A

Just give me a second while I just try and find my slides um on Google Drive here for something I was meant to show um one second here: I just need to I restarted my computer just before all of this, and it's logged me out of everything so, um but let me just see here: slides I'll bring these up uh uh yeah.

A

A

Let me try and uh one second, let me just try. Hopefully this doesn't cut us out again right there. We go okay, um you'll, see here, I think I have a diagram in here; hopefully, I've just realized. This is probably the wrong talk. I, don't think I do okay right so much for that one um one! Second! Here, let me just try again then um and see I did another talk.

A

Oh actually wait a minute. I know where this diagram actually is one second I know if we go to on Boost. Actually, if you go to boost.filecoin.io and I think it comes under tutorials, is it uh uh no getting start architecture here we go. This is a diagram I wanted to show right uh so just to show the Flow, by which a storage deal the way in which it goes on. So we've got client we've got web server. In this case we used uh lighthouse's uh service, but you could run a web server locally.

A

We have Boost, which again, we've got running locally and the blockchain itself, which again we've got running locally, so the client boost and the and the actual blockchain itself. We have all running locally in this example here, so uh the client adds funds to the chain. uh So that's those funds going escrow uh to pay for the the storage deal they're going to create uh they upload the car file to the web server. That's what we did with uh lighthouse. We created a text file.

A

We uploaded that and and lighthouse's data warehouse created that car file for us. We proposed a deal. We proposed that using the client contract uh contract itself, so that was when we ran that command here.

A

um I, don't think I've got it still up here have I, yes, I do so when I ran this command here, this was making a storage deal proposal, so this was calling our smart contract and our smart contract in turn contacts boost and gives it the details of the of the storage deal boost checks.

A

The funds accepts the deal downloads, the file so boost, went and downloaded the file from um lighthouse's data warehouse, and then we clicked publish the deal, so it then gets published to the network, and then our client contract can actually check and see whether the deal is published and there is actually a command. That's just been added I, don't think I've got it on the copy I've got here, but there is a command. That's just been added to the um starter kit that allows you to check the status of a deal.

A

So if you give it a deal, ID and I think a PC ID, possibly it might need it will actually go and check and see whether that deal is published. But we can actually see it here with the Boost yeah, so we go seal proving so that means now that the the the deal is is published and that it is active and the network is continually every 24 hours having to prove that data. Is there so there we go I think that's uh probably it I think that's most.

A

What I can think of in terms of FAQs? If there's any.