Filecoin Bacalhau - Compute over Data, 5 Sep 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Open Data Hack Powered by Filecoin: Using Lilypad for Decentralised AI and ML

Description

On Tuesday, 5th September 2023, we were delighted to host the 'Using Lilypad for Decentralised AI and ML' workshop for the Open Data Hack powered by Filecoin! A big thank you goes out to Ally Haire from Lilypad for holding the workshop!

Ready to build? Check out our Hacker Pack on Notion for all the resources, workshop dates and bounties:
https://encodeclub.notion.site/Open-D...

Open Data Hack powered by Filecoin:
Join the Open Data Hack Powered by Filecoin and showcase your skills! This three-week online event provides opportunities for developers to improve their skills and work on projects in the Filecoin ecosystem. Technical workshops and mentoring sessions are available for all levels of experience. Complete challenges and compete for prizes!

Useful Resources:
Questions zone: https://discord.com/channels/70579992...

Join the Community
🐥 Encode Twitter: https://twitter.com/encodeclub
🗣 Encode Discord: https://discord.gg/encodeclub

About Encode Club
Encode Club is a web3 education community learning and building together through fantastic programmes with the leading protocols in the space. Together we organise programmes like education series, bootcamps, hackathons, and accelerators both online and in person. We then help our community get jobs and funding for projects and startups. Read more here: https://www.encode.club/

About Filecoin
Filecoin is the L1 blockchain and storage network poised to power the open data economy. The Filecoin Virtual Machine brings programmability to Filecoin and has gained traction with thousands of developers who are building exciting new projects and companies!

Timestamps:
00:00:00 - Introduction
00:01:20 - Lilypad Vision
00:03:00 - Lilypad Technicals
00:05:00 - Lilypad Applications
00:06:24 - Getting started with Lilypad
00:26:00 - Ideas for Hack
00:27:00 - Community and Resources

A

Alrighty cool well, thank you so much Rory for the introduction there and hello everyone so good to see so many people in the chat today for open data hack, uh which I'm super excited about so um I will just sort myself out a bit, um and here we go right. Here's my slides, okay, so I'm, Ali and I'm super excited today to be sharing. What I think is a really valuable and I believe super impactful project that I've been working on for the past several months called lilypad.

A

um So today's agenda first, what's Lily Pad? What is our vision for Lily, Pad um and I'll quickly chat about some technicals uh and why? Maybe, before diving straight into a live demo of how you can use lilypad to run Ai and ml jobs, which um you know is what I'm more excited about? And it's not just you know, hype but anyway.

A

Oh moving on I then want to chat about some potential applications of lilypad, and maybe some project ideas for the hack as well and by the way I'm happy to take questions throughout the presentation or I can answer them at the end. I may not see them so just in case I, don't see the your question. uh Please ask me again at the end as well, try and save some time for that.

A

uh Firstly, though, what is this Lily Pad thing? uh Well, let's head on over to governing France? No, no wait! That's not right! We're not going to Monet's Castle um with Lily Pad. What we want to do is we want to build a trustless distributed, compute Network that enables truly internet scale, data processing, AI, ML and other arbitrary computation to be callable from Smart contracts, with those amazing blockchain guarantees that we get so as well as like being able to do these uh other tree computation jobs such as Ai and ml, inference and training.

A

We want to be able to unleash the idle processing power uh in both the filecoin network and in like a lot of uh crypto networks to enable this new wave of data and compute economies in web 3..

A

So we believe projects like lilypad are going to be essential infrastructure for web3 and the next Frontier in development and I am a bit biased, but I really think this is true not just because you can run Ai and ML on them, though that's a cool reason, but also because computation is really one of the Three core components of the modern internet and so for a truly internet scale, decentralized, internet uh we're going to need a decentralized compute platform uh that matches this kind of scale, and it's going to be necessary for web3 to be able to create these, like scalable decentralized applications that can take over the world, which is what you know, we kind of hope.

A

We all hope for. So we can get rid of some of this. uh You know decentralized problems, we're getting with traditional technology, um so much like filecoin, uh so lilypad was into the technical sorry lilypad was built uh leveraging some of the key components of the back. We are project for those of you might have heard of it. It's um an arbitrary computation project, which was uh spearheaded by David aronchick, um and it still exists as an open source project. We actually live in the back of our slack now as well.

A

So what we did, though, is we want to combine uh this open source peer-to-peer background, Network nodes uh with on-chain guarantees uh and crypto economic incentive models, so kind of like filecoin is to ipfs lilypad is to bathia it's bringing these incentives and guarantees to the network um So. Currently we have a a tester which is awesome. uh It's called the lilypad Lala Chooser test net uh I think that's an owl, but someone's probably going to connect correct me when I looked at Lalo cheeser.

A

Actually it was funny how um chief engineer called it la la Tuesday. Apparently it's a kids kids program in uh the UK and I. Looked it up and it was a story about a witch um who haunts uh people in you know who had died and is now haunting children, so I, don't know which one it is.

A

But anyway, this is what we've called It's called The Lola choose a test net, it's built on, uh go ethereum and it allows for arbitrary untrusted nodes to join the networks and for users to run compute jobs via these Lily Pad modules. So we've Loosely, based this test net consensus on a paper called modicum as well, which uses a mediator approach, and you can kind of read more about this in our um docs.

A

If you're interested in in the game theory behind it, uh we've got a light paper in our docs as well: research documentation in there that that really delves into the game. Theory behind what we're doing, if that's of interest to you.

A

um So the exciting thing, though, is that lilypad can be run from both a command line interface and also from a smart contract.

A

um The ability to perform these off-chain decentralized compute over data from Smart contracts really opens the door to a multitude of possible applications, as I was kind of inferring a little bit earlier in the talk as well and both for web free and for traditional technology I think so we believe projects like this are essential architecture, and can you know unlock things like inference, Ai and ml training jobs, which is what we have started out with, uh because you know they're fun and also you know, there's so much you can do with them as well.

A

So um they're really um kind of changing how we operate with things, so we can invoke and support generic ZK computations, which is definitely of interest in in the current uh web. 3 space I know, there's a there's a lot of um stuff happening around ZK um compute networks as well um so cross chain interoperability uh to complement Bridge protocols is possible. We could utilize inbuilt storage on ipfs, there's Federated learning consensus that could be done eventually. Iot and sensor. Data Integrations, for example, I'd love to see whether uh XM.

A

If any of you have heard of that, it's kind of a a weather station you can set up anyway a really cool project. I I encourage you to look that up.

A

um It could be helpful for their data, uh so it could provide a platform for digital twins, supply chain, tracking and Analysis and ETL and data prep jobs as well.

A

um But let's get started with lilypad: let's go to the fun part so to get started with the lily pad. All you really need to do um is to add the lily pad Network the lullaby Network to metamask.

A

Now uh We've named our currency Lilith, because some of you might get the joke of when I looked up, La lechuza there uh I decided to call the symbol Lilith and just stick with the whole theme anyway.

B

A

Easy uh anyone that's played in the web 3 space probably knows how to add a new network to lilypad. So there it is.

A

um The next thing you can you need to do is get some test net funds, so you just go to testnet.lillypad network I've got a typo there, sorry about that, lilypad Network it is.org and uh get some test net funds for your wallet.

A

Now, let's have a look at playing around with the clear. So akali is a command line interface, which looks exactly like this. Here's what I prepared earlier just for you guys. um So this is the clay and it's really simple uh to download the lily pad interface. You can just run this curl command. One line of code and you'll have lily pad up and running in your terminal, um which is pretty cool. uh The next thing you do need to do, though, is you because Lily Pad does run on a blockchain network.

A

You are going to give you need to give this a local command line, your uh wallet, private key and so that it can approve transactions for you. uh So I've already done that in my clear interface here, so I'm not going to do that again and by the way like that, doesn't get displayed to anyone else. No one else can see your private key You're, simply running it on your local network, so uh you're not giving it away to anyone else.

A

Now, we've decided that our first module in lilypad and by the way modules on lilypad are kind of job specs. So each module runs a deterministic lily pad job and they're deterministic, because in a compute Network like this I, you need to be able to prove that the nodes, for example the supply side nodes, are doing their job right and not just saying yes, we've run the job.

A

Please give me the money for running that job, um so each module in Lily Pad is therefore currently deterministic, because if they get spot checked, if someone goes I, don't think that's the right. uh You know I, don't think that's the uh right, compute answer. We need to be able to check that and we do that.

A

uh Determinism, just in case um you're interested in Lily Pad, means that the output hash, so the CID output hash is the same given the same inputs for a job, so it doesn't matter which way you get to that output hash. As long as the output hash is the same.

A

um So that was a little background on determinism in lilypad as well. So anyway, our first module that we decided to do is uh the Run hello, cow, World module. uh So maybe what should we ask? What should we ask this to do um so? Pretty easy I've already installed: lilypad uh I'm, gonna yeah. There we go. That's actually a good one to say: woolly pad, run cow, say Vision 0.0.1, just in case we decide to update these in Futures and I'm, going to run a hello, open data hackers.

A

Now this will uh submit this job to the lilypad network, so I loaded, the private key uh registered a drug Creator. It added a mediator because each job is going to have a mediator in case this job um gets checked. So a mediator is like a ticket inspector. Basically, um the.

B

Chance is just sorry just to let you know, we can only see your slide. I, don't know if you're trying.

A

A difference, thank you for saying that um I will. Let's, yes, I will happy to share all the resources after this as well. Let me see if I can?

A

Okay, huh okay, maybe I needed to go with the entire screen. Thanks for thanks for letting me know that there you go well, that's a lot more helpful, uh sorry guys! So, basically, I've run the first job, which is this Cal State Job our Engineers have a sense of humor.

A

Clearly this was our first module, so Lily, Pad, run cow, say I've, given it the prompt hello, open data, hackers uh and you'll see that it fetches these results, and these results are an ipfs CID, basically, which you can then open up in your terminal, which I've done here to be like I can verify that this is the same thing by typing this in for you one second, because they're deterministic it'll be exactly the same, but I will I will show you that working now this is brave browser which means I can just do ipfash uh colon dash dash- uh and this end CID here and uh sorry and brave browser automatically knows that this is an ipff CID, which is another cool thing about Brave browser, if you're not already using it um so and then I can go to stand it out here, and here we are hello, open data hackers, that's the that's the result of that job.

A

B

A

Go um cool righto, let's see if my screen yay it moves. Now awesome um Cassie I made us if you want to have a look at the code uh behind this module. I'd recommend having a look at the docs I haven't, got it in here, uh because it's quite long, but the docs do have the entire module in there and also resources on how to make your own modules. If you wanted to make your own module, um so you can definitely go ahead and have a look at that in the docs all right.

A

What's what's next um Now cal state is fun and all, but it's probably not really gonna do much for you, I guess you could use it for slides, occasionally, I, don't I'm, not even sure what you would use it for. To be honest, um it's just a bit of fun proves the network is working, oh by the way, of course, because this is a two-sided Marketplace. You can also run your own node and get paid for jobs. Currently you're only going to get paid Lilith.

A

But you know, if you want to run your own nodes, you can do that. You can also run your own mediator, so if you don't trust our mediator, which is what's uh getting used by default fault in this network, you can go ahead and run your own media there as well all right. There is, of course, more Lily Pad modules, so one of the cool ones is, of course, stable, diffusion and we're running sdxl uh 0.9 here.

A

um So let's go ahead and run this uh sdxl uh actually uh does someone want to give me a prompt for what I should put in what image should we make for for this one? What do you reckon Rory got any ideas on the text to image generator.

B

I see some recommendations in the chat, so I see some people typing.

A

Alley hole in the world on her shoulders, I, don't know if it's gonna know who I am so, maybe we'll go with our Turtle holding the world on its shoulder. Let's go with that I. Don't know why I turn off, but there you go thanks for the prompt and uh whoops I haven't finished the quote or something there. So I'm just gonna go ahead and run a cute cat would have been good as well next time next time or you can try it yourself actually now that you know how to run it.

A

um So I do have a few from earlier. I was playing around with it an astronaut riding a unicorn uh and here's an example of the output. From that astronaut riding a unicorn. uh You have to change the seed if you want to get a different image, but because, like I said before, it's deterministic, um so if you only use it it's using gpus in the background, uh so the I idea is that uh so Ally just asked the question: can you use GPU or CPU?

A

Only um it does need gpus for stable diffusion uh images. um So let me just while that's running carry oh here. We go it's ready beautiful now. Sometimes ipfs does take a while to propagate, so you might have to wait a little bit for this image to actually come up, but let's take a look yeah, oh, and that was pretty quick. Actually, uh so it will come up in the outputs folder here image: zero. Let's see what this Turtle holding the world or the shoulder.

A

Ah That's great I love it I'm gonna, see if I can post this CID the chat here. Actually so you all, could you know verify yourselves as well? There we go if I give the same prompt again tomorrow, you are jumping ahead. You know exactly where I'm going. uh If I gave it the exact same prompt because modules are deterministic, uh you will get the exact same image, so you could prove you could give a quiz for this and go.

A

um You know, go and run this prompt and you could prove if someone had actually done that, because you would get the same image back now. If you do want to get a different image back, because that's definitely something that you might want to do. You can add in the seed I'm just going here.

A

I might just clear this oops. Let me just clear the screen: it's getting a bit messy there, so here I'm, running willypad, sdxl, 0.9 and then I'm, giving as a Json input, The Prompt, uh a unicorn writing an astronaut or let's go a turtle holding the world or shoulders. Was it well a cute cat? Let's go cute: uh let's do a turtle hole in the shoulders.

A

Which is a long queue but great one um now I would have to give it a different seat and the default seed is zero. So if I give it a different seed, it can be any number and then you will get a different image here and again. I, don't know why I didn't finish the quotes or something, but there we go um so yeah I've. This is this: is free image generation with sdxl, oops I, don't know what happened there.

A

I must have put the wrong clearly oopsies, sorry, something something went wrong there with my input.

A

Cell 0.9 that doesn't look right, I think I've definitely done something wrong. uh Tell you what I'm just gonna see if I can? Let's have a look at what my slides say, because I actually do have this on the slide, so yeah same prompt same image, you've got to change the seeds, so run S6. Oh, it is brute version. 0.9, that's interesting, I feel like that's wrong.

A

No, that's correct! Okay!

A

um This should definitely work I. Don't know why it's not I think I've just got something wrong with my um quotes in here. Oops.

A

Maybe missing a quote: one of my quotes. Look: do they look? Alright, we'll! Try it again, prompt! Let's go with the Unicorn. Writing an astronaut, see what happens yeah that one that one went through better so anyway, we won't wait for that to finish it could take a while again, uh but yes, basically change the seed great question all right.

A

Moving on, where are we so yeah super fun uh by the way uh I've been using this for kind of we put out a Blog report every week on what we're up to with Lily Pad, which is that blog.lipadnetwork.org and I've been using uh this to generate our header image, which is great, so you know eventually I'm gonna see if I can I think uh stability AIS are releasing a new uh version of stable diffusion, which is going to include like the ability to add words to your image, and things like that.

A

So as soon as that's out, we'll definitely add that to Lily Pad and things like that as well. So it's going to get better and better I um I can't wait.

A

um So the next thing that we've done with Lily Pad is introduce fine tuning and really this is kind of a really really uh cool feature for Lily Pad, because not only are you kind of um you're training uh new models on the lilypad network? Basically, so for those of you that might not know what Laura fine-tuning is, you can have a look at the dock.

A

I've got a whole spell on what uh it basically does, but what it means is, instead of having to train a whole model on all these, you know hundreds of compute uh nodes and servers like Google or the you know, stability, AI or these big uh people producing these models need to do.

A

uh Instead, you can kind of just tweak the parameters a little bit by adding your own input data, and you can do this with llms as well, not just images um and it it comes out with a model attuned to your specific data set. So, for example, if I put a whole zip file of Claude 1A artworks uh into this, and by this is broken at the moment, I would love to show it to you, but um we broke it today. Of All Times uh and now we need to fix it.

A

So apologies for that I'll be running you the next day or two. uh The engineers are on it, um but you so you will be able to use this for for the rest of the hackathon. uh But you know this is really cool because you can just give it a ipfs CID of say a zip folder of Claude Vanessa artworks, uh which I did upload and zip somewhere. Actually I, don't know where it is now I've been playing with unicorns and astronauts too much, obviously anyway.

A

So there's it's in the docks uh you can play around with it yourself when this is up and running, uh and it will attune uh this uh stable diffusion model to Output images in the style of Claude, Monet artworks, basically, which is pretty awesome um and then so.

A

You'll use that fine-tuned bottle by uh then running lilypad again uh with Laura inference instead of Laura training and using the CID result from uh the one before with whatever prompt you want, and this should come out with an astronaut riding a unicorn, but in the style of Claude Monet, for example, which is pretty awesome and that'll be out soon. We've also got a filecoin data, prep module, so for those of you that might be deep deeply involved with the filecoin network. You might know how difficult it can be or not difficult uh challenging.

A

It can be to sometimes on board uh large amounts of data onto the network. So we created a filecoin data, prep module which basically um chunks data from an S3 bucket, an Amazon S3 bucket, and and turns it into a car which is what's needed to then upload it to um the filecoin network. So that's another module that we've got out there as well. uh Oh, so that's just trying from a clue. So obviously you can also use lily pad from a smart contract.

A

uh So I do have remix set up here and hopefully still running because sometimes remix likes to freeze on you and you just don't know when uh so, let's just check if it's running or not, which is sometimes hard to do, let's check with this one Yep looks like it's still all up and running great, so there is uh both an sdx caller client contract, an example client contract in the GitHub repo, and these are both in uh the docs as well.

A

So you can go ahead and just like click, a link in the doc, so we'll open it in remix for you and you can play around with it yourself, but you can call lilypad from a smart contract deployed on the Lala Chooser Network. So just make sure that you are on the lily pad Lola Chooser network uh and on remix you've got your injected provider metamask on uh to make sure that you're interacting with that Network. You can then go ahead and deploy this contract. So basically all it is, is it interacts?

A

I actually have some yeah, let's see yeah, so you connect your whoops. What happened there?

A

Sorry guys, so you connect your client contract, which is this contract that your smart contract that you're writing for whatever purpose you have for it to the lily pad contract, which is in this uh first, this it's called modicum contract at the moment, uh because we based our code base of a paper called modicum, and then this uh just connects to it. So here I pass in the modicum contract uh address, which is also in the docs, and it's also in this example client just to make it easier.

A

So when you're deploying it, you would just deploy it whoops. You would deploy this contract um with a Constructor of the modicum contract here so that you can connect to it. And then all you need is a received job results, uh function implemented in order to get the job results back from uh the modicum contract, and you can run your sdxl stable diffusion function to uh get get a stable infusion image from a prompt. So you can imagine.

A

If you had like a front-end app and just an input, then you could go ahead and hit this back-end client contract with run sdxl and return that image to the user. In your front end, for example, um so I can show you that working as well- let's just go to sdxx caller here um so well, I've got a prompt, more rainbow, unicorns and astronauts. So I'll just show you here that we're up to job ID, 330. I'm, going to transact with this one, because I haven't actually run this uh command before it.

A

You need to send it to uh Lilith. That's how much every job is on the lily pad Network at the moment, so go ahead and make sure that you have that in your value up here in remix to Ether, otherwise, you'll get an error saying that it didn't work uh and then we are just going to need to wait for those results to come back, so the transaction might have finished. But we need to wait for a node to run this job and then return those results.

A

So basically we're looking for this job result ID to change to a new number and.

B

A

B

Going on I hate to interrupt you, um we do just have a minute or two left just to.

A

Let you know: okay, no problem, no problem. Let me just quickly check out what I did have left on the slides and see if there's anything specifically, that people would need to know uh so running sdxl on yep, so hanging on Lily Pad, there's some I'll hand out these slides for later, because I'm obviously way over time. Sorry about that, um there's some great ideas in here on what you could use lily pad for um and we'd love to see.

A

um You know some more dapps that leverage some of these uh Ai and ml inference models that we're providing on Lily Pad. Are you who created JavaScript wrapper for the lily pad clear that would make it really easy for other devs in future and we'd love to see something like that. If you're interested in that kind of work, you can write your own Lily, Pad module they'll! Warn you that kind of is POD at the moment.

A

So uh just a warning that if you do go down that route, uh you might be up for some uh roadblocks along the way um and yeah. There's a project showcasing our docs as well. Have a look at that. If you do need some help, uh our main way of getting help is uh the backyard project slack. That's where we live, I, try and keep on top of the Discord, uh but I. Don't always pay attention to it. So um bitly slash back it up. Project slack you'll find us there as well um and I.

A

Think that's it for Me Maybe. Do you think we have 30 seconds for some questions Rory or we got someone else coming up.

B

um I can see our second speaker already in the lobby, so yeah. Unfortunately, I'm.

A

B

We don't have time, but.

A

Yeah I just I'd.

B

Say on behalf of everybody, thank you so much.

A

B

Thought the camera module was really cool, but I think with the other images as well so yeah. Thank you very much um for that and to all the participants. I'd encourage you to reach out on all of the platforms which Ali is sharing on the screen here and, of course, a recording will be available as well for you to recap. So one last time, thank you to to Ali and uh yeah. We look forward to seeing what everybody builds with Lily. Definitely.

A

B

Hacking, thank you, cheers.