Filecoin Filecoin Liftoff - the Main Event, 24 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How 5mb Was Built

Description

This talk will be about a tool we built for simplifying importing of large directories, such as package archives, into Filecoin and for retrieving and serving them.

Join us for Filecoin Liftoff Week, an action-packed series of talks, workshops, and panels curated by the web3 community to celebrate the Filecoin mainnet launch and chart the network’s future. https://liftoff.filecoin.io/

Events take place all week, October 19-23, 2020. #FilecoinLiftoff

For more information on Filecoin
- visit the project website: https://filecoin.io/
- or follow Filecoin on Twitter: https://twitter.com/Filecoin

Get Filecoin community news and announcements in your inbox, monthly: http://eepurl.com/gbfn1n

Chapters
00:00 OB1: Tools for IPFS and Filecoin
02:10 Why build on Filecoin?
03:15 Walking through the stages
07:10 What's Next?

A

uh Let's just pull up the slides, as you said, it's I've been um I've been working on, wait. You know with ipfs. It feels like almost from the beginning. Now and now you know kind of be interesting to be working with filecoin, also in its infancy and stuff like this.

A

So um this is talk is just gonna, be a little tool that um me and uh brian built um here at uh our company, ob1 um brian came up with this 5mb name, you know, naming stuff is always like real difficult, so I just I always refer because this is dealing with packages. I was just referring to it as a amzm on the command line or mzn, um but uh so brian calls it 5mb.

A

um So we'll just talk a little bit about you know who we are open, uh ob1. You know, we've built uh quite a few things um open bazaar, which is a desktop app. We build haven um which is a ios and android app. um Both of these have been using open bazaar really from um very early on. So this is ipfs on on mobile for the haven um and then a bunch of other stuff, with it too, um like our search engine and uh some social stuff in haven and all of this.

A

So uh problem uh that would that were um this tool is kind of intended to solve. This is a problem that um you know, other people might have too, but it's also we're working on something kind of a little bit beyond just this tool that we're going to be utilizing this tool, for that, has um you know a similar problem. You know you have package uh repositories can be massive.

A

You know very large data sets that we want to store and we want to make sure we can do that kind of efficiently and we have tools to kind of you know, pull out the certain parts of the data that we when we need it, but not necessarily have to retrieve like the whole thing. um You know why build on filecoin. Obviously you know we love you guys and they have a really good reputation um in the community.

A

um uh I mean that's a big part of it um and, of course, for you know our purpose here, we'd like to have ways of incentivizing uh data, storage, uh infrastructure and sort of a sustainable way, which of course is what filecoin attempts to do and so we're um you know, hope we can build some tools that uh can help people out in doing this. um So this is uh 5mb. It's a little project uh that we've put together.

A

It's a tool um for basically managing like really large um data sets and getting them into file coin and helping retrieve them and kind of go over. How how we're doing that. So um you know we've got, we've got this command line tool, there's the amz I named it. um You know, there's a web server involved, we're using powergate and lotus. There's a database involved ipfs this sort of stuff.

A

um So basically, first thing um we're doing is when we we have a we're gonna, essentially like r syncing, a large package archive um and we're taking it. This could be like you know many many gigabytes of data.

A

For example, we've got like the ubuntu archive, and things like this that that it's basically a very large directory and what we're doing is we're kind of splitting it up um into like different sort of buckets and segments and we're kind of ingesting that um each of these, uh like buckets, are being ingested into while we do it into powergate, so it's they're being like staged in ip ipfs and like a hot storage and then they're also going to like long-term storage um in in filecoin, and um then the tool will also manage um like making the deals in filecoin monitoring them dealing with any uh errors that are handled um and making sure that you know everything ends up.

A

You know fully staged there in filecoin and it saves all this information in our database like a mapping between um like the path of the files that we're looking that we're interested in and like what deal into filecoin it went to so that we can um ultimately retrieve it later. If we know the file path and then the last part is just um retrieval.

A

So what we've done here- and this doesn't have to be a um use, the file coin or the ipfs ui, but ipfs has this nice little web ui for browsing through directories, and so we've kind of modified this, so that when you start clicking through different directories and subdirectories and files, it is doing a check to see like do.

A

We have a cache copy in like hot storage and ipfs, and if not, then it's going and it's setting up, um uh you know connecting to file coin network in order to try and retrieve it, and so it's notifying the user that, like hey, we're pulling this from like long-term storage. So you can come back later um and this file will be available for you. So we can like look up the mapping between the the path that they're looking at and retrieving it from.

A

uh You know the individual kind of bucket or slice that has the file that we're interested in from the file coin network. um You know challenges, um you know, we've got you know, file coin, being a new network, um you know, especially working with test net, was maybe a little bit challenging every time it. uh You know there was an upgrade and stuff like this um user experience. uh One thing that we would like to improve on is the ability to uh provide sort of more transparency into the the user.

A

As um you know, like the status of retrieving that file like how long they can expect it to be. uh You know when they should check back like right now or just have a static page, that it just shows and says: hey we're retrieving this. You can check back later and and we'll get it so um upgrading that user experience is good too and then you know still a bit of a steep learning curve.

A

I think you know a lot of the tools and stuff are still kind of raw and kind of learning how filecoin works under the hood is a little challenging too like as someone I'm, you know, I'm um I'm like a bitcoin cash developer as well sort of in my spare time. So I work on you know full node software and I'm pretty technical in that regard, and it's still even for me, like pretty challenging to just kind of understand what the heck is going on under the hood of filecoin.

A

So there is that kind of uh you know steep learning curve there in terms of it's a pretty unique blockchain architecture that you don't really see elsewhere. um So what's next, um you know we.

A

We are going to be using this tool in like an application that we're building um that's going to provide sort of like a little almost like patreon thing for um storing packages where people can um or not packages, but files and directories, and things like this, um where people can um you know like subscribe to different packages and and uh incentivize storing them and paying to have them stored and stuff like this, so the tool this tool would is something that we're going to use in that, and maybe hopefully, it'll be useful for other people too.

A

If they want to use it, um I think that's it so.

B

Someone asked if you have an api that they can test.

A

um Right now it's a command line tool, um but I can we can show you how to use that for sure too, we'll we'll publish something that may be on our website about it too on how to use it.

B

Well, thanks a lot chris. This is fantastic, uh I'm sure. If people have questions they'll reach out on slack, which I know you're very active on or check out your website for some documentation thanks chris.