IPFS Tech Demos, 6 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: sqlotus preview

Description

Uploading this mini version here because one can speed up on youtube. DrRobinson - let's trash this when the better versions come.

A

Yeah uh cool uh next up, we have peter reba uh talking about uh sequel, lotus, uh I'm giving us a preview. uh This is what will be the back end of insta lotus, which uh maybe you can provide a bit more context on, but also I recommend, reading the proposal for it, because it's really good uh hey, we will go off.

B

Yes, let me see if my screen share shares.

B

Everybody can see a terminal right, yep, perfect, okay, cool, so uh this uh journey actually started like way back in december, when I needed some sort of very up-to-date, essentially a real-time uh sentinel, so to speak, and we didn't have this at the time. So I started digging into things, and this is a combination of about like one month of work. You know spread out over time, so uh I will just walk through what you see on the screen and it will become obvious like what this is all about until later.

B

So what you see right now is uh a lot of demon actually running. uh You know it is sync to chain it's like five, five, two, nine eight. If you go to fuel fox, you can see that it is all well, it's 297 because you know fuel fox itself is lagging behind a little bit, but you know everything works correctly and uh it will at right now it's 30 second mark. It goes to 99 so and so forth. Now what is interesting here? This is an entire full node.

B

But uh if you look at our uh free space, uh we basically take only 7.9 gigs uh and um no tricks. They're, like no uh no disks or anything like this attached to this thing, uh in fact, if we go and look in the lotus folder and look in the data store, there is no chain uh directory at all. This entire thing works fully under network, taking a completely remote block store and using it. You know, as we go.

B

One of the advantages of not having a badger in store. This is a full node that you know that takes like no uh amount of ram at all it just you know, sits there and runs and does its thing and on the server side uh to which I just switched.

B

We have uh basically just post wrestling there and it uses also, essentially no memory at all now uh made the um the demo gods be kind. What I have here on this box is another lotus which once again, is the same database, but with its own state and everything, so it shares the entire block store with the one that is running here.

B

That is, you know, uh continues to operate, but it has been stopped quite some time ago, so we are now at 555.302 and I shut down like a 214., so I'm going to put it back up and it's going to start working and uh you may notice it starts instantly. There is no, like you know, time to open badge or anything like this. It's like there right away and uh right now it starts from. uh I did not resize something here.

B

One moment we want to see just the top oops, just the top two yeah, so it's starting from you know five five, one three, where I pretty much shut it down and uh what we're going to do is while it's been down, the other node has been constantly populating the block store, so we're just going to jump forward without uh you know having to do anything else and let's see if this works, uh let's see if you, where are we now?

B

uh Let's take this too guys. So, ideally, if I do this- and if I do this.

B

It should- and it just took it so we're now uh four uh blocks away from being synced, and uh if we do this in about in about uh like this, current epoch is weird, ah I remember we actually need to restart photos after we do set head because it will not interrupt the thing that is going right now. So now, if we start it, it is going to be at the right place. Yep there.

B

It is uh and now to basically say that it needs to do four and it will be ready to go, uh and that is almost always the insta lotus uh proposal that I that I wrote, although uh I still have like uh stuff on disk and so on and so forth.

B

So it's not fully fully in place, uh but that is the part that basically would allow somebody to have a replica of you know of a database like this and just stream it to something like uh will's graphql interface, which here we literally don't run anything aside from a postgres, uh a replica that is streaming.

B

uh You know all the changes from this uh main note in and just here, stateq server, nothing else, and this thing will tell us that we're now at 300, where is which is where we left uh this uh node at and if we refresh now, we should see once it gets to 302. The graphql is one block behind always so once this goes to 302 the graphql will, uh you know, do things for us uh as expected and yeah.

B

So this entire, uh like contraption, is basically to allow places like euro, for example, to run a bunch of nodes out of the same data store which, because this is uh like a grown up database, so to speak. It is designed for multiple reads over few rights and it is extremely optimized for huge data stores. So we can like throw all our three terabytes of data into this thing and it doesn't bad at night. It just continues continuous working uh and yeah. Now one more thing that I want to like go over here, real quick.

B

uh If you see here, we scroll that it took about nine or seven seconds to validate the tip set. The reason this happens is because, uh in addition to uh keeping track of all the blocks that go in, we also keep track of the chain. So we actually keep track of orphans and stuff like that within this very same store, so that you can stream out the replica, and somebody by just looking at the database knows exactly where the head is what orphans are like.

B

uh You know what the chain looks like from the perspective of this particular instance that has gone into uh right into this database and each individual instance, like both this instance and this instance arrived to separate places in the database. They have their own instance name spaces so that you potentially could run.

B

uh I don't know, like 20 notes that all right to the same place- and you will see all of them- how are they seeing orphans, how the chain evolves from their point of view, aggregate in the same place, and the other thing that we do here is that every individual block that is sent into this block store is actually parsed on the fly and its entire ancestry is being recorded in the database as well.

B

So you can actually do graph walks through the chain or through individual blocks or whatever, within false graphs, with just recursive queries and nothing else, and uh this actually has implications also like uh if we decide to reuse this, for something like ipfs.

B

There is a way to retrofit this for garbage collection, because then you have a reverse index of I'm a blog and who actually is mentioning me and if nobody is mentioning me, I'm just going to delete myself and then you do the scan again and you basically go level down a little down level down, at which point you basically will have garbage collected here in the star store without any locks without any without anything uh and yeah. All of this is already on uh github.

B

uh I need to grab somebody from lotus to basically figure out how how to massage some of this stuff in uh will has already written a uh front end for this, so that tommy is trying to like get um uh sentinel to run off of this up-to-date data store and um yeah. That's it any questions.

A

uh I think questions in chat have already been answered. This is fantastic work. um uh How many of our project partners are already aware of this, because a few of them will like this eh.

B

Yeah, so uh very little actually uh yeah uh it. It wasn't clear until the very end that it's all going to work and perform correctly, but yeah I mean clearly it does so yeah. We can start like talking about how to productionalize it and stuff like that.

A

uh Yeah, I think uh I think that will be important to do yeah a lot of people will probably want it uh and if, if it's working well, I think I think we have something great here.

A

uh The pm team will probably be really interested in knowing about this as well.

A

If they aren't already.

C

It feels like it's definitely like you know. This is this is big news, capital b, capital n, so it goes goes to all the peeps and then eventually blog posts and videos and everywhere. So people are aware of the new options for them to do the things better. I love it. Thank you very cool.

A

uh All right uh last call for questions. I don't. I hope I didn't miss it just on me yell, if you have any questions.