Status Swarm Mini Summit, 23 Mar 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: [Swarm Mini Summit] Deep SQL and NoSQL Blockchains for Provable Database Storage

Description

Rodney Witcher - Wolk

Wolk has developed a set of code-complete deep blockchain services: Layer 3 SQL + NoSQL chains anchored in a Layer 2 Plasma Chain, itself anchored in a Layer 1 Ethereum stack. L2 Plasma Chain has evolving states of Plasma Tokens and its child chains’ latest roots stored in Sparse Merkle Trees (SMT). NoSQL Chains have evolving state roots of keys and values stored in a SMT, wherein values are chunk hashes kept in Cloudstore. SQL Chains have evolving state roots of table root hashes and owner root hashes, wherein root hashes model all key database structures (owner, database, table, index, record chunks); SQL transactions utilize a SQLite3 Virtual Machine to execute opcode instructions that interface with a Remote Cloudstore. The Remote Cloudstore receives both chunks and minted blocks from these L3 NoSQL/SQL chains via SWARM+PSS GetChunk/SetChunk and resource update interfaces, (that, in turn, interact with SWARM, a L2 Plasma Chain, or L1 Wolk Cloudstore, see below). All chains utilize abstracted consensus interfaces (RAFT, POA) to mint blocks. Upon minting each block, the “minter” sends Anchor Transactions to the L2 Plasma Chain, wherein the L2 Plasma Blocks record usage of storage and bandwidth along with Anchor Roots to L1. Critically, each SQL and NoSQL transaction submitted to a L3 results in a “Deep Merkle Proof” wherein provenance is derived from multiple SMT proofs: (1) a NoSQL key or SQL owner/table key stored in a L3 SMT (2) a L3 block hash stored in a L2 SMT (3) a L2 block hash stored in a L1 Smart Contract. Immutability is derived from L2 and L1 and censorship-resistance supervenes on the chosen remote cloudstore. Wolk shall present a working demonstration of our L3 and L2 deep blockchains starting with a Plasma deposit on Rinkeby and share current throughput/latency results.

Rodney Witcher, Mayumi Matsumoto, Alina Chu, Michael Chung, Bruce Han, Anandarup Ray, Yaron Asher, and Sourabh Niyogi (Wolk).

A

A

Database services and deep blockchains and I'll give a bit of an understanding of what we mean when we refer to those terms.

A

We talked about provable data services, so over the past year year and a half now we've been working on. How do we enable people similar to what eland mentioned? Build?

A

Databases builds apps that don't require the learning of all the solidity and all the other blockchain technology, but leverage the knowledge that many of us already have in building applications and also keep all of the same properties that we like about collections and what we really focus on is how do we make sure that whatever data is being stored, it is actually proof of provable that it's being stored and that over the last year has evolved into a certain kind of architecture that we'll go into that.

A

We think gives not only provable data services but also infinite version control. So, just like a blockchain where you can go back and look at the state of the universe over certain blocks or block numbers, we want to be able to show people almost like you view and github.

A

The version of your data at a certain point of time, and that leads us to what we call a deep blockchain deep block chains, are essentially layer, 2 and layer, 3 and layer in block chains built on top of each other, that we call deep smart proofs and smart stands for sparse Merkle, anchor root, trees and all of that, and so all of that really goes into the deep blockchain architecture that we think that allows you to store data on a blockchain or a database service.

A

That is, that is not D, centro de centralized, but can have proof ability of blocks as well as values all the way down to your layer. 1 in this case aetherium root, so try to. If we have time to give a brief demo and then also talk a little bit more about what we're calling our smart proofs and so like I just mentioned provable data services.

A

We also believe that the solution will be very high throughput and is blotching, but you'll also gain that same provenance that you're looking for as I mentioned earlier, infinite version control and then the deep blockchains concept. This pretty picture here talks about what we've come up with so you'll, see at the very top was what we call our layer, 3 chains and so later, 3 sits on top of a layer, 2 chain which we're using plasma, cache and then layer. 1 is etherium.

A

Give you time to take the pictures of our pretty picture, but well we're coming up with right now. In terms of demo, we have available no sequel trained in sequel chain and then we're gonna also have just a standard storage change. It says a as proof of concepts right now and what happens is that the layer 3 on the no sequel chain takes key value pairs, so you can have a key and just a simple value or some sort of JSON object.

A

Think of it as what you might do in Google's, big table or DB in sequel chain. You can run sequel queries, we've taken the sequel, Lite virtual machine and created what we call our walk virtual machine that takes the sequel statements and processes them and stores them based on the opcodes that we've developed specific to us, and so those acts essentially as block chains.

A

So over time, as you have certain sets of transactions that go through the layer, 3 block chain, those blocks for lack of a better word are submit to the plasma chain and we're calling those now anchor transactions. So that's where the anchor root piece comes in, and so we essentially saying layer. Three is its own chain. It can have whatever consensus algorithm it at once. It submits the anchor it anchors.

A

It connects itself to the plasma chain layer, two through a submit block or submit anchor transaction, that we've developed layer, two acts as a plasma chain, and so it's job is to essentially gather all of the chains, not just one well all of the chains that are connected to it and submit those to aetherium, and so, at the end of the day, if I say that I have a key that is Rodney's email and the value is rodney at wall comm you'll be able to have confidence that you can provably show that, after block one when it comes back at block one or block 100 or whatever block that transaction was submit, the value of Rodney's email was Rodney.

A

I walk now, I change it to say, Rodney, Widger, I, woke and that's block. You know, 1053, you will still be able to go back to say, 1052 and then 1054 later and see that my value on the layer 1 chain has been. Success is successfully stored. Where swarm comes into play. Is that all of this? We took the chunking process that is in swarm and just use it here.

A

As one of the data stores, we also have, because we're so keen on the prove ability piece, the ability for people to forage for data to be stored in a cloud store, whether it's warm or one of the standard services is one of the key innovations, and so my colleague sera will talk about that later and in a different, I think lightning talk about how we're able to prove that data on some other storage mechanism has has been actually approval, provably stored in whatever chunk story it is with swarm like we said, the hash shows that it's immutable.

A

You can show that the data that is stored is attached to the hash of it, but we develop them that of using proof using others, data storage layers in the backend to prove that so now, I'll just give reef overview again of what we're talking about so no sequel chain how it works. You take a set key operation and it just has a key value. All the keys and values stored in swarm chunks, the transactions are in a sparse Merkle tree. Are people familiar with sparse, Merkle trees?

A

Is it something that people know so I won't go into much detail here, but I have some other colleagues in the back that can answer the questions about it, but a standard Merkle tree is generally balanced in full.

A

A sparse Merkle tree just requires that if you have a depth of 64, it looks just like the binary tree, but some of the leaf nodes or many of the leaf nodes will be empty, but it allows for us to have proofs of an inclusion exclusion, along with combined with bloom filters to achieve that things are verifiable on the main net. We have a paper on our github.

A

It talks about deep block chains and has an implementation of this and go so we'd love to hear people's feedback on how they're able to potentially leverage or improve on what we've come up with here. But we think that it's a really important for the solutions that we're coming up with and it gives support for this proved ability that we talked about so the SMT route, just like any kind of Merkle tree route as data goes into it, the route changes, but it does. It supports the data provenance piece. Also I mentioned.

A

We have sequel chain sequel chain is a solution that takes a concept of owners, databases tables also with the indexes and the records of what you would think of as a standard database. You can have all of that in a way that works, just like I was just say, a database like sequel Lite, so we took sequel Lite, as I mentioned there, Virtual Machine took the opcodes and just made all the map to set Chunkin get Chunk. Essentially.

A

So if you look at where we have our set chunk, no sequel and sequel, both leverage set chunk can get you operations in the sequel chain case, though it's just a little bit more complicated in that you're running through actual what we call wvm opcodes to do those operations, and then you commit to a new state DB a route for each of four route of the account kept in the sparse Merkle tree and that really is used to support sequel transaction data provenance.

A

So you can run insert select, update, delete all the same same types of operations there.

A

This goes into the proving piece. I'm just gonna. Go ahead right now and try to do a quick demo.

A

Our github is know if you can see a github walk. That's last walk DB. This isn't yet public. We have a few changes that we want to make before we actually make it totally public, but what you end up doing is steps one through three is you just have docker and node installed, and a local webserver, and so with on my terminal here. I hope.

A

It's not showing up there right now, but it's got a web server running and it's got the code on it. You can clone a repository and then eventually you start up our server. So I have this running and I have the UI running, and so, when I go to sign up. So this is all running locally, so you're not connecting to anybody's.

A

You know, centralized server or anything you're running code and a package that runs on your your laptop or desktop, and so I'll give the blockchain a name and I'll just say: yomo I'll make a no sequel chain, give it a password, so there's crates, and so what create is doing right now, since we're using plasma and maybe you're familiar with plasma, not we're creating a new account, so this new account will be used to make a deposit on a theory.

A

Um's may not to get you a plasma token, and so the plasma token that you receive is what you intend. Chily will use to pay for the storage and the bandwidth of the database that we're building here, and so we have it hooked up to rank B. This new account is set up. We have a faucet that is giving it some s for you to use, and next you'll see this deposit happening on Rigby.

A

So am I going here so like I mentioned, we have a contract on Rigby right now that accepts some F provides you back with a token. That token is then used to set up and.

A

Initiate the blockchain- and so that's just talking to our plasma plasma operator that we have running our eventual goal- is to have that fully decentralized and allow for others to join in and become plasma operators or themselves, but for now it's just talking to ours. So now we have a token. We have a new address, that's created. Typically, you would just download the keystore like any other key store. We have an option where you can login.

A

If you have an existing account that you'd like to use, and now it comes to activate the blockchain, you see we've, given it an ID, that's associated with the name. These ports and files are for the docker file. We pass in to the docker file, where you want to run so if, by chance you have something running on these ports, you can change them.

A

This point. I already have the docker image on my box, so it's really fast, but it may take a little while to download it for you, and then you activate so now we can test it so you'll see here. This is a new wallet address that we created. This is a new token, and this is the balance now, if I want to say give something. So I was using this before so I say, let me say: I'll just say: Rodney email and then I me at walk, calm create.

A

So what this is doing, its submitting a transaction to the layer 3 chain, it's saying: do a set operation set key and I, don't know if the why it's taking so long Wi-Fi is jumping out on me, but it it's setting a just calling the set key operation and that set key is actually setting the value to Rodney I, walk.

A

Okay, so that one went through, and so what you'll see is that it started off at block 0. Now it's at block 1 and so that first block that it just did. It's sent an anchor transaction to plasma. So it says in this case I only have one transaction I'm a bill to block on my layer, 3 and I'm, going to submit that Blanc and say: hey plasma I have a new block.

A

Take it and then plasma in the backend is working with to collect all of the other transactions on whatever its time frame is for the plasma chain to then submit down to the main net. So now, I'll make another one and I'll just use my son and say he's got a job at 4 and create that's another one right, so we're actually doing blocks every couple of seconds like we're. Going really fast here and I can also go and change it to say. Rodney.

A

And then create so now, I have three transactions and you'll see it's going. 3 blocks right, so we could have multiple transactions going to be multiple going into one block, but for the purpose of this example,.

A

So now, if I go here, this is doing a get key, so this is getting is going to get if you leave this on our demo empty, it's going to the last block. So when we come here- and we say, I want to know the value you'll see as of the last block, this value is Rodney which I woke, and we have this proof that comes along with it.

A

So I don't have a demo of this piece right here yet, but what you would want to do is take and say how do I know that on block in this case 3 the key rodney was associated with Rodney we're at walk, and so that's something that we're going to release in a little bit. The other piece, you may say, is well what it would happen is Shaw. Does he have an email? He does? Let's say I said I can set shot.

A

One I block one not found so Shaw did not have an email on block one. You can see what about Rodney on block one.

A

Rodney on block one, you see it as Rodney awoke and you'll see that it's got a much shorter proof and that's the uniqueness of the sparse Merkle tree is that the sparse Merkle tree instead of having to everything, going all the way up the tree with real values, there's a concept of default hashes, and so when you only have one value, it's just hashed, all the way up to the default value, and so your proof actually can be empty. In that case now, the next pages I want to review is the proof.

A

So I talked about the proof of the values consistency, but how do I know that my block is actually being published to a theorem, so I want to know that, even though I'm working on whatever DAP this might be that the data that I submit is actually made it to its way to the main net and the way that the reason this is important, because in aetherium for plasma cash, if you do an exit or you do some sort of withdrawal or a challenge.

A

If the plasma operator is being malicious or it doesn't or has some sort of downtime, you can exit and retrieve your your money, and so for this again.

A

This is really raw stuff and we haven't had a chance to really make it pretty, but this is an example of what the anchor transaction hash would look like. So for black 1, 2 or 3. There is a hash, that's associated with it, to say, here's a hash of my block, and that gives me an opportunity to actually go back to the slide here.

A

The slide here so in the picture here you can see layer three is at the top layer, two plasma chain at the bottom in aetherium here, and so you can see that just take block 303 thousand to that hash or the details of that block or passed to the plasma chain and the plasma chain is contacting and sending Merkle roots down to layer. One blockchain, so this picture here just shows that layer 3 as it's going through its life.

A

It's constantly in contact with the layer, 2 plasma chain and layer 2 the same way, and so you can see well where the anchor roots that we call them make their way through, and this is why we call it a deep merkel proof, because you can prove eventually that you know Rodney and Rodney awoke, has a value I think it has a key value pair that matches that that can be walked all the way up to the top to a root.

A

Then you can come here and do the similar thing to the to the root on the layer 2 chain, and then you can come here and do a walk up. All the way to that to the root on aetherium, and so that's why we call them deep and that's why we call them sparse Merkle trees, because all these are leveraging sparse, Merkle trees and it's essentially anchors to each of the chain. So you can anchor your layer 3 chain to layer, 2 and layer. 2 is anchoring yourself to layer 1.

A

A

So I come back here.

A

We built it to where you can communicate with the anchor root. I mean with the plasma chain to get the anchor transaction proof so you'll see I, don't have time to go through all of this, but you can see the different pieces that you would want to use to have the proof we have the contents of the plasma block and then, after, if you come up and you.

B

A

To us we can walk through this, but we have 60. We have a depth of 64 in our sparse Merkle tree and so you'll see that this h0 is the level zero of the root the leaf and we're essentially walking up.

A

So we have the hash of the leaf or the Leafs Ranka root with default hashes, and so the D on each side is essentially a picture of how it's going through the proof all the way up, and so, when you get to the bottom, you'll see that the 63rd or 64th gets you a smart proof.

A

That's matched to the anchor root, so I know it's probably hard to kind of grasp and see in this showing here.

A

But we're really excited about the ability for us to have this provable data database services that are powered by all of these innovations, and we think that the smart proofs that we have will enable developers to be able to not only just build apps but provide their users the confidence that their data is stored there and it's accurate and that they can actually verify it and not take anyone's word for it and enabling us to have other people participate as plasma operators as well as cloud store operators to contribute to the decentralization of applications and particularly database services.

A

I. Think that's all my time. So if there's any questions, I gladly answer do.

B

We have any questions in the audience.

B

Does this structure of three layers, or in the in that in the in the third layer data base off the data file ability problem? So if I restore some data, there do I have the warranty that this that is going to be available in the future.

A

Yes, I mean that's the that's of the intent of the the plasma operator and apply in the cloud store to work in I, guess conjunction to ensure that data is available. That's what the proof is meant to show. We have some innovations on just the data cloud store later that ice rod will talk about to kind of go in a little bit more detail about how we're looking into things like Starks, to show that the state changes of your data, not just the initial data, is being maintained over time.

A

So I know there's a lot of kind of questions about what you mean by data availability, but the the use of plasma puts a punishment. Essentially that says that you told me this data was here.

A

I have this proof that this data should be there, but when I go to verify it, it's not there for you get punished for it, so it doesn't guarantee it, but if we're ensuring it and then we're trying to provide, you know deep proofs to be able to provide some level of confidence that your data is there and going to be available.

B

Any other questions.