Graph Protocol The Graph - Featured Videos, 30 Nov 2018

Previous Meeting

⏯

youtube image

►

From YouTube: Designing a Decentralized Query Layer for Web3 with Brandon Ramirez from Graph Protocol

Description

Brandon Ramirez, Graph Protocol's co-founder and Research Lead, gave a presentation at the "Building the Free Web" event in Prague on November 3rd. This is the first presentation outlining our decentralized network design and future research.

0:00 Intro
0:25 Issues on the internet
3:30 A naive design
5:17 The design
7:42 Building a subgraph
8:20 Market discovery
11:10 Query processing
12:51 Exploring the trade-off space
15:05 Mechanism design
18:06 Back to query processing
19:18 Tactical Opportunities
21:50 Future research

Follow The Graph on social media
Twitter: https://twitter.com/graphprotocol?s=20
Instagram: https://instagram.com/graphprotocol
LinkedIn: https://www.linkedin.com/company/thegraph/
GitHub: https://github.com/graphprotocol
Website: https://thegraph.com

#web3 #dapps #thegraph

A

So like freed I'm Brandon Ramirez I'm with the team called the grass we're building a layer, 2 protocol for querying public blockchain data, and today I'm, going to talk to you a little bit of our progress in designing this query layer for web 3.

A

So before we get into the design of the protocol, I can I want to talk about what the problem is that we're trying to solve and, like all things on the Internet it kind of starts and ends with cats.

A

So crypto kitties is like a great dap example, because it's one of the few ones that like tried to hit scale and failed, and one of the problems that they had of doing everything on blockchain, is that they've got millions or over a million crypto kitties minted at the time that I made this deck and in the application they want to do these queries like query, for you know, kiddies sorted by time or sorted by these different attributes.

A

That's not the way that they're stored on the underlying blockchain, and so, as many of you guys, know and probably know in the audience. This is a pretty technical crowd.

A

What you would usually do for something like this is use a database index, and so that's exactly what most apps are doing today, they're using like a sequel database or some other database and taken advantage of like the native indexing capabilities of that database, which is fine, but the problem with this is that, like all that infrastructure is running on a centralized, you know AWS cloud or you know Google cloud or whatever. So it's not really fulfilling the dream of like decentralized applications.

A

I'll note that the daps that aren't doing something like this are usually doing something like this locally on the user's machine and the UX is terrible right. The user has to like sync a bunch of events from the blockchain just to do something really simple in the app. So this is problematic because for those of us that believe in a web, 3 stack. That's fully decentralized like this is falling short of the vision right. It's creating single points of failure and both in terms of infrastructure, but also economics right.

A

If a business shuts down, then the app stops working and it's incentive incompatible for a lot of developers and a lot of projects right, like a team like augur, doesn't want to spend or pay for you know: dev ops costs for the lifetime of the project. They want to be able to something usable to the community and then be able to walk away once that protocol and that application is self-sustaining.

A

So, like I said before we're designing a decentralized query layer for DAPs to mitigate this problem and I'm not going to go through all the properties on this list that we're shooting for, but I will touch on the first one, which is data as an interoperability layer out the graph.

A

We really believe this is one of the core promises of blockchain that web 2.0 was about proprietary api's that were cloud-based, and we think that block chains are going to enable this new paradigm where applications and developers interact with the common substrate, which is data and that's the interoperability layer. And that's what we're hoping to kind of take. Blockchain and crypto that, like final hundred yards to get there.

A

So before I get into like our our protocol design, just as like kind of a strawman I want to like go through like what may be a naive design, would look like so I kind of took the same infrastructure from a couple slides ago and I just put it into a query node. So that's kind of what you're seeing in the top left and we put in a decentralized network and like let's say you can pay for the queries with micro payments and problem solved right.

A

Well, not really so you could I guess you could improve upon it a little bit and you could like throw some like pseudo-random to kick random verification. You know maybe X percent of the query. Responses need to get validated on Shane, that's maybe better than nothing, but it this Falls dramatically short right for one validating that many queries just consistently is going to be extremely expensive.

A

Each of these data sets is is different so that the actual cost to a verifier of like validating these query responses would be pretty high you're, not really getting that much of an economic security margin right. You recognize economic security margin is, is pretty low.

A

Does everyone know what that term means generally economic, so economic security margin is how much an attacker a participant in the system would have to spend to lie to you in the context of the protocol right if a query node, for instance, stakes, $10,000 worth of ether, graph tokens and that's what they're gonna get have slashed.

A

If they provide you an incorrect response, then your economic security margin is 10,000 eath and that's important as you're designing your system, because you know for different use cases like if your DAP is in exchange or it's gonna cause users to make decisions that are, you know economically significant. Then you want to know like how much of a guarantee you have that the data you're working with is correct.

A

So, let's, let's move into the actual design. So this is the working architecture of our graph node. This isn't the entire system, but we'll start here and so I'm gonna kind of take you through the whole loop. So at the top you see that we have the DAP and it's kind of a CQRS pattern for anyone. That's familiar with that, so transactions are going to the ethereum blockchain and did the smart contracts that those smart contracts are emitting, events which the graph node our query, node event sources inside of a some runtime, that's the mapping!

A

So it's a deterministic ETL process in turn, the data, that's the output of that ETL is indexed, and it's added to this. What we call in an index chain and I'll touch on that more in the next slide, and then we expose that back to the DAP through like a graph. Ql API is what we support today, but in the future it could be any number of query languages. And what I'll note here is the index chains I'll touch on here?

A

Actually, so the index chains are basically layer, two block chains where the root hash of amortized database index is committed to each block in the in the index chain, and so so already, your economic security margin is getting a lot higher because, instead of just one query, node kind of telling you like, oh yeah, this is the right result. Now you have the guarantee of every query node in the system. That's formed consensus around this index chain so like.

A

If you have 10 query nodes, then your economic security margin is like 10x what it was in the last in the last construction. The other nice thing about this is that you no longer have to choose like make a trade-off between choosing a query node for, like for the Economic Security margin, like the amount that they've staked versus, maybe other things like their latency or their performance, characteristics or proximity to you right like if you had to make that trade-off.

A

You know you might not end up with an optimal UX, but in this way like you can get a proof that the data you're getting is correct and has the economic security margin of query nodes that are distributed across the globe, while choosing a prairie node to give you that data, that's you know, maybe has, like other characteristics that you're interested in looking for, and so these are a bunch of properties of like Oracle AERS database indexes I'm not going to go in depth into too many of these.

A

But if you want to like dive into that during QAM, I'm happy to so real quick, the experience of building the sub graph, that's what we call the data sets that we index on the graph. That's there kind of like you still main specific data sets right, so you might have like a sub graph for crypto kitties, a sub graph for 0x they're. All the definition of those sub graphs are in what's called a sub graph, manifest where you define your schema.

A

You register mappings it's stored on IP FS and that the the hash of that kind of uniquely identifies your data set right, and so that's going to uniquely identify what you're querying in the network and, like I said we can create with graph QL today, but in the future we will support more query languages so, before I get into the query. Processing workflow I want to talk a little bit about market discovery right because we're trying to create an efficient marketplace for querying public blockchain data right.

A

It's not enough that we have the technical capacity to do it. We want to make sure that there's actually a robust network of query nodes that feel incentivized to provide this service in a verifiable way to the network, and so an important part of that is market discovery. Right like how does a query node even know that the crypto kiddies data set is valuable to index or that the 0x data set is valuable to index chances. Are that they're not going to have like domain-specific knowledge of every DAP in the universe right?

A

So we need to have a participant in the network that that does have that kind of private knowledge that can signal to the rest of the network.

A

Hey this datasets gonna be valuable in the future and I'm willing to stake money on that fact, and so that's what the curator is doing so so the curator signals in the staking contract first particular data set and curating successfully is gonna entitle you to an annuity based on how successful that data set is in the future, married measured by the query volume to that data set that kind of overcomes the bootstrapping problem. The after after that bootstrapping happens.

A

Then, like query notes, can look at actual volume on the network right they can look at, like you know the actual transaction volume. Sorry I'm just checking I'm doing on time here we're good and so perrenoud. When they're. Looking at like the the marketplace, they're gonna say like okay, this you know 0x data set. Has you know a ton staked behind it? So maybe I want to sync that Index chain and I'm gonna start validating on that index chain.

A

The final participant I want to touch on this is a construction called delegated stake in which, like you, might be familiar with from a lot of different protocols. But what this gives us the opportunity to do is get participation from a wider selection of token holders in the network right. How can we avoid like speculators and like free riders in the network that are just holding the token to like hope that the work other people are doing in the network is gonna like drive that token value up allowing encouraging people to delegate stake?

A

You know allows all tokens that are in the network to do work and the way that we encourage that as a mechanism called participation, adjusted inflation I believe dug from live peer was the first one that offered offered up that construction and it's basically, you set a target participation rate in the network and the further you are away from that target participation rate the higher the inflation is going to be there, thereby diluting the holdings of users in the network that haven't staked the tokens and those inflation rewards go to users that have stake the tokens and are doing work in the network.

A

So the query processing. This is actually believe it or not. A simplified diagram that the query process is a little bit more complicated, but the things I want to draw your attention to here is that when the gateway nodes are querying, the hurry nodes for these read operations, they're, not just getting the data back, but they're, also getting a merkel proof back.

A

That proves that that data came from the index chain that the Gateway node specified, they're, also getting back an attestation, and so the out to station is something that we can use for like dispute resolution, and- and so these at the stations are potentially valuable right to like a fisherman or anyone else in the network, because if you successfully file a dispute against one of these query nodes, that's malicious, then you can get a portion of their state tokens. It's like part of your reward.

A

The other thing I'll note here is that fishermen, in addition to verifying like at the stations, they can also just look at the index chain directly, which is something that that, like a naive construction, didn't have right, like there's a huge difference between a fisherman, just collecting query responses. After the fact and saying, like okay, I'm gonna like validate some like subset of these and like, hopefully with this like battleship approach, we can make the system safe in this construction.

A

The fisherman can a priority validate these index chains like in real time, and then you know if adapt user comes on later, they can get that. You know Merkel proof that the data came from this index chain and it carries a lot more weight in the system. The other thing I want to point out is that curators and delegate errs, as I mentioned, before, get a portion of the stake staking reward and potentially a portion of the query fees as well in the system.

A

So these are kind of areas of active research for us, but I've kind of walked through already, it's like the design that we have so far, and so a huge part of what we want to do is we start prototyping, like the fully decentralized version of the protocol, is to kind of explore the trade-off space right, like I kind of give a hand wavey.

A

You know explanation of why I like the first like naive, construction, wouldn't work and I think you can take that it like at a pretty good face value, but like there's a lot of different designs that are, in that general space and, broadly speaking, the way that we kind of think of the trade-off space is like at one end.

A

You have, you know high centralization and high performance and at the other, and you have like really really high-security, but like performance takes a big hit and, and we want to figure out not only like what does that curve look like? Is it like? Is it the curve on the left? Is it the curve on the right, but like what's the most tasteful trade-off that we can make as a layer, 2 protocol to kind of deliver the most value to users at the greatest security? And maybe it's even something that you know we make tunable.

A

You know in the context of the protocol, based on like the particular application. One other thing I'll say about these diagrams. That I don't hear talked about a lot. Is that, like almost every step on these diagrams, except for like the snarks and Starks proofs, is relying on some level of economic security right like even even using like Merkel as data structures and consensus, like consensus, like Nakamoto consensus, is still relying on Economic Security right, like you're you're, relying on the fact that people can't like buy enough hashing power to like overcome the consensus mechanism?

A

Proof of state constructions also rely on like a level of Economic Security right. That doesn't mean that it's bad. It just means like. We need to be honest about, like the trade-offs that we're willing to make in a system, and in some cases you know we might want the stronger.

A

Like cryptographic, security of Starks or snarks, but our intuition right now is that the right trade-off is gonna, be somewhere in, like the Merkel eyes, data structure in like and consensus space and that's kind of the index chain, construction that that I explained earlier so on the mechanism design side, there's a ton of parameters in the network that were that we need to tune that we need to select and I'm, not gonna, go into each one of these, but I am gonna.

A

There was a question in the previous talk about like how to set some of these parameters. I think it was someone sitting over there and so I'll kind of give my framework for how I think about these things. I put them into four broad buckets, so I think you can have these parameters set at the network level right like it can be fixed in the protocol and it's something that can only be changed through like a hard fork in the protocol.

A

You can have them set as like a variable, a variable value, and so that's similar to like the inflation that was that's determined by like the participation rate, you can have them set through free market mechanisms, and so that's like you can kind of think like how live pure, for example, only allows you know 15 or 20, or you know, X validators and like people are competing to stake enough tokens to be in, like the active validator set, like that's kind of a free market mechanism right like you're, not setting at the network level.

A

What the staking amount should be it's it's kind of this competition. That's setting the value I actually think. That's the most desirable way to set a parameter if you can set it because it's kind of like the network reaching its own equilibrium right, but it's not something that you can use for for every single parameter in the network, and so that's kind of where governance comes in I haven't heard. This talked about a lot but I kind of see. Governance as like a cop-out like it's kind of like a lot of projects.

A

That kind of like defer to governance and say, okay, you know it's kind of saying like we don't know how to set this parameter and- and that's not a bad thing. I think a lot of these parameters like we're gonna figure out how to set them through.

A

Like empirical observation of like these live protocols, but that's kind of what that's kind of what you're saying, because, like you're kicking the can down the road you're saying that you know someone who's, not us is gonna figure out how to set this parameter, and maybe that's like the wisdom of the crowds and so I.

A

Think in some cases you don't have a choice, but I think it's important to call that out explicitly and so an example of that would be like the target participation rate right like again going back to the live pure example, which I only keep bringing up the real, only one to know that I've actually implemented this mechanism in the past, like they chose 50 percent as the target participation rate that wasn't like a scientific choice.

A

Right, like that, was kind of like a gut choice and it's you know something that in the governance mechanism, so the protocol people can change over time.

A

One other thing: I'll actually touch on that's kind of kind of a tricky like active research question is doing forced errors for for the index chains, and so so this I've actually talked with a few protocols about this. That have all kind of thinking about this. I haven't heard a good solution proposed for this, but going back to see it's not so so going back to this diagram.

A

I kind of didn't stop on this, but like I, have this fisherman here right and he's verifying the the layer to index chains and, like you, guys kind of took it for granted that, like this fisherman, would exist and would feel incentivized based on like slashing malicious peri nodes, but like an equilibrium, we would all expect for the network to not have malicious party nodes right like if the quarry nodes learned that they're gonna be slashed every time they you know, do something malicious, like the equilibrium should be that no prairie node is, you know, committing malicious blocks to these index chains.

A

So then, the question is like: what's the incentive for the fisherman to keep validating these index chains in the network, so for other constructions that use fisherman's for like different roles in the network? Usually this is something where you rely on like what's called forced errors right, like you basically have in the protocol specification that you know, there's some mechanism for creating a forced air and then the fisherman will catch those forced errors and still get a reward.

A

It's a little bit harder to do for layer to block chains because, like people are relying on these things to be correct. So this is an area of ongoing research for us and if you know anyone has any, you know novel thoughts in this I'd be happy to talk to you afterwards.

A

So I want to talk a little bit about our philosophy at the graph and like how we've been approaching development. You know I've talked a lot about like our decentralized query. Protocol design and I've talked a little bit about stuff that we've implemented so far. So I want to get like a quick update on where we're at now and like how we're approaching the path to decentralization. So we take the approach that we want to create value for developers as quickly as possible and on an ongoing basis. So each iteration that we do.

A

We want to make sure that it's creating value for developers and that's in kind of, in contrast to like a more waterfall style approach where we say you know we're not going to ship anything usable until the network can be fully decentralized. So what we've already open sourced and it's honored github you can check it out at graphic protocol- is an indexing node that has the lazarum runtimes. The deterministic ETL can listen to Theory events, you can build a subgraph and you can like run that locally or deploy it yourself and like host it.

A

What it doesn't have yet is like the p2p networking like and like consensus layer and some of these other incentive mechanisms built into it yet so like right now, it's something that you can run yourself and you trust it because you're running it, but on the path to decentralization there's all these tactical opportunities where we can like start centralized- and we can. You know, move to the fully decentralized network over time and so like the very next step that we're gonna do.

A

Is we're on channel hosted service early next year, but then there's all these other dimensions as well.

A

So, like governance, for example like it can start with this centralized multi-sig, but it can move to something where we're using dowse to signal that small group of people and that eventually it could just be the dow itself, that's making protocol decisions and like setting parameters, I'm, not gonna, go into all of these, but like another one that I think it's like pretty interesting is like you could have you know a centralized registry that either, like logically centralized or like architectural, be centralized, and you can move to like an overlay.

A

Network over time and use like I like the node node discovery of that network, as a registry and payment channels actually are also pretty interesting, like you can do a hub-and-spoke payment channel topology, which is like what spank chain is done, for example, to like be the first I believe they were the first ones to actually launch a payment channel network on on aetherium, and they did that by making a really pragmatic choice that they weren't going to try and solve this like really difficult graph.

A

Traversal problem that, like the rate in network, is, is trying to solve and making good progress on. So that's just kind of like our our philosophy and like moving to the decentralized network like create value first and like decentralized slowly, and this is further out I haven't we haven't done a ton of thinking interactive research here yet, but you know in the future, we want to support multiple block chains. You know we're deciding on like what the next layer, one that we would support, is so, if you're part of a layer.

A

One project, please talk to us. You know we're looking at what our story is going to be for, like indexing, specific charge or specific plasma side chains, as well as like indexing, like private data- that's you know maybe encrypted on chain and that's it. Thank you guys very much.