Graph Protocol Graph Day | 2022, 5 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Graph Day 2022 - Indexing Improvements via Substreams, with Sebastian Siemssen of The Graph Council

Description

Subgraph developer and The Graph Council member Sebastien Siemssen described the world’s past struggle for data access in web2 and how subgraphs change how we interact with data in web3. Siemssen also reflects on how scaling indexing performance is a matter of parallelization. Consequently, Siemssen introduced substreams, which are fully parallelizable, infinitely composable, highly reusable & cacheable data streams, that are about to increase subgraph syncing speeds by orders of magnitude.

Read more about Substreams and how they will improve indexing speeds over 100x:
https://thegraph.com/blog/substreams-parallel-processing

Watch more Graph Day 2022 talks, panels, and announcements here: https://www.youtube.com/playlist?list=PLTqyKgxaGF3SWt8ipQw-Ul9HQvmxulbUu

Keep up with The Graph on social media!

Time Stamps
0:00 Intro
0:15 Indexing improvements
6:33 Substreams

Twitter: https://twitter.com/graphprotocol
Instagram: https://instagram.com/graphprotocol
LinkedIn: https://www.linkedin.com/company/thegraph
GitHub: https://github.com/graphprotocol
Website: https://thegraph.com

A

All right everyone, I can't actually see you it's quite bright up here, so indexing improvements. I hear um that the beans have already been spilled by yaniv earlier and also in follow-up presentations.

A

My talk got rescheduled, so here we are at the end of the day talking about indexing improvements that are about to land and come to a decentralized network near you. I am a subgraph developer. Let me tell you something about data heavy applications. They are hard always but especially on the blockchain for the blockchain. It's just not a primary concern to make data access convenience access at all.

A

To be quite frank, so we have been in this struggle me and my team to solve this problem, and we have been in this struggle and it was painful. Let's call the dark ages. We experienced this firsthand. The four subgraphs were a thing and before because by default, data access on the blockchain is slow because you have to do multiple roundtrips. If you want to fetch data, it has.

A

It is complex because you need really intricate knowledge about the smart contracts that encode this information on the blockchain to be able to do something with it to extract it and then make sense of it, but also it is just really inaccessible.

A

So let me emphasize on this last point: complexity, top graphs, actually, not just don't just index data, they also encode and enrich that data with the knowledge and the context from the various very smart contract developers who build these smart contracts from which we are extracting that information.

A

That is a very important aspect that I think is underappreciated, so smart contract developers and subgroup developers make this data accessible by exposing it in a manner through the subgraphs that is more digestible in its format.

A

Application developers, but also third-party consumers, can now use that data without requiring them to study the intrinsic details of these smart contracts, so subgraphs empowered us, they gave us a second wired. They made the data accessible to us. They made it fast now exposing that data through a single, simple http interface, through which you could fetch the data in a single round trip from the server from the decentralized network, and they also made it simple, because now you have a simple self-explanatory data formats with which you can work.

A

We realize that there's still work to do, and we have heard you so we want to support alternative use cases in the future. Alternative use cases could include, for instance, streaming real-time data.

A

We want to enable and further foster reusability and composability of subgraphs, and we want to make the more performant. This is a subgraph. This is your perfect case study of a traditional etl pipeline.

A

You extract data from the blockchain e, you transform it and then you load it into a database. This is currently all done in a single monolithic unit of computation, your subgraph, your mapping code and the subgraph.

A

This is where that happens, and because it is a single unit of computation and because it is a constant, read and write operation on the shared database, which holds the state it is, it has to be done linearly and linearity doesn't scale, especially as the blockchain and the data that has to be indexed grows, so it becomes clear. We have to parallelize this process.

A

Furthermore, for alternative use cases in order to cater for them, we have to enable other consumer interfaces, other consumer interfaces for, for instance, real-time applications like notifications alerts, maybe uh something that could be um like an on-chain relay, centralized network of the graph uh reading data and indexing the data in a subgraph and then pushing it back onto the network.

A

Currently, as I mentioned, these etl pipelines work in an isolated manner. They are connected at the consumer level, so you can consume data from multiple of these subgraphs and then display that in your application and join them together at the end of the pipeline.

A

But there is no way at the moment and it has been attempted to connect the data at the indexing layer.

A

So if you build a subgraph, let's say um to to show pricing information, that's coming from a uniswap t-web and you are aggregating that, from from the uniswell subgraph, there's no way currently in your own subgraph to read that data and make use of it within your indexing operation.

A

So how can we get to a point where that becomes possible and we can share these subgraphs at a much lower level than we currently can?

A

And the answer to this, in my opinion, is to create this universe of interconnected and yeah, also uh of of intragalactical and intergalactic connections between these smart contracts and the subgraphs that they represent and enable communication with all of them in this universe. That is the graph, and that is something that is, in my opinion, very true to the vision of the graph.

A

The graph means one unified data source, not multiple individual subgraphs in isolation, but this entirety of all of these subgraphs combined and within that lies the real power of the graph.

A

So today um we have already introduced it before, but we're introducing the substreams to um a decentralized network near you soon, substreams are fully parallelizable infinitely. Composable and highly reusable and cacheable data units- and this is how you define such a substrate.

A

So a sub stream is essentially also just like a subgraph, a yama, manifest with which you define the other substring modules from other developers or maybe from other protocols, even or from the subgraph dao, and then define and yeah declare how that subgraph is supposed to work with those you also, if you are reading this now, can see that there are protobuf files in there. So you declare the data sheet. The data schema for the data in motion within these substreams using protobuf.

A

Furthermore, you'll also see that there's rust referenced in there, so these substreams will be built using rust, there's a command line interface through which you'll be able to package, build and redistribute these substreams and share them with the community, just as you can also reuse, some stop streams that have been developed by the community.

A

This modularity of the individual substreams and the modules within these substreams enables us to fully parallelize the indexing of individual data streams within such a module and any other substream can inherit them, and it makes it possible for us going forward to also enable sub streams like, for instance, a fully generic esc20 token substring, currently unthinkable.

A

So you define your protobuf schemas for the data that you're handling within a stream and then in your substream code in your module. You expose that data and return that data for others to consume.

A

These other modules can then also read that information using the same protobuf schemas and further refine it within their own sub, graph or substream, and that leads me to the next question: how does this fit with the current model? So substreams are not a replacement for subgraphs. In fact, they are what what is going to feed subgraphs.

A

So in the future, when you are building a subgraph, you will consider that as the final indexing and aggregation layer around a series of substrings combining the flow of the data from all of these different sources into a human digestible format that can then be consumed by your end user application.

A

So, as you can see here, this gigantic network that I see will manifest itself um with generics. Generic substreams maintained by the subgraph, dao or very specific substrings maintained by individual protocol development teams will be exposed to you and you can reuse all of that. Conveniently.

A

So tomorrow, there's a session by alex from streamingfast it's in the connects breakout room at 10 30 at 3 30 in the afternoon, and there you'll learn exactly how these subgraphs are supposed to work, how you can build a substring and how you can use it in your application.

A

A