Graph Protocol Core Devs, 1 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Graph | Core Devs Call #15 - September 1st 2022

Description

The Graph is the indexing and query layer of web3. Anyone can build and publish open APIs, called subgraphs, making data easily accessible. On this latest core dev call, learn all about the Gossip network proposed by core developer GraphOps, and other key developments that are being worked on to improve

The Graph Core Devs coordinate work using the publicly accessible The Graph Core R&D Workspace: https://thegraphfoundation.notion.site/The-Graph-Core-R-D-5dca927e67eb4e459539f5bd6d0a7fb8

Timestamps
00:00 Intro
03:05 MIPs (Migration Infrastructure Providers) Program Overview
11:20 Intro to Graphcast
24:00 @Live Queries
32:13 Building Subgraphs for Tendermint Chains
42:05 Substreams Development Update

Link to external sources mentioned in the call:
MIPs Blog Post: https://thegraph.com/blog/mips-multi-chain-indexing-incentivized-program
Graphcast GRC: https://forum.thegraph.com/t/grc-001-graphcast-a-gossip-network-for-indexers/3544
GIP-0035: GraphQL Live Queries: https://forum.thegraph.com/t/gip-0035-graphql-live-queries/3432
GIP-XXXX: Substreams Into Subgraphs: https://forum.thegraph.com/t/substreams-into-subgraphs-a-simple-integration/3542 | https://substreams.streamingfast.io/

------

Need help migrating subgraphs to the network? Here's a complete tutorial: https://thegraph.com/blog/how-to-migr...
Feel free to reach out if you need help: migration@thegraph.foundation

------

Follow The Graph on social media
Twitter: https://twitter.com/graphprotocol?s=20
Instagram: https://instagram.com/graphprotocol
LinkedIn: https://www.linkedin.com/company/thegraph
GitHub: https://github.com/graphprotocol
Website: https://thegraph.com/

A

A

Thank you all for joining hi everyone. um It's our 15th call September 1st good good to have you here. We have a good agenda. Welcome all back, um as you know, we're having these calls on a monthly basis with the graph core depths. um All of these calls are recorded and uploaded to our YouTube channel. So if you want to find it, you just need to Google the graph protocol.

A

Youtube and you'll see a dedicated Channel there for for devs calls, and we also have a dedicated core devs workspace and ecosystem calendar that you can find. If you follow this link you're, seeing in my slides to graph.com the graph.f foundation, the workspace will, let you know the different working groups. We have the you know, meeting notes recordings, um there's a kind of on board in some cases and you can follow along with the latest updates and there's plenty happening.

A

So for today's call we have an interesting agenda. We're going to go through the recently announced, nips program that the foundation um has put together. We've announced it last week we will be hearing from graph Ops and and the networks Ops working group on graphcast. The uh Gossip Network they've been working on I, believe there's also a GRC in the Forum. We can post it in the in the show notes and in our chat, so you can take a look at it and Dotan from The.

A

Guild will also share some updates on the a few things we've added to graph node and graph clients which are pretty neat um I, believe we're going to go over child filters and graph clients. Some some um like some support for live, queries, graphql, live queries and then barters from figment. um We're gonna. Do we're going to do I'm.

A

Sorry, I'm, still blading people in we're going to do a workshop, a quick workshop on what it means to build a sub graph for tender, mint based chains and we've done something similar with subgraph developers a couple days ago and I think it went fairly well, but might as well take the time and do it with other core devs and and give the opportunity for other folks to ask questions. I think it went fairly well, so um my yeah, it's worth doing it again.

A

You know with the different audience and uh lastly, Adam from edginode.10 api's working group as well and updated on substreams. Also a fairly recent post, we've shared Adam is shared in the Forum. I can also link it in the chat and show notes. Okay.

A

So let's do let's talk about mips, um quick overview, the whole point so, as you know, the graph we're pretty serious about this centralization, it's always been in our roadmap to you know, have multiple networks supported in multiple stream, supported in our Network, and you know that we are deprecating the hosted service, which is the centralized and free service that most subgraph developers, use and love, and this hosted service when compared to our Network.

A

um It supports more than 20 different chains, whereas on the decentralized network at the moment, you only have one that's ethereal minutes, as most of you know, and um this means like deprecating. The hosted service means that it will go away in about nine months from from now, but not the support for all of these chains, and so there's a process in place.

A

We're working on you can follow up in the Forum there's some interesting threads like the block or Oracle, where atomies has been updating on what it means to add support for a new chain on the network and what the process will look look like, but um now that we have mips, we can see it as a complementary Pro program, really where, where the foundation is incentivizing indexers to bootstrap all of these chains um in a nice way and by nice, I mean with nice quality of service and so yeah bootstrapping an exercise Supply, making sure that we can serve these changes properly.

A

B

A

Net environment first and then moving on to to our Network or mainnet.

A

um Yes, so you might have seen the incentives we've allocated 50 a total of 50 yield in GRT for indexers that that's 0.5 of the GRT Supply, but also 25 million GRT. For my migration grants just to make sure that we can migrate most sub graphs over the decentralized network and help out our community going fully decision to product decentralized, quick breakdown, quick, um uh send me for information about the breakdown and this distribution of this GRT.

A

So we out of this 50 million we're going to be allocating um a specific amount um for any given chain and we're going to be announcing the specific allocation um by Midway through the program. That is once we get support for the for the chain. We'll.

A

Let you folks know uh how many, how many GRT we are allocating to that chain and by the end of the cycle and you'll, understand what these Cycles are in a bit uh by the end of each change cycle index that will get access to 30 of their budget that we've allocated to that to that chain.

A

The other 70 will be will be investing and uh participants will know more uh on how this whole thing will work out when it comes to vesting contracts, I'm just doing a quick overview now so um going talking a little bit about this leaderboard, the way we're going to be Distributing, the uh rewards is based on a leaderboard, as I mentioned. We do want to make sure that we have a nice quality of service in this in this chains, and so um we're going to be keeping track of a few things.

A

Some things we can monitor on chain like allocation deficiency. Some will be of chain like uh you know, if you're doing serving queries with, you know low latency. uh You know we have like nice data freshness not being too far behind blocks, for instance, so and and other metrics like uh providing support to other indexers participating in our forum, testing, L2 and participating in workshops that is going to be held that we're going to help in indexer during the future index of office hours.

A

So the higher you rank in this leaderboard, the more GRT you're entitled to or you'll you will be eligible for, um as I mentioned, we're going to have different changes, uh we're going to be announcing new chains, uh or rather the future chain. Once you get a new chain uh supported in the program we'll, let you folks know with two weeks notice, so you can prepare your infrastructure start. Syncing your archive archive nodes and um yeah, um as I mentioned, the GRT will be distributed after each cycle and I'll show you what diagram next on?

A

What what that means? The first chain is going to be noses chain. We're just uh we've just announced it. So we have some participants already syncing up their archive notes, so a quick over overview of what I've, just what I was just mentioning, so the focus will really be on quality of service monitoring stuff in the test net. So, as you know, we have girly. That's our test net environment. We're going to be adding support for these chains.

A

There indexes will start serving subrops that we will be deploying there we're going to be injecting uh traffic, uh starting with noses we're going to be monitoring qos and once the foundation and core devs agree that there's a a good enough quality of service. That's when we'll be signaling um that this change should be now moved to Mid mainnet.

A

This participants will still be serving those subgraphs over mainnet we're going to be monitoring just to make sure that they are still keeping up and that's when the recycle ends. Will you know all the all indexes will be scored based on all of these qos and how they participate in the program and by then we'll know how how many GRT they should be eligible for once that cycle ends we're going to repeat the same process, we're going to add a new chain going test net monitor qos go in mainnet, monitor qos repeats as optional.

A

If you um participate in other things like testing our L2 Bridge, the arbitrine page, or even share feedback on YouTube new tooling that we are working on, such as Auto Agora and the allocation of atomizer. That I believe we've already covered here.

A

um That's gonna also give you some extra points, and so yeah I hope. That's clear. Let me know in the chat, if you guys have questions or as mentioned, we've had a whole workshop on or an intro and I've covered all of these as well during previous office hours. So, if you want to know more, you can re-watch.

A

You can watch that um recording it's it's been uploaded to YouTube and you can join future index of office hours to know to know to know more and if you're, not a participant, yet I highly encourage you to apply.

A

um You can still apply until the the 19th. The sorry the 15th of September uh I'll link I, will add to the chat, a link to our page, where you can know more about the program, the different type, the different phases and uh yeah. You can still apply and we would be glad if you didn't do so.

A

Okay, that's it uh I'm gonna! Stop sharing! We can talk about this or do some q a at the end, giving you the opportunity to ask questions, um and so we can move on to the first Topic in our agenda. That is graph, cost and I believe that you can already share your screen and talk right. Yes,.

C

Yes, uh thanks Pedro uh first, can everyone hear me? Okay?

C

Yes, you can cool cool awesome. uh Just a little heads up. uh My network has been a bit shabby. So if you lose me at any point or if I start cutting out uh just uh say something and I'll I'll reconnect um cool. So let me share my screen, one second, um great okay. So now you should see uh my window with the presentation right with the graph gas presentation.

A

C

To be sure, awesome awesome, great um I'm, Gonna Keep, the chat, open cool, so yeah, hello, everyone, I'm, I'd, go from uh graph Hops and today, I will be giving you a demonstration or more uh more, like a high level uh presentation on graph cast, which is the graphs gossip Network. um Initially, the headline was um a gossip Network for indexers, but uh seeing as that in the future, this um gossip, Network idea and graph cast can be used to facilitate applications geared towards other network participants as well like delegators or curators.

C

That's why we think that the graphs gossip Network sounds a bit better, um so I'm just going to give a quick um overview on what graphqest is uh and by the way I'm going to send in the chat uh the GRC, because it has been published yesterday. So there you go and what I'll be covering um in the presentation is going to be more or less um very, very similar to the GRC. So you can follow along or um yeah.

C

So, let's start um graphcast is meant to be a decentralized and distributed peer-to-peer communication tool so something that is off chain. It's. It's not a part of the on-chain protocol protocol operations and what that communication tool can. Allow is basically at first for um indexers to exchange messages with each other in real time, and those messages uh can be really anything so um information about their operations and anything else, that's that's relevant. uh It is.

C

It is very important to note that this this layer is uh completely optional, so you can add it to your existing indexer stack or you cannot, and so the the whole reasoning behind why graphcast is needed according to to us is because right now, the only way for uh indexers to exchange information is on chain. So the the network coordination happens uh on Chain by submitting transactions which obviously cost gas, and so it's not very efficient depending on the type of communication, so naturally, this situation makes it so that some messaging remains.

C

um Some messaging is just uh in a way priced out because it's not maybe worth the gas cost. So this is what we hope to solve with graph cast um and that's because the cost of exchange PHP messages is uh near zero, um which is ideal for uh coordinating without a concern for cost.

C

um Obviously, this um graphql being off chain comes with its own set of trade-offs.

C

um Like you know, graph cast cannot ultimately guarantee the uh truthfulness of the message, um at least by default.

C

um We are adding we're we're going to work uh really hard uh to add uh things that um guarantee that, after uh basically uh the messages during the exchange of the messages so uh but yeah, all of that is covered in the GRC. So again, I'm I'm gonna. Try to keep this brief. uh So how will this actually work so at first we're going to develop a graphcast SDK?

C

So that's software development kit that can be used to develop the so-called radios, so the radios are um applications uh in the network and the specific logic of the application of the radio can be pretty much whatever so the graph cast SD K allows the connection to the network so connection to other participants in the network and also provides a interface to interact with the network. But the radio is the place where all of the application logic happens and when different participants run the same radio.

C

Basically, that's when the communication happens so I'm going to move on to the next slide, which is yeah the motivation.

C

We talked a lot about that, so it unlocks basically um new forms of communication among indexers that are that are not um available today in this manner, um of course, as well with that comes easier experience for ecosystem, tooling developers, so the tools that indexers are going to be using, which are built on top of graphqast in the form in the form of radios, we're going to make it a lot easier to build those with the aforementioned graphcast SDK.

C

um That whole thing will lead um well ultimately to the to the democratizing of the underlying data, because that way, it's not like some team is building a um application for indexers and they keep it on their back-end servers and they, and only they can leverage it. This is going to allow for a much more open environment for build for building these applications.

C

Of course, as always, this as well comes with the sets of trade-offs, but again I'm not going to dive deep rinse them here, because I want to to have some time for a live demo. So um I've listed here some of the use cases that can um that that can be realized using graph cast in the future I'm not going to uh read all of them.

C

You can just uh see them in in the list, but of course, it's very important uh to note that the only limits to the future use cases of graphcast is one's imagination, because essentially everything can be in the messages that are being exchanged and the logic in the radios can be pretty much whatever. So anything that can be um beneficial for indexers to to share in a distributed and fast manner um can be done.

C

So now, let's touch a bit about the tech stack, so uh we currently have a proof of concept that is using typescript um entirely and we are relying on the white group protocol for um the decentralized messaging um features functionality. Basically, um of course, we are very open for discussion about Alternatives. After all, we are building all of this oops. Sorry, we are building all of this completely in the in the open, so um we are more than happy to receive feedback to participate in discussions on how this can get better and everything.

C

Everything else like that. um So the idea is that the typescript SDK apps abstracts over waku, and it provides um first and foremost it provides identity resolution which is really important, because that's um what I said before that the messages don't have any um default truth truthfulness.

C

This is at least a way to make sure sure that the sender is who they say they are so this includes things like message signing so that the indexer signs the message and when another indexer being a receiver receives it, they can that they can know where that message come comes from and they know that uh it is in. In that sense valid um as well as identity resolution.

C

It exposes a rich interface for for building the radios because it kind of takes all of the great um wacko messaging features and wraps them in in a layer such that they are geared uh specifically towards the graph protocol and in this stage um indexers, but in the future. As I said, there can be applications on top of graph gas that are uh suited for other roles as well.

C

So we, as I said we do have a proof of concept running which is a POI cross Checker. uh So that's a proof of indexing cross Checker that um I'm going to link as well in the chat.

C

It's it's in our GitHub, um so what it does just very basically, it's uh that Aggregates normalized pois, so each and each indexer uh sends their POI for a given sub graph deployment on a given hash, and that way there is a real-time view of um these pois and each each POI attestation is um backed by the on-chain stake of that indexer.

C

So that way, when you want to ensure that your POI is valid, you compare it with the POI that is backed by the most um aggregated stake so uh and and also it provides um handling POI Divergence uh by updating the cost model for the diverse subgraph deployment. And obviously this is. This will be very useful once it hits some more uh stable version, uh because uh it will, it will enable a very rapid detection of Divergence cool.

C

um So just moving on to the yeah. uh Just very quickly, next steps obviously collect feedback from indexers and as well as the wider Community, like all of you today uh publish the graphcast SDK as in the form of a library so that it can be used to develop radios and uh as well uh further develop the graphqas SDK and the POI cross Checker radio.

C

um So uh how much time do we have left Hydro um just want to fit the demo.

A

Yeah, you don't have a lot. You have already passed a 25 minute Mark, but how much is going to need like five minutes? Oh.

C

I can I can do it in two minutes. Perfect.

A

C

Okay, cool cool awesome and yeah thanks. This went a bit over uh what I was initially planning but I. Just wanna, I I think that the live demo is is gonna, show uh very neatly just one sec, because I want to share a different window. It's gonna share, uh it's gonna show the workflow um one. Second.

C

Literally gonna be two minutes: uh okay,.

C

Cool there we go so uh this is a the terminal where I'm just gonna run the oops I'm, just gonna run the um proof of concept.

C

ah Of course, building is gonna. Take a bit more than no I know what uh yeah my network is. Yeah, my network is being is being shabby. Sorry, um yeah, I I will be sharing a record.

C

um I will be sharing a recording of the proof of concept and I will share it um as fast as I can on the Discord. So so you guys can check that out as well. uh Yeah I hope that makes sense. Thank you all and that's it for me.

C

A

Good thanks, Petco yeah um I didn't mention before, but we're gonna have some q a at the end. I just want to make sure that we can go through every agent item and at the end of all of these uh presentations, let's say you folks can just react, raise your hand. I will allow you to speak, and you can ask your questions away. Okay, all right! So moving on um the time graph, node and graph client feel free to there. You go sure.

D

uh Hi everyone I'm Dotan I'm, the CTO of the guild, uh we're working uh on the entire graphql air, both uh in graph node and in graph client, which is the actual query layer and the client for the graph uh today, I want to show uh I could do a quick demo. I know that we're kind of short in time for two new features that we recently added.

D

um So we will start with the graph client I'm recently in graph client. We added a built-in support for live queries, so live queries is a concept in graphql. It's also um in RFC level, 4, the actual graphql spec. The idea is that every query can be a live query. So that means that you can take a query and just annotate it with the live query, and it doesn't really matter what kind of implementation we have under the hood.

D

The idea is that we can make the query live so by leveraging um really cool transports like SSC, and things like that. We're able to do a lot of cool stuff on the graphql layer, specifically in graph client in the latest version live, is already built in and under the hood. The implementation that we have at the moment is based on polling, but the uh the general idea of having live queries is not only to pull and fetch latest data.

D

It also means that um you declaratively, say that the query is live and it means that you get live updates and it's being updated by doing that on the graphic graph, client layer, uh we're basically decoupling the the graphql client that you might use. Maybe it's a polo client Erica relay it doesn't really matter so some features are no longer depends on the graphql client that you're using and they're part of the execution layer and the network layer, which is graph client. So if you're using graph Clan, you can specify the live directive.

D

I'll just run it in background just to see here. The data I just have like a subgraph here, it's on the hosted Service uh just for this demo, I'm fetching transactions and I just asked for the first five. uh What happens under the hood is that we're doing polling- um and this is uh automatically updating. So if you have graphical or graph graphql clients that already supports live queries, which is most of the graphical clients today, you'll get that out of the box.

D

So instead of getting just single response, you'll get a stream of responses when data changes. So in terms of the point of view of uh adapt, developer or front-end developer, you don't really need to do anything. You just specify, live and uh um behind the scenes, you'll get all the data being updated, cache updates and so on also supported. It's a bit different than polling, because polling might say that you want to get the latest state with live. We can do more stuff like getting a list of items and then update it.

D

But this is just like an initial step, because later we can do things like stream and defer, which are also New Concepts in graphql that might get into this spec at some point. So with the things like live and defer in the future, we'll be able to just ask for partial data, or only the first, uh an amount of Records in an array, um and this is something that is decided completely by the client. So we can do crazy stuff with graphql uh in the future.

D

So live is just one step, so yeah I, don't know if you got that uh yeah, but uh this subgraph is updating frequently so I'm getting all the transactions here. So I I always have the latest state of the first five. um So this is really cool and it's updating uh very frequently. uh So that's all with live. uh If you're using graph client, you can already use it uh right now with the latest version.

D

um Besides uh for live queries, we also have uh really cool list of features in graph client. uh You can do um composition for multiple subgraphs. You can customize the fetch strategies use fallback for the hosted service if you're migrating um to the decentralized network. So you can do you can do a lot of stuff with birth clients. So if you haven't tried it yet um go ahead and try it, you can use it in any front-end application with any it's JavaScript.

D

Basically, and you can integrate it with any graphql client that you might already use I mean just acts as an execution layer that gives you an enriched graphql experience. So all of these are supported and a lot of cool stuff, so yeah just give it a try and feedback is always welcome, um yeah. So that's all we're live. uh The second demo that I wanted to do today is um for the nested filters, so nested filters. uh I think this was the most requested feature uh for graph node in terms of the graphql layer.

D

So this is just landed recently like a month ago and I'm going to show here a cool example of how am I querying a partial set of the data based on complex filters. So, let's say here we have uh the Life beer, subgraph and I'm querying it for the pools entity.

D

So um let's say that I'm getting all the data here, I know that before we already had uh the wear filter that gives us the option to filter some parts of for some parts of data, but now for every linked entity that we have um within this entity, we're able to add more advanced filtering. So what what does it actually means? We can filter the pools based on the round entity and based on the value of the initialized field.

D

So, let's see an example here, uh the query that I have here is basically filtering the pools based on the initialized field.

D

So if I query, it you'll see that I'm only getting uh the pools entities that has the initialized value uh set to True uh if I'll go to the a bit more to the extreme, I can also do stuff, like um let's say, I want to get all the pools that are linked into a round of uh a specific value. But let's take this one.

D

For example, yeah I can just do here, um yeah ID, as for a specific ID yep, and now that's all the data that I'm getting and the ability to filter the root entity based on some nested Fields, basically means that we no longer need to fetch tons of data filter it on our end and then decide what to do with it.

D

We can just query the graph directly for the data that we actually need, um so this will probably reduce cost and will probably make things a lot simpler for everyone who consume data from the graph so yeah. This is already live uh and live so feel free to try it um and uh yeah just like with graph client. Any feedback is always welcome.

D

Yeah I try to keep it short. uh So if you have any questions, feel free to ask in chat or open an issue on GitHub.

A

Thanks right on time, we're only two minutes late, which is which is good: um okay, cool moving on um partosh. Are you taking lead on.

E

Umes cool yeah, exactly so hello, everyone, I'm Bartosh and our team match pigment. We've been working on the cosmos integration which basically brings the graph into the cosmos ecosystem, which I think opens up like a lot of a lot of opportunities. So let me briefly like really briefly talk about this integration and then I'll walk you through the the whole process of creating a subgraph building and compiling the deploying and so on.

E

So first of all, like the most important thing about this integration is that it uses the fire host framework developed by streaming fast and by that it gets all of the great features like substream is built in. So that's one thing. Second, uh we the way we implemented it. We we've done it on a tender level on the level of the tendering library.

E

So, as you may know, the tendering library uh is basically a consensus mechanism used for all the different chains in the cosmos ecosystem, which means that by doing that, there's actually only one integration that allows us to integrate pretty much all the different chains that exist in this ecosystem. So there's one fire hose that we need.

E

There's one graph node integration and the only required step to bring in like a new chain is instrumenting the the binary like the nodes, binary and so far, we've done it for Cosmos hub for osmosis, and did you know it's in the works too, but actually we've seen community building their own like the Integrations by themselves, which I think is really awesome. So uh yeah.

E

Let me move on to the to the example all right, so here's by the way, here's like I have a local Firehouse setup, which is running a cosmos, Hub node, which is connected to and then I have like a craft note, also running locally, which basically is connected to to our Powerhouse. So I'll show you one of the examples of graphs uh that we created or specifically for Cosmos and by that uh I'll probably demonstrate like what are the key differences between like ethereum, subgraphs and custom sub graphs.

E

So it will be this this one uh I have it already here.

E

Not this one, this one! Okay! So let's see the Manifest first, so here's the software final Fest and we can actually compare it to to the ethereum one. This is the gravity example.

E

E

Oh yeah, okay, so yes, as you see like first of all, the different kind of data source here interior in ethereum subgraphs, we have this ethereum uh contract kind, which is actually important because uh in case of the cosmos integration, there's actually one data source per each subgraph, because uh Cosmos chains are pretty much like smart contracts on ethereum. So this is like the key difference between those two ecosystems, so we can think of a of a chain and a cosmos ecosystem as a smart contract or, if you like, maybe with the exception of Juno.

E

But that's that's one of the one of the exceptions. So there will be most cases. There will be just one data source for a sub graph and it will be just like the network data source. So, as you see it's Cosmos kind, also the network is specifying because what's up here and the other difference that is important are the uh the handlers. So in ethereum there are free handlers, block Handler transaction and block Handler event handler and call Hunter, I think, but in uh in the cosmos, Integrations actually also free but different ones.

E

So we have a block handle which is pretty much the same. It gives you all of the data related to one block, there's an event handler, which is a bit different and I'll talk about it in a second and there's a transaction header which actually comes with all of the data for for all the transactions. So it contains all the data that that you can pull from from the chain. So let's go to the to the example and let's see the mapping.

E

So, as you see it's pretty similar, like the example I have here, is basically uh indexing. All of the validator rewards in a very simple way, so it's mostly to demonstrate how it works. It's not like a real use case, but yeah, it's just an example sub graph, so it takes like event data. So this is a where it's. It gets different from from ethereum, because event, data and tangerment and all like Thunderman based chains is basically a list of attributes which are which have keys and values which are both strings.

E

So this is like the data that we get from when we use the the event handler. So, as you see, I get the event. Data I'm actually interested in the reward event only which is specified here like this is the event type that I'm interested in and in the mapping actually pulling the data here and getting the reading the attributes, which is the the amount and validator images stored in the database in the software. So that's pretty pretty much it now. I think we can compile it okay.

E

So it's the usual like the flow is very similar, almost identical to the uh to the one. That's that you have for review. So it's.

E

First, like generating the code.

E

We'll link a subgraph, and now we can first like create it locally in our local instance, and then we can just deploy.

E

Yeah like it's, it's already started indexing in the in this local instance, and if we open up the uh yeah this uh this thing we can actually query it. So I have a like a really simple query here, defined which basically pulls like one uh address, so the rewards for one one violator and it's actually two returning all data so yeah. So this is how how the event handler looks like if we have a bit more time, I can demonstrate the other type. I, don't know Pedro.

A

If we have I think we have two minutes markers if you, okay, but.

E

I can fit in yeah sure yeah, so I'll show just another example, so I said about event handler, but there's also a transaction handle which gives you all of the data. The one caveat, though, is that here you need to decode the data on the subgraph side. This is like the major difference from from it here where you go to get all the data up front, so decode it up front. So, um let's see actually how it looks. So the definition is here in the Manifest files are pretty similar, pretty easy.

E

So it's it's just like specifying the it's a transaction Handler which basically Triggers on every transaction. uh So there's like one one um Handler function, 100 method or for the for the whole subgraph and here's how the mapping looks like. So we have the Handler function. It receives the transaction data which then leads to like the data that we get and specifically the messages. So messages are kind of like there are multiple messages in each column: image yeah, Customs transaction. So this is how like decoding on the subgraph side looks like for this.

E

We use a library called as Proto it. Basically generates the bindings like protocol bindings for assembly script, uh and it's actually like the decoding boilerplates that we are like generated beforehand. It's actually here so so yeah and it's basically imported in this file and used to decode the in this case. It's like a delegation message which is done later on to start like just in in the previous example, so yeah.

E

So this is how it looks like it's actually one more thing so I the whole, like the coding part, is on the subgroup developer end, but to make the subgroup developers lives easier. We actually uh created a library which comes like with all the all those buying bindings like protocol bindings out of the box. So once you like, this is the only the only thing that you need to really interact with with this chain, basically to decode the messages, all the messages for that particular chain.

E

In this case it's just Cosmos, Hub, uh Network, so yeah so I think that's all yeah. Thank you.

A

Perfect thanks, Bartos I'm glad you did decide to walk us through this last bit. I think it's important for folks to that are mostly uh um used to build Sub, sub graphs for evm base chains, and you know smart contracts, and then you get the whole data model that you know how to work with. It. I think this one is fairly different, but I hope that's clear now. So.

E

A

Thanks, thank you. Okay, Adam! uh You, you ready, I, don't see.

A

Adam, hey there, you go.

B

I'm sure, let's share some slides. um Yes, this is very much a um I guess. Work in motion, um I, guess for a bunch of the other stuff is, is excellent. Work already done. um We thought we'd be um good to run through um so streams. Obviously, we've been talking about these uh across the ecosystem for a while um uh we're kind of in the uh and now I guess into the fun bit of um building out on building.

B

On top of the original, um the original implementation stream faster than making great um great progress, um leading the charge there um and I guess what we're going to be focusing on um today is more the current collaboration between uh streaming fast and Edge node, uh where we're working on essentially uh taking sub streams and streaming them into um into graph node. um So, essentially, your sub streams uh process can make its way into a sort of an object generated, subgraph API.

B

So um hopefully this can be quite discussive, um uh but we'll see how see how that goes. I think the um in terms of overall status, so teams internally next time, you're experimenting sub streams, which is really cool to see um across a bunch of networks. uh One of the beaches of substreams is that if you've got a FiOS and now we've got fire hoses um across um obviously evm chains uh near Cosmos um Solana.

B

If you have a, uh if you have photos, you can run subscriptions, which is a really cool cool unlock for all of those ecosystems and we're seeing experimentation across across the Beast um in draft with them. It's starting to see some of the limitations or hurdles that uh the teams are hitting up against.

B

So one example within um evm mix specifically, is essentially just having to write a lot of boilerplate code um when extracting uh say events um which is sort of done uh or can be streamlined um via sort of Cogen uh from the sort of a small contract API.

B

So um those kind of help with those kind of learnings are are coming up and it's called see, I think I think Matt uh who's not on the call I'm jumped on that and as a as a draft um PR, um which should hopefully just make it much easier to go from xero to uh sort of working substream without running so much right and there's a actually really good documentation at substrings.stringfast.io. Don't know what the Ampersand is doing.

B

There um but that's um a great place if you want to sort of get started and I guess have more notes on that at the end, um so that substream's course stuff um sub streams and sub graphs published uh the Gip for this on on the Forum earlier this week.

B

um At the outset, this is a really simple implementation in graph mode, which essentially Imports the sub streams package um uh in in the Manifesto It's, a new, manifest format versus the um versus the formats um which I guess most support drives to be familiar with um essentially Imports of a Sub sub streams package and uh persists the data from um from that substream um into postgres, and it's then queryable via the usual subgraph API.

B

um So that's yeah, as I mentioned business of collaborative effort um between uh between teams, Philippe bism along this from Edge node, eduard um and Julian from uh streaming fast, and so uh the point that we're currently at is that uh we've got the sort of uh a working uh like working example, end-to-end, which is really cool to see.

B

There's some uh rough bits around the edges, which was um which we're ironing out, um but essentially we're at the point where um we want to start to push that proof of concept a little more um and I guess just wanted to raise. A couple of the discussion points which came up in yesterday's sort of working group um working group calls so um I know a couple of people are on on the call um something for Loopers and I, think Edward and Charles from streaming faster.

B

There, too, um so do feel free to jump in, but just wanted to raise some of the things that we're discussing um so, firstly, um validating the initial graph. Node integration um I think there's the initial step, which is uh going from and I believe they can correct me here, but I think it's it's kind of a pretty simple block, Handler that we've been using um for the initial testing to um essentially importing and seeing how more complex substreams perform uh so obviously amms.

B

So your uni swap um uh types of dexes um are um really applicable to sub streams, because there's a lot of data um and uh subgraphs have often um struggled to uh process them uh in as performing and as one they're driving things behind. um Substream so I'm I think I I think that's one of the places we're going to start I'm going beyond the initial hello world example.

B

um uh I guess then, the next sort of like as as well as just making sure that things work, um there's a sort of ensuring correctness during live operation.

B

um I think that's everything from seeing how the whole system handles when uh the whole system reacts when one or both components restarts or loses connectivity, uh making sure that the whole thing works and recovers, gracefully um and sort of, as is expected. So I think things we've seen um I guess over the months and years um is just making sure that uh failures.

B

Work is expected that when you lose a connection, maybe when you're part way through indexing, a block that you sort of start in the right place um and and those sort of detailed, detailed things are often things that I'm like uh really important to get to get right um at the early stages.

B

um So those sort of correctness things during operation, Wiki key consideration, testing determinism, so I think this is a.

B

um This is um obviously something that's really important, um but essentially the simple fact that a substream um being imported into a subgraph should uh essentially have the same outcome, whoever's running it and it should have same outcome every single time. um So we'll want to do some sort of testing there and yeah those things I mentioned before around handing failures and um and restarts, and those kind of things will um will need to be part of that testing um performance, but I think I.

B

Think then one other consideration is obviously always checking versus the chain and checking versus what you'd see in a sort of equivalent subgraph will be interesting benchmarks for us um and then uh once I guess we have um that confidence in the sort of correctness and determinism getting into performance benchmarking, scalability benchmarking, because that's how does it? How does the whole um setup perform when you've got hundreds of substream-powered subgraphs um being indexed by one graph node?

B

um How does it work when you have a particularly heavy um substream connecting to um connecting the substream server? um That does it? Does it perform as as expression, so I think that there's a bunch of local um and sort of simple testing that we'll do before I guess we move to I'm testing and I'm, also posted, hosted environment? Is there anything I guess I've missed from the people, people on on the line as we're sort of thinking about validating the initial integration.

A

E

I think you're good cool thanks.

B

Adam um so then um I think one interesting discussion, that's kind of come up. Is that sort of tooling um to build and deploy so super sub streams.

B

um Your folks be developing um substreams in Rust using a sort of rust tool chain, um but obviously the familiar means of deploying a subgraph is with graph CLI, which obviously in JavaScript, and so it's npm and I think we need to work through uh what that looks like, because this is something that we I think discussed yesterday, where I think the working assumption was that um it was the graph that was that developers would expect to be able to deploy these sort of substream powered subgraphs with graph CLI, um and so we need to update graph Clyde support substreams, but I think that Counterpoint, which Felipe made, which I thought was a really interesting um was really interesting, is was that if substream developers are like working um uh in the sort of substreams context and they want to deploy, they weren't necessarily want to import this sort of npm package.

B

It might be a whole um different workflow that they use um for develop, deploying subgraphs normally.

B

um So uh this is a proposal that actually you could take I, guess a more rust native approach for a sort of cargo, plugin um and so I think it really it's an interesting point which I think we need to kind of kind of work through. What is simple makes sense from a um from a sort of substream and subgraph developer perspective, um yeah, Philippe I, don't know um because I know we discussed this yesterday if you're thinking has sort of evolved but um on this tool.

C

um No I was like I think the I think an interesting thing that we could discuss is.

A

Starting to slowly move some parts into rust and then just create some bindings on graph, CLI and so reuse. So we don't have the choice.

C

Of you know one or the other, but instead it's kind of considered.

A

An option where we do the work and reuse it on on graph CLR.

C

um But yeah I think basically mention everything else.

B

Yeah, if Vincent's still there I think you were, you were pretty open to I, guess the rest based approach from a sort of developer perspective.

B

A

So, listen to you, you can talk. You know.

E

Oh uh yeah, sorry I, wasn't able to speak. uh I might be missing some compact on this one, uh but I setting up a call with Matt.

A

Afterwards or next week to to talk more so.

E

Far I'm open to other options, yeah.

B

It makes sense um cool um so then um I think on, like a like on a related note, I think there's an interesting discussion yesterday about school unconscious of time um about sort of schema graph about definitions.

B

So this is essentially there's one slightly gnarly piece between substrate and sub graphs, where the sort of um substream-based sort of Proto, above definition um needs to then map to the sort of skew graphql schema, that's used by graph node, um it's obviously we don't want that to be necessary in the long term requires sort of manual manual typing by developers and mapping, and ideally we have tooling um that helps there, but I think there's an interesting question as to whether you sort of start with the schema and that then um informs your sort of substream scruff out definitions um or the other way around and you'll graph out sort of has an auto-generated, graphql, schema, I, think I think the direction travel is that starts, starting with the schema is maybe makes a bit more sense.

B

um I think this will be an interesting area discussion that may be informed by some of the tool chain um tool chain stuff. We did um we just mentioned um and then uh very quickly, um yeah I think that's.

E

B

Maybe I think there's one missing: there's a slide which I was expecting which isn't there, but that was more about. There are some interesting challenges around um productionizing, the substream um substream server, um which again comes back to some of the testing, we're talking about I've like how to handle scale, how to handle load, how to degrade gracefully, um and so those are the sort of things that will only sort of start to see, as we start to test test more and more subgraphs and sub streams, um so I think in terms of time.

B

I think we're just at time now Pedro but I'm. Getting involved, tricep streams out um substreams.streamfast.io review the GOP on the Forum um and get in touch if you're, trying out substreams if you're hitting adults in the Discord, the substrings channel um and yeah engaging on the forums also I'm strongly encourage so or on GitHub as well. So um lots of channels, maybe that's more confusing, um but uh we're excited. So many people are giving it a go and um yeah we're. Looking forward to getting this out in the world foreign.

A

But I want to give folks the opportunity to raise their hand and ask questions if they want, assuming that some of you may stay for a couple more minutes.

A

There you go Matthew all over okay, let's start with Matthew yeah sure.

E

Thanks uh just a quick question to Petco I'm, just wondering: what's the roadmap, looking for you for the next steps for graph cast and yeah.

B

How do you see this evolving.

C

Thanks Matthew, uh that's actually a great question. um The next steps are basically pretty much are basically finishing up the uh proof of concept, which is now on GitHub. After a proof of concept, we are looking to develop a MVP that actually Network participants will be able to use and to give real feedback, because right now the proof of concept isn't really usable.

C

um So that is expected to happen sometime within the next month um at Max. So we'll definitely have a announcement when the proof of concept is stabilized is sort of moved some a more MVP stage and um yeah, then it will be usable by indexers. So that's like the short term roadmap and then once we get some actual feedback on the MVP, we're going to develop a much more formal, um formal sort of duck on the roadmap after the MVP I hope that answers your question. Yeah.

E

Absolutely awesome stuff.

A

Matthew uh Oliver, you can speak now, hey guys, also a question for Petco here um with regards to the gossip Network. Do you see and if so, how would you assess potential regulatory concerns that a gossip Network could facilitate collusion amongst indexers, such as price fix.

C

ah That is another amazing question: Oliver thanks a lot for it.

C

Unfortunately, that is way too big of a topic to unwrap here, um specifically about the price fixing, because I know it's a very um interesting topic and I I just want to say that um ultimately, uh the gossip Network won't stop people from doing that, but we will uh do our best um with with the help of The Wider community, of course, to um kind of well not enforced, but um add some guarantees and some rules and ultimately A reputational system that that kind of makes sure that everything is um as Fair as it could be.

C

In this context, but again yeah, that's a great question and we we might actually write a blog post or something on the topic, because it, it really is huge.

A

Okay, thanks again, I think great question. Oliver thanks for sharing um and I see. Edward is also sharing a link to substream stocks. That's substreams.streamingfast.io, I believe Adam also mentioned it and I don't see anyone raising hands anymore.

A

If we don't have any more questions, no I don't think we have okay, so we should wrap it up here again. Thank you all for joining this will be uploaded to YouTube and yeah I'll. See you soon. Thank you so much foreign.