Graph Protocol Developer Highlights, 5 Apr 2021

Previous Meeting

⏯

youtube image

►

From YouTube: Developer Highlights #1 - “Future proofing your subgraph with dynamic data sources”

Description

Juanmardefago will teach us how we can utilize dynamic data sources (templates) to make our subgraph a bit more future proof. If your dApp deploys contracts using a Factory or Proxy patterns, this video will be invaluable to you! Through a practical example, you will learn to leverage templates in your own subgraph, as well as learn the limitations of this method.

00:00 Intro
03:46 Using templates
17:42 What handlers can be used?
19:43 What are the benefits of using an event handler?
28:48 Can you filter which contracts are indexed?
31:35 Using templates with proxies
32:42 What are proxies ?
35:13 What if a new datasource need data from prior blocks?
37:36 Possible workaround.

The Graph Webpage: https://thegraph.com/
The Graph Discord https://discord.gg/vtvv7FP
The Graph Forum: https://forum.thegraph.com/

A

Welcome to the first developer, highlifes photograph with me: today we have a an active community member and a subgraph developer, beast juan. How are you.

B

Hi there guys I'm fine thanks and you.

A

Yeah, I'm great so I'm like excited to have you on. You have been working with uma, orgo version 2 and compound version, 2 and also the graph network. How is that? Well.

B

It's been a while right. Let's say it like that right, so lots of work, especially as played with the the graph network subgraph there's been lots of work, and there still is a lot of work like going on, probably behind the scenes for many people, but there's a lot of work going on yeah.

A

So how did you first get involved with the graph.

B

Well, I actually started working with the graph when I first started working in web 3. Actually because I was I worked in, I started working with protofire and like the first thing they wanted me to do was to start working with subgraphs and I didn't know much about web3. But at the moment like I was just like reading stuff about the ethereum and blockchain and whatever so yeah. I basically went like straight into the support development side of stuff, so I kind of started working there.

B

First, uh of course I didn't have much solidity expertise yet so I kinda studied. While I was developing subgraphs, so it was like trying to learn two things at the same time right so how to develop sub graphs and how to read the code that I have to like support with the subgraph, so it was kind of like it was challenging for sure, but it was like really really fun.

A

Yeah we had stuff getting or finding resources to learn working on subgraph development, at least as early as you were. So we hope to be able to supplement that with the with these videos and more so.

B

Like there wasn't much documentation more than like the official docs right, so it was. Thankfully, I had like a few colleagues that were a little bit more experienced than me in the subgraph development like category or whatever and they've been working a year prior to when I started so like two years prior to now. So they had a lot of experience.

B

But of course, the graph back then like a year a year ago, like was still not even it didn't even have a test net yet, like most staff were just being developed for ethereum and on the hosted service, and it was a little bit well, not a little bit, but a lot smaller than it is now. So the documentation was only like the first versions of the official documentation.

B

It was useful. You could do a lot of stuff with it, but you kind of have to like have a little bit of uh say, curious brain. You need to be really curious and and how to try stuff yourself and then see if that works or not.

B

Thankfully, like we had a lot of support from the chordograph team, which is now called each note, but back then we knew it as the graph and they they were like the draft team, so they they were really helpful like teaching us how to do stuff and what not. So that was really really really helpful.

A

Yeah, what what are we going to learn about today.

B

Well, we're going to try to at least try to right to cover like some ways to future proof or future proof, your subgraphs using templates.

B

uh Basically, the idea is that you could like harness the factory and proxy patterns for solidity development and like use them in a way that you don't really need to update yours, your called so often as long as there's there isn't like many big changes to even signatures, or whatever, like you, could pretty much get away with just deploying a first version and unless there's like a really big change at some point in the in the protocol that you're trying to support, you won't really have the need to update it.

B

So yeah, that's what what we are going to try to cover today. Let me share my screen.

B

Okay, so I'm gonna use like one of the subgroups that I made in the in the past, which is the lubrin version 3.1, and basically this one uses a factory pattern right, so we'll go first showing the this one one. Second, so to be very, like brief, with the domain that we're going to cover, basically the lubric contracts, like you have like the main lubricant contract, which is uh this one, and this one is basically a factory, it will be creating or or actually it's more like a registry, but it's basically a factory.

B

What this means is like this one is going to be uh triggering events right whenever a new exchange or a new contract or whatever is created in this case. uh This one actually tells us when a new exchange is being created so and the exchange is the actual contract that we actually want to track.

B

All the events from so we're going to be harnessing the factory type of pattern that this this contract is using to be able to avoid having to hard code new exchanges whenever they become available, and we will have all those exchanges data available for us. So, basically, just to give you a quick run rundown.

B

These are the exchanges like there's only two, but there could be many more. So um it's basically the looping, this internet exchange and the wii the central exchange. I think it's called we or whatever, and if we go to the implementation that I prepared this one is like the example.

B

Without the the template, it's just using a data source uh not dynamic, but rather a like fixed data source, a static data source, and whenever you use static data sources, we basically have to define a name for the data source which avi is going to use in this. In this case, it's just the exchange version 3 api that we're going to use the other is for said data source, and this one is optional. But it's really useful.

B

If you need to like speed up your syncing speeds, uh you could actually like tell the the subgraph d block that it has to start syncing for that specific data source. So, basically, this field is like it lets. You avoid having to sync 9 million blocks before starting because those nine minion blocks won't be having any data that we are actually going to be using. You can actually check that this is a small pro tip. I guess, but you could check the the start block for any contract going to either scan.

B

This is useful for for each network, but it most likely any network has an explorer like etherscan, so you could probably use it on an others too. So you go to the contract that you want to check when it was first created you go to the transaction, you open the link and you basically have the block here. This is probably it's too small. I don't know. Well, you have the block here, so you could copy paste this like just copy here, and it's gonna be this one.

B

Of course it didn't intend properly there we go and well basically without the templates. What you're going to want to be doing is like defining the event handlers as you would normally and you will have basically everything set for your manifest right. You have the address. You have the avi all the abs that it's going to be using it's also using an erc20 api, because it's tracking a few transactions here and there uh or actually it's tracking, I think the decimals for the tokens. So we're gonna see that later.

B

Don't worry so then like we could see that we have we're going to be tracking accounts for set exchange, all the tokens registered for set exchange and something on the block which is using the print. Basically it like batches a lot of transactions, it's a rollup, a layer to roll up it patches a lot of transactions and like transfer them to a layer, 1 ethereum in this case, as a form of a block with a lot of transactions inside.

B

uh So basically, you will be. We will be tracking, like account, created account, updated block, committed and token registry.

B

uh With all these events, we could like generate these different entities here, but one of the drawbacks that we're going to be examining right now with the templating and whatnot. Since we are not using a sort of a dynamic way of indexing stuff, we are just like defining what we want from this very specific contract.

B

If we had like a clone contract of this one another exchange, for example, we would we would have to like copy paste all these data like so well. I have it here already and paste it and change a few things like, for example, changing the name and the others. If you see here, this address is not the same as this address and, of course, this name is not the same. It has to be unique just so we can actually index also the widex one.

B

As we see here, we have this contract, which is lubric one, and we also had this contract, which is also an exchange which is available on the low bring um lubric registry contract or, however, you want to call it factory, contract or whatever, and so we actually want to avoid having to manually add anything to our manifest because, as you might expect, if we don't have this, we won't be able to index this exchange in particular.

B

But if we wanted to start indexing in the future, we would have to re-index everything we will have to deploy manually, create this so manually like copy-paste this code and actually deploy it again, so we could start indexing, so we will be like having to manually, update everything and it really doesn't make any sense when we can like do it in a better way.

B

So, the way we are going to be refactoring this cod, this call- I actually already did it because we want to save a little bit more time, but the way we are going to change this is to use something like this. Let's go to the other country. So, as we said earlier, we already knew that there was a factory contract which was called blueprint, version 3, which is this one. I guess yeah this one has a specific event. If I'm not mistaken, it was uh how was it going?

B

Let me check exchange initialize, so we could actually look it up here. This event will disappear. There we go. This event will allow us to like know the addresses that we want to index for all the exchanges that are going to be registered in this in this factory or registered contract, and since we have an event, then we can use as a as a hook, and we can listen for these events. We will. We will know whenever a new exchange is created or actually initialized.

B

In this case, we will know the address for that event, and this is exactly what we need when we are using templates to be able to, let's say, future proof, our subgraphs and have to up and basically avoid having to manually, update the support of code. Whenever a new exchange is available or a new contract, it doesn't really have to be an exchange, it could be whatever.

B

So, if we look at this implementation, we are, we could be actually looking at a lot of other events. The this loop ring factory contract has a lot of events, but the only one that we are really interested here just for this example is the exchange initialize right. So, let's see what we are actually doing with that specific piece of code, so on the handler I don't know if you can see the code well is.

A

B

Big enough, it's good cool, so in the handler we are going to be like creating an exchange. This is just to track to have the entity, so we like have some persistence of the data. This is isn't really needed for the example to work, but we are doing in it anyways.

B

We basically get a few a few parameters and save it as a some fields just to have a little bit of data and then the really important part is this one: we're going to be creating a that is a dynamic data source from the template. This one is a template. If you, if you go here like we are importing exchange v3 from the generated template. If we see like the file structure, we could go like generated templates and we have like this folder, which is template for the whole exchange v3 right.

B

So what this allows us to do is to call this specific method, which is create, with the exact address that we want to dynamic dynamically index at this very specific block, so just to put in in context every handler is called in a specific block. So part of the context of the handler is the block that is being processed at.

B

In this specific scenario, it will be called only in two locations, since we only have two exchanges and it will be called exactly in the block that that the exchange was initialized, which are the two blocks that we had in the other manifest which is like this one, nine million and something and the other, which is also nine million. Something, and so basically, what we need for for data sources to work is three things right: the avi that it's going to be using, which, in this example we already have, is the exchange p3.

B

Then we would also need the address, as we as you can see here, in the in the fixed or static data source declaration, and we also would need the start block. This is optional for the static static data sources, because, if you don't use it, it just assumes that it's zero, but since we are using dynamic data sources, it's been like inferred because of the context so like we have the avi here. The address for the set a contract here and the block is the block that this specific handler is being run at right.

B

So it would be like nine million something nine zero: zero, zero zero.

B

It's inferred because of the context right. So when we, when we call this function, it will basically generate for us this kind of stuff, like this static data source implementation, using this template right here. So it basically uses copy paste this and fills the gaps here. Templates don't have an address and maybe an uh start block, because it's used, as I said dynamically in the code, so it basically fills all the gaps that you need here like the other is here, fills it and the start lock here fills it by context right.

B

So the only thing that you actually need to define for templates is everything except that, so you define all the apis that it's going to be using all the handlers and functions that it's going to be calling.

B

But you don't really need to define the blocks the address and start blocks because it's going to be like inferred by the context and but whatever parameter you use for the others.

A

Will uh will the initialization always be an event, or will we sometimes need to use block handlers or another handler.

B

You can use whatever you want. You could use a block handler. You could use an event. You can use a call handler if it's supported by the that the change that change that you want to use, because some chains don't have call handlers and whatever, but there's no like difference.

B

You could call it wherever you want as long as it's a handler of course, because it's the only place where code is executed and the only thing that you would need to know is to have available the address that you will be using at the time of the execution.

B

So basically, we are using this specific event because it allows us to have the address available, as we said before. If we go to the browser, we knew beforehand that this specific contract would allow us allow us to know like the specific address for set exchange. So we could use it as a dynamic indexing. uh Let's say entry point right, so yeah, that's basically the the only like caveat you need to know the address that you are going to be dynamically indexing beforehand or at least within the context of says handler.

B

It could be a call, it could be whatever you want right and if we actually check it, it's probably being emitted in a call right. So yeah this is being emitted in the initialize exchange function and that function also has the exchange address as an input. So we could actually use the call handler if you wanted, it's probably not recommended. If you have an event.

A

What are the benefits.

B

A

An event handler.

B

It will be a little bit more performant and it's more compatible. So, like call, handlers need to have very specific um nodes underlying nodes uh for ethereum or whatever, to be able to use call handlers. You need to have an underlying node that has traces available and only like open ethereum versions of the underlying nodes work for that a specific purpose.

B

So other chains, like I don't know, binance or polygon or whatever, don't really use open three nodes on the under underlying graph node. So you don't have the call handlers available for those change chains. So, in that case like using an event, it's uh like the best option, both as uh performance improvement.

B

I mean it's not going to be called a lot right because it's only going to be called a few times whenever a next, a new exchange or a new contract is, is created, but events are usually easier on the indexing computing performance, let's say, and it's more available for any chain that you might use. So it's like um a rule of thumb right you it's better to use even hundreds.

B

If you have them available, if you don't have have them, have them available, you could use call handlers or block handlers or whatever, as long as you have the required data to trigger a dynamic indexing right so yeah. I guess that might like answer your question cool. So if we go back to the code now that we actually triggered this create function, we will basically have let's go to the manifest a new data source. With that address that I used there, which will be indexing everything here.

B

So all these handlers will be called or should be called at least if they, if these events are triggered at some point in the exchange and we will have like- and there we go, this handler has been called and this code being executed, as we would expect in the hard-coded version.

B

So if we go to the examples I already deployed them, uh it's the one with templates is still indexing, because it's a little bit more heavy. I guess well, both are kind of heavy. Well, as you can see, the one without uh template is a little bit faster.

B

uh One of the reasons is that it's only using one of the two exchanges, since I didn't like copy paste, the other one, so it like managed to go to 90 when the other one is only at 83 percent and they are most deployed at the same time an hour ago and 44 minutes ago.

B

So, of course, if you don't have a solution that scales with time, it will most likely be handling less data, so it will most likely index faster unless the data sets are similar right, the amount of data they have to handle, um that's one of the let's say, differences between both solutions. But of course, if you don't really have like the automatic scaling or automatically dynamic scaling that templates allow.

B

You of course, are missing on a ton of data, and if you would want to like index that part of the of this of the contracts, you would actually need to manually deploy it. So I would have to like go to my code, which is here right. I will have to go. Where is my code?

B

A

B

More, oh, there we go it's here. I would have to go to the manifest, as I told you earlier copy, this one save it and then I will have to like generate the code just in case I don't think it's needed, but let's turn out the code and then deploy it right.

B

No, no! No! No! It basically builds the files that we need. Hopefully it works all right since the hosted service has been having a few issues as of late.

B

It's deploying awesome, it deployed perfectly awesome and we will have to refresh this and oh, I had already played with the other decks. Well, never mind, uh yeah, it's it's basically 90! uh Well, yeah, that's mostly it. uh You would actually need to manually, deploy and, let's say, there's a new there's. A new exchange here that we wanted to have that we don't already index would have to like copy it again and paste it fixing anything because it didn't paste it right.

B

Here we go and change the name here, blah blah blah and change the address and whatever and check that everything is working and then deploy again and indexing with resume from the very beginning so yeah. This is really not an ideal situation. If you have like a specific contract that creates a lot of other contracts. Let's say it creates like 10 contracts per month.

B

You would have to like update it at least once every month, if you like, wanted to have it at least somewhat up to date, or you might even need to manually change it 10 times every month. If you wanted to have it extremely up-to-date and you would suffer from all the syncing time every time because it would start over, basically, you.

A

Would also suffer from one's curation goes live on mainnet. The developer would also have to cover half the curation tags for the curators, who are following the latest version. We will get to know more about this pretty soon, and uh that would be.

B

A

Enormous cost, as well for developers if they were to upgrade from every new factory initialized contract, so.

B

Yeah, I actually haven't thought about it, but yeah you would basically have new deployments like every time. A new, a new contract goes live from the factory one uh which you wouldn't actually have to do if you were using like templates to cover this particular use case. So yeah, that's a really good point, it's incredibly useful for that specific thing, yeah, so that pretty much covers how you would use templates. It's not really that difficult. Once you get the hang of it, it's just like calling a simple line in which it's the create one.

B

It obviously have some implications on how you will debug it further because since you are using like let's say you have like a hundred new contracts uh deployed on the factory one, then you would have like a hundred data sources that could fail at any point and you would have like a hundred contracts that you might need to check um so yeah. It's like it's not perfect.

B

You might like go into some issues that you might not understand at first, but it's not really that different from deploying uh static data sources right. So the only difference is that you would be using templates and you would have to understand that there are many contracts that you are actually using, even if your code on the states a single address right. So in this example, this is the hard-coded one. We have two addresses, so we know if something fails, it's either in this address or in this address the issue right.

B

But if you have like templates, if we go to this manifest, we actually don't know the addresses beforehand. Of course, we we are, as I told you earlier, we are saving those addresses here in the exchange entity.

B

That's one of the reasons why I'm doing it, because I want to know the address of something if it fails, but you also only really know beforehand the address for the factory contract, but you don't know every address that might be implicated in an issue so yeah it makes things a little bit more complex, but it's pretty needed for these use cases right.

A

Can you in any way filter which which contracts you want to index, say that there's a factory contract that creates a lot of creating new contract for each pair, but you only want each pair. That includes usdc, for example. Is that possible to do? Is it possible to do some complex logic on which contracts you yeah.

B

Yeah yeah for sure you could, as you see here like since you are basically triggering those dynamic data sources inside a handler which is basically a piece of code you could like use like. I don't know, if um contains usdc.

B

Right I put this here, and it will only index it if it contains usb. Of course, you would have to like code yeah.

A

B

Contains usdc means uh you could you might be, you might need to like bind this address. Do a few contract calls check that this specific exchange has usdc in it or some like weird logic right like that right, but you could totally index it in a filtered manner and so yeah. You basically have full control of what you actually dynamically index and what you might want to avoid.

A

B

B

Yeah, it's really it's really useful. It can be used in many many different ways and it's quite a powerful tool. Yeah and you don't really need to update it much often well, at least unless something breaks right, but if it breaks it will work anyway. So you will have to update it manually in that case, but yeah it's. It's really really useful when you have like- and this probably isn't.

B

The best example is one of the examples that I could find easily and that I could like change easily, because the lubrin 3.1 subgraph, the the actual one that we deployed is incredibly more complex than this. I basically trimmed it out. So it's easier to to understand, but yeah there's, there's some other subgraphs that were like you might have, for example, in an exchange in a uni swap or whatever.

B

If you were to deploy different contracts to manage different, I don't know pools or farm strategies or I don't know pairs. You might really really find it really really useful to use templates for that specific thing. uh So long as you have like different contracts that you want to index right so yeah, it's it's incredibly useful for that specific tool yeah.

B

It might also work with like it might also work with like um proxies, but it really depends on whether on on, like the specific proxy implementation that you're, using like some proxy implementations, trigger all the events in the proxy address instead of in the implementation address.

B

So whenever that happens, you don't really need to index the implementation addresses, because it's it's not triggering everything. Anything actually so yeah in. In those specific cases when you're dealing with proxies instead of factories and like your mileage, will vary right. Sometimes it it's really useful. Sometimes it's! It doesn't really make sense, because all your events and calls will be triggered in the in the proxy address. So for proxies, it's really dependent on the exact proxy implementation that you're using, but it might be useful for, for anyone that tries to like index proxies but yeah.

B

A

About it so, first for for anyone who doesn't know what proxies are, could you just give us the oh.

B

Yeah one minute for that yeah yeah. So basically, as we've seen, factory implementations basically create new contracts uh under specific, like uh conditions like it might create under. Whenever you call a specific method, it creates a new contract or whatever proxy implementations, basically allow you to upgrade the logic on a smart contract.

B

So you basically deploy what is known as a proxy, which the only actual thing that it does is like send a call to another contract somewhere, which is where the implementation resides right. And since you basically have a pointer to an implementation within the proxy, you could change the implementation that it's pointing to at any time and in that way, like upgrade smart contracts and have like a specific address in smart contract. Others that you might change the implementation every so often to like update some critical bug or whatever.

B

In these specific cases, you might um want to use templates or not depending on how you are sending the transaction to the implementation from the proxy implementation right. Because if the events and the calls are, let's say, um emitted right in the proxy address, you won't really benefit from using templates because there won't be any events in the in the implementation addresses.

B

But if you, your proxy implementation, allows the address the implementation others to have their own events submitted, like I think it it has, has to do with delegate calls or whatever I'm not really sure. I'm not a solidity developer, I'm mostly a software developer, but sometimes it triggers in the events in the in the implementation address and in those specific cases you could treat them like factory contracts and do exactly the same thing that we did here.

B

You basically use templates and you pretty much have it all covered so, every time a proxy implementation changes, you just create a new dynamics, data source from that implementation, with the new address and url set.

A

What if a new data source needs data from fire blocks.

B

Yeah, that's a that's a really good question, like that's the main drawback with templates. Actually, since you won't be able to define at which point or at which block, it will start indexing right, the dynamic data source- and it only like allows you to do it from the block that you triggered it.

B

If you need prior data, like let's say, 10 blocks behind, because the registry contract registered it like not at the exact same time that the contract was created, you won't be able to have access to that data, because the way that subgroup indexing works is it did with deterministic. You can't like generate a new data source that has to go to the back, uh like let's say a hundred blocks back or I don't know a thousand blocks back and change the entities for those blocks.

B

Since you like this state of the whole subgraph needs to be the same at any point in time, you can't like change it with dynamic data sources. So that's basically one of the main issues with dynamic data sources right now, mainly not so really complicated for factory contracts, but it might be an issue with registry contracts since most of the time registered contracts. My might like um register the the exact implementation or contract or whatever it's registering at a later point than when it was first created.

B

So there might be like a gap of data that you don't have access to. That's one of many issues. You could argue that if you have access like, if you are developing these smart contracts, you could like take that in mind and make sure that whenever a contract is registered in a registry, it's registered at the point at which that specific contract started existing in the blockchain.

B

So you don't have this issue. Of course, if you are only a subgraph developer, then you don't really have a way to change how the contracts work and whatever that's one of the main limitations that I can think of right now. So yeah, it's a really really good question.

B

Yeah, it might be a possible workaround yeah.

A

B

It might not work every time because, let's say you you, prior to the point that you dynamically started indexing right. You would need to reconstruct the state from the past basically and the way that you would handle it.

B

As stated in the docs, it's like trying to query the contract itself to check the full current state of everything that you actually care about in the contract, which might be something really heavy computationally speaking right, but even if it's not like it won't be that that specific data set that you try to like support from the past, won't be created in the same way as you will for the all the data that will be coming in events or calls or whatever.

B

So you might not be able to reconstruct it like a hundred percent accurately and a hundred percent full. Let's say there might be some missing data points or some missing.

B

I don't know some missing fields for the entities that you try to store because those specific fields only appear in the events or the calls, and it's not like present in the in the contract state right, because you have to remember that um subgraphs, don't only like take the blockchain data and store it, as is in a postgres database and then like, allow you to query it in a graphical manner.

B

uh Some graphs also process data so in in an event, hunter and a call hunter. You might change a little bit of data. You might calculate some extra fields you might like, I don't know, generate some analytics data or whatever, if you have like a really really complex, upgrade and all those pieces that you will be able to create by handling the events and and having everything at the moment that it actually appears in the in the blockchain, isn't exactly the same as just like.

B

Reading the current state for the the contract and copy pasting it to the to the database right, because you will miss like all these. These fine details, like the analytics data and like the the calculated stuff or whatever so yeah it could be a valid workaround.

B

If you absolutely need to index this in a dynamically way, and you need to like support it with templates and unfortunately it's a registry and unfortunately the registry isn't exactly.

B

It has some some missing information, because it's like registered at the later point and you need to to generate a workaround because you need it to to to index right. I mean it's, it might be decent enough. It won't it probably or most likely won't be exactly the same.

B

So as long as you're fine with that it works, and as long as you like, taking into consideration when you are like trying to consume the subgraph data, but it's probably better, if, like you, could have a factory contract that actually does it the right way, let's say and of course no one's perfect. There might be some changes when you are deploying smart contracts, so so workarounds are some sometimes needed, and this one is a really really decent one.

B

But you have to remember that it won't be like exactly the same as having the data available at the expected time.

A

uh Thank you, everyone for being on that was really insightful and we've learned about subgraph using dynamic data sources for subgraphs. So thank you and see you all in the next episode.

B

You're welcome guys bye-bye see you in the next.

B