Graph Protocol The Graph - MIPs Program, 30 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Graph - MIPs Workshop - Running a Gnosis Chain Archive Node with Nethermind

Description

The Graph is an indexing protocol for querying networks like Ethereum and IPFS. Anyone can build and publish open APIs, called subgraphs, making data easily accessible.

Follow The Graph on social media
Twitter: https://twitter.com/graphprotocol?s=20
Instagram: https://instagram.com/graphprotocol
LinkedIn: https://www.linkedin.com/company/thegraph/
GitHub: https://github.com/graphprotocol
Website: https://thegraph.com

A

Hello and welcome back to another mips Workshop. Thank you all for attending I'm excited to get into today's conversation.

A

um We have the gnosis team with us today, which I'm very excited about. They are going to be teaching us about the best practices for running an archive node in the gnosis chain, and so we have Ricky here who's going to do an introduction, and let us know a bit about himself.

B

So hi guys so yeah um has already said basically I'm involved with devops at niosis.

B

We try to do our best to keep uh niosis blockchain service is running and helping out other teams to to achieve their goals with projects around the news's chain. Yep.

A

Wonderful great well, um let's get into today's conversation, so I'm just going to go ahead and share my screen, so folks can see the presentation that we have just bear with me. One moment.

C

A

A

Great awesome cool well um also, if jackmo's still here, but um if, if not then um Ricky, if you could uh take us into the presentation, that would be fantastic.

B

So, yes, sure so running so, as you all know, guys, uh ethereum is uh now merged. This does mean that basically, the beacon chain now does do the staking on top of the execution layer.

B

This is something that hasn't happened at the yet on the gnosis Chain, because, as you know, it's uh it's a little bit different from ethereum.

B

We run on a different algorithm called Aura and we need to take all uh the proper consideration and we are doing all the proper tests with Shadow Forks Etc in order to do the merge properly and we hope to achieve this- uh I mean as soon as possible, because the merges are really important because and for any blockchain really because I mean a lot of Advantage so scaling and also is much better for the environment. So the sooner the blockchain will get there. The better is oh yeah. We can.

B

We can continue with the next slide, yeah, okay, how to run an archive node onions. So since we are still premiered, we will focus on the execution layer. So, in this case, with nethermind, you know there are different kind of clients that you could use in order to to run the execution layer on a blockchain, and we will focus today on nethermind and we're prepared for you um a database snapshot that is hosted and yeah. Basically, since we need to provide this database in a secure way, we have all studied behind the CDN.

B

So in this way you bought us- and you will will basically be able to to get that snapshot that possibly quickly and also in a way that is secure and also we needed to create uh downloader app uh due to some limitation of the CDN that will not allow to download the ADB. That is more than 30 gigabyte. But currently our database is around two terabyte. So, given this uh Baseline, we basically needed to find a solution that was a user friendly for you and it was also, let's say, cost effective and savior for us.

B

So, let's start with the requirements of your machine, so you could run another mind in any in any news we possibly will with Ubuntu, because it's the the deals that is mostly used for running anode, then CPU to course Ram 8, gig disk. It should be a higher than this. So in order to in order to be able to to get the the database and running properly, so we usually have a disk of around at least three terabyte and uh yeah.

B

C

Sorry, let me let me visit me yeah.

B

Okay, so yeah, let's continue with uh with the other slide. Please please, if you want to add anything, feel free to to chime in in the in the discussion. Okay,.

A

B

Great so uh yeah I cannot see now the screen I've lost it.

B

Okay, okay, so hey I, I, don't know why sometimes I see the slide and other times it disappear. uh Maybe there is some issue.

D

I think you need to click, watch stream, yeah.

B

Yeah I'm in the watch stream, but sometimes I lose the the screen. It becomes all black. So I, don't.

D

B

It's I can see it so yeah perfect. So now I cannot see it again.

B

Yeah, there is some issue with this, like guys with the Stream.

A

um I could I can send you a copy of the presentation um and.

B

I can actually let me let me open in full screen, and maybe it's my browser. Okay, now I should be able to see it Okay. So, okay. So uh in order to to prepare for the merge, because, yes, we are not yet there, but uh if we are prepared this better. Basically, what are the client provider doing? They are embedding all the specification for the chain, so the the Genesis file uh inside and changing things inside the the actual client.

B

So, for instance, there are two events that should happen in order for the merge to succeed. First is the Bellatrix event and uh is what basically ethereum has done on the 6th of September and then a a terminal at the terminal. Total difficulty will be picked up. That is basically the point in time where the consensus layer is gonna start to lead the the charge. Basically, so it's gonna start to send in information to the to talk with the execution layer and uh and uh yeah basically is gonna pass.

B

The execution layer is gonna, pass the ball to the the consensus layer and we will start to talk about the staking and not anymore, about mining. Basically, so can you use CL client, I I? Will try also to to to answer to some question. Meanwhile, can we use an ecl client or is Lighthouse they only so a I. We are testing several ones.

B

We are testing Lighthouse prism, we are testing load star and techu and I would suggest to go uh with any of these four with the one that you with the one that you prefer so yeah, but uh I personally, like uh the the the couple, never mind the lighthouse so but it's just a personal taste when merge, ER I cannot give a date um because we do not yet know it and I'm, not the the person that it is gonna decide when it's gonna happen.

B

You should know that we are working relentlessly in order to achieve this goal so yeah, hopefully not the months so yeah.

B

Okay, so, uh let's see make sure so make sure you're running version of the execution layer, client that supported the merge yeah. So you have to update your execution layer and so, for instance, um I think we never mind are at 1.14.2, and this is the latest one so be sure that the execution layer is up to date and then make sure that the execution layer client is fully synced.

B

This is really really important, because otherwise, when you kick off the consensus layer and it's gonna start to talk with the execution layer, basically the consensus layer should have already a good overview of what is happening in in the blockchain. So the execution layer must be synced, and this is really important. You cannot to start them, maintain them without the execution layer being already synced.

B

So that's not a requirement for ethereum right.

B

C

B

E

B

C

B

Want my kids, it's on okay, perfect, so let's continue okay, so as soon as the execution layer is synced. What you have to do you have to uh so, basically, in order to allow the execution layer and the consensus layer to talk in a secure way.

B

There is a mechanism. Basically they do use a Json web token. This Json web token can be generated by you and we will share I mean one sec. I think there is a push to talk for Jacob. I will tell him.

C

B

Okay, sorry guys, so, let's continue. I was saying that, in order for the consensus layer and execution layer to talk, there should be a security mechanism that will allow them to talk uh in in a single way. Sorry for the repetition. In this case a the client have chosen to use a Json web token. This could be generated by you in different way.

B

There is no no proper way. We do use open SSL, for instance, to generate it on the Fly, and the one important thing is that one execution layer should be paired with just one consensus layer. So this is really important, no multiple consensus layer to one execution layer or vice versa.

B

It's a one-to-one relationship be between them, so other than this on either a API has been built by the execution layer, clients that is called the engine API and basically this is a is actually the the the way in which the consensus layer will communicate with the execution layer so having the engine Port opened and the Json web token set up for both the the nodes is going to allow them to talk and in the consensus layer, argument or environment variable you will specify to which execution layer should communicate your consensus layer node, basically so, and is.

B

It is basically the only difference, setup that you will have post merge so Json web token, plus opening this engine airport on the execution layer plus pointing and use the same Json web token on the consensus layer. That's it so get prepared for the merge, okay, so yeah we can go to the next slide. Thank you.

A

Next slide is a q, a I believe, okay, perfect! That's.

B

A

Right, um let's get into questions I see that people have already been asking questions on the way which is great um I'm. Just gonna unmute uh Giacomo see, if he's able to also hop into the conversation um and um yeah Ricky and Jack mode for free to answer any questions and everyone else feel free.

C

A

All right, I see a question from Chris in the chat.

B

Okay, is there any reason why I can use a standard, HTTP client call to download the archive snapshot? So yes, there is. uh There is a reason, so, as I was saying basically for security, a reason since it's a big file and we could get the fluid of requests, so we could be attacked because uh niosis is uh I mean sometimes I can do. Take us as a Target, so we needed to secure it in a proper way. So the file has been put behind the CDN and unfortunately, CDN have got a limitation.

B

In this case. A cloud front does not allow you to download the file cached that are bigger than 30 gig. So, in order to achieve a full download, we needed to provide I mean it's possible also to do with code, and but you will need to it- wouldn't been a user friendly, so we provided the download manager, and so this download manager does chunk the file in multiple pieces and then, after you have downloaded it, it merged them together. So it was just to facilitate you guys to download it.

B

The only thing since it was done quite quickly in a few days, we didn't have much time for testing it, but it's a work in progress, It's, a public repo. So for any issue, please write an issue and we will take care of it as soon as possible. Exactly.

B

So we can do a stream download an extraction with two weeks exactly yeah. There should be a little. You are absolutely right. There should be a little bit more space in in your disk yep.

C

But what about effects or.

B

Hey I mean we we could we, we really had the uh I'm using to use it you're right guys. uh We, uh we will gather all this feedback and provide a better solution. That is uh I mean that is good for you for the community, so yeah. We are learning by doing at the moment, so yeah.

C

And these are the form from phase zero. These are girly interior matters of your.

B

Indexer I will leave this possible all the graph related question directly to wobble a few Apple can chime in for those that would interest you please.

A

Sure have a look yep.

B

So, actually, you can start from a specific part, uh I I'm answering to a Chris dottete. If you add a dash p and let's say you want to start on a part 230., you can restart from there.

B

I know I know, I mean I, know that the software should simplify our life, but really that the tool was put together in a few days. So it's uh it's not any anywhere near uh being uh uh robust, but official request. We will take it in consideration and we will. We will try to do our best and really it's a public repo so feel free to rise issue. Also there, because I guess they would be taken, I mean I. I will leave it there and then you know the act on it. Basically, yeah.

B

F

Do you have or do you plan to have sport for the Nimbus consensus layer, client.

B

So a yes, we will for sure, because we will be. We would like to be on pair with with ethereum on that sensor. So, yes, I can say already. Yes, okay,.

C

Zero impression.

B

Okay, I I'm, taking notes guys at the moment, because our very great feedbacks.

C

B

Also guys, if, for any one of you that runs on AWS their nodes, it could be good to I mean we, we could even share the EBS snapshot that it will be like instantaneous. So you will never know the running in a few minutes really. So if that is an option for you, we can take it in consideration and share with you an EBS snapshot to your AWS account. So yeah.

D

That's awesome, Ricky. um How can people reach out for that like? Is there a um is, would you make the EBS snapshot broadly available and people just need the EBS snapshot identifier, or would you share it with specific accounts, I.

B

Mean, let's see because uh we could actually think uh on this with the graph team and base it on your feedback. We can find that the best solution that you think it's suitable, so yeah.

D

Okay and if, if somebody wanted to you, know bootstrap an AWS archive node today, would they just reach out to you? Is.

B

Yes, yes, yes, yes, you can reach out to me, but um I will need to to speak with uh other team and uh yeah. So I do not think that today will be possible. Maybe tomorrow we can have something set up so yeah, but anyway, as soon as possible, because we want you guys to run uh your nodes as soon as possible.

B

Okay I can see. Giacomo is also answering to the questions. A nice.

F

What is the refresh rate of the of the snapshot? Is it like? You made one's a snapshot and that's where we keep going for the next several months or is it like a daily or weekly uh fresh one? So.

B

We didn't uh actually talked about this, but also really guys your feedback will be gold in this case. So, based on your needs, we can decide like um a time frame which we are gonna upload. The new snapshots so yeah.

C

C

There's no, that made I.

B

Think uh yeah it was made in August. It's a it's quite the reason that so you really by using that snapshot, you should be able to be up to speed uh in uh really I mean yeah really quickly.

C

B

Hey guys thanks a lot for all your feedback, because uh you are helping us a lot.

B

Also, we we tried it actually and yeah. We.

B

I would say so: I would advise a five terabyte, at least at the moment, with the downloader. A medium five terabyte, but Jaco Morgan can I mean give his opinion. Also on this.

E

Yeah, so if you, if you, if you sing from scratch, then uh let's say the current size of the database of the Netherland is around 1.9 terabytes.

E

uh So at least, if you think from scratch, I will say three terabytes uh yeah no less than two terabytes but yeah. If you download the the copy of the database from yeah through the downloader app, then uh yeah, the double of the site, so yeah around five terabytes and also yeah uh having a more distributed version of the database would be great. So if you, if any one of Yeah Yeah from the community, uh should start a torrent. Also please reach out, and let's see how we can also contribute and also I, know.

E

uh I know that there are plans to yeah for the Netherland team to support uh to introduce support for uh snapshot, sync, which actually is only supported by GIF uh clients on mainnet, so hopefully in the next future. uh uh Yeah the Netherland client will support uh snapshot sync as well, so this will lower the amount of time needed to get up to speed on on the network.

E

Yeah, a snapshot sync is basically uh Chris is basically instead of thinking from scratch to basically download the copy of the database from appear.

E

So you don't have to go through the downloader app in the case, since you will be downloading the the copy of the database uh yeah that that would be automatically handled by the Netherland, Clan, client or well, better and better, say the the client and the specific client in this case, guess which is the only one supporting it at the moment, and so the process is quite um straightforward, since you don't have to do anything. I just set the node to to operate on Snap sync mode, and once you download the the latest snapshot.

E

Let's say this depends on obviously on how often the the other nodes take snapshots on their end. So once you die, you have downloaded it. Then you will start syncing on top of that. The snapshot so yeah the overall the overall time to to get to the latest Block it's uh as you can understand. There is really little.

E

It depends on on how it is configured, because uh if, if they are I mean if it really depends on the configuration, because if these snapshots, yeah probably there from the guest team and the Netherland team, will also try more more information.

E

A good point to bring up to the graph team.

E

E

Okay, are there any um any other questions uh relate to yeah how to run yeah nodes on Northeast chain. uh Otherwise, if there are more questions able, uh please could you collect them and also provide them with that to us. I will be happy to to answer them and also yeah.

E

We also got a lot of feedback um and many many many hints and points for improvement that would make yeah our lives easier many times many okay, so such as, let's say the SnapChat thing, uh yeah having a snapshot sink for archive notes, probably would be also interesting. Let's see how teams can Implement that but yeah, let's start with telling them that there could be also a good feature to include in their nodes.

A

Yes, we'll definitely um message over any questions that aren't fully answered here so that you guys can come back to them. I see that stake stake. Machine is typing.

A

um Oh Chris is typing if any yeah, as as we mentioned, this is kind of uh the last call for questions. So, if you guys have any questions either in the chat or voice um feel free to ask away um I feel that voices quicker. So if anyone wants to unmute feel free to.

C

E

D

Have a quick question um so we're we're seeing that um Aragon and ericon derivatives like akula are emerging as kind of the next generation of evm uh execution nodes, um and it looks like you know. Certainly a lot of the ecosystem is recognizing that.

D

um Yeah, a lot of the improvements that that Aragon brings to the table are like pretty important for these chains to continue to scale like particularly with things like State bloat um and yeah I'm. Wondering like how the gnosis team is thinking about that and whether um there is an intention to bring Aragon support to nurses chain.

E

Yeah sure, actually there was an announcement today from the Aragon team around the topic um so yeah. Indeed there, the ergon team is working hard on yeah, leading up to speed and yeah release on.

E

Really soon on uh yeah on office chain, and that version will also support, support The Merchant of this chain, so yeah um they they are really I mean the air gun team is really committed uh to uh yeah, releasing uh softwares that will be embracing, also noses chain and yeah.

E

Not this chain on the other side is also committed to help everyone or never mind uh Eric on and other teams, clients and then developers on um yeah or boarding them on the chain and helping them building and propose any any kind of solutions that would be beneficial for for everyone and for uh the whole ecosystem, so yeah to do to answer your question. Yes, we are supporting uh edible, such as we are supporting other teams and clients on on a building on top of nurses chain.

E

Thus, let us answer your question. Chris.

D

It does uh thank you so much and uh super excited uh to see our support. Coming for nurses, too yeah.

E

And thank you for pointing out the this. Does nothing for yeah for archive notes. uh Actually, the last time I'm I used, uh yeah I said um and a full note to to think of nerves and snap thing. You know it was an opening video one and then, after that, I I didn't I. I didn't go through yeah, the latest updates, but yeah.

E

Hopefully, hopefully this um you know this uh snap thing for archive notes could be probably also the the community would could also potentially help on that, and uh you know with um yeah proposing features and so on so uh yeah. Let's see we let's see first, if they they come with something from uh for the nurses chain. Let's see if it's uh just a full node a Snapchat thing, and uh if that works first and then uh let's try to get it uh to the next stage and also enable snap sync for archive notes.

E

Yeah, unfortunately open your video is well actually a few. A few providers I believe, are still running open field notes um not on an ethereum chain, of course, but yeah on on other ones, but uh yeah, it's deprecated. So if you have any uh any opening Freedom notes running somewhere on your machines, taking the account to to migrate to another client.

D

um I guess another question that that's maybe a little bit less related, uh but I think is maybe somewhat relevant for for node Runners, um so I understand that like uh um gnosis chain today, uh you know is obviously um you know just an execution like the you know. It's a monolithic um chain today and and like the merge, will move it into.

D

You know the the kind of knows to speak and chain, and then uh the execution client will be separate um and I I know that today, obviously you guys don't use proof of work, you have a custom consensus mechanism um and I'm wondering about um you know. It sounds like that custom consensus mechanism will basically be deprecated when the merge happens um and the gnosis beacon chain will then be the consensus mechanism for gnosis chain.

D

um I I'm, conscious that um if I remember correctly, the nurses Beacon chain has block times that are half that of the ethereum beacon chain. um Is that right, and does that mean like we can expect stake growth um to be? You know roughly twice um what we see on Main I guess it depends on utilization as well, but um yeah. Can you talk a bit about that and maybe like the pre and post uh gnosis.

E

So that's a really good question Chris. So um as of now, you know uh the con. The current consensus algorithm for uh nurses, Chinese uh aura and this algorithm uh doesn't doesn't run on uh yeah. It doesn't rely on proof of work proofs, but instead uh there's um a set of validators that basically are enabled to validate transactions for the chains or kind of uh proof of stake, with some yeah minor changes.

E

um The way nurses chain will work after the merge is basically um the consensus layer where basically, the the beacon chain, the nose is Beacon chain, will take over the consensus, part and uh yeah validate transactions and send them to the execution layer for final execution. So the idea is to- uh and indeed the validators that um yet today are currently validating transactions on nurses chain will become worthless.

E

Once we merge the nurses chain with the beacon chain, since the consensus layer will take over the you know, the um the consensus for the for the chain, uh so noses will will be migrating towards uh yeah, definitely toward the proof of stake, a fully proof of stake a chain after the merge and also there are plans. Actually there are some discussions around how and when and we we should get rid of the aura consensus in which way.

E

So there are some ongoing tests um helped by the community and by nethermine by gnosis and other parties. uh That's why uh yeah we are confident we can make the match really soon, but still we we must be cautious, because uh we must be hundred percent sure that after the match happens, everything will be working. Fine, um so uh yeah. To answer your question and, to recap yeah: we will migrate from Aura into yeah.

E

Let's say: approval stake, consensus, algorithm and nurses chain yeah Beacon chain; sorry, yes, uh the uh uh the amount of uh so basically the the epochs uh with slots uh slots to be specific are half of the half of the timer. If you compare with uh ethereum ones, so it means that, uh ideally, the change should be two times faster, so that's the current setup and yep.

E

So, as you know, it's all being tested yeah the beacon chain actually is already available and anyone can can join it and run validators uh so that you you can you you can join the network at at any moment. You want uh on the docs, uh there are um there's a guideline um explaining how to deposit genosis chain and uh yeah enable your validators if you want to stay in the start, validating transactions on the Bitcoin chain, but yeah for the moment.

E

uh Probably this is how to scope for this, uh for this call, if you have any anyway, if you have more questions, you can reach out to Able able Abel, which will then forward all your questions to us.

E

Actually, I don't know uh it's around 13 15 I guess something like that.

E

It used to be either in the past. I. Remember.

E

Both static peer configuration prognosis chain, execution nodes, you mean like uh setting up a boon notes. What do you mean increase.

D

I mean I I'm sure that uh you know, like the X guy under like the X-Type configuration that nethermine ships with specifies. You know built-in boot nodes and so on, but I'm just wondering like with your experience operating these these nodes, um you know, do you, um do you pass any like static peer configuration like if you run maybe five nodes? Do you specify each? You know each of the nodes as peer nodes for each other, or do you kind of just let the node figure it all out.

E

Yeah, okay, this depends on how fast you wanna uh you wanna, learn how to uh to bring up and uh and start thinking. You know uh as far as yeah, based on my experience, you shouldn't take more than a couple of minutes to start thinking but yeah, um even though even so I saw, uh let's say multiple times notes uh unable to to start thinking, uh because uh no peace were available. So probably it's just because you know the algorithm is distributed.

E

The routing algorithm requires some time to uh to reach out to yeah, to to peace. In that case yeah. If, if you by chance, have um more nodes running- or at least you, you know, someone who runs nodes and you can ask them to provide you with the uh the enote.

E

Well, the the yes, the E note, and then you can add that to your configuration, at least for sure, you know that uh at one peer and it will be available when you start your node, so yeah for most of the configurations, I saw I, seen um no static. Ipa, at least at least, was uh was needed by yeah in in a few cases when nodes were not bringing up uh yeah, we had to set up static, IPS.

C

Yeah cool thanks.

E

You know because sometimes, if you run, let's say two three notes, then you you already know your e-nodes and you can set the the set them on on the other notes. You are running and they they all will bring up faster.

E

E

Chain, actually, uh there's no faucet, since uh basically I mean the faucet will be someone who sends you, some general stockings actually actually not and there's no faucet for the beacon chain. Sorry.

E

Okay, they're not I'm, running a Tom instructing for a beer, yeah, also um yeah. There can be many reasons um uh after you restart the node. If you still, if you are still experiencing a beating issues, uh probably will make sense to to specify and the static aesthetic, IP yeah.

F

E

Address and and restart the node again and see whether they, whether it can can start thinking uh I, know uh at least on open ethereum. You could specify the amount of peers, the minimum amount of speeds uh you you wanted to maintain and making obviously the node to consume more run resource, since uh he had to do more discovery on the network and not sure about nethermine whether it supports that that feature, but in case they they do I will say you can also increase the amount of um the amount of peers.

E

You wanna you wanna You, Wanna Be, connect to.

E

Yeah, um generally speaking, uh it should not be a problem to find out peers. It's just a matter of time. You know between the moment. You start there and now then, when the node finds some available appears but yeah. If you, if you are aware of other index indexer providers, you can yourself uh exchange your e-nodes addresses and and put them and configure them on on your node.

A

All right well, as I mentioned in the chat, we're at the top of the hour, so we'll just wrap up with final questions comments. Anyone feel free to unmute your mic or uh type in the chat and um and then yeah after that, we'll wrap up today's conversation.

A

That's really that's really nice to hear um Derek, they're, very excited for uh indexing nurses, data and a nice meme there Chris great well I'll, take this opportunity to wrap up today's conversation. I just want to say a big thank you to Giacomo and um Ricky for joining us for today's conversation, answering everyone's questions and thank you all for attending and also asking your questions as well.

A

So um as we normally do we're going to have a workshop again next week, Tuesday and Thursday so be sure to lock into the calendar so that you can stay up to date with everything. Thank you all for attending and we look forward to seeing you next Tuesday take care.

E

E

A

Thanks bye-bye, everyone.