Ethereum Cat Herders PEEPanEIP, 12 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: PEEPanEIP#78: Prysm & The Merge with Terence Tsao and Potuz

Description

Topics covered
-----------------------------
0.52 - Client introduction
1:31 - Terence’s introduction
2:42 - Potuz’s introduction
4:44 - Presentation
05:05 - Agenda [Terence]
05:38 - Prysm
6:27 - Client diversity
8:10 - Strengths of Prysm
9:07 - Protobuf +gRPC
11:25 - APIs under Prysm
12:05 - Bazel
13:37 - Bazel go-ethereum
14:00 - Prysm v3
15:44 - Testing -e2e (Prysm)
17:00 - Testing Hive
17:44 - Testing - existing testnet
18:28 - Testing - shadow forks
19:00 - Testing others
19:30 - What we need more testing of
21:02 - What we need spend more time
21:47 - What’s left?
22:30 - JWT authentication [Potuz]
25:10 - Suggested Fee Recipient
27:58 - Storing blind blocks
31:45 - Vectorized SHA 256
33:43 - A story of bad coordination
37:35 - Q&A
37:57 - Which testnet event was the story of?
39:10 - Main difference between Justified blocks/slots/epochs
41:46 - Is there a fixed time interval from being justified to finalized
43:35 - What are the permissioned and open validator sets?
45:55 - What does a validator missing a proposal mean? Will that affect a validator if running a single client node?
49:30 - What are the noticeable points for debugging when something goes wrong on the client side?
52:35 - How do I debug what’s going on with my validator? Why missing attestation and losing money on beacon chain
55:40 - Any specific spec change since Rayonism to Kiln?
58:47 - Any significant change that is being brought to beacon chain clients/users/validators?
1:00:56 - Special prep for Goerli - Prater testnet merge?
1:02:07 - What do you think of EF client support?
1:13:17 - What can the community do to support Prysm?
1:05:20 - Any advice for aspiring validators?

Resources:
-----------------
Slides: https://docs.google.com/presentation/d/1SLLxz6VsUHPydvEAKezF6HBYQw5Gfu4DWh-nSD1VZDw/edit?usp=sharing

TheMerge playlist - https://www.youtube.com/playlist?list=PL4cwHXAawZxqoLxXqZqT4hcYhoHoP6w12

KnowYourClient - https://www.youtube.com/playlist?list=PL4cwHXAawZxoruie1hmYWJfiakpYivXft

Altair upgrade (Prysm) with Raul J. & Terence - https://www.youtube.com/watch?v=92BeeDr1Nhw&list=PL4cwHXAawZxoruie1hmYWJfiakpYivXft&index=5

PEEPanEIP playlist - https://www.youtube.com/playlist?list=PL4cwHXAawZxqu0PKKyMzG_3BJV_xZTi1F

Check out upcoming EIPs in Peep an EIP series at https://github.com/ethereum-cat-herders/PM/projects/2

Follow at Twitter
-----------------------------------------------
Terence Tso @terencechain | Pooja Ranjan @poojaranjan19

Check out upcoming EIPs in Peep an EIP series at https://github.com/ethereum-cat-herde...

Contact Ethereum Cat Herders
---------------------------------------------------
Discord: https://discord.io/ethereumcatherders
Twitter: https://twitter.com/EthCatHerders
Medium: https://medium.com/ethereum-cat-herders
GitHub: https://github.com/ethereum-cat-herde...
Email: support@ethereumcatherders.com
Website: https://www.ethereumcatherders.com/
Music

A

If you want to be a validator on the ethereum network, it is essential to know which client providers suit you the best to run a node. We want to help users make an informed decision, thus presenting the know, your client series to learn about ethereum's, el and cl clients and what they have to offer after the merge.

A

Welcome to the mud, special p, penney episode, 78, I'm puja, ranjan and with me, are talented, ethereum developers, portals and terrance to talk about the prism client. Thank you for joining us on the show. Thank.

B

You for an invitation, thank you for having us a.

A

Quick introduction to the client prism is a go implementation of ethereum protocol for consensus, layer, aka beacon chain, as per the official documentation prism, contains a full beacon, node implementation, as well as a validated client for participating in blockchain consensus.

A

Terence has been on our show earlier to talk about prism at the time of altair. The first beacon chain upgrade, and this is the first time we have photos with us, so welcome you, both and before we get into clients detail.

A

Let's have a quick introduction for our listeners.

B

Sure I guess I can go first hi. My name is terence. I am a court death at prismatic labs. I have been around the space since um early 2019, so what got me into it was. I was reading this reddit post by vitalik about wanting people to work on scalability and then on the same ready thread.

B

I found wrong with fast and they're also looking to do the same thing and that's how I got started basically, so I first started just working on sharding and uh that actually didn't go anywhere because we're trying to do sharding through the smart contract and that wasn't gonna work and at the same time there was this casper by danny ryan and then it was like cip1011 or something, and instead of doing like proof of stake on like a smart contract, they ended up um going to do proof of stake on a new blockchain and that's what the beacon changed today.

B

So that's how I got a story just working on those like cool r d initiatives, so yeah super happy to be here. Thanks for having me.

C

uh So hi I'm I'm an outsider actually to this community, uh I'm a professional mathematician. I just had some times during the pandemic. I guess I entered this community more or less in 2020.

C

I've had just read about ethereum. I wanted to learn more. I got excited about the merge and the idea of the of proof of stake, so I that was a good way of saying. Well, I'm just going to be locked on this. I bought some it. I immediately put it in the contract that I bought some it at the time of the medallion incident and uh started to read. I started to learn how it worked.

C

I I just entered this community through the easter egger community, as many of the validators have and uh well so I chose prison as my clients, so I entered their discord started.

C

Seeing that I wanted some functionality in the client that I that they didn't have, for example, some login functionalities for linux, so I started opening some pr's. Then I started to get interested in the specs and specifications.

C

I've learned a lot from some members of the community and it was very it's a very welcoming community if you want to learn people like even down from anyone that is in charge of the spec I just want to. I just don't want to drop names, but everyone as high as you can go, they're going to be very well welcoming and open to open uh to answer questions and engage in discussions and well so that got me here. Essentially, I started bothering the mother and mother more until these guys lured me.

A

That is awesome. Thank you for the introduction, and now you are a part of core contributors, so uh this is very interesting for new people who would be willing to join the ethereum community. I hope your story inspires them.

A

Well, I understand we have got some presentation uh so without further ado. Let's just begin.

B

All right, so I can start so hi guys so like we did an intro already. So it's really exciting to be here- and I think last time was here- was like the uh tear upgrade. So it's been a minute. Okay, so yeah so today, products and are here we're going to chat about the merge, especially what does the merge look like for prison right? So as for the agenda, I'll briefly go over the client, and then I go over like our next big release, which is v3.

B

What are the high level features? Are there and I will go over some testing updates as well and porters will take you on the second half he'll go over the detailed features and then he'll. He will also tell his very story, which is like what happened couple weeks ago when there was like a seven block, reorg so yeah. So let's get started so pretty soon.

B

That's why we're here and prism is one of the five production cl implementations out there today it's return in go, and I do want to emphasize like prison britain and go and that's why, like there are certain like features that exist in prison.

B

It doesn't exist in other clients which that you, as a user as developer, you may be interested in and the features such as grpc and such as we also use bazel as well, and then prism is also nice in a way that we do have this like pretty natural interability with execution client in particular, yes, and that's something that I will also go more in that thing but yeah.

B

So this is prison and uh before I start like talking about prison right, I do wanna, like mention client diversity, because it's a very big thing as a ethereum core deaf, that's essentially what I do really care about. I'm really passionate about the client diversity has gotten much much better since earlier the year, prism is roughly at uh forty percent now, so it's less than fifty percent. So that's good and then the um the association client obviously needs a lot more work and the step is something that a security.

B

We should probably pay more attention to in the next following months, but yeah. So I'm really happy with the client diversity right now, and uh I also do want to encourage more people trying out minority client and see if there's like features that you you like, there's features you don't like there and then bringing feedbacks across playing right. That's very important, just like it's all about like being on the same level across all the clients, something that you like on one client, a you. You also want to implement in client b.

B

You should definitely like open issues and help developers, and most people are really really happy to like talk to you about it right. I also wanna. I mentioned like just to be cautious right. It's like it's, it's okay, to try every clan combos on the test net, but when it comes to mended, if you're staking real validators with real money, you should do whatever is most comfortable for you right you.

B

I probably wouldn't try some other clan combos that I don't know as of today, so definitely play around with the test net, but for men that be very, very careful with what you do, and um so every client has its own like strength and some weakness right. So this blog post by uh michael laughs with this couple weeks ago, it's a new one as well, so I'm just quoting what they're saying basically they're saying: hey prism has pretty good ux.

B

So I'm really happy to hear that it's because, like ux always has been in our heart for the last couple years. We want to be the client, that's good ux, but also for samsung downside right. For example, we implement http server on top of our grpc server, so the response time may be slow there and it may not be ideal. That's something that we're working towards as well, but yeah. I highly encourage everyone to uh read the post and then and then and then just see the trade-off between clients and then yeah.

B

Let's just talk about like something: that's more like prism specific. I know one thing about christmas: specific that you don't see on other clients is that we are heavily we're heavy users of put above and grpc right, and then we always we often get like. Why are we doing this? Why don't we just do like the standard like api rest approach?

B

Right so like I must admit, like I'm kind of like a grpc maximist so like I might be biased towards your pc right, but, like I like grpc, because, like I think in technology right, things tend to go in circles between like strongly type um communication approach and weekly type communication approach. Right, I think like for me it's like when two machines are communicating with each other right.

B

There should be like a very clear contract with strong types online on on just like on what those machines are talking about, and grpc is basically that right, you define the contract and then you auto generate the client and the server code. So any changes can be propagated very quickly as possible without human trying to parse like documentation and understand how to write the code right.

B

So, in summary, like grpc, give you this spec, that's actually used for for generating client and server code, and then it's also a stack that provides like forward and backward compatibility changes. It has a more like structured metadata pattern and then it also has better commun connection primitive, for example, bi-directional streaming and concurrent requests over a single direction. It's very very nice for on grpc stuff right and that's. Why, like we are using grpc like like a lot of like cloud instrumentation companies like kubernetes dockers, they'll use like golen and they'll use grpc, and that's.

B

Why, like we are using and that's why we love it and that's definitely, I think something that we're really really proud of right and then what I said right.

B

We do have like two different apis because, like I think when we first started like implementing our own like api, like there was no official api yet just because I were slightly like ahead of like the time, so we have our like prism grpc api, which, like some people, use this, but at the same time we also use using the standard json api, that's defined on the e2 api ripple, because we do understand the standardization is very, very important.

B

You do kind of like you want like someone to be able to use multiple clients using the same api.

B

So then another thing, that's very, very prism specific is that we use baso right and then we also get a lot. So why are you guys using baseball? Why don't you use like make fire right? So we use spacer, just basil, provide this reproducible build and the sandbox environment that ensures that everyone is building. Bazel has exactly the same experience right and that's very important, for example, like my friend may be like um building our beacon client on their own, like os on their own machine language system, and I'm doing it on my own system.

B

So there may not be some sort of like reproducible between us right, so bazel enables that and then so, which is nicer, say today you have a bug introduced right as a developer environment being different than other. So we don't have that. We don't have to worry about but being introduced because of like developing environment being shipped different than the user, and then we also don't have to worry about a messy dependency management when dealing with a monorepo with multiple programming language.

B

This also solves that and then at the same time, base also has remote caching, which is very, very nice, to run like ci and test in like a massive code base, and then he also has advanced features such as. Like dependency analysis. You can you can query like dependency graph and stuff, and that's why we are using bazel as of today, and we also have our like basal version of gold ethereum, so we fought through ethereum.

B

We turn that into a basal repo, that's very nice, because that allows us to use gold bindings in a way. So it makes like our code base a lot cleaner, a lot, a lot less duplicated code because of this right. So let me just quickly jump into what shift gear call about what prison bd3 is right, so we're gonna launch prison v3 at some point before the merge and what is b3 so v3.

B

So personally I like this. I like major release because major release is, we can say it's not backward compatible, so you can like introduce breaking changes and to me, that's nice, just that's like the perfect time to do cleanup right. You can clean up all like the things that you don't like in the code base. So what's what is vp so v3?

B

We, of course, we kind of have bellatrix and the merge support and then we're gonna have support for netboost because sounds like timeline wise, that's what everyone will be ready for, and it's also important for all the clients who have mapped support at the same time. So therefore, you don't create this network, a cemetery where everyone will use the clan that ledbc gives more profitable block right and then something that raul has been working tirelessly on is trying. We do want to like have better separation of header and payload.

B

Currently, it's kind of silly that you're storing payload on two different places, which potentials will go more in depth on later and then we also want to deprecate all the like outdated flags and apis and things that people don't really use. But we cannot delete that today, just because it's not backwards compatible right and then we're gonna clean up our code base to make it more idiomatic, better us, better, better login monitoring and just really happy that we get to use this ability to basically make our whole base much much much better.

B

Then, let's talk about testing, that's a fun stuff right, so I I think we mentioned this oops. Sorry. I think we mentioned this in previous update as well that we have our end-to-end test. This end-to-end test is more prison, specific, that's how we implement in-house, so we launched local vehicle nodes, client, validated clients, execution clients and slashes right, and we recently kudos nishang.

B

We recently added black clients support, so it's kind of multi-time right now we're able to catch bugs not only on prison but also like sometimes our lighthouse, and we get to alert lighthouse, hey. You guys have this issue and- and I think they're you should be pretty happy when we do that and then this is trigger before, like pr gets merged into developed branch and then how it works is this: we have this evaluator at every epoch. We basically monitor matrix api logs and then make sure like they pass certain threshold.

B

Otherwise we fail the test. So this is essentially besides. Unit has unit test is probably there is zero. So I would say this is like the first layer that we find bugs, and then he has caught us, so many so many bugs before, and we're really appreciative for that right and then there's also the hive test.

B

So the hive test is coordinated by um ef by perry and mymar and and by mar real and all those gentlemen, and then this hype test is, I would say, similar to prism into e, but it's more like engine api focus, so it doesn't really test scenario with like slashing like deposit adds it, so it has more like client, big, consistent client and as your client and their interactions between each other. So it's very important. It's very it's very useful advice.

B

It's more engineering, api focus and I'm really really happy that we have those in place so hive test. Definitely something definitely deserves a shout out right and then testing. Of course, the existing client right we are, we did all the amphira we did the like. We did all the private test night before and then with the eruption and today support the emerge. So we're like knocking out the door for every single one right and the next one, something goldie then the minute.

B

So so this is literally like probably the best like this is the best playground for testing right today, we found like certain like peeps searching like exchanges pools, did not use the right config right and because of that, like there definitely some hiccups, but that's what testnet is for to like find to find like to find situation. Issues like this, so this is how we can improve as a community and there's also the shadow force.

B

Shadowforge is very, very underrated in my opinion, because it allows us to test what the mainnet merge will be like without actually merging it, and then we have done like so many of those and then I I would say personally I'm not really impressed because I feel like before from from the start, we're barely working, but now we're working flawlessly.

B

So we can see that all the improvements throughout the time frame and yeah shadow fort is great right and then there's other type of testing such as spa scenes which, like we definitely so like.

B

Shout out to lighthouse guys they're, like they're the ones that's been doing this differential fallacy between all the clients to make sure that, when something failed, you feel the same way and then we have gotten like a feel like pain about that and that we we definitely used to face a few issues right- and this is stress, testing performance testing, kutosis, manufacturing and there's just a lot of testing testing that goes inside.

B

So what do we need more testing off right so, like I do think like now with like macbooks, we need more testing around macbooks.

B

I ideally, I would love to like incorporate metaphors inside, like hive or inside our end, to end which we'll work on very soon, that's very, very important to make sure that uh to make sure that that boosts reply correctly and in the event, if it doesn't reply correctly, you default back to the local ee right and then I would love to see more testing of rewards, for example, like what happens under certain real conditions and then fix all the proposal boost related issues that happened before and I'd love to see more testing on api throughput, for example, I know recognize it for prison.

B

Certain api response are much much slower and then there's always like things you can improve there. So that's definitely very, very underrated area to test and then p2p analysis and just like typical, due to analysis right so that so so like what does the chin look like when we shift from 12 second block time to sorry 13 second blow time to 12? Second ball time? And what does it look like once you have steve's law? It's definitely something that we probably need more analysis of, and also the guest name it. What happens now?

B

The validator can actually set their own guest limit. That's something that's very, very interesting that we probably should think of more as a community.

B

So what we should spend more time on, I think ux is probably the most important thing right like today. We saw issues because of conflict right and then like this is going to happen, probably going to happen.

B

When mainly comes just like the less configuration we have to set the better and unfortunately, today we have to run two pieces of software and maybe more if you wanna extract mev, how do we make the u.s better to avoid like issues with like different conflict mismatch on the field, and then we probably want to look at like what this size looks like between different client, combos and then I think that's something that's very important and documentations and tooling those come a long way.

B

So, what's left before the merge very high level right, I definitely would definitely want to harden the map to support and the documentation to make sure it's solid people want to use macbooks. You can use macbooks like comfortably. We want to store headers of the payload, that's what we're always be working on. We want to redundant, remove all the redundant if one endpoint supports right now we can like use. This can pass like multiple if one endpoint and then that's not gonna, probably not gonna, work too well, post merge.

B

So that's something that we should probably remove, and just more testing and more ux improvement. So yeah, that's it that's what I have so I'm gonna pass the ball to potus.

C

So, thank you, so I think I'll I'll be very brief and I'll just focus on things that uh that that really just matter to our users, to our current users and some features of our clients that are changing uh one of them is that currently you're, you have been communicating your beacon node with your execution clients they need to communicate because the beacon node needs to know about deposits.

C

So when you propose a block, you need to to have some information about deposits. This is not so essential and that's why people can divert this. This need to places like in four hour, alchemy or some providers uh after the merge. This is not the case. You are going to be required to run an execution client, and this is why it's great that we can actually we are going to be on board in a lot of new execution nodes into the network. So that's why client diversity?

C

We are hoping that it's going to change on the execution side, uh but apart from the improving crime and diversity, one of the things that's going to come with the merge and the need to run an execution client is the fact that it's going to be more dangerous. If someone grabs your execution plan, they can actually steal your. They can steal your map. They can steal your earnings. They can steal your transaction transaction fees.

C

So, in order to prevent this from happening, if you're running your execution client over http, you are going to require to have an authenticated connection and that the system chosen is jwt the communications. uh Well, they just need to be secured. That's why they were chosen. Unfortunately, the ux is not good. It's not great uh you're not going to be allowed to just pass a password you're going to have to pass a 32 byte and go to string as a hex string.

C

So this is going to be a bunch of characters that you need to put into a into um into a file into just a text file, and this string needs to be passed to both your beacon, node and your execution client in the case of prism. You're gonna pass this flag, which is this jw secrets, uh converter uh just to the location of that file and what the flag is going to be in the case of your execution, client. Well, that depends on the execution client. I just put there uh the flags for nethermine.

C

If you haven't tried neither mine, try it it's great. It's working, fine, it's ready for the merge and it's not guest, which is a great client get, but we need clients that are not get a nether. Mind is a great option, so it's best by the way I mean there's plenty of plans that are ready for the merge.

C

All right, other feature that most users would be interested. Is this suggested fee recipient? This is the ethereum address the current execution ethereum address where your earnings are going to go, not the validator earnings, but the transaction fees. This is a fee that currently miners are taking and that, after the merge, it's going to start going to validators.

C

Validators will get access to those fees immediately as soon as they propose a block. They are going to get those transaction fees into whatever address they put in their configuration files or in the configuration flats.

C

We have been got getting really weird questions regarding this suggested fee. This is not an address that needs to be tied up with your validator in any way. This is not an address that you need to own. You don't need to. You, do not need to have the private address the private key for this address. If you want to donate your fees, you can just put vitalik.e probably tell it, doesn't really need that much either, but you can put whatever you want. uh You don't need to own that address.

C

uh What else can I tell you about this? um There are several ways of setting their address. Their different users will have a different uh different needs uh say a small sticker like myself, I'm just gonna use this uh on the I would like to use it on the beacon node. Unfortunately, you need to pass this on the vehicle on the validator clients so on the validator client.

C

There's this this client flag, which is suggested free recipient, and you just put whatever address you- want there to send all of your bits for any validator that proposes some other validators that have more validators. For example, they just don't want to show to the network, but they have plenty of validators they're, going to be using a json file where they're going to just assign different addresses for each validator, for example, and a default one for all of them that they weren't assigned a particular address and even more advanced, valid validators.

C

They would want to have a server that is going to be sending post messages with a changing address whenever they negotiate, for example, where the the fees are going to be going so those three different ways: there are different ways: take a look at the documentation which one is for you. uh If you don't put anything these addresses, these fees are going to be burned and there's been internal discussions as to what is the burn address. I had a pr open to send everything to potus.ed, but it was banned and it was not approved.

C

Not even they didn't review it, so I think we're using a burn address, we're using zero hash as the burn address. If you don't put a configuration flag, you are going to lose all of your piece.

C

Okay, so the next. uh The next item I want to talk a little bit is something that already turns covert. This is storing blind blocks. uh This is one of the unfortunate features of the merge. The way the merge works, I'm not. I don't want to get into technical details, but we are now broadcasting beacon blocks, we're broadcasting consensus blocks on the consensus, layer and execution blocks on the execution layer, and they are completely unrelated, except for one intersection, which is the information about deposits.

C

After the merge, the full block containing transactions and everything is going to be broadcasted in the consensus layer. The consensus client in this case is prison is going to send to the execution client the execution payload that contains all the transactions.

C

The unfortunate feature of this is that both the consensus, client and the execution client need to have access to the full set of transactions, so the full set the full block of execution because they both need to broadcast it internally from the execution side because of the way they sync and the consensus layer, because well that's what we need to validate. We need to validate the full block.

C

This uh turns out that it's going to carry an extra 260 gigabytes at the current search and usage, and we expect this to be actually larger after the merge per year and just data that shouldn't be stored because it's in two different pieces of software, both the execution layer and the consensus layer is going to store this so we're moving fast. Actually, this raul from our team just took over this responsibility and he has a working pr he's already been thinking.

C

uh Chestnuts, only storing the consensus part of the blog and, of course this carries the problem of like having to assemble back the block. If someone asks you for the full block now, if you're just a user, why would you care about this? You probably don't, except the fact that your hard drive is going to be less full, but uh there's some sort of connections about this and pbs of proposal build separation and the way people are going to extract math.

C

uh The way most validators are going to want to run their notes to get more money to earn more money to earn more transaction fees. Is by not preparing the execution part of the block themselves by but outsourcing this to another another builder, and to do this, you need to send this blind block that doesn't contain this. uh This exhibition payload itself, so it's useful to have this separation until the consensus layer, the natural object that the consensus layer is going to have to work with. Is this blind block?

C

So here's another thing. So this is our contribution to client diversity without switching clients, uh we have a completely independent uh port choice, implementation that has essentially nothing to do with the purchase implementation that president has by default. Today, you can run this fortress implementation by using that fee. That is the second one there it has been tested for the last four months. We have never had a consensus, failure between prison and prison between prison and lighthouse and within prison.

C

Now, in the last few months, within prison and anything uh following uh without using that for choice, uh it's much less back prone than the default port for choice. That's why we decided, because it's much easier to read the code in this implementation and we're hoping that this implementation will be default soon.

C

uh So that's one feature that you may want to test. You may want to give it a spin and, as I said, this is a way if you're, uh if you're, just too lazy to to choose a different client. This is a way to add a little bit of plain diversity uh to the mix.

C

This is perhaps the reason why I'm here, uh I'm actually personally proud for this. This is uh a feature that we stole from different clients. uh This is something that happened, a technical digital detail that happens in bitcoin. It's the way they hash their blocks, they hatch merkle trees.

C

They realize that they're hashing blocks of a constant size which is 64 bytes and all of the shy implementations. All of the hashing implementations out.

C

There are optimized alar to hash large buffers, not just 64 bytes but length buffers, uh so we just told her that from bitcoin we just took that uh and used the optimization of knowing that we're hashing a lot of the times 64 bytes, and we also stole some uh I'm seeing start stealing, but just mixed code of intel uh that uses spiralization when you're hashing several things at the same time, and these two things are very uh impressive: they're they're having very impressive results in the way we're hashing, which is one of the most expensive things that consensus node, does now uh we're seeing up to 10 times or 12 times faster on newer machines than the regular hashing, for this is a very optimal setting.

C

This is hashing like very large list, which is not the case regularly, but still we're seeing like uh we're sinking twice faster by just uh turning on that flap, so that flag, enabled vectorized hdr is a flag that is potentially dangerous, because it's a new library in assembly, which is people developers, are usually scared of these things. But we've been fuzzing that library, the html foundation has been passing that library, uh people, the personal people, have been passing that library without any problems.

C

So we're very happy about this all right.

C

So here's the last thing I wanted to say one of the items was to tell a story, a success or well, not so much of success story, and this is a little story that I don't want to give technical details. I just want to say how being just a little optimistic and not really looking into the details, uh brought us a problem and a perhaps a an advertising problem of having an eight, uh a deep, eight org on the network.

C

So the reason this happened is because of a feature. This feature is easy to explain at a high level, it's called proposer boost and is a feature that would allow a node when it sees a block arriving early, say: oh, this node is much better than any other one that I have now. So if you need to choose between two blocks- and you saw one that was exactly on time- it was it was fast. It came to you fast and you have another one that should have come to you 12 seconds ago, and it disappeared.

C

You're gonna vote for the node that you just have now. So this feature was coordinated among every client. It was deemed that it was fine to just ship it in whenever we wanted and for the most part it was.

C

It was just a combination of another picture, and this other feature is a picture that said before you propose a blog just before you do anything for personal blog, just check again, if you've seen new attestations check again, if your head has changed and as soon as you do, this propose your block. So these two features were going to be shipped at the same time.

C

Essentially, no client got the second one on time prison had it released just right before the right um at that time, so we managed to ship it in, uh but I think prism was the only client that was shipping. That and uh half of the nodes were updated, and this half is the key. We got very, very unlucky that half of the people updated if no people had updated it or if all people had updated it.

C

We would have been fine, but just half of the people updated the node, and this created this sort of like conditions where it wasn't really clear which no, which block to vote on so some notes that weren't, updated were uh were proposing for the wrong in the wrong chain, uh but the other blog could still be had.

C

The fun fact is that uh prism was the only client that had shipped the new feature. So if any proposer uh was a prison uh prison node this this, this would not happen not not, because prison is great. It's just because prison happened to have these two pictures together.

C

And uh so when I, when I looked at this, I got the alert that we're having a huge fork. What the what the hell happened. I went and looked at. I looked at the blocks and said: well, this happened, but uh but but I don't believe this, because this must be an attack, because if any prison node is there, this fork is impossible and I just computed quickly. The the the the probabilities- and this is something that we get confused all the time yeah.

C

This is not so probable, it's only one in five thousand times that this is going to happen. If you have a fork that you're gonna get eight uh eight notes in a row that satisfy these properties of like not being prison of not being updated or whatever it is, but the problem is that we have a lot of slots. We have one slot every 12 seconds, so one in 5 000 is going to happen every couple of days so yeah.

C

This is a situation that happened it's unfortunate and now we are about to ship new properties and we're very mindful among all clients of talking is it is it safe? Are we going to do it or not? uh We've learned a lot of that with that event. Okay, so let's just stop right. There.

B

I think that's all the slides that we have so yeah open to questions or any feedbacks concerns.

A

All right, thank you so much. Thank you so much for sharing all these details. It is really interesting to see inside of how client development is going on and about the last slide, the story that what was shared about the bad coordination or the proposed true story, would you mind sharing, which particular incidents it was.

C

I'm sorry so could you just repeat the question so.

A

I was wondering which uh incident it was. Was it for any particular test net or.

C

Oh no, so this this was in mainnet. uh It was a few weeks ago. uh It was uh an eight deep blog uh fork uh that sort of like uh yeah. It's been raving on twitter as some like the merge killer, but it was just an unfortunate series of events of people carelessly upgrading their notes and hours carelessly. Shipping updates without without like coordinating it.

A

Well like when there is not so good experience, that is always a lesson learned, uh which is ultimately good for users. So definitely we are happy that situation is under control. Thank you for sharing that story. I have a few questions which we have collected from the community.

A

I can probably take uh some of them. The first one is is around the basic understanding. I understand the terms that we use in day-to-day basis may be very easy and simple for developers and many technical people, but there are many users uh who are maybe willing to become a validator in future.

A

It would be useful for them. So what is the main difference between justified and finalized block slots? Epochs.

C

uh Okay, so the way the protocol works is that I mean as a first approximation is this. You would say a finalist block. It's never going to be non-canonical.

D

C

Finalized vlog, it's never going to be reverted. A justified block is very unlikely to be reverted, uh the technical. So that's the first approximation, that's not true for either one a finalized block might be reverted, but to revert a finalized block, it would require a lot of people losing all of their like two-thirds of the validator set will lose everything they have.

C

If you want to change a finalized block, we do not want to find ourselves in a situation where we need to revert a finalized block, uh because that would mean that two-thirds of the validation set would lose all of their money now technically, what is the difference uh when two-thirds of the validators vote for uh a block they put yeah, I'm just gonna, simplify it in a block when two-thirds of the validators votes per block. That block is called justified.

C

If just so happens that two-thirds of the validators vote for the child of that block, then well, the child becomes justified and then the parent becomes finalized and the algorithm we follow for poor choice, guarantees that we're not going to ever change a finalized block and the only way of changing a justified block for something like a contending block.

C

So to revert a justified block would be if there was a network partition and a large chunk of validators went offline and suddenly appeal say china's wall that china's uh firewall just blocks a lot of validators from china, and then they suddenly open it. We see a long chain, okay, we might revert up to the justified point, but we're never going to revert the finalized points.

A

That make a lot of sense. So uh another follow-up on that like just like, we have 12 second slots. uh Do we have a certain fixed time? Is there any uh like fixed time from a slot being justified too finalized? Oh.

C

No, so not at all, however, when the network is behaving correctly, uh what's going to happen is that every 32 slots you're going to justify a new app you're, going to justify a new block and you're going to finalize your previous justified block. So if there are no forks going on, if the networks will behave and I'll remind you, this has happened in maintenance since its launch. It's never behaved in a different way since it's launched every 6.4 or since for whatever 32 times 12 seconds.

C

So this is about a little bit more than six minutes. We justified a new block and the previous justify block becomes final, and this has never changed since the beginning of the beacon change.

A

That is good to know.

B

I will also add, on top of that right, I think most of the viewers may be familiar with tanzania, for example, so any sort of bfg bft chains they kind of favor safety over lightness. They have this concept of a single run right. So what happens in the single run is that block proposal proposed and the validator does pre-commit and and also commits, so there's basically, two runs of both right and then, in our case, our first round of book is is kind of like justification.

B

The second round of vote is kind is basically finalization, basically right and then, instead of like doing this, every single slot, we do every single epoch. So therefore we favor liveness or over safety, and that's basically the trade-off that we favor.

A

Interesting and thank you so much for sharing. I know these are basics, but it's helpful for new users uh talking about new users, so I understand this beacon chain and the pos version will be mostly about validators, so I heard about validators set when it comes to test net. Sometimes it is permission and sometimes it is open, sepolia, the merge that happened today and the godly that we are expecting in few days.

A

uh So could you maybe talk about uh what our permission and open validator set for test nets, and why do we have it the way it is.

B

ah I can take this one, so um post merge, there's going to be two main test net there. There will be long running and that will be kept alive. So the first one is sepodia, the second one is 30 or or or you can say, prater as well. That's the that's the consensus layer side. So the support here is the permission. Testing by permission I mean that not everyone can join, but you can join if you submit a request, so we basically want people to sign up for it.

B

They know what they're doing and they can like sort of verbally promise that they like they will do their job nicely and not just keep up on their validator after like a few weeks right and then why- and why does it matter? I think it matters to have this very like smooth running beacon, chain or a consistent layer chain or the merge whatever that people can test their dab song. So this is basically the solid test, grunt right and then on the other spectrum. You have goldie, which is more free that anyone can enjoy.

B

Anyone can leave- and that will be slightly more bump here, but it's also important to test your depths on that as well, just to make sure like, at the worst case scenario that you're deaf still survive. Basically so yeah. So there's basically like different end of the spatula and then it's highly encouraged people to test both.

A

That is good. uh We understand that both of these are public test nets and those should be open for uh dap testing and other purposes, but having it permission and open uh gives flexibility to developers to maybe test it. The way it should be. So that's good to know talking a bit more about validators so and uh like their questions, concerns, maybe sometimes, and mostly it is related to the slashing right and losing up the deposit.

A

So what does it mean um about a validator missing proposing like will that affect a validator? If running a single client node.

C

So uh about a validator is missing a proposal, oh well, so so that, because I mean you mentioned slashing and missing a proposal, this I'd say one thing about slashing: just don't run two validators and that's it you're not going to be slashed. If you just don't run to validate, you don't run two clients you're not going to be slashed period. There hasn't been any other kind of slashing everyone that has been splashed is because they tried to be fancy.

C

They tried to like not lose time, not uh not me be offline and they tried to do this in a clever way and they were not as clever as they thought so don't run to validate your clients, you're not going to be slashed now regarding missing a proposal. Okay. So if you are a very large staker, you will miss proposals eventually, unfortunately, the network has some hiccups and if you run like thousands of validators, you may lose one or the other one, and it's not so bad.

C

If you're, a small staker like myself like, I propose one block every two months or three months. If I lose a proposal, I try a little bit and so far it hasn't happened to me. Luckily it would be a concern if you don't propose too often, and you look missing a proposal- it's a concern. It's uh it would depend on the situation.

C

If you see that you miss a proposal, the common culprits are that your execution, node was uh was behind you or you're running within foreign and and you had some problems with your internet and that node needed to include a deposit. That would be a case. So this is going to be diagnosed case by case, but it's not common to lose proposals, and you know we know that it's not going to lose proposals, because the participation rate in mainnet is almost 100, so almost all slots are full.

C

So we know that people are not missing proposals. uh Reasons to meet up misa proposal are your computer is too slow for some reason. Like you're running on a raspberry pi- and it's stressed then it just you were about to propose a net a block when there was an nft drop and your raspberry pi couldn't take it.

C

This kind of things happen.

C

Click internet uh also affects this, and I can't really think of many more reasons for uh proposal means, except just that those those are the main goals.

A

And it is always said that do not run one validated key on different machine. Is it just related to machine and uh nothing related to like running.

C

It's it's the first thing it's. You would never want to run two clients at the same time, if you change your validator key from one machine to the other, one make sure that you burn and you drop your other machine up out of the window and you crashed it, make sure that those keys are no longer in that machine as long as you're, absolutely certain that your keys are in only one machine, then you're, absolutely certain that you're not running two validators. At the same time,.

A

That is a very good piece of advice for new validators.

A

uh One question is about debugging like we, we are doing multiple testing right. So what are the noticeable points for debugging when something goes wrong on the client side.

B

That's a good question. I think I can add mine and post maybe can follow up because he is like the master at debugging, so yeah, I think from my end, like I look at logs, I'm kind of old school, I'm like not fancy like some people on my team are super fancy they use like burgers. They don't they. They look at this like aggregated mattress on cabana and stuff, like that, I just look at rocks like I didn't just for me just I look at large enough.

B

It just comes more like natural to me, so I look at all the error a lot. So I look at all the info logs. I I I basically just do control fine, that's how I do it and stuff like that, and then it's also like nice. Just there's like, for example, like tooling, like beacon, champ block explorer out there right, so you can use website as well log into their website. You search your validated id, then you look at like if you miss anything for the last 48 hours.

B

If you missed something that something may probably not be right right. So just a very basic level of debugging and stuff, but other people do something fancy I mean, that's probably not for me and I'm sure quotas probably have some other inputs on this.

C

uh Okay, so I'll continue on the line of terence and then I'll, ask you puja a question uh about the question, but continuing with the terence, I can tell you how I debug that kind of problems that terence is saying, which is the problems when we're coding. I just called nisha- and I asked him nishan what's wrong with this nishan is the master debugger in the team and he'll just gonna find the bug in whatever you have. If you don't have any underground.

C

I think it depends a lot on the kind of problems that you're you're you're facing. There are some problems that naturally call to like say, run through a debugger. If you're working on uh a hashing library in assembly, you're gonna have to spend nights and nights on a debugger, seeing where one of the assembly instructions changed a byte that should not change uh if you are hashing like if you're comparing spec tests. That's also the case that you, you you're, trying to look block after block exactly at which line it failed.

C

Other kind of problems require knowing the area of code a little bit better and knowing what kind of error happens so most of us. I think all of us agree that the the way we're doing this fake debugging, it's just recompiling, with a lot of logs in our own code. We should probably leave those logs but uh but yeah, so we sometimes those logging like they come up in our in our pr's and terence, has already caught a few curses in spanish in my in my prs, because I forgot to remove them.

C

But now my question to you uh when you, when you asked about debugging- and I asked I'm assuming that as a user asked this question, are you referring to this kind of debugging of our code or debugging of say? I am missing at the stations and I'm losing money on beacon chain. How do I debug what's going on with my validator? Is that the question or was the question that we just answered? No.

A

Actually, you answered my question pretty well. Actually, I was curious to understand, like uh in the situation of emergency, how does developer or the development team react and how they start looking into the uh debugging of what went wrong.

A

So that's nice, but if there is any piece of advice for users like how they could debug like what went wrong on their side, why they missed something. So that would be really nice to add on here.

C

I am very strongly opinionated on this, so I'll just leave my advice, first opinion and just stop polluting our discord, servers about validator effectiveness, effectiveness as measured on beacon chain, uh both the the most famous explorer. It's just completely useless. Just don't look at that measure and just measure your effectiveness, then a different way either. Look at how many votes you got correctly or how many good head votes target votes and these kind of things validator effectiveness.

C

uh It's not a good measure of validator performance. uh I am unaware of one number that is a good measurement of validator performance, so my advice for users that think their validator is not doing well is not doing. Well is looking the explorer whether or not their validator without proposals is doing poorly compared to others. So the ranking in the in the explorer is good. It's a good measure, another one.

C

If you want to go deep, is if you're missing at the stations and just randomly missing some of the stations is go and look at the block that you were supposed to attest. If that block had very little votes, it's most, probably the proposer of that block. That happened. You were unlucky and you had to propose in a bad blog that came late for everyone.

C

If those two things pass, namely you proposed in a blog that was had a lot of posts. uh Well, then, a discord server and ask and come with the locks. That's the most helpful thing to tell me is grab the logs of just don't, come and say ah I'm losing money and no description. No, no explanation whatsoever. Just come and say I'm losing money, and here are my logs and if you do this, we're happy going to take a lot of time. Actually looking in your case, because you might have found a bug.

A

That is super helpful. I I know that the team are always eager to help validators and this card server sometimes gets spammed because of that. But if you come to the server discord, server with information that are useful will be helpful to both team and the user who is seeking for help.

A

My next question is about the devnets. I know the testing of mud started in summer 2021 with the rayonism followed by amphora pithas, killing kitsuki, and I think prisma have participated in oil testnet, any specific spec change or improvement. That is brought to the client since realism, to kill just net uh that you would like to mention, like you thought something else in the beginning when it was reunion and when it is killed, it's all together a different story.

B

I would say on the high level wise, like this concept of optimistic thinking, was definitely pretty new to me for its I'm, not sure to others to me since late emperor and it kind of took me a while to like wrap my head around it and then in terms of implementation. That's probably the single feature, or I wouldn't say it's the future, but yeah the single orientation. That's probably taking up most of our time in terms of implementation and testing, I mean I'm so happy to have pulled this on board.

B

I mean he is doing 99 of the work there and the four stories so yeah, that's just my two cents. Do you have anything? That's like something! That's interesting, not.

C

Really it's a good question. um uh No, I I think tara just made it optimistic think I think, was the biggest change. We didn't really have it or we weren't supported.

A

Well, this is also good to know because that means the initial thought process was in the right direction and we continued building on top of that and if I understand correctly optimistic think it is not still finalized. It is still under discussion, or are we clear on that? Oh.

C

No, so it's the spec is final, that the the thing is that it's optional. It's not, uh even though I think every consensus client that is open source uh implements it. uh It's still optional,.

A

Cool, maybe we will have someone on the show to talk more about optimistic thing in future.

C

But, but something something that you mentioned this is it's important to actually stress uh the original vr. It's uh it's one of the original vrs by by mikhail.

C

I mean lots of people worked on this on the first vrs on how to do the merge and it really hasn't changed.

C

So people say we're rushing through this, but it's it's a few years that the main idea of how to merge hasn't changed at all and the only changes are in user experience, uh optional features or the appy of how the engine and the plan and the consensus client is communicating, but the actual way in how the merge is going to happen and how the process is going to continue after we merge has not changed in. I would say a couple of years now.

A

That is incredible, kudos to all ethereum developers for like thinking of it and then continue working on. On top of that one question related to beacon chain in specific, is there any change that is being brought to beacon chain clients um after the march like? Will there be a significant change in penalties or anything new that wasn't there? I know there can be minor enhancement, but anything in specific.

B

um I will say so: I will answer this in two parts. The first part is user experience, so the biggest change for user experience is now that you must you must. You must run an institutional client right. You cannot be lazy and just assuming that you will never propose an event. You propose there may not be a deposit, so that, for your block, is still valid, so you must run as usually a client, but you have five five.

B

I think five different essentials that you're trying to choose for and that's the biggest change, and then you also need to specify a fee recipient like potus said. Otherwise, your e will be burned right and then on the implementation side, I will say just optimistic thinking is probably the best. Probably it's probably the biggest thing that we have.

B

We have to do so far just to like cover all the edge cases because, like unfortunately like there's, five different el clients- and they all kind of like do something slightly different than the other in their own good way, and there's always trade out between them so being compatible to every single one of them. Is probably the biggest challenge yeah.

A

That is nice, um like yes, I remember when we were having a conversation for altair upgrade there was this flexibility to not run the execution plan, so now it is a must and options are available. With this know, your client series, we are trying to share information about both execution layer and consensus layer client, so you will probably get to know more about uh uh options. Other than get obviously get is also there. That is also coming up with a lot of new features, but basu nethermine and arigon climb too so yeah.

A

We have a lot of options for you to choose.

A

um The last test net is going to be godly platter march, which I believe is going to be very critical with respect to specs, as that is going to be closest to the mainland and also the fact that godly and practical both are long-running testing. So are there any special preparations for garlic? Platter doesn't merge.

B

I don't think so I think, what's nice is that printer is already an existing beacon chain, so you don't have to launch another beacon chain. Our infra team, tehran, president, is probably happy because we must get more money on launching more prod notes but yeah. So I think we just need to decide um a balanced, merge e part and then, after that, we've decided on like a ptd, it's proper authority, so yeah it's it's going to be easier.

B

You don't have to like set the tv super high and pray that miners do not show us, and I said it longer again, so I think the process would be a lot smoother in terms of timeline. I'm not the best person for this. I would default that to tim and danny, so they will have to decide on that.

A

Awesome: okay, um my last set of questions couple of more questions to go that is about uh the community support. I understand that uh prism may have been lucky to get support since beginning like a lot of support. What do you think about the ef's client incentive program? Was that helpful towards prism and other clients as well to continue working on merch yeah.

B

For sure, definitely I think, that's definitely very like I think the money is nice, but besides the money, I think like people in general, wants to feel like rewarded right and, like, I think, like that's just a very nice way, to show that yeah like care, is like every single cordette out there, and I think that's a very, very nice thing to do right and then I do.

B

I mean we're super lucky to be in the position that we are so thank you yeah for doing that, and then he also allows us to have like a very like um sufficient wrong way to make sure that we can keep doing what we're doing for the next like three to five years so yeah. I'm I'm really appreciative for that.

A

And other than that, in terms of general uh community support, what can ethereum community do to support prison or prismatic law.

B

I would think just more bugs try to try prison more. I mean you should try all the clients. Don't try to do some more, but you should like. I would appreciate, like people just testing it and then give us feedback and just tell us like we suck and like and like which part we suck right. I I I do like I I'm. My biggest worry is just like everything smoothly, but post merge something happen so like I.

B

I want all the issues to happen right now and then happen really like really rapidly fast and put us anything that you want people to do more.

C

I just want to repeat what you said: uh I I I was a user. I am an user of prism and one of the things that uh strikes me as fantastic of this community and other clients have the same thing is: if you want something you they don't have it for it, and if it's deemed useful, they will make it quickly.

C

uh It's impressive how you can get to talk to the primary core devs in essentially every broad, every serious project that I've that I've had interactions so far and prism is one of them we're happy to take any advice. So this is one of the good things that you can actually help. Just try the software.

C

If you find something you don't like or if you in the documentations you you find something that you didn't understand just come and ask us, it doesn't make any sense to in order to use prism that you're going to be reading other sources for information instead of the prison uh webpage. Our documentation portal is revamping uh now uh we've just onboarded someone that is doing a great job in our dogs mick, and uh I encourage everyone to go and read them.

C

If you find anything that you think it's not clear, just ask for it to be changed. That's that's going to be very helpful for.

A

Well, you partially answered the my next question, which was about any advice for an aspiring, node runner for someone who wants to be an epidural validator. um I know it's time to wrap up, but if any one of you may have final words to add for our listeners.

B

uh I just want to thank puja for having us here. It's such a wonderful time to like just to show people that we have been working on and yeah just keep testing five bucks and then we'll get there soon.

C

So if you ask about like uh what advice for a new validator, I would say first decide if you want to validate, because we are going to merge, but withdrawals are not implemented yet and they don't have a date. So do not think that you're gonna just start staking and you're gonna be able to take your heat away immediately. That's not going to happen until withdrawals are implemented and we do not have a date for this uh other than that.

C

uh I have no advice, we're not supposed to tell people that we hope that it's going to be fast. We hope it's going to be very fast after the merge. It doesn't look like a particularly difficult implementation, but better not to give days.

A

I understand thank you so much for sharing this. My hope is, like withdrawal will be enabled uh with the shanghai upgrade, which is expected after the merge, but it may take time because there are other proposals in line to be shared with the shanghai upgrade potus and terence. We appreciate your sharing insight on trying development work and merge preparation. Thank you for your time.

A

C

Thank you for having us.

A

On this note, thank you to everyone watching or listening to this special episode on merge. Should you have any question, leave a comment or reach us at eatgathers.com check out description for links to useful resources and guest twitter to follow stay tuned for more on know. Your client series keep sharing your love with ethereum cat hardest and do not forget to subscribe to the channel, cheers everyone.