Ethereum Foundation DevCon 6 - Layer 1 Protocol Track, 14 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Hybrid PBS from CL's Perspective by Terence Tsao | Devcon Bogotá

Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/hybrid-pbs-from-cls-perspective/

The talk seeks to address the following questions:

- Why does PBS matter from CL's point of view? What happens if we completely ignore it on the protocol layer? What if we wait to release it?
- How did we implement hybrid PBS on CL clients from history to functional products?
- What do they look like in code?
- What are the gotchas and the trade-offs?
- What are the remaining questions we shall seek before moving to full PBS?

Speaker(s): Terence Tsao
Skill level: Beginner
Track: Layer 1 Protocol
Keywords: MEV,consensus layer,PBS

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

A

A

Pbs instructions, layer, I'm, Terence, I'm, a core Dev and uh I saw earlier this week. We are product of chain Labs, so I had to change my name real, quick before I get in trouble. So let's get started. So this talk is about consensus, layer interfacing with hybrid PBS. It's not so much your like Searcher Builder, your typical Mev Towers.

A

So we want to understand more like what does it mean when transcends this layer interface with hybrid PBS, from like from the it's it's tonality, such as latency file and censorship, then we'll also talk about mitigations.

A

Just to reiterate what I said, everything in the purple we will go over. It we'll have three layers matte boost, which is a relayer aggregator. We have consensus, layer, client, a solution, layer, client ability to the client- and this is what we have been me and my team and other teams have been working on over the last few months so background. Why are we here right? So there's options and options are nice, so the first options which most people use these days is normal blog processing.

A

So as the consensus of their client, you know which validators you're serving you know when validators are proposing a Blog, so you will build the blog for the validator and post merge. We utilize execution layer, client to prepare payload, so consensus of that client uses attitude on their client to prepare payload. They put a payload in the blog that passes to the validator validator signs. It return the blog and consensus their client broadcast the blog.

A

So here the separation concern is nice here because, like it's, usually a client and because this is their client they're, both like very complicated piece of software, but they only need to know each other through the engine, API and so the separation. So the separation of concern is very nice. The second option which people are starting to use more and more now, is that I can Outsource blood production right a consistently client if I want to participate in the Mev game. I can utilize the relay Network I can hey hey relate that?

A

What can you propose? A block for me and a Blog are usually more profitable and also in the background consistently. Client can also talk to the attitudinal kind of like hey. Can you make me a backup blog in the event that the relay network doesn't work? I can still use it, and this is kind of the Paradigm that we're heading towards, and it's important to understand like what does this mean? So this is today's number, and hopefully you guys can see it I capture it. This morning there are 55 of the network.

A

Participation is using Mev blog production, it is outsourcing their blog and then out of the 55. 81 percent are dominated by flash boss, relayers and the Seven active three layers. So let's talk about the first risk, latency right, so when, when you propose a block normally under your local setup, you have your consensus, layer, client, execution, client without their client. So how does this work? You ask educational client to prepare a payload, and then you pass the payload to the other client to sign it.

A

Well, their client sign the payload, and then you broadcast the payload simple, easy, very easy to reasonable right, but with Mev blood processing, it's a commit and reveal approach right. Why is it coming in reveal because, like as a relay as a builder, you don't want real data client to steal your um transaction? If it's? If it's, if it's in the clear text, so you made them sign it first then return the signature and then then you give the full transaction.

A

So, given it's a community approach, there's a more like Steps in the middle right, you first you get a header, the relay Network return, the header, then you sign the header, and then you submit the sign header and the relay Network return, the full payload, which is transaction in clear text. So, as you can see, there's a there's like two more steps here right and besides that it's just like it's also on a different network right, it's not local anymore.

A

So, ideally like you, want to relay Network to broadcast the block as fast as possible, instead of giving it back to you and then you broadcast it so hopefully, like most of the relayers have been like publishing blogs like by themselves, instead of just passing back so for that you do save some latency, which is nice. So, let's ask yourself right: do the additional run through and latency matter, because, like I said before, when you have your El and your CL, they are both in the local setup. It's lightning fast! It's!

A

It is reliable because they just go through Hardware. They just go through electron circuits right, but if the relay network is slightly different, you are actually talking to some like um infra provider, gcp or AWS on some regions, and um so this is I was able to capture some numbers.

A

Unfortunately, I haven't been home for like like the last two weeks, so these numbers are slightly old, but on growthy, with a thousand validators they're they're, not mainly validators, but yeah, just a thousand thousand validators, but given it's worthy, the network topology may be slightly different, but I mean I hope we can get the picture here right here: I'm, not even using Mev Bulls I'm, just talking to the relay directly. It takes about three times slower to propose a block given the additional latencies right.

A

So what does this mean when you, when there's additional latency, because when you propose a Blog, you shoot it how it works? It's like you run a four chairs to get ahead. Then you build a block based on the head. Then you get the payload from the execution engine. Then you broadcast the block right and then at the four second mod, which is one third of the seconds per slot. The testers will vote right.

A

What is the head of the chain and if the testers did not see your block, then your blog may get orphaned and that's not ideal. You do not want to lose a lot right, so this is what we don't want. We don't want something that's taking up so much in the Middle with the gay header, second header and suddenly blind block right. So this is this. Is this to me is worrying and then let's look at some more numbers, some block arrival, latency differences. This is actually capture on mainnet.

A

This is my home setup at home with 300 Mega bandwidth. So this is a home setup as possible and we capture over 15 000 samples and map, and then the MVP block took about 500 milliseconds longer right and um and what does this mean from the uh some big anticipation timeline? It just means that if the block takes longer to reach to to arrive, then a testers will unfortunately miss it like that right, you don't.

A

If you're waiting like in the front like for that long, then you would essentially eat up the time that you have. So we don't want to be the lazy block. We don't want to be the late block because you would get orphaned right at the top example, Block C was supposed to build on Block B a and then because the blood B was laid on.

A

The bottom example of block e instead of building on block D because of proposed boost or something it was supposed to be the head, but it's not the head, so block e built on Block B. Therefore, C and D got orphaned right. So another set of numbers, 50 of the orphan block, actually came from relayer from September 17 to 27 and there's our the orphan plus law, the relays they're using validator ID the entities right and that's unfortunate, I mean I mean you could ask that okay!

A

Well, maybe it was gonna, get off and one or another we don't know, but still like 50 off. They come from the relayers. It's not just like sunshine, rainbow and Butterfly right. This is like like this is I mean there is risk to this right. I mean like we often tell people hey, we should use the relayer because it's more profitable, but it also comes with risk right if using a real layer, makes your blood two to three times more profitable, but you get offering ten percent of the time. Is it worth it?

A

That's something that you have to ask yourself about, and another risk I think with latency is your centralization right. The whole point of Mev Booth is like to make validated decentralized, because now we can extract Mev at home, so it gives everyone equal access to Mev. You don't want to look like this right. You don't want people to start realizing, oh well. If, if I have a better latency, then I won't get orphaned, so I'm gonna move by a home stating node close to the relayer, so it it becomes a negative externality.

A

This is what we don't want. So what's the takeaway with this right, the takeaway is latest thing matters both for hybrid or even maybe, for a little PBS right and latency can lead to centralization risk, which we have known a long time already, and it's actually really hard to optimize towards Network latency right at the client level. Like I, don't think, there's much. We can do I. Think on the map boost side. There's talks about like instead of Marshall and Marshall Json with emotional module, SSD the there's something improved from there from the real perspective.

A

I hope like they have really good Network config. They have a lot more peers, I feel like they have a robust like infrastructure. There I think that's as much as we can do, because, unfortunately, if someone just have a slower internet connection, they want to use a real layer. I mean this is going to be affected right and it's important. We educate these risks besides. Latency there's also thoughts too right. So what does fault mean? Fault here means that sorry, the stock got a little mess up there point.

A

So when you ask for the header, the relay Network failed to return the header. So that's the convey phase or the happens when you submit the header, the relay failed to replay failed to reply, the payload, so that's the second fall. So we'll focus on these two type of faults here right so the first so the first category of faults is just gear header false when you get header and then the relay net will fail to reply and they can be categorized as, like. You have a mail form header.

A

You have a consensus invalid header, you have a payment, evaluate header or you have a non-conforming header, so I will go over them one by one. So what does it mean when, like a head? Is male form right? It just means that it is syntactically invalid. It has an invalid structure, it has to email, the signature. Can the consensus can? Can the CL client detect it? Yes, it can right, which is when you are more show it. If is not the right structure, then? Well, you know it's run.

A

You also can verify the signature, so this type of so this type of like false, we can detect and we can mitigate. So this is fine. Another type of fault is just consensus, value header, so that just means that the block hash is reality, the transaction similarity, but for this we unfortunately cannot validate, because we cannot see the full transactions right. We are. We cannot calculate the block hatch ourselves. So that's something that we just have to trust blindly, that's unfortunate, but it is what it is. Then you have payment invalid header.

A

So at this type of fault it just means that well, the Builder was promised to pay the validated proposal some like some. If but it failed to pay so consistently consensus that your client cannot detail that type of file this which we like relay, we we trust, relay to simulated, that's why relayer is trusted for us and then there's also the non-conforming head. That just means that when the validator register, it basically say hey, this is the guess limit I want to use, but I guess the name is incorrect. The time stamp is incorrect.

A

The block hatch is incorrect. Right can consensual client detect that, yes, he can. So those are like what we say. The commit faults so now we're jumping to reveal false. So this is the set second type of false right. So what are some like reveal faults, so the payload could be invalid or or the payload is available and keep in mind. There's no falling back for this, like if you just you sign the signature ready right. So at this point you can complain on Twitter. You can probably like complain something else, but there's not much.

A

You can do here to basically make the block as a whole header. It just means that it is invalid, like the block the flip Halo does not match the header. Can the consensus layer client validate? Yes, it can but hey it's too late right you at that point. You lost the block already, there's not much you can do and it's the same with consensus in value payload. You can validate the transactions now because we see everything but hey, it doesn't work too late. Sorry then unavailable this.

A

This means that the relay Network just went to sleep when malicious decided to turn off. It did not reply back the header to you, so it did not fulfill its commitment right and then you know when it does that, because he never replied, he never received the payload and there's someone and there's still there's nothing. You can do there because you already signed something your signature is out there. We have this concept of falling back to a situation, layer, client, just if gear header goes wrong. You can produce real local execution. Client, that's that's!

A

That's probably fine, but if the gay payload fails, then you cannot produce with your local assistant client, because you don't want to double sign. So the return header the first commit can fail two ways it can either fall or time out right. We do prefer for better.

A

Just because, like you get a response right away, you can start proposing the blog right away if the timeout it kind of sucks, because you have to wait for them to time out you wait for them for a second, then you lost a second of like your apprecious time there. So, let's go through some many incidents. I, don't want to sound like I'm like like pulling out their fault incidents, but I do think it's important to go over this type of incidents so that we can learn as a community right, the first type of incident.

A

So the the first incident is um September 16th flashbacks relay they fail to our Marshall deposit for the payload reply, the damage we missed, three blocks. The second one blocks rewards relay this. One is when the this one is going to relay, did not validate the block and then and then they rep, and then they reply it, the Constance smelled it. We miss Ada blocks here. The third one is velociros again, this is consistency, value payload and the damage of that is 15 blocks was missed, so this type of things do happen.

A

Right first happened and, and then it seems like it's mostly happened on the on the commit and then also the reveal phase. So we need some mitigations like the circuit breaker and, for example, as a beacon client I can detect when there's a liveness failure and then, when the life has fitter is determined by clients say if you miss three shots in a row or if, if or the chair misses eight slots over 32 slots, if that type of things trigger, then we just default to local X3 engine right.

A

So this is to prevent like a dominant relay, Builder go offline and it doesn't solve, like the cases I mentioned before. Just because, like those are happen, maybe like 0.1 percent of the time for this, this happens all that. But but for this, like this is a stronger defense. Then you have your relay monitor, for example such that you can monitor, relay space on performance such as behavior and such as Behavior, which is like safety and liveness and performance, which is latency and people can see how the relays are performing and people can figure out.

A

Okay, based on that do, I want to connect with every day so did it just makes the information more available, then we have features like beef filtering solos. So as a proposal you can, you can say, hey I only want to use the relayer if they give me something that's over this value right. So there's some like so there's some nice things. You can do there. So what so?

A

What is takeaways right, we're still early, but we need more robust relay and also we need a way to hold relays more accountable right, for example, a simple idea: I, just you monitor all the missing slots and the offering slots live, and then you pull the API. The relay API and then, if you see hey the missing slots, is coming from the relay okay, I'm gonna, just like shout out loud Tweety or something so that people know hey they're like they're.

A

It's the incident right there is happening, live you need to turn off your relay or you need to switch local processing in terms of just like faults, time out, I think I prefer Gear Head defaults. Then that perfectly had the timeout. Then I prefer forget Halo false just because, like with gate header files, you can propose the blog just like still and then it's it's most likely. Fine and I do believe that relay quality will improve over time.

A

Just because, like it's still relatively early-ish, so I mean they're still learning and we're still learning and something that will proof over time. So, okay last section censorship, that's something that we have been talking a lot about so as of today, um let's see whatever set 49 of the many blogs have some sort of effect, compliance building and that's unfortunate because that's already over half right. So we have to understand like who is censor right like who can censor. Builder can sensor right.

A

If you're a builder, you don't want to build blocks, they can send like offer compliant transaction. You can censor or the relation sensor as well right. So so, therefore, it's really hard. It's really hard. What's the problem here problem here is the enemy boost is a neutral piece of software.

A

It doesn't care about censorship right and then the ux of just figuring out like how do we defend and censorship is still early right now it just essentially everyone chooses the relay, that's non-censoring, that's it right, but then it's hard to figure out who is censoring at the given time? There's it's like you, don't know who is censoring I mean we just look at the news with the real search and that's it. We need more like information there, so potential Solutions right.

A

These are very instrumental ideas that I just have been thinking top of my head right. You can have some active encryption, such as MacBook C or this you can have some solar, essential issue or censorship Oracle. So for active inclusion right high level how it works. It's just like a proposal. You express the intent to like force transactions into the payload right and the relayer has to present those slide transactions to the Builder and then proposal will only accept those transactions if they are included, or the block has been full right.

A

So for this you do require some sort of multi-proof to make sure that hey the transactions are actually included and then consistently client can do the validation. So what's the downside with that right? That's not that we just like there's more timing, because now, as a proposal at the previous slot, you want you do need to send the transactions. You want to send a transition transitions during the previous slot. So we need to figure out. What's the timing for that, and now the proposals also need access to the mempool to the institutional area.

A

State there's also some latency complexity here, just because, like now, it may take longer to propose a Blog and then yeah, and then you can also have like this is like a poor man. Version of just like censorship like um filtering. Basically, like like proposal, you just walk into the mempool for like the top earned transactions.

A

That's that's that's based on ksv at the given time and then when the reply, the relayer is also have to show a proof that hey this top and guest transactions are included unless that the guest is at its price at a given time right. So we don't really trying to force transactions inside, but we just want to make sure that the top end transactions are actually getting in and and then which, when here we're assuming that okay, if someone's in the sensor, someone will probably use a higher dsv and stuff.

A

So it's not ideal because it kind of loses like this inclusion control, but it's probably easier to implement. Then there's also like ideas like censorship Oracle, you can introduce like a new actor to produce censorship, kind of like relay Monitor and now, as a proposal I mean boost when I receive a header. I can add this Oracle this new attribute. Hey. Is this a sensor one? If they reply?

A

Yes, then I won't use it right, but then again you're putting Trust on like a new actor and that's obviously not ideal because like who, because who is going to monitor the censorship Oracle and there's always that problem. So there has been a lot more research going on which I'm really happy to see right. We posted a research a couple weeks ago on just like how do we constrain Builders without bringing blood proposal burdens to to the proposal?

A

There's ideas like using blood, prefix and stuff and I highly recommend you to take a look at that and then there's like from Barnaby I think it came like um last week, which I haven't had time to read it. It basically enforces that proposal, commitments on chain and then I think that's very neat. I definitely want to take a look at that very soon.

A

So what are the take care of ways right, I think they take away with this, just like it's important to figure out like who can censor and then who can feel the censorship, because there are so many actors in the future. There's proposals. There's there is some Meb boosts and then there's also the relayer right, it's important to figure out who does what and then I do think like. We need to leverage the Builder API a lot more, because the Builder API is probably the best thing we have today. It's the best defense.

A

We use Builder API a lot more. We can like provide more ways to like to basically organize defenses against censorship, such as like inclusion, this block prefix, and then there is like a spectrum of solutions and then we're out there which we're thinking, but the simplest solution typically has like more trusty assumption.

A

So I'm not sure, if that's the way that we want to go so some final thoughts, I think like for me, I think censorship resistance should be the highest priority, a size from scaling and withdrawal like what is ethereum if 50 of the transactions are censored right, that's something that we have to ask ourselves about right and I do think. The hybrid PBS is basically our best two bars to defend against that, because it allows fast, iterations less and then before we enchant into full protocol PBS.

A

Then we kind of lose the step, because everything will be hard for based right for high PBS. You have to Builder API, then you can play around with that and then you can. We can figure out like what works best and what doesn't work best and then I know like people that have been working on high bpbs such as Med boost relays Builders. They have been getting a lot of like bashing and stuff I I. Don't think it should be that way.

A

I think we should be like working together as one team and to like to basically Advance this um Advanced. This um censorship thing forward so yeah definitely um shout out to other teams that have been working on. That I mean they are the real heroes, so yeah, that's all I have today. Thank you so much for having me and yes.

B

Thank you clearance. We have.

C

Some questions for you hi, uh so regarding latency, the numbers you showed would more or less the same as Dennis in the first day. So, if you have 50 of the network proposing through three layers and 50 of the orphan blocks are coming from, relayers I would have taken that as a sign that latency did not matter. That would be the expected. Yes.

A

But I do think that we can do better from the orphan point of view right, because now, if you look at like 20 plus kids orphaned a week, if like if, let's say just have temples they get orphan, then it's probably better than 20 plus video orphan right. It just I'm coming from more like the orphans perspective, because when something orphaned, then it's obviously not ideal, because that transactions could be included, and there may be some U.S concerns.

D

There, hey Terence thanks for a super informative talk. um Can you say a little bit more about the three relay faults that have happened recently like how were the malformed, headers or or payloads generated and like how was that mitigated? How do we prevent that in the future.

A

Right I would say, like the relay landscape, there's still a lot of work to do there, because I need more testing and these bad tests and more end-to-end tests and in terms of the fault right, the first fault is Flash ball, and then they fail to Marshall the deposit, and that point is too late, because someone already submitted the header and stuff there.

A

You have a signature, so there's not much, you can do there and then, but then they fail to our Marshalls because they did not test um the payload with a full signature embedded and the second fall, and the third floor are basically. This is the second incident and the third incident are basically the same and I think the plot relay just so.

A

The block Rod relay did not validate the payload, which is what they're supposed to do they're supposed to make sure the payload is welded before they pass it to the proposal, but they just did not validate yeah, hey.

B

Turns thanks for the talk, uh I was curious about the out of protocol CR lists and the proofs of transaction inclusion. uh Have you done any Research into it? Is it practical like? Would the builders or the relayers be actually able to calculate the proofs in time.

A

Yeah, there's something still under research. I think like Chris from flashbots does high just open the pr, so I need to look at it, but high level I think like how it should work is just that say: you're proposing a slot n at slot, M minus one. You have access to the main pool, and then you see some different sessions. They get filtered you basically I started minus one.

A

You send those you basically present those transactions to the relayer and then relay will also present those transitions to the Builder, and then the Builder will essentially build a block. Just include those transactions, real, quick and then send it back to the real layer and then like so at the end after the signature is done.

A

When you get the payload right there, so they they can include like they can include, like the multi-proof, to basically prove that the signatures, sorry to make sure that the transitions are indeed like are are basically indeed included in the payload I. Don't know yet that something's still under research, yeah.

E

Hi uh Alex made coinmetrics. Thank you so much for your talk really loved it just wondering if you would comment on your opinion about maybe non-public relays do they exist. Is this a possibility just wondering.

A

As for now, I, don't know any and I'm probably relay I only know the seven relay that was present in MVB, boost.org and I also think that if well, if there is non-public, relay It's, probably hard for us to know just because it's coinbase is using a relay they're extracting Mev they're not open to public I mean that's, not something that we can easily find out.

F

um Why do we think that proofs help in that like what he was talking about before peripheral inclusion of transactions? Why can't you just if the things that are not included once you see the actual payload disconnect from that relay, because I mean the relay, even if there is a proof, could always just not release the block and order the block could be invalid or there could be all kinds of faults that make the proof kind of meaningless and what you end up doing is anyway disconnecting from the relay in either case. So just.

D

G

F

Make it simpler on ourselves, and just you know you you say: please include these transactions if they don't want to disconnect from them and next time that won't happen.

A

No yeah I think yeah I mean I, fully agree, I I think that's possible and I think there may be a better solution. Yeah I think yeah, hey.

H

Yeah um I apologies I've already missed this before, but you said that uh 50 of the network is making use of Mev boost. Do you have um more like fine grain numbers in terms of like whether or not uh like what the numbers of institutional stickers versus like homesteakers um I, just know like an institution that I work at they're security considerations, it may be boost it's sort of like a centralized.

A

Yeah yeah actor at the moment. Yeah there is a site for that I'm happy to share with you after but yeah, there's a site that actually I typically track. Like example, Lido coinbase Finance like the top three like like just like what relays are they using and stuff like that and what percentage for each layer yeah we there? There is active tracking on that, but I don't have the data in this slide. Sorry yeah, but they're yeah, but there is a site for that.

G

Given all of this a validator, that's deciding whether they should run Mev boost or use Mev or not. What would be your advice to them?.

A

I will say this is a hard question for me personally and then me and my team have different debates. I think for now it's too early to be using this type of Technology. Unless you know what you're doing just because we don't have any like public infra out there for I would give you an example just like, for example, for the second incident right when the block Rod relay has support.

A

It took them six hours to turn off the relay which they could turn off right away, but as the public validators like we don't know right, I, don't have the access information be like hey, there's something wrong: I need to I I, basically need to shut up MVP boost people. The best thing you can do that people can go to Twitter, but I'm, not sure. That's like the best like media for these type of things. Right so I would say, wait yeah, that's that's my advice.

I

So you mentioned at some point that if you can't reach the relayer, your execution, client you'll, the sorry, the consensus client will reach out to your execution client and build your own block it'll default to that. So that makes sense if you can't get the header right, but you mentioned, there are two situations, one where you sign the header and return it, and then you can't get you know a receipt of that. Is that not a slashing risk? No.

A

You cannot get the right so once you sign the header, you, you basically pass the header to the relayer. You you don't want to like. You, don't want to use your local blog anymore, because now it's a slashing risk, because now you're assigning two blocks.

I

So so you don't you don't default to building your own Block in that situation.

B

No okay, I misunderstood yeah good. Thank you very much. There is amazing talk.