Everything EOS EOS Node Operator Round Table, 21 Jul 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Configuration changes and Feature 5.0 - Enhancing Transaction Efficiency July 19th

Description

The EOS Node Operator Roundtable meets every Wednesday at 14:00 UTC to share and discuss feedback to improve the Antelope protocol.

00:00:00 Welcome
00:01:30 Configuration changes
00:22:42 Feature 5.0

The EOS Network Foundation coordinates financial and non-financial support to encourage the growth and development of the EOS Network. We’re harnessing the power of decentralization to chart a coordinated future for the EOS Network as a force for positive global change.

// EOS Network Foundation //
https://www.eosnetwork.com
https://twitter.com/EosNFoundation
https://www.linkedin.com/company/eos-network-foundation
https://t.me/EOSNetworkFoundation
https://discord.gg/eos-network
info@eosnetwork.com
#EOS #Web3

A

But welcome everybody July 19th another note operator, Roundtable and I'll pass it over to Brian.

B

Excellent, thank you Daniel good morning, good afternoon, good evening and good night to all of the antelope node operators.

B

um Thanks for joining again today, we do have at least one known agenda item that I want to make sure that we cover, um which is um related to some recommended configuration changes and coordinating um the process of of testing those configuration. Changes um which are is going to be facilitated by Eric and Kevin, who are on the call today before we get into that. I do want to have a quick open floor just to gather any potential topics.

B

um We won't go into conversation on those topics yet because I want to make sure we have time to cover the uh um cover that primary item.

B

um But yes, any any topics that folks have on their mind that they want to uh discuss. If there is time available today,.

B

B

All right, I think we've given sufficient time for people to find their unmute button. uh So with that I'm gonna hand it over first to Eric and Eric. You can sort of facilitate uh from here on on the topic of configuration, changes.

C

Thanks hello, everyone yeah I did want to talk about some configurations and changes uh with a lot has changed. We've been doing a lot of work on the and we've been doing some performance testing. Along with the scene, uh data in the live environments. We wanted to recommend some configuration changes, I posted them yesterday, but I'm gonna go over them again. Here um you know. The reason for the changes is: we've noticed. Some transactions are bumping against up against this 13 millisecond time limit that they have.

C

In addition, we introduced the read Ali transactions and there's some specialized configuration settings for those the purpose. These changes is to uh give us more Headroom on transactions, so we don't hit that 30 millisecond time limit and also to make the full use of read-only transactions.

C

So, specifically, we've got the read only window time, which is in microseconds and we're recommending setting that to 165 000, microseconds and the, uh and that read only window is to uh support the read-only transactions.

C

The other chains, we're recommending is the max transaction time to 151 milliseconds and that's just 151. This is in milliseconds, uh we'll be rolling. These changes out to Jungle. First and as you guys may know, we asked jungle to make some other changes that we're testing and those who are multi-sick changes or Max transaction CPU usage and Max block CPU usage. We're going to ask them jungle to roll those back the match.

C

What's on the mainnet, uh we want to test, you know incrementally one thing at a time: I'll also I think the multi-sig changes should really come after we've done these first set of configuration changes so where I'm going to ask jungle to roll back those multi-sig changes, roll out the new configuration changes and then we'll see how things go.

C

uh I think we'll want to test it, for you know not too long, because these changes look pretty benign uh to to me and most other folks on leap, team, I'm thinking a couple days, maybe a week so five or seven days, that's what I'm thinking uh after that, we're gonna come back and ask the node operators here to roll out the changes.

C

I think we're going to want to have these um to be somewhat coordinated and and here's the reason why, if we're up in the time limit for transactions and not everybody, has the same time limit and some node producers would accept transaction While, others could hit that time limit and then transaction not be allowed. And then you have uh different blocks and you have a little bit of weirdness consensus and we don't want to see confusion happen at all. So we do want to roll that out. I I, don't know how.

C

Often we are bumping up against that 30 millisecond time window for our transactions. It's I, don't it's not too much uh so I'm not worried about getting a precise rollout time for everybody, but that is a reason for us to have some diligence around this and if we do want to go forward with this change after evaluating a jungle, I'll roll it out and not somewhat coordinated fashion, so I'll pause there. That is my Spiel. Does anybody have any questions? Any concerns, uh any other related things that you want to bring up.

C

So I'll be posting uh the same note in the uh jungle forum and we'll have the discussion there. uh Making these changes and I'll get a time stamp for those and communicate that back in our forum,.

D

So uh is there any Benchmark results you can show for internal testing related to whatever you're changing already.

E

So you know it in in this. In this case you know we we because of some of the concerns that was I, think about last time or time it will. Last about the consensus level, Max uh block CPU usage. You know so we're not recommending uh those uh those changes at this time. I still think that chain could handle it personally uh and I. Think that's something we could do I think there's more concern in terms of like what uh what kind of additional load that might put on History, Solutions and other things uh Downstream.

E

What we're specifically asking now for is just the um you know: the Mac, uh the subjective, Max transaction time or node, so it's not going to be um Illustrated in terms of like throughput increase or anything like that, because um you know all this is is the point at which you allow a transaction to propagate or, or you know, um pass speculative execution on the Node or ultimately uh to be placed uh in a block, so it just gives that larger, breathing room for uh EDM uh transactions, so it wouldn't be so much like a performance sort of um uh increase in in that regard, unless you're reading performance in terms of like the amount of CPU it that makes it into a walk, in which case you know, there's you know at least the potential for that because of these, these larger transactions.

E

So the I think the more concern is just in terms of like um you know whether it's a subjective billing is is sufficient for failed transactions that run longer than than 30 milliseconds right. So that's I think that's the the major concern here is the fact that you know, apparently, as most people have it and figured at 30 milliseconds.

E

Although I think some VPS actually have that slightly larger than 30 milliseconds on mainnet these days, but wherever that is whether it's 30 or 50 I, don't know I think somewhere around 50, then um you know if you've got a transaction that would happen to run for the full say, 50, milliseconds or 30 milliseconds. You know, then, that that amount of time that we're subjectively built for uh and if it happens on a VP, then they're.

E

You know it's potentially uh Batman amount of block time that is quote unquote, wasted uh effectively executing uh well, not transaction, so I mean that's all kind of information. You all know so the you know the the big thing here is a long way for me to say it's difficult that they kind of test these sort of scenarios uh in any sort of like contrived, controlled environment, sort of testing.

E

um And so you know, that's why you know we're wanting to like roll down the jungle, get a feel for it there, but even on Jungle right I mean we can realize that um you know it doesn't have the amount of load that you know the mainnet has, and so you know and like right now we're watching mainnets having a huge amount of transaction flowing and that sort of thing it's like it's during this sort of time period like right now on Main Network we'd, really get any sort of real sort of feedback for what happens to the network, allowing these larger transactions to flow through I, I I think the impact would be a small on the the biggest risk here, of course, is somebody's doing a a uh a transaction.

E

That's designed to fail by using up all the CPU it can um and ultimately that's going to fall back to our subjective billing protections for those types of transactions. Somebody can do that. There's nothing. You know preventing someone from doing that, obviously, except for subjective billing, and so the only difference there is. You know the subjective billing. If you will kicks in maybe a little bit quicker with the 150 millisecond seconds, because they're on their quote, unquote: waste 150, milliseconds or they're going to build for 150 milliseconds of subjective time as opposed to 30 milliseconds.

E

Oh oh subjective tire, so we'll play that maybe addresses some of what you're asking there.

D

uh Sure I guess the other question is: is a y-151 and not 140, yeah or 120 or.

E

Yeah yeah yeah, no, no excellent question, so we we picked 151 because the on-chain can census value is 150., so 151 puts us above the subjective limit and bait and effectively gives us the objective limit on chain and and the reason for that is just the fact that then you get a knock. You get a slightly different air message, which is a nice. uh You know which is telling you that it's the injector cellular as opposed to the subjective failure and so forth.

E

We thought for users that provides us, maybe a slightly better kind of indication that uh watch blocking this transaction is not any particular node operators configuration or really the on-chain consensus value, which I think at some point, maybe sometimes soon we would be back talking to you about changing the on-chain consensus value. You know we're not asking for that now, but potentially you know we might entertain increasing the on-chain value as well, but certainly going from 30 to 150 gives us a whole different uh uh magnitude of what's possible, particularly you know you know.

E

Do we want to be? You know, particularly in the EDM realm, where you know the a lot of little transaction because they are running in that environment or uh you know, are heavy, uh and so that gives them some more breathing room to uh to to run out with them. Those you know within that timeline.

D

So currently I have the subjective, the max transaction time set large on the BP node and smaller on the Sentry node.

D

So to prevent uh cases where you know it uses up a lot of CPU time fails and then you get those empty blocks so trying to catch them, so they don't even make it to the BP and that was set up when Kevin you and I were discussing the empty blocks. You know like two years ago.

A

D

With this recommended configuration, you know that goes away and because everything is bigger now right so just wanted to be out any all right. Let me know configuration versus the block production, yeah.

E

I mean that's, that's a really good point, um one that I haven't given a lot of thought to is the hardware the same on your Century as on your BP yeah.

D

E

um Yeah, well, that's that's interesting. You might uh because of that yeah. You might want to consider uh like on your API note, going with the the 151 um potentially um or or you know, maybe slightly lower on on your API in your Sentry new, um just to give that maybe a bit better feedback.

E

If that's what you're thinking and that would be fine like if you want to leave those or say uh uh 149 or you know one maybe 145 or something, um and then you know allowing the the BP at 150, you know 151, which pretty much gives it, which then gives it to consensus value of 150..

E

So I think that's that's that that's perfectly acceptable and inappropriate, um and you know- and maybe we could uh maybe we could discuss um recommending that for for everyone as well, because you're you're right that does give you potentially a better feedback to the user.

E

um You know the the fact that okay, you know that this held in or you know it. It succeeded until I've got pretty high like with it. Then it's going to see on the other now I will say with your nice uh transaction tracking, though uh you know, users are able to actually see. You know see that as well, and you know, I would think that there would be very few transactions at least initially, uh as people were, are getting used to the higher values. I would think there'd be very few.

E

That would be on that edge of that amount. um You know, because.

D

Yeah I mean this was back at the time where somebody was targeting trying to hit the 30 millisecond boundary your 29 millisecond right, and then it would just sometimes right slightly over and sometimes slightly under in the slightly overtimes. You ended up with empty blocks. You're.

E

Right right- and so you know the the Hope or the thought here again is that the subjected billing is our. Is our guard against this sort of you know Badness if you will right and so in a sense that it makes that more expensive um in terms of like you know, heat failure is going to cost them 150 as opposed to 30..

E

um So you know they're the.

D

Cpi is cheap, yes, yes, it is right, so you know we may need to look at the price of CPU on the network like charge some real dollars. You know gas fees on ethereum. Are, you know quite a bit higher than CPU fees on EO like if every, if every failed transaction, all of a sudden costs you a dollar?

D

You know that changes the cost benefit analysis. So doing all these fire transactions.

A

E

Well, the other thing is, you know, you, you've got a lot more tools now with the subject to billing. In terms of like you know the window and and how many you know how aggressive you are when uh they let hit the the three strikes or whatever limit you want to pull out. That's now configurable as well, so you know you can put a three strike limit over a dozen blocks right and that makes that rather expensive. If somebody um you know, is trying to to get those still transactions.

E

uh You know into your block and using up your block, tie yeah.

D

I've set it to two for now, rather than one just to yeah, don't go too extreme to set it to 12, but right right, but so far it's a two.

E

Yeah and those sorts of things that we may need to reevaluate when we um you know when we roll this out I mean ultimately, we know like it's, it's extremely difficult or uh well impossible to really simulate what happens uh in in, like us main net in terms of of the amount of different accounts and transactions that are flowing and the interaction between all of those and so yeah we may have to uh you know we may have to Pivot.

E

We may have to get maybe even a little bit smarter about our subjective billing again, uh hopefully there's enough knobs there that we can, uh you know, react to appropriately. If need be um and and then ultimately you know we if we roll it out and it all hell breaks loose, then you know it's, it is a setting you can.

E

You can dial back if you need to or hopeful that that won't be the case, because you know we're we're hoping that you know the US may now becomes a place for people become and run their large transaction s.

D

Yeah, okay, so I think the initial messages you know were a bit short on these kind of details, so we're gonna roll things out. Maybe we need to think about well what happens if things go wrong? What are the implications? What are the other settings? You know these kind of messages to the block producers.

D

This is a good time to educate them about the new stuff that has been included, which maybe they don't know about. So maybe a blog post that's longer than you know, a one paragraph.

D

Telegram message would be appropriate.

E

Yeah thanks excellent feedback.

D

Yeah so just provide context right, because that was what was missing. It's like hey. We want to make these changes. Okay, sure.

D

Well, that would be my suggestion anyway,.

C

Yeah, it's interesting on the um going back to the performance, Benchmark testing, I think with the performance and The Benchmark testing has really given us a sense of confidence about how much Headroom there is and I think we've also with a performance testing focused on down on the changes that are needed to you know really increase our throughput are those multi-sig changes and Max transaction CPU and Max block CPU usage, but going back to testing like the per transaction CPU limits we're not really doing failure testing in the sense of we're not really creating these long-running transactions and trying to run them against the node.

C

When we go back and look at getting a variety of transactions to run we're mostly looking at replay stuff and, of course the replay stuff is only going to be the successful stuff. So I think that's why we don't have as much of the empirical testing this particular change. But at the same time we still have the confidence, because, based on the testing about how much Headroom uh we've actually got and uh knowledge that there's other up, there's I, don't know. I would call Upstream.

C

But there's other configuration changes that are the real bottlenecks and then this one will be model on bottleneck, but there are some on into always. You have to watch out for unintended the consequences and you know Matthew. That's why I'm glad you brought that up. I think a blog post is a great idea. I, don't know how much like really techy blog posts we do, um and so that would be a fun discussion for me to have with some other folks in in.

D

Enf yeah I mean the lot the last blog post was about subjective building, and you know our team wrote that one and posted on the enf blog. It's.

A

Nice excellent.

C

D

Or more it was Ghost written I guess.

C

Secret's out now, yeah.

B

Yeah all right I need final thoughts on this or next steps that we should be capturing.

B

Hey well Eric um and Kevin did uh have you covered everything that you needed to cover in this? uh This topic.

C

I've been taking notes, I was gonna post a little mini recap in that note: operator's a telegram.

B

All right, um so we didn't identify other topics at the top of the call, so I'll have uh sort of an open call for that uh last call for topics to cover otherwise we'll close early today.

E

um Like talk a little bit about um feature 5.0, that's um some, maybe some other what we were just talking about sure.

E

Okay, so you know we were talking about uh subjective billionaire and uh empty blocks and the possibility there. You know one of the things that I'm excited about and the 5.0 release that is already merged in May is the uh Improvement to uh block time. So I haven't talked a lot about this um I, don't think, um but so what we have in in Maine, which will be in in the next release, is that right now, when blocks are produced, they're always they're always started on the half second interval.

E

So now they may finish at uh various times, either at the at the very end of that 500 millisecond interval, although that is unlikely because most people are using uh either the default uh CPU effort amount or they provide their own CPU effort amount.

E

I, don't think anybody is probably running with a hundred percent CPU effort under under block there's, certainly not for the last block, because that causes Havoc, but potentially they might be for the other walk, uh so the default there is 80, and so that you know that that gives a little bit of time where the block is a stop, whether it's full or not, um giving it time to sign it and ship it off over the network and uh and then it it effectively, then pauses until it starts the next block time right until that happens, for each of your 11 blocks and then there's the same sort of thing happened for the last block, except for there's, as you know, there's different configuration options for the very last block uh in terms of like how much time you'll be allow to run so again when you're producing a block, um there's two limits that you can hit.

E

That then indicates that that block is complete and it will ship it out over the network.

E

The first limit is that CPU effort, so once you you hit that time limit according to your CPU effort and and block offset same there's, two configuration options for the same thing um and- and it was mentioned, I think the last node operator Media or, if not the one before that, that we really should collapse that down into to to one setting um and just have one actually, that's already in Maine I'd forgotten about it and that's part of what I'm talking about.

E

Actually all those settings are gone and all that's left is a CPU effort, a percentage that is uh taking um and there's no longer a last block setting there's only the CPU effort uh setting that exists in Maine I'm, going to explain how how that works now so before every half second interval is when the block was started. Now what happens um or you know and I'm sorry then there's two limits. Then you can hit that then finishes out the block and sends it all over the Block versus xcp record.

E

The second one is the on-chain Max uh fee of CPU block, which is one of those consensus, changes that we that was changed on Jungle, that we were originally asking you to change on on Main net that we, you know, kind of pulled back on at least temporarily. For now, that's currently set at 200 a millisecond.

E

So what happens? Is you can hit one of those two uh limits? If you hit the the 200 which it means that you created a block that has 200 milliseconds of billable sleeping type within it?

E

So when you hit that limit, you can also hit the net limit as well, but I, don't I, don't think we ever heard two, but you can it's possible to hit the same limit for net as well and I uh it's set at one megabyte, um so you can hit either one, but typically you hit the 200 milliseconds, so buildable CPU tie for a block.

E

So what currently happens on on 4-0 and 3 and 2 and 2 1, you know all the previous versions, uh once you hit that 200 milliseconds you'd finish the block and you ship it off and you effectively. Then sleep, you do nothing, you basically just wait for the next half a second interval. So that means that if, when you start your block production, you've got a number of unapplied transactions that are queued up, which often is the case uh and I, don't know what off their ears, but sometimes let's say: let's go with that word.

E

Sometimes it's the case that you actually have enough UW, an applied transaction that you can fill up your first two three. Sometimes the complete weld blocks with with transactions before you've actually drained your unapplied transactions. Basically you can.

E

You can spend your entire round doing that um and and that's done and those little chunks like you, you do 200 milliseconds work and sleep for 300 milliseconds and you do 200 milliseconds worth of sleep, and then you do 200 Mills I can work and it- and it goes in that that route until the last block, which they have configured typically get configured smaller than 200 milliseconds, and so it runs through 100 or 10 or whatever you got it configured for and then it's done it and shits off now.

E

What happens in five O's we've changed all that so, instead of having the configuration or the the 11th block and the different ones with the 12th, now we have D configuration, which is the CPU effort and the way this works now is that it starts a block and if you've got enough transaction to fill up the 200 milliseconds of the consensus of value of what's allowed in that block. It will actually finish that block ship it on the network. That's all the same. The difference is in 5.0.

E

It will actually begin the next block immediately and it will potentially fill it up and ship it on and if there's more it will start the next lot, and so you could actually within um say a two block round, which would be one second, you would typically would do a Max of two blocks. It's now possible to do um three blocks.

E

uh No four four blocks. I think that right, I, don't know two uh three and a half uh you can. You can do, thirdly, to three blocks right yeah, it sits under milliseconds you can you can do those those three blocks within that amount of time, and that keeps going so you could actually finish all 12 of your blocks. Assuming you've got all these transactions, you can get up ready to go within um 200 milliseconds times, Well 204 2.4 seconds.

E

You can actually finish your entire round of blocks and shipped along, and so how how that comes into play with the increased Max transaction time that we've been talking about is the fact that since you're doing all that early, if you happen to get a 150 millisecond fill transaction in that time. Well, all that's going to do is kind of push that out. It gives you a lot more breathing because you're starting to walk things earlier.

E

So if you waste 150 milliseconds uh on a transaction, let's say well yeah that pushes everything out before that might cost you almost the entire block. Now it's going to cost you time within that 12 second window.

E

um Hopefully I'm explaining it right. So it's it's less about wasting an individual walk time as opposed to wasting 150 milliseconds out of your complete um uh route. Your complete six seconds of tide that you're producing the block, and so um anyway, I kind of want to describe how that's going to work with the with the 5000 difference in the way black shark is sure.

E

Russians welcome.

D

Cool I guess the only concern is, if you have 12 150 millisecond failed transactions, then all of a sudden you're back to the old thing. Basically.

E

That's true I mean there's, certainly the there's a choice just like today. Like you know, um if you've got whatever the number of 30 millisecond real transaction, then yeah. No, nobody can. Can you eat up if they're clever enough to get those dirty uh millisecond uh do nothing transactions into your walk production window, then yeah I mean if somebody can show up a thousand of those, then you can.

D

Yeah, so the only thing is you remove the last block percent time so now, with the new way of failing, we might Miss passing off the block to the next block producer, because now we don't have control over how big that last block time is.

D

Based on what you described, so if we have the failure case, where we're we use up all 500 milliseconds of time for each block, then we keep the original schedule and then the last block. Well, maybe it would be empty because it all has the failed time. So maybe it wouldn't be that big, but it's still, you know we're missing that control for the last block to get to the next block producer.

D

So in that scenario, where you're using all the time, you still want that last block to be shipped early.

E

Yes, um all right, yeah, it's an excellent point: I'm gonna, I'm gonna go look at that exactly how that's how that's working today, yeah.

D

Okay, how do you stupid failure case? That was should never happen, but will happen so um yeah I.

E

Mean it's probably much less likely, but certainly is possible. uh You effectively push because of of uh fatal transactions. You effectively push yourself back into the old schedule. uh What happens with that? Last block uh currently I believe yeah. It's going to be according to the CPU effort, which is going to be probably good to your larger than um the way you would typically went for that last, while yeah.

D

Because, like on some networks, we have that block last block that you know ten percent or something it's rightly very tiny, because we need to get from you know, Africa to you, know: Brazil yep,.

E

Yeah well the nominal case, what this does and we we look at that all. We can certainly address that by overstation moving back the last block configuration, but the um the nominal case, though, is that, like you're not going to need us more or less yeah.

D

Exactly so only time, if that last block is scheduled well ahead, then we don't need this. It's only. If we get to the you know: 11.5 SEC, you know time window and then the last block is scheduled. Then then you know, so it's really like time that the last block is allowed to be released rather than you know, percentage of time, so you know always leave at least x amount of time at for the last block to propagate to the number yeah I cancel the option. You want cool yep thanks.

D

uh The other thing that we discussed before on this is you know currently tracking uh metrics, in terms of how long it takes for Block produces blocks to arrive and I provided a suggestion that you know. Could we track the time that the block was actually produced and I'm, not sure I? Think you took a note about that, but I'm not sure if anything happened on it.

D

So, like there's the scheduled time when the block is supposed to happen and then there's the time when it actually happened, and these are two different things, and so if we can measure the time when the block was actually produced, then I know the latency between how long it took for that block to arrive here. So then I can measure the latency between the block producer.

D

Otherwise you have no idea, because you know I have no idea where you know when it was actually released, because you don't know how much failed time took up in that block production.

E

Right, that's true today as well, although it's it's Amplified with this new setting for sure.

D

Yeah, the new one is not I mean the current one is not so bad because you know it's always within 500 milliseconds, so you're just checking the Delta between the scheduled time and the block time. Now you don't know the scheduled time anymore, because the schedule time is based on when the last block finished, which you don't know as a remote person right.

E

D

So it's just for monitoring. It's not really, for you know, on-chain consensus. You know that's not important, but it's just peer-to-peer when this block actually get produced.

D

All right what about great I, I love? This um should make uh you know being able to have lots of transactions faster. My only worry is that this is going to give a variable more variable, experience, I, don't know if that's right to the users, because we're going to chop create all those blocks in two and a half seconds and then do well for the next three seconds. There's no block, so you know here submit a transaction and you wait, however, they're probably waiting anyway, because there's a queue so right, yeah, that's.

E

That's where we yeah that's, where we landed in that it's actually not any worse, because the queue is already there and you're only going to chewing through it. The exact same uh amount that you would be it's just you're doing it sooner so you're, giving a better Legacy to the people in the queue but yeah giving the same. Latency wood wouldn't make it anyway.

E

D

D

I love having these discussions early before the release, so we can get those three back in so again another opportunity for a good blog post on this.

D

How are we making it better for transactions on the network.

B

Yeah I, like those call outs, by the way, those are um always inviting ideas for ways to highlight.

D

um Yeah, this is a great Innovation right and I. Don't think we spend enough time talking about all the great things that we're we're working on in this Tech stock.

C

D

B

Be honest, um any other topics we should cover.

B

Excellent well, with that I'll hand, it back to Daniel thanks everyone.

A

Thank you, Brian thanks everybody and yeah. We will I just sent out the cancellation for next week. So no no meeting next week and we'll see you all again a week from next take care guys.