Everything EOS EOS Node Operator Round Table, 6 Sep 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Node Operator Roundtable - September 6th 2023

Description

The Antelope Node Operator Roundtable meets every Wednesday at 14:00 UTC to share and discuss feedback to improve the Antelope protocol.

00:00 Intro Leap 5.O upgrade
01:06 Transactions Plan for Block Producers
02:20 Early API Node testing 5.0.0
13:37 Test Plan Prerequisites and Evaluation Steps
18:08 Disable deferred transactions on Jungle and Kylin
20:47 Antelope eosio.deferred Substream
25:06 Indicators of Progress
26:54 Documentations: Highlight Reel

The EOS Network Foundation coordinates financial and non-financial support to encourage the growth and development of the EOS Network. We’re harnessing the power of decentralization to chart a coordinated future for the EOS Network as a force for positive global change.

// EOS Network Foundation //
https://www.eosnetwork.com
https://twitter.com/EosNFoundation
https://www.linkedin.com/company/eos-network-foundation
https://t.me/EOSNetworkFoundation
https://discord.gg/eos-network
info@eosnetwork.com
#EOS #Web3

A

All right welcome everybody. It is September 6th back for another node operator, Round Table uh upgrade Edition again we're preparing for leap, 5.0 upgrade and um the Focus right now is. We are disabling deferred transactions on the test, Nets getting ready to disable them on mainnet, and then we've got also some uh early API node testing ongoing. So maybe we can talk about all those topics and, if anything else, to share either on the progress side of of software development.

A

If uh maybe, we could update from enf on how that's going if that's on track and uh and we'll leave it open for any other questions, comments, topics that come up from from the group on the call here um so yeah before we hop into it any anything to add um on the software development side or anything.

B

Well, so, on the topic of um deferred transactions, the you know, the the plan we published um has, you know asks for uh block producers on EOS to begin disabling deferred transactions uh on the eighth.

B

If you're a developer and you're uh hearing this and you've got a application deployed on ES, and that means uh you should make efforts to confirm that that is not going to break you prior to the seventh, which is tomorrow um so we're down to the wire I, sent a reminder um to the Antelope developers sort of group the other day I could I could certainly use help from this group to disseminate that I'm going to send another message.

B

um Sorry I sent the reminder yesterday I'm going to send another message today um and and would appreciate help from folks on this call to sort of disseminate that information out to um to app developers who have apps on EOS.

B

um On the you mentioned early API node testing. We do have instructions for that and we're looking for people to help with that. Yes, so Daniel is I'm sharing that here, I've also just uh at least the URL for that in the zoom chat. Here um it's on the antelope bio leap, repository in the wiki on the right hand, side you can see the you know: Pages, there's early API, node testing for leap 5.0, some context here this is this: is a tag release. So it's not a release.

B

It's I shouldn't even use the word release. It's a tag of um the features that have been developed so far uh and sorry merged into this main. So far, so that means it does not include the instant finality um features.

B

But what we're trying to do with this is, um you know, reduce risk for the 5.0 by testing all of those non-instant finality features as soon as possible, rather than waiting for instant finality and and just doing all the testing in one go.

B

um So what we're asking is that, folks, you know, follow these instructions. It kind of gives some a template for like recording your testing results and uh step by step. You know instructions on tests, we're also leveraging the The Wharf and Wharf kit test Suite um here so um yeah, so there's instructions on on what we want to do here, namely we're testing around uh read-only transaction parallelization and ESPN OC, and just confirming that one software stable and two that the resonant memory is indeed smaller and that the CPE utilization is indeed lower.

B

um I, don't think I have more to say on this I don't know Kevin if you have anything to add Eric's unable to be here today. For this call.

C

um No I, don't think I have anything to add. uh You know looking just for any feedback around anything, you know run it on API nodes and relay nodes and whatever and let us let us know if you hit any uh any issues.

D

C

Essential thing.

D

And then I asked the question we don't know like what is this Wharf kit test Suite how to run it like? There's, no install okay, but it's over yeah.

E

Okay, I'll not even have a weight yet.

B

Exactly I'll update the yeah, that's a silly oversight. I'll update the um Wiki article here with uh links to that and then um send a message in the Antelope um nude operators, Channel saying that I've done so.

E

um And I'm also curious, I mean what's kind of and Stan even asked. You know what is the general scope of where we're trying to test it. I guess first thing, I did was fire it up on NES? No, no! Obviously I'm not running it on my production, US News, because we don't know so I've got a dedicated node that I'm trying to Baseline and and figure it all out, but I mean that's EOS.

E

Well, I can run a test kit against it, but you know I mean some of the true test is going to be. When you see I get a 110 million API requests a week right. That's the 256 thread benchmarking, where it chew up all the memory and see if it's better so I'm curious, you know, I, mean sure and and going forward right, so we released three finality features for just API.

E

The next version after that is, is it so.

B

At one thing, I'll say is that at this stage this this tag does not have any um like consensus breaking stuff or anything like that. um I wouldn't jump straight to like putting this on a production, API node, where, like you're, actually serving uh critical uh requests at this point, um because you know we don't have the confirmation that it is indeed stable at this point um so right now uh having it on an EOS node is fine, because it's not it's not block production. You know.

B

um Excuse me well,.

E

The thought was that state DB is way. Bigger, I mean that that's really, where we're looking right, I mean yeah, and that jungle is like two megabytes or something yeah I.

B

Would even encourage doing and Es um API node, but but in internal testing, only one for at this stage right um now. Of course, that does mean that, like we're limited on, unless, unless somebody has a capability to do uh massive sort of parallel or sorry, massive uh traffic Generation.

E

Yeah I mean I'm just banned the out of a lot of nodes, or maybe here it's even got some cannons from from that can do it but happy to even provide the API for you guys to smash on. If you have a methodology or better tool right as a provider, we do Adam.

B

We have the ability, with some.

D

Test harnesses.

B

To do uh head heterogeneous, um no homogeneous, that's the one homogeneous uh um uh sort of transaction uh loads right so, but but we don't right now have a great way to do. um You know a variety of different transaction types.

F

B

Stage also, those tests aren't, aren't really necessarily uh they're not updated for read-only transactions. At this point so.

E

So what the forecasts all be a valid test on EOS, that's I mean I, don't I, don't know what it's expecting. I guess! Sorry, I didn't hear the first part. What was that? Well, these Wharf test kits yeah. So.

B

With the you know, what the wharf tests will provide is is more of a general usage um uh overview and yeah are those valid for uh running on EOS and maybe not right. So that's a good point. I hadn't thought about that aspect.

E

um I, don't know what it takes, so I mean I'll I'll smash the hell out of it with a bunch of git infos. Just because now I know, that's a reason. I don't see that.

B

That, however, is is an.

F

B

Thing to do with with this version on ES, because that should be more more performant. Now, with this early 5.0 version, and in fact it's one of the one of the steps here is get infos and get blocks uh and and like spamming, spamming with those.

E

I can do that yeah.

B

That would be a helpful early step uh to to confirm.

E

um That's what I was just so like say: overall, it was a general confusion of boom. I got a note online I'm Baseline it on 404. Just so you know it's a moving Target to even try to Baseline this memory usage when I launched it after a snapshot, it said like 17 Meg. Will it eventually climbed to like 20, gigs or something I mean.

F

E

Takes a long time for that memory, usage to ramp up and stabilize, and it never actually stabilizes I'm running a watch every two seconds and Let me poop. It changes it changes it changes, so you can get it to a general idea, but and then like note, OS percentage utilization, it drops to 3.9 and spikes to 15.6 pretty broad range and it's just an idol. I'm not even smash.

B

Yeah, but it's it's definitely a valid point it might. It might make sense to split these instructions into uh test, to run on an es, node versus test to run on a test net uh node.

E

So what are you trying to get you're trying to smash the API to get performance results right I mean that's that that seemed like the idea, but a I didn't know what tests are running and B I'm curious. Well, so.

B

The parallelization part in theory, it doesn't really matter because you could. You could smash that on either Network right the chain based size thing. uh It's definitely relevant what network you're running on it and yeah.

E

I'll, take it I mean discuss it, but that gave me some of the pieces of info at least I'm looking forward. I can even manually fight through it to generate some statistics on EOS I can get blocked in info. I can smash it a little bit and at least reduce my own results, regardless of the testing kit and make sure it runs. Stable right, I mean it'll stay in sync, with main Network anyway, yep.

B

Yeah I think generalizing this to the two like most important elements: um it's one parallelization so um the way to test that is get infos, get blocks and custom read-only contracts right um and then the other one. The other category here is uh just chain based size and Resident memory size um and that one I think is more just letting the node run right.

E

Yes well and that's semi-related to the apparently the number of threads right I mean. That seems to be the number of these.

B

It is related in that, in that we had an arbitrary limitation on the number of threads previously because of um because of memory, overhead related with multi-threading stuff and now we've lifted that limit, because we've kind of overhauled uh that parallelization code, so that it is memory efficient so and.

E

Basically, we could never get above eight right, I mean even didn't matter what you did, they would lock all the threads were locked at eight right and now we're just sending them through the roof. Exactly.

B

Yeah yeah now it's up up to 256.

E

Or I could play with that knob. So that's why I was asking I've got zeros and I got threes by default. What are you looking for? So we can definitely get a bearing on how things react at 8 or 64 or 256 I mean I only got 23., so we'll see how it takes to get you sick, but that was just a little more guidance on what are we trying to focus on in tests versus just run it, and whatever this Wharf kit is yeah awesome. Looking at it, morphin's awesome thing.

A

So is any of that you you mentioned. If you gave some guidance there, Brian that sounded pretty good. Is any of that captured in this test plan? What.

B

What is a bit erroneous about this plan after this conversation? Is that the the plan doesn't really account for what you would want to do where right um so I'll take another. You know pass through it there um as far as some of the the high level steps they are, uh they are captured here. I might add a um section to this.

B

That's sort of the you know, tldr, like I, don't have time for a formal test, but I at least want to like smack the thing right um section so that we can get a broader audience on it. So EOS.

E

Usa version of those goals- I, don't know yeah.

B

Yeah, exactly if you could also I'll add a section that says if you're Michael, you do.

A

That cool well any other feedback on the test plan or early results. If anyone has been doing any testing to share.

B

Yeah, even just like who has attempted to build this, did the build work for you, not that I expected it wouldn't and, and uh has it at least, you know, started and run without crashes, for you.

E

They're working on baselining next will be bill.

E

All right name, Bill, Barrett, yeah, I, assume right, I, haven't looked at the release, notes but saying is 40 same kind of manual build process.

B

Yeah so there's a there's um like two um two important configuration changes that are documented here in the plan.

E

I did like those like the EOS OC or the OC I.

B

Guess yeah and you ask build instructions. So that's.

E

Kind of correct: that's what I mean just the standard. It's it changed in four yeah. I assume, that's the same, see make.

B

Yeah um I am not aware of changes to the build instructions. So if you run into problems, let me know, because you know it's possible- that there is a change here that I just you know slipped past me that I would I wasn't aware of so.

D

Well, there was Bill yeah I'll check that changed in the middle of the 4.x somewhere and those got reverted so probably they're back so probably there's some build change. What.

B

D

That sorry say that, again in the middle of the 4.x releases, there was a build change at one point, which then got reverted because I complained that it changed the behavior that I was used to. So probably that changes in 5.

E

It may change back, it may change I.

D

Don't know I mean I didn't look at it. Yet that's.

E

Why they haven't gone and chased that.

C

If I remember right, I think that was a script that we removed I, think we put it back, I, don't I, don't think we've since taken it back out, so I think you should be fine there uh if I'm remembering the right thing, if not I mean certainly let us know and take a look at it. uh The biggest change probably related to build, and this stop that Sim 5.0 is the fact that we're now unlink boost along with uh leap as opposed to being an external artifact. That was pretty much under your control.

C

Whatever wanted that one uh 5.0 there is a boost, that's included, and that's the one you use uh and you'll notice that when you pull the branch, when you update the sub modules, it takes a bit to hold down all the Boost so that that's one change.

B

A

All right, unless there's any other feedback, Let's uh turn to disabling deferred transactions. Lucky the moment.

A

All right um so both jungle and Thailand uh we're we're past the target date. We were aiming for uh September 1st, uh there's still one BP EO Sweden on both jungle and Kyle and who is yet to at least give us any word whether they're aware or whether they have disabled for deferred transactions.

A

um I, don't think, there's a way we can just we can check, but perhaps it's at the point now where we can start discussing if we want to expedite this test. This is in both cases on Jungle and on Kailyn.

A

Sweden is the BP? That's missing action, so we could coordinate to you know, give it one one more chance, but I think I think I saw Michael just the other day, not today uh poking poking them again still no worked them again. So we'll see.

A

um So if we still, if we don't hear anything, what we can do is you know essentially coordinate vote them out, put someone else in who's, who's active and then we'll we'll get our test.

A

um So that's the current status on a jungle and kylan um this was raised. We've got the monthly EOS block producer call with the top 30 Block producers, I believe uh in attendance, pretty good um attendance on those calls.

A

um So we had one of those calls I think it was two weeks ago where I raised this on that call that this is coming.

A

um So everyone is already aware: we've shared it also with all the top 21. um there's already been I. Think one or two there's already gone ahead and disabled it, but we will starting Friday is is the date we want to start start disabling on EOS and uh and so we'll ramp up the the coordination on EOS mainnet at that time, but I feel pretty confident. We should be able to take care of that by the by the deadline.

A

uh Anything to add.

D

Yeah I was gonna, say: maybe someone can go, run sub streams on those test Nets and see how many different transactions there are yarrow, just random, yes, right, yeah and so which block producers are still not right.

A

You can see you can see the who produced that uh yeah. So speaking of that um yeah yarrow has built the substream for measuring or for tracking the Deferred transactions. We ran a test on EOS mainnet on the last 600 000 blocks, and we found that there was only 16 contracts, pushing deferred transactions um and 99.99 of them are all coming from. A single art. Bot uh baby is coming as the account name did.

E

They even work did you check their uh execution status, I.

A

Did not I did not. uh We could do that, but there's, let's see here we go.

A

um This is public if anyone wants to go and look, I have a look and analyze the results for yourself. um Currently, this has been shared in the node operator. Roundtable um telegram, Channel I'll, actually I'll create an issue on our on our board here with all the details and links, and so anyone else wants to dive a little deeper. We can do that and uh yeah to Matthew's point I.

A

uh We we are, we running our fire hose on both kylan and jungle, so that someone could use our endpoint if they wanted to and uh and run us run this substream or uh yeah. Maybe we can ask yarrow to check there too, once we've settled the final BP.

B

And this uh this spreadsheet is from.

A

What network this is: eos, mainnet, okay, cool! Thank you!

A

um So yeah on the on the topic of you know, Outreach to dapps it's a limited audience that are actually looking that are that are using this right now um and yeah, mostly our our bot. So this R bot will need to address their strategies um anything from uh Dario. Your support. Have you I know you guys have been doing some Outreach with developers any new feedback coming back since, since last week's call.

F

um Yeah in prospector seems to be ready and we received one more information that also U.S Royal is working or let's to be ready. Otherwise, I'm not aware that someone isn't ready. Okay,.

A

I wonder if those guys are on here.

A

Don't see it anyways, maybe ready means we were never using it anyways.

A

uh Okay, thank you for that update, uh so yeah. We will follow up uh offline on this and, if necessary, we'll make the required changes to get the BPS who have removed deferred transactions in the running on the top 21 on these test. Nets uh and yeah feel free to poke Sweden in in the meantime, if you haven't I think uh yeah. Thank you. Michael sounds like that's just been done.

A

um So, like I said, yeah I'll create another issue on the board, with these details and um yeah we'll our. um If you do want to try running this yourself, talk to us right now. Our our fire hose endpoints are not publicly advertised, but they will be soon and anyone can create a substream either for this or other things run, the substream uh get some good data. This is kind of cool Dro uh whip. This substream up in a couple of hours and ran it on our on our endpoint and there you go spit out.

A

These results super.

B

A

Anything to add on that topic.

A

All right- um and we talked about early in node, API testing talked about disabling deferred transactions. Anything else do we have any any indication of progress and being on track or not on track with the um hot stuff stuff.

B

Yeah, so the the overall we're still working on um you know, timing for 5.0. um You know this is just overall. So there's here's uh a few items here where, where dates are, are still being worked out. um So we should have updates on that soon. Hopefully, next week.

D

A

uh And then otherwise, those were all the topics I had unless anyone else has any topics to add. For today,.

B

Yeah I'm Wrecking, my brain.

E

E

On the documents I am trying to go through, I think it's Nathan that works on them right, the EOS, Network Docs and pick out what I can that he's put out there in relation to all the different parameters and rude beings and there's some interesting info I found out there when I get time, I try to go back through there.

B

Yeah speaking of documentation, too, there's also um I'm over the next week, uh I'm working on what I'm, what I call the Highlight Reel so to speak. So this is. This is not going to be the exhaustive release notes, um but rather the you know, the the major features that'll be included in 5.0 and sort of their benefits and, and that sort of thing um it's a first step towards complete release, notes- um and you know- should be useful both for marketing purposes and for support purposes, um leading up to the release.

D

A

That's expected of the 12th is that on on track.

B

Well, that is a very valid question. um I've got you know. Basically, two features uh with with high level notes uh written up. uh There's a ton of features. Part of the work here is actually just sort of like semantic clustering.

B

If you will, because you know like what what things make sense to sort of bundle up into talking about as one theme rather than as individual features, if I tried to talk about individual features, um there's literally hundreds of little things, you know what I mean that you know most of them kind of group together into like just a small number of logical things, um so yeah, that's part of the work, so it's I'm actually not at a point where I like it's clear to me, how close to done I am, um but you know this this being a highlight reel and not intended to be a complete uh set of release.

B

Notes. There's also, you know an element of um good enough if that makes sense, so uh it may slip by a couple days, um but I'm I'm don't intend to let it slip past next week because worst case I'll ship with whatever we've got. You know, as far as that document goes cool.

A

All right so highlight reel coming soon. Anything else and I see you have support tagged to to send that out when it's ready right, so Dario support team standing by.

B

Cool yeah and the attention of that by the way is, is um just to you know it's one thing to tell people you know: 5.0 is coming get ready. uh Ideally we can tell them. 5.0 is coming and here's why it's important get ready.

A

You know what I.

B

D

A

Any other topics.

B

Not on my docket.

E

Related to the upgrade nah I can always talk about tech.

A

Well, we have 25 minutes left, we can wrap up early or shift gears.

E

I'll, let you all decide.

A

Well, I'm happy to rap there's nothing pressing and we'll, uh like I, said, coordinate offline about getting these last, the last BP upgraded on kylin and and or last BP disabling deferred transactions on Kailyn and jungle, and uh we will check back in on that next week and in the in the chats sounds good. Have an awesome day. Everybody see y'all next week.