Everything EOS EOS Node Operator Round Table, 5 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Node Operator Roundtable - April 5th, 2023

Description

This week, the EOS Node Operator Roundtable presents a Leap 4.0.0 live demo from the Leap Core Developers covering Read-only Transactions, Snapshot Scheduling, Performance Harness Tests, Auto Peering for BP Nodes, Prometheus Exporter Plugin, Log Splitting, and much more!

00:00:00 Start
00:00:18 Leap 4.0.0 Live Demo
00:03:30 Read-only Transactions
00:15:35 Snapshot Scheduling API Design Specs
00:28:30 Performance Harness Tests
00:41:30 Auto Peering for BP Nodes
00:45:11 Prometheus Exporter Plugin
00:46:18 Log Splitting
00:48:44 Faster Block Propagation
00:50:50 Multithread Support
00:55:55 Subjective CPU Improvements
00:59:00 End

The EOS Network Foundation coordinates financial and non-financial support to encourage the growth and development of the EOS Network. We’re harnessing the power of decentralization to chart a coordinated future for the EOS Network as a force for positive global change.

// EOS Network Foundation //
https://www.eosnetwork.com
https://twitter.com/EosNFoundation
https://www.linkedin.com/company/eos-network-foundation
https://t.me/EOSNetworkFoundation
https://discord.gg/eos-network
info@eosnetwork.com
#EOS #Web3

A

Hello, everybody it's April 5th. Today we have a special edition of the Ankara brown table. We've got uh some friends joining the call from the enf to give us a demo of beat 4.0 so I'll hand it over to Brian to uh to introduce that.

B

Yeah I'm super excited to have some of the elite core developers here today to go over what we've been working on for 4.0.

B

um You know just as a sort of refresher that everyone on this call uh in watching this video probably already knows es, is always sort of you know, Excel, that performance, scalability and reliability and and those things have been kind of um reinforced with what we've been working on with 4.0 uh I. You know I just on a personal note.

B

This is my first major release that I've been involved with the the whole way um I'd like to thank everyone for the warm welcome to the community, and you know putting me in I'm really proud of what the team has put together here for uh 4.0. uh There have been some lessons learned on my part.

B

You know so um I'm I'm eager in the next release to have more direct Community involvement earlier feedback on the problems and opportunities for prioritizing, and, of course you know uh the proposed approach to the solutions that we're developing um and, of course part of that too, is demos like this one. So I'm really excited to be doing that today. Some of the themes for the release just to kind of set the context before I hand it off to our uh first to demo.

B

um Participants uh are that in 4.0 we built features to provide even higher performance. um Then you know yes is known for with with some multi-threading features as a common uh thread.

B

uh We uh we also built features to reduce latency and enable faster block propagation um and features to provide more control for node operators around uh well data control and visibility, um and then you know, lastly, just kind of a an odd assortment of quality of life improvements if you will for node operators.

B

um So those are the big sort of buckets of themes um as as we're watching through the demos. You know see if you could guess which ones were targeted at what things, um but with that I will hand it over for, for our first um sort of demo presenter, which is uh Lynn and Kevin I, don't know which one of you is gonna sort of run but have at it.

C

Yeah, either to the first part and the Kevin will do the second part right. So can you hear my screen.

B

Yeah yeah Daniel might have to give you permission. You are good.

A

C

B

C

C

So, okay, so you guys can Okay can see my screen yep. So you let me go through that. Okay, this way I don't have to memorize. Okay,.

C

So in photo zero, so we Implement a new RTC in the point for sending read the only transaction and then we went on to pronounce it and then we will not replace all the read-only tasks. That's our send I mean that's this three features, and so, firstly, so I'm going to talk about a water or how a real utilization look like so our read-on translation. So basically it's similar to redisin, but it has some difference.

C

First so, and it's actually I mean the actions when you uh in a transaction cannot modify States and we have our search in place to prevent accidentals these changes. Also, we don't allow any authorizing or signature in the with only transactions this way, so the transaction will not it's not to the other node and also we make sure there's no such side effect after with honest I think it's just one: it's executed it, so it will not be put into the block and because for this Rhythm transitions they will.

C

We make sure trade, the action Trace always return, and also we because it's uh we don't transition. So we don't do keep in mind login and for riddle. Also in this list we.

C

Transaction but there's two types of translations: we provide the separated transaction timing. Summary also. We play the separate producer, Choice login. So now let me show you her power. This look like so I already have contact, so here is just a regular content, so yeah this is just for. You know like so this one is a person. So that's all that's similar, so I the person is, the buyers are like uh I have the idea that has K age. So basically it has so yours you can get.

C

You can insert update and get the data from this table. So this is the counter. So let me show you so now. I already have before the stop before the demo. I already have no use start. It's here also I already set up an account and the Creator account and I already have a built, a read-only transaction. So now let me show you how it can look like.

C

So here this is.

C

Okay, so you can see it's in this transaction, so we don't have authorization, and so this is just the action is get age and also so all other players are empty, so I know so now before I'm going to try to so we will now I'm going to use the careers to send the reading transaction in the careers. So we already updated careers. We added the one in a new option in the center option or in the push twice again under in the push it action. We have that so here.

C

So if you just now, if I don't if in the other way and if I don't say so also now, if I do this, so actually it's a yeah, because so this will fail because I don't have any acceleration. But here if I say I put the reader.

C

Here's if you use read option. So this is for centers with the only translation here you can see so now. uh No, yes! So now you can see the return, but because I don't have any record for this. So then you will say you can see. There's recorded outside exist, but it's you for now. I insert a regular there.

C

I can do okay, like my Sunday, is not work anymore. So sorry, so now I needed to look.

C

Let me see what's why is this one? That's too quick, yeah I can do a quick one.

C

So I hope, sorry and then it has already started.

C

Yeah because I started it at 40., sorry about it,.

B

um Perhaps while you're doing a moment of debug.

C

Yeah, sorry, yes yeah, but also I can talk about the parallelization sure, yeah, sorry yeah, so I'll come back to. Thank you.

C

So and so after we finish the read-on transaction, so we went on to Paradise so to Paradise the rhython after that also uh Kevin, and also notice that also we can even press ball. So now we pair as a read-only transaction. Also, we paralyze all the reader. Only tasks I mean Edison, which can be run as in parallel with Paradise. So that's uh here, so we divide the order translation as Transit into negative two uh two categories.

C

So here so with yeah, we identified the uh net tasks which can run in parallel among them so like get info, get black all most all.

D

C

Of the majority of the owners get which can run in parallel, so we mix make sure the minute they can run in parallel and the idea is we divide the execution into two windows and read the window on the right window, the right in right window, so it's similar to the regular window.

C

So both the read-only translation and the rhython read the reader write function can write, but at the right in six in security and the right on when thread, but then in the right in the reader window we only allow read-only transaction under read, only task run password. They are right in parallel and there's one in multiple thread. The bases are allow for this. We have a uh we, so the operator needed to configure three parameters.

C

So you need to communicate how many threads you want to run. So you put the threads, but as if the number of thirds is zeros, then the feature is not enabled. Also you need. You need to specify how long for the red window and for the read window last so yeah. We you need to provide this three parameters and so yeah also for the pronunciation, so we for now we only allow it to run on April node because yeah, so we don't want to uh Interruption with production, foreign.

E

Yeah yeah it looked like your wall got locked on you there do. You need to unlock your wallet. uh Oh yeah, a little bit more there on um what Lynn was talking about that read-only uh windows and the the right windows. So you can spot you can configure those uh the defaults are are just kind of numbers. We we picked out. I think you probably uh be interested in tweaking those.

E

So if you're, an API provider, for example, you might want to set the read-only a right window time to be rather large, like you could set it to be, maybe 400 milliseconds and you can set the right window to be tiny. Really you may get uh really. You can make it just one uh for that matter, because uh the the block when a block is received, it will be applied which, which means implies that it will switch automatically to the right window for you.

E

So if all you're interested in doing is providing um get calls to users, you can set a very large read window in a very small right window. Now, of course, your right window will be used for things like push transaction and well yeah. Basically, that and compute transaction put transaction compute transaction. So you know, if you got to know that's doing both a serving uh push transactions and get calls like your table rows. Then you know you you Google separate. Now.

E

If you proxy those two different nodes, then you can set up one node, that's in the right window, always so you disable read-only capability on that node and it can serve push transaction and then you could proc the all. The git calls to a node, that's configured for a very large read window. So you know that might be one way that you might want to, but do that and then you could.

E

You can crank up the number of threads on the uh on the Node that started on those reads and it will provide um like uh get table roll calls. Multi-Threaded then, which is pretty cool. Actually. So that's that's.

C

It yeah yes, yes thanks Kevin yeah, you know testing. We noticed the performance gain is quite a big uh except before yeah using one of our tests. We have a lot larger number for uh like transactions and all the read-only mess with the tasks so before that my test was on like what I've around the 500 400, maybe 450 seconds. So now with this, this is like 300 seconds, so it's big again for what I get uh actions not only a lot, only with only translation but also the other ones yeah.

C

So now then yeah I already said. Let me show so now. If I say, uh canaries uh send push transaction get my this. This is my transition, then, if I say is read, so it will just read so now you see I pick so yeah I put a 80 to 15. yeah well, for my insert so yeah this one. So this is insert record so also and yeah, and also you can just use the so you can do that so so uh so, okay yeah.

C

So that's all for my demo and thanks Kevin for the explanation, yeah, and we feel also this feature. It me yeah, or that is read and actually yeah quite a bit.

B

Right, okay, thank you. um If, if anybody has questions, please take a note of them, um I want to make sure we get through all the demos. So take a note of them. If we have time for Q a at the end, we'll do that and if we run out of time um you can send them to me on um telegram uh it's just at be Hazard on telegram. So um next up we have Vlad.

D

Yes, uh I'm ready to share the screen. uh I think it should work. Let me see.

D

Yeah, please, let me know if you can, if you can see my screen.

B

Yep looks good.

D

Okay, so um yeah I kind of initially planned for five minutes. I'll try to stick to stick with the schedule, uh so uh I'll cover it. This is probably hopefully uh in that amount.

D

um I I wanted to show you uh one of the additions that we made for 4.0, uh which is um uh requested for from some time feature about ability to schedule a snapshot for a few snapshots for a future time and uh also ability to schedule uh recurring snapshots that would be taken uh through certain intervals automatically and ability uh to manage. Several of this schedules uh cancels and view them, and we uh pretty much introduced uh three additional um API calls which uh allow you to do um all of this.

D

uh We, uh this is the first uh revision. It doesn't, for example, have uh pagination for a snapshot request, so if you have a lot of them, but we're not anticipating that there will be too many of them, and also um one of the uh sinks uh which is really important here is that uh this uh snapshot schedule is being stored uh on a disk as a Json file. I will show you right now and you can edit it manually. You can see what what is scheduled there and, of course it survives um service restarts.

D

So uh it's uh your schedule is not lost when uh node is being restarted. For some reason, I actually created to simplify things and make it smoothly. Yesterday, uh just a simple repo, I kind of share the link to that which contains um just a very simple examples of uh this API usage and uh I'm, just going to just briefly show since here or so, for example, this kind of description, I'll number it from zero to uh four I said five and uh it has uh kind of clean up.

D

Then it runs an audience with a basic uh options. Here then we will take. We will do one time snapshot and let me kind of make it a little bigger. So this is uh one of the API endpoints that I showed you before so, which is called uh right now we had create snapshot here and we haven't deprecated it.

D

Yet it's still working, uh but this new uh actually endpoint called schedule snapshot and it's you can post, um for example, um in a body um Json with uh parameters like start block, number and block number and also I would say, block spacing, which is for curing snapshots here, for example, this one. It really um kind of uh your your asking system to create snapshot, that's a specific height in the future. uh It will be created and this schedule will be automatically updated and won't be uh active anymore, because uh it's already executed.

D

It's a one-time thing uh for a curing snapshot. I for Simplicity I put here that you can specify also block start blocking at which it will. uh The schedule will start at the end and the spacing between the snapshot here. I specified only spacing. That means that uh it will execute as soon as possible uh and we'll be repeating every spacing blocks so yeah enough. For that talk, let's kind of show something uh so I have this script.

D

So, let's, let's do clean up and let's start uh nodius and we can see the noise is running here and uh blocks are being generated and um let's try uh first thing: we will go to the local folder and see that there is no snapshots available here. There is no schedule, nothing, nothing here available. Let's, let's try uh to schedule a one-time snapshot and, let's see which page we want to make it happen. Let's do 100.

D

Okay, so you're getting as a as a result, uh the confirmation that snapshot is being scheduled, uh yeah, it's already 100, so so it already happened and it we can see that the schedule already cleaned- let's just uh I- wanted to show that let's schedule it for just a little bit more like 180.

D

Okay, so if we can go here, we can see that uh there is a one item in the schedule that is waiting for a block number 180, and at this point there will be a second snapshot or you can see it here. It's already taken. So here we go. uh That's uh how the one-time snapshot for the future uh date is kind of is working. uh The most interesting feature is uh procuring snapshots and let's, let's, let's do that so um and oh one.

D

Second, just first of all, um if we uh I have a really kind of simple script here, is that uh take the snapshots and the folder and calculate them to Json using lip with TL I'll just show it in a few minutes. Let's run it quickly and just to see uh uh what block numbers are being associated with a snapshot, and you can see that this is exactly uh 100 and this one is 180., so uh they've been generated properly and um let's try to do uh recurring snapshot, which is most interesting so.

D

Spacing 20 blocks and if we go to this folder, we can see that uh we have this SnapChat. Only one snapshot request and it starts immediately, uh and that was a block number at which we posted this request, and the spacing is two in the snapshot request. Id is too so, and you can see that this folder is being populated with with the snapshots which are happening, uh every uh 20 blocks and uh that's uh basically, uh since I haven't specified the end.

D

This will kind of I would say uh process until you terminated right and for that there is a request that I mentioned before, which we called unschedule snapshot, which is a very kind of simple script here.

D

That is calling unscheduled snapshot with uh just a single parameter, which is uh ID of the snapshot that you want to unschedule Let's unschedule it right now and uh it's a four right and uh it shows which snapshots are already scheduled and we want to on schedule number two done so uh if we will go to this folder right, so there is no more snapshots are being generated and uh it's all done. We don't have anything in schedule. Nothing.

D

And uh at the end, let's kind of uh convert all the snapshots to the jsons and we'll see that uh 504. This one.

D

This one is 4 84, so they are 20 blocks apart. All of them. uh That's yeah, that's pretty much it for uh scheduling, snapshots functionality. uh It only has three requests. uh uh Just one contraction just keep in mind.

D

There is a document here posted uh which uh I think in one place needs updated uh updates because um it's uh the names uh uh kind of that uh this request are expecting it's a start, block, Nam and block Nam and uh actually a block spacing- and these are just kind of three parameter responses for the scheduling and uh yeah. That's yeah! That's police,.

B

Yeah, sorry, sorry to cut in I uh so we're uh you know to keep on Pace, um move to the leap, util enhancements and and probably keep that one uh reached. If we can yeah.

D

Yeah I can do it really really quickly. So uh that's for snapshot and also uh we, uh as you know, we have a new tool which is called lipo, teal and um basically uh I, just kind of in in one minute.

D

I can I can show a few things here, so lipotil is getting more and more uh functionality built in and uh you can please go ahead and explore it if you haven't done so, and we have a kind of a variety of options here which are exposed with a helpful command actually also uh we're updated, uh uh clears to support help in much more convenient form here.

D

So it's really kind of I would say you can understand what your kind of what what does three of sub commands are happening and also, if you've done installation that everything properly uh there is after complete working, and uh you can do things like, for example, that which I'm pressing the top button and it's uh doing stuff. uh We also have uh incoming feature which will be uh kind of reporting for clean on clean shutdown, which is right now in pull request, which is getting into liquid seal and yeah I.

D

Think I'm I'm done so that's.

B

That's pretty much. It awesome thanks blood. uh So again, if, if anyone has any questions, please take note of them and we'll um Circle back to them. If we have extra time at the end or you could send them to me on telegram at be Hazard and I'll, make sure to um funnel them through and get you to the right people for for answers all right with that, we'll go to Elite Peter next, let me make sure yes Peter next.

F

All right, can you see my screen, I think so awesome.

F

So I, don't know I'm here to talk a little bit today about a piece of work that a utility that we've been working on in the background and kind of bringing it to the Forefront today to give you some idea of what's out there and kind of the utility that it provides um for the past several months, we've been working on a performance harness test bed, and this uh was originally thought to give us a a better way to kind of run some performance metrics against uh different operational modes of nodiost, especially for developers to see how their performance or their improvements may be, affecting a performance in the ecosystem.

F

um So I'm here, I'm not really going to demo much today, because um the the reality of these tests are that they take quite a bit of time to run, but I have run some of them and I will walk through um what what the outputs are.

F

So you can see them, but I wanted to initially start with walking you through, what's involved in in the performance harness what's there today, um so they're they're kind of three different levels that you can interact with the performance harness and starting at the kind of the basic, the lowest layer, is the performance test basic uh script and that uh script is there to create and run a single performance test. What you can do is it um it targets a configurable TPS setting.

F

So we were looking to measure as an initial performance metric, the transaction throughput that we could support um in in a block producer mode, and so what the simple basic test allows you to do is configure how many transactions per second. Would you like to Target?

F

And how long would you like to generate that load for and then at the end it uh it does some analysis to to determine success factors and then also some um reporting on the analysis to give you some of the performance metrics at the end, and so this is kind of the lowest layer of the performance. Harness and um I will call out here quickly that if you want some examples of how to run this, there are some examples in the the readme that's provided at the top level of the performance tests on directory.

F

There's a good write up there on how to use this uh tool. So, if I leave something out today, that's a good reference for you to go back and look at, but there are also in our CI CD. There are several of these being run.

F

Both some top level performance test runs as well as some lower level performance test, basic runs and you can kind of see how we've set those up in CI, CD, we've kind of curated them to be a little bit shorter and just to kind of prove out that we haven't broken the interface and and how the the test Suite runs.

F

But that gives you a good idea of kind of what typical configurations might be interesting um to to mess around with, but back to this, how does the performance test basic uh uh achieve this, and so it has a couple components that we've developed to also help in this manner. So the first one is um there's a script called launch transaction generators, and this has been added to the test, harness package, python module- um and this is really useful.

F

There's been a lot of work going on in our integration testing uh our test harness, and it makes it a lot more usable, but we added this in there as well, and what this allows you to do is given say a TPS, uh a transaction per second load that you'd like to generate. It will do some calculations for you, based on what you think, a single transaction generator on your specific system can handle and you it might be um up to you to determine that for your own system.

F

But what this will do is given a large TPS it'll fan that out across multiple transaction generators to generate that load in your system to actually do some of these performance tests, and that brings up the transaction generator.

F

This was a new piece that was uh developed recently as well, and this kind of steps in and replaces the transaction gen plugin that so I had some questions about how well it was working in the and how we were using it in the system, and our goal here was to really try to own in and like zero, in on being able to test components without you know, duplicating the the environment, so we didn't want to use nodeeos to test nodios or um so by having to load up a plug-in in nodeeos to then test nodio, so it was kind of like you were double dipping and um not getting kind of a clear-cut performance measurement.

F

So this is a pure C plus plus implementation of a transaction generator. There are about two or three components in it.

F

um My my bar here is blocking my ability to see that um so the in here, in the main, you can see that this is configurable, so you can give it you know appointed at where you want to Target these, and currently we in the transaction generator sorry the transaction provider. Here this is what determines how these will be sent out on the wire and So, currently we're using uh and supporting the P2P interface and so anything that the transaction generator um creates.

F

It will send out on the P2P interface directed at the nodeeos um uh instances that you're using so you kind of configure it to point at if you've got multiple instances. Maybe you want to Fan those out, um but but that's how it's currently uh working and we're looking to expand that to support you know the HTTP API as well, and then in the transaction generator itself. There are several different types of uh generators that we've created.

F

uh The the basic one is just a a blind creating of transfer transactions that it will send out and create a whole bunch of token transfers to do that basic measurement. That was the first step and we've since expanded that to allow you to um provide some simplistic uh transaction uh descriptions in a Json format that allow you to kind of expand that and see what uh or to test different kinds of transactions. And so there are a couple in the code base right now.

F

To give you some examples, if you jump over into the test performance test, there are um a ram transaction, there's a another data. So it's pretty simple. You can kind of see how it's done there and there are some abilities here to um like, for instance, testing new accounts new account creation. We wanted to get some performance metrics around that, and so you can see how that was specified here.

F

I'm not going to go into a huge amount of detail here, but I'll leave it as an exercise for users who are interested to go and kind of dig in, and if you have any questions, let us know um so. Those are kind of walking back up through the transaction generator. The performance test basic once again uses that to create a load and then measures uh the information coming out the other end, and so to give you an idea what that looks like I'll jump over.

F

This is a larger report of the full test Suite running but inside the test, Suite itself inside the report that the test Suite builds when it finds what the kind of the max sustainable load is. It will include the low level test report in full, for the specific run that it determined is the best, and so here you can see I wanted to kind of walk through the types of information that are available in that report.

F

For you to um to see so here you can see that you've got your start and end times of the test run. This is useful to kind of zero in on what logs may be associated with this run, and also to give you an idea of how long these runs are taking for for planning reasons, um you've got some uh information on the Block size during your run.

F

Some statistics around that this blocks guide is information about out of the totality of the blocks that were generated during the test run, which ones were used for the analysis. So we do some work to kind of zero in on that steady state of block generation and transaction consumption, so that you're not getting kind of those outliers. When things are spinning up and things are just getting going, that will affect your performance numbers, um then you got some uh some details here on the TPS.

F

So your transaction throughput, your min max average, how many you actually got what your? uh What you're configured transaction load that you were attempting to achieve. This also gives you some specifics on the. How many transaction generators were spun up to accomplish this and uh and how many transactions they were each given as far as what their load was supposed to do um and then coming down here. We also for the transactions that were analyzed um looking at what was their CPU usage.

F

What was their latency and the latency here is measured from send time until uh block inclusion time, so you can see uh how how that was working out. You've got your transaction net statistics and we've also got information here about dropped blocks, uh production Windows during the run and forked blocks. Forked uh blocks count, and these are have some additional work being done on them as well in in future releases.

F

We intend to continue to build this out, so this is even more informational and then to to help, because this is a test suite and you may want to be able to reproduce, runs like this and see. Maybe how your performance is improving or degrading as you do work and create features. We wanted to be able to um log all the the parameters that were used to do this log generation, and so we did a very detailed view of all of the different um configuration parameters.

F

No, not all of these are specified, but we actually do the work to pull out nodeeos defaults for all of the plugins and provide that in a full readout here, so that if something in nodeeos were to change, you could actually see kind of what the defaults, how the defaults are changing and how that might be affecting things. But that's a look at the the low level kind of the performance test, basic report and then, if we step up one layer and more, we've got the performance test, and this is kind of the high level.

F

This runs a suite of these tests, and so what do you? What do you get here?

F

um You get a a full run of using the performance test, basic to kind of search the TPS space, and so we do kind of an initial binary search through a large range of transaction per second load and find out kind of in some a little bit shorter performance runs.

F

What the max sustainable TPS is and then follow that with a a smaller window kind of zeroing in on the sustainable TPS that your system can support and do so longer runs to really zero in on what the kind of the max sustainable transaction throughput your system can support, and there are some additional parameters here that you can tweak like how long you want the test to run.

F

um What what is your max TPS to test? You can kind of shorten the window of how long this takes to run, and then there are two or three other test capabilities here. um There was some question about adding threads to the different plugins in nodios and seeing how that affects performance. So we actually provide some automated uh ability to run through the number of threads that you're allocating to each of these to kind of zero.

F

In on how they're affecting the performance and give you kind of an ideal, it's kind of a naive implementation, but at least it gives you an idea of how the number of threads are or are not affecting the performance, and once again this is all kind of summarized in the readme here. I'll leave you with that um and yeah that's kind of what I wanted to get through today. So thank you, excellent. Thank you. Peter.

B

um Exciting stuff, uh looking at the time, we probably will not have time for Q a today.

B

um But if you uh please take note of any questions, you have and send them my way at be Hazard, um as Steven pointed out with twozies ph-a-z-z-a-r-d on telegram, I'll make sure to bundle those through so I can get you answers with that. Evan um you've got a lot to cover and you've got 15 minutes if Daniel, real, quick as a housekeeping thing. Are we good to run over by by a few minutes.

A

Yes, you are yeah, um let me see here my calendar I might have to drop them no arm. Actually, I can I can hang around a little longer too. All good.

B

E

Right, thank you, Kevin Grace, great, so I'm all attempt to go uh quickly here um and and not in in any great depth on any of the topics uh that I'm gonna recover. So again you know uh it Brian or me up on Telegram, and we can we can answer any questions you have so.

E

The first thing I want to talk about is the auto appearing capability uh for BP nodes that allows you to configure um a a b uh BP accounts, along with their uh connection, uh their endpoint that you want to connect to and then what I'm going to show you here is what I have on the screen is just an example of the test that actually we have for this- that the integration test we have from this, just to kind of show you uh how this is working.

E

So here we see um I scroll up just a little bit here. So the producer is esio at this point and then we're promoting the schedule uh to a new set of producers, I think it's a through U or something anyway and polar new set of producers. That hap that's happening right here. uh Promoting that proposed schedule and then you'll see if I find the block- that's being produced- uh okay right here, so I'm actually with esio. But if I go down a little bit, I should see a block. That's not esio yeah!

E

Here we go producer deaf producer e, so you see that the the producer schedule has changed. So if I jump back up to that, where that was was changed so now, the net plugin reacts to a proposed schedule. Change like this, and it says oh, my pending Downstream neighbor for this particular producer. So this producer this producer here, it's Downstream neighbor for the next two producers in the schedule are R and S um they're pretty sure are different F.

E

So that's the depending neighbor what's been configured with those endpoints for those two producers, and so what it does. Is it attempts to connect to those- and that's what's happening right here at any connection before adding new connections for five, and so after that there's hand shapes it's. So it's it's a after this point. Wherever the actual message is doesn't matter, it's then connected to those additional uh endpoints that are part of that schedule and then later on, uh I won't show you here for time it it would um if it had connected now.

E

In this case this is the first from going to efio to a set of producer, so there aren't really any previous connection. But sir, when the schedule changes again uh r s, let's say: RNs are no longer Downstream it'll, actually disconnect from rnf and connect to the two nodes that are Downstream now, if, if R is still Downstream, but that says that you know it'll keep at but lose the other, and so it massages that connection as uh the schedule is changing and that's all according to the configured uh BP schedule there.

E

So that's um that's really kind of all I wanted to show for that. I'm gonna, pull this over here I believe you should be able to see that I think I'm sharing my whole screen.

E

um The next up is the Prometheus exporter plug-in and the not really on the show much here, just a couple notes on it: the fact that it is ipv4 for the 4.0 release. Now we do have plans to make um that support uh ipp6 in the 50 released, but for now we're focused on ipv4 uh for the four hour release, and it does now, uh as requested, allow uh a different port to listen on our initial implementation.

E

We actually added on the same HTTP, or does everything else requested, actually move that on to a separate Port that is now possible.

E

You just specify the Prometheus export report along with the plugin, so to enable it we simply enable the Prometheus plugin, like you, can do other plugins and then add to Prometheus export report for the given Port that you want it to uh to listen on, and then it will serve that that endpoint out which, which is B1 Prometheus a matrix, is, is that and that information is in the in the release, notes all right uh quickly. Moving along here uh launch lady.

E

We talked about this a little bit last time, uh but just time to talk about a little bit more related to the to the splitting out of logs is there's actually a new option that has nothing to do with that per se. But I put it in the same category. We do allow you to specify a different state directory now for notice.

E

So it's in the same vein of being overput, a very a persistent data and various locations on your file system, and so the state directory allows you to put the state somewhere else and so that that's useful.

E

If you want to run your state on, say a high high-end SSD drive, but once your your block log and maybe your ship log, maybe a slower, uh Rusty, hard drive, you know you can you can separate those out of query and so in uh 4.0 we have the ability of specifying a blocks, log stride and a state history stride. Here we go straight history, stripe and those are it's the same thing, uh one for Block log, one's for the State history logs, uh and so it's the same for the capability.

E

You specify a stride, which is the number of blocks that you want to have in each file. So the stride is it's a file here we have 10 now, obviously you wouldn't want to understand you, probably wouldn't ten thousand hundred thousand, maybe a million I, don't know uh whatever you number of walks you want to put in each file and then you can indicate if you like, the number of retained blocks that you want to to uh keep around and those are within the retained directory.

E

Now this is uh relative to the um the blocks directory uh if or it can be an absolute pass. So if you find absolute path, it won't be relative to block directory. It can be in a completely different location, completely different disk. If you will and then there's the archive now, the difference between the retain and the archive is the retained or why so? Nodius has access to the retain and will serve those out, even though they may be in a different location.

E

The archive are not available to know these once they move into the archive directory. No, these will no longer uh know they existed. You will it kind of shuffles them off there and then forgets about them and that's the same same Festa, same specifications for the State history, same behavior, okay, um continuing on here, so uh the other thing that we have in 4.0 is a wider validation for Relay nodes and what we mean by that is a faster block propagation.

E

So there's a number of things in fortnite, oh, that that makes the block propagation faster. uh One of the things is that we've just optimized that plug it there's a number of optimizations there and what ways that it that it handles blocks and with it it sends those out. Those are somewhat micro optimizations, but you know hopefully they'll make a little bit different. The big one is that we actually propagate blocks now after a header, validation, and not only that that hinder validation is actually not even done on the main thread.

E

So block can come into a node. Now it can validate that header um all off the main thread and then propagate that block to all its fears without ever uh disturbing the main thread that's over here, working on, let's say, processing transactions, and so a block should be able to propagate now much quicker, uh a 4.0 environment, as it simply does header validation and propagates that along so in terms, uh you know, give you an idea of what that means.

E

You know, let's say on mainnet on the the max block: CPU is the 100 uh 200 milliseconds currently, so that means that, typically for a full Block, it's going to take you about 200 milliseconds to validate that block.

E

So before this change, uh every hop along the way is a 200 millisecond delay as you validate the block and then pass it on now that has been reduced to whatever it takes to do a header validation, which is you know, maybe one millisecond, um as opposed to 200 uh milliseconds, so huge Improvement in terms of time as it as it jumps and, of course, for each hot before that is 200 milliseconds and then 200 milliseconds and then 200 milliseconds, as a block made its way across a node.

E

Now each of those are just going to be, like you know, a handful of milliseconds as it does block a header validation, so much much faster block propagation.

E

The other theme of the release and bride was talking about, is multi-thread support. That's done not only in what Lynn was talking about in terms of providing parallelization during the reading windows, but even if you don't even have that feature enabled. So even if you just go with read-only threads at zero uh and and only have a right window, there's still a number of multi-threaded improvements in 4.0. uh Two big ones are the block log and the fourth database uh are both traded safe now, which means that we can take advantage of that.

E

The net plug-in can now serve out blocks, for example, so uh a net plug-in that's serving blockchains peers can do that completely off the main thread, so it can treat the block package it up and send it across the network, while allowed ever disturbing the main thread, and they can do that concurrently across multiple connections. So you so, let's say you've got 10 tiers. All wanting different block ranges.

E

It can be serving all those uh at the same time, without disturbing uh anything else that that's running uh included with that we've also optimized to get Block call. That's a call that uh it's been a very slow in the past and one of our test cases the actually the one that we were using as a as testy.

E

We saw four x Improvement in the performance of git block, so rather FD Improvement in the terms of getting that, and even um it fill it faster period, plus uh more than half of its time has been moved off to main thread so before all that processing was on the main thread. So now it's four times faster, plus only about half of that time, if actually on the main thread or I thinking less than that um some fraction of that time, it's actually the rest of it had been completely off the main thread.

E

It's now running in bhcp thread pool so giving your HP thread, pull probably more threads and 4.0 is going to be, is going to be useful because it would be taking advantage of those threads of four watt, a block rate of thing, a lot more things included with that are two new endpoint get wall block and get Block header These are nice because they don't do the expensive things that git block does. That is converting it all into um expanded Json. It simply Returns the block unexpanded uh and allows you to do that.

E

Processing on the client, um and so those endpoints are now available in 4.0. Hopefully those will be very useful, particularly the get Block header. uh We anticipate that being very useful for um a light. Dilation work or um you know, work along the lines of the uh new, instant pan out and stuff that that doesn't work. uh Ship locks have been made um prepared, safe. That allows us now to for the ship to serve out its data without interrupting the main thread.

E

So now, Chip can serve all of its clients uh without interrupting uh the main thread, so it now, the ship is not so there's only two threads there.

E

The the ship thread that serving over its clients and then the main thread, that's writing to the the ship logs, uh but that the one thread that's serving all the client doesn't interfere with the others, and so it's not multi-threaded in terms of a thread per connection, but it is has a separate, dedicated thread for serving all the science which should see uh should help the the put there.

E

Along with those sorts of changes, the guilf longing um turns out that that was pretty expensive and it was all on the main thread now move that off the main thread. So the gel's logging should should not interfere with performance like it does in fb4 4.0.

E

So it actually slows the note down quite a bit, um which was a bit of a surprise to us, but uh anyway, so that's all been moved off into its old thread, so it doesn't look very with the domain trailer processing, along with that is, we've made uh various optimizations to the HP plugins, so not only, uh instead of taking better advantages of the thread, pull that you're giving in, but also just in terms of the processing that it's doing so we we uh it's doing less string copies, it's doing uh faster, uh Json, uh parsing now and I'm hopeful I, don't know when we'll get it done, but sometime soon we hope to move actually to a better Json implementation to Json application.

E

We have is not great. um It's back to you, know, 1.0, a version of nodius, saying Json partial, that's been there forever. It's not a great implementation, uh there's better ones out there that we want to live to, but even if one we've made some improvements, one example that we we saw was that you know a set code to be a vial system as an example, um the Json parsing for that has been reduced from 3.5 milliseconds to 1.4, so uh pretty pretty good Improvement there on that particular one.

E

Hopefully that's, um hopefully that's across the board for all uh all Json. It's not just that color! It's like every Json uh interaction that you're doing every uh get Trace every get block. All that stuff uh is now uh slightly improved with the better Json portion, huh okay, uh additional things that I've run out of time, uh subjective, PPU improvements and four out at all. There's some there's some tiny kind of improvements there. We now enforce uh the subjective CPU on transaction deadline, it's a pretty small little kind of corner case, but it's there.

E

Oh, we also apply subject to billing now onto producer. No, during the production window, we used to not do that. We now do uh so. It gives a bit a bit better uh Bookkeeping on subjective billing on the producer note itself. So again, pretty in a narrow uh Corner case, but those have been taken care of them 4.0.

E

uh There are there's better logging throughout, so not just Prometheus for it, but just in terms of various logging improvements, uh one big one there is that the CPU usage now tells you why the shipping was exceeded, which I think will be useful for users give them a better understanding of what limit did I hit uh when I ran out of syncing was its objective? Was it uh was it consensus? Was it known configured those sorts of things?

E

Look now tells you which of the which of the various uh configuration options your particular transaction has exceeded, as opposed to before. Just saying that you ran out of something uh good luck figured out what it was. um So now it's much more descriptive uh and then. Lastly, um there was the terrific a lot of talk on this call about the resource, Monitor and saline 4.0. There is the new resource monitor space, absolute setting, so you can set at F oops. You can do that.

E

You can set an absolute setting for uh the threshold as opposed to a percentage and that's been I. Think again, talk about in time for on this call, and so now you can set that, as as the gigabyte set eight, uh you know whatever you want that to be, and that's going to be per all the various nodes that are disks, that it's that's monitoring and that will override uh the the percentage setting.

E

Okay, so I've run out of time so I'm going to in there um yeah.

B

So I said excellent thanks everyone for uh for your demo and overview today. um We are over time so I'll hand it back to Daniel after one last thing, which is just again. If you have any questions, um please by all means reach out to me on telegram at be Hazard, that's b-h-a-z-z-e-a-r-d be eager to get answers for you thanks. So with that Daniel.

A

Bryan and thank you to everyone for the demos looking forward to uh to sharing that more broadly of the community, so yeah we'll post up that video recording as soon as it's ready and uh and we'll see you again next week, I guess next week we're back to continue the node special purpose. Node discussion we'll see you all next week to continue that discussion.

A

Thanks everybody bye.