Everything EOS EOS Leap Upgrade Group, 29 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: EOS Node Operator Roundtable Meeting #41 (Nov 23rd) 2022

Description

This week the EOS Node Operator Roundtable updates the community on Software Development, P2P Improvements, and discusses State Database Trimming, focusing on RAM overuse and how best to solve this problem.

00:00:00 Start
00:00:03 Recap and this week's topics
00:01:43 Software Development update
00:06:23 Defining a process for topics
00:11:02 State Database Trimming technical discussion
00:44:08 P2P Improvements
00:53:33 End

Summary

🟢 Leap Software Status

Recent release:
Leap 3.2 Final Release
Antelope software/documentation updates on the way:
Document on how crypto extensions inside CDT to utilize
crypto primitive host functions
System contract updates
Release of DUNE in the next month

Technical roundtable for EOS Node Operators
This week, we hosted a technical discussion on state database
trimming, proposed by Michael from EOSUSA.
Reframed - The problem: we use too much RAM for storing
state history, and it keeps getting bigger and harder to manage.

Question posed:
What are the trade-offs we're willing to make in
performance vs RAM size?

Answer: willing to trade-off 1 block production cycle for
loading data into RAM (like a warming block)?
this comes with risk of abuse

Short term opportunities
An RFP is being drafted by the Antelope coalition to research this
issue.
Make Heap mode startup and shutdown faster
Is there an opportunity to make the tmpfs more out of the box?
Leaving account queries disabled can save on RAM (opportunity for
node operator configuration, already possible)
Store account queries to disk (maybe 4GB for roughly 14M accounts
on WAX of ram in savings here, probably not worth it)

Long term opportunities
Can smart contracts be incentivized to specify RAM vs Disk storage?
Hardware vendors need to start offering very fast CPU cores with
high amounts of RAM

Next week
For next week, Matthew from EOS Nation has proposed a discussion
on what statistics should be added to the prometheus exporter in
Leap 4.0.
nodeEOS is a black box, we have no idea what's happening,
let's provide some statistics
Bring your list of what you'd like to have in prometheus to the
next call

Participants (14)
Max_Cho :: KOREOS Owner-South Korea
Randall Roland
Matt Witherspoon
Kevin Heifner
Brian Hazzard
Stephen Diesel
Daniel Keyes | EOS Nation
Corvin Meyer auf der Heide
Jannis - Rakeden (Jannis)
Matthew Darwin
Patrick - Aloha EOS (Patrick Burns)
Ross (EOSphere)
Corvin Meyer auf der Heide - cmadh | liquiid.io (Corvin Meyer auf der
Heide)
Dario | EOSsupport.io

The EOS Network Foundation coordinates financial and non-financial support to encourage the growth and development of the EOS Network. We’re harnessing the power of decentralization to chart a coordinated future for the EOS Network as a force for positive global change.

// EOS Network Foundation //
https://www.eosnetwork.com
https://twitter.com/EosNFoundation
https://www.linkedin.com/company/eos-network-foundation
https://t.me/EOSNetworkFoundation
https://discord.gg/eos-network
info@eosnetwork.com
#EOS #Antelope #Web3

A

All right welcome everybody. It's November 23rd! This is week eight since EOS Independence.

A

um Last week we had a pretty productive discussion talking about uh net plug-in improvements, um P2P improvements more more broadly um the outcome of that I'm, not sure. If, if Stephen there's been I've linked to the I know, we had this um GitHub issue here, tracking that um and I know you were taking some notes during during the call yesterday are those already now captured here or is that still a work in progress.

B

um That that's still a work in progress, uh so Brian and I are collaborating on next steps around this item um and we should have an update in the next week or so cool.

A

Cool, so the we've linked, so this was sort of the problem here that we were discussing yesterday. Lots of great ideas came out of the call last week. Some potential new features for the backlog, um so Stephen and Brian will be updating this issue. Accordingly, with the outcome of that call from last week, um and this week, um suggestion from Michael EOS USA is to discuss State database trimming, uh it's Thanksgiving in the US now he's unable to join.

A

Today um we can, we can decide whether we want to continue having that conversation or table that for another date. We have folks on the line who can speak to that, but before we get into that, maybe we want a quick check in on uh on a software development side of things for leap Antelope. Last week the 3.2 final release was expected. I believe that's out now correct me. If I'm wrong, Stephen.

B

Yeah, that's correct um and I might actually hand it over to Brian, since he uh handled the release this time, um but I'll fill in anything else. If there is anything else to update.

A

B

A

Take it away Brian.

C

Yeah, so the uh the leap V2 sorry v3.2.0 um final uh is, is now released on the antelope. Github um I saw that some block producers have already upgraded.

C

um This is a this is not a consensus upgrade, so um it is optional and- and uh you know, node operators can upgrade or not at their Leisure um yeah and if you know, I'm I'm always available on Telegram and the in the Eos block producers channels or in the Antelope um uh uh upgrade channels. uh If there are any questions about the release, I I keep an eye on this chance.

A

Great so I'll draw a link to that in the updated agenda for today make sure everyone's got that handy and uh thank you Stephen for sharing that in the chat here. I'll pull that up. Your other release, notes and yeah like I said, though, we'll share that any questions from anybody or comments on the 3.2 release.

D

I haven't actually used the leap you tool program, um I actually wasn't expecting it I probably wasn't paying attention. The last call when, uh when you spoke about it, is anybody here actually used it.

D

I have not I like the way it. It says somewhere that it's got order complete I need to I need to feel that Kevin I don't want to feel that.

E

Yeah, it's pretty cool.

A

Do we want to talk a little bit about leap utility? Is there and is there any like now? We've got some node operators on the line here. This is a utility targeted, two node operators. Do we want to get any feedback or testing on this from from some of the people on the line here.

B

um I I think it'd be helpful just to get feedback just in terms of like the experience of using it and if there's any uh like prioritizations of commands or just organization of things that you wouldn't expect. The only thing I'd say about that is that um we are actively making some updates here. uh Some of it is going to be adding a few commands.

B

I know we're working on integrating some snapshot related functionality, but I also know um there is some work related to CLI 11 that we're refactoring I I'm, not as close to that work, but I I don't want to send people too far down giving us feedback. If uh some of the nav is going to update inside of that utility foreign.

A

So it is useful to Loop node operators as a few of us on this call. Yeah.

D

I use I use kiosk all the time right, but but like some of the kiosks that I'm using is from The Arc. You know you know like when it's just kind of there on the desktop you're just using it all the time.

D

Not even an old version I got.

B

B

With a couple more updates um Beyond this, you know uh we we've still got an RC out for CDT, uh we'll probably be looking to finalize that pretty soon and we are still working on a few minor improvements to Dune. So we don't have a delivery date for that, just yet to kind of close up this kind of three segments of our primary release process, but uh that and system contract updates still don't have uh an actual due date. Yet but they're things that we're making progress towards right.

A

So these three here are still in progress. This is out.

B

That's correct cool.

A

All right any other comments or questions on that.

A

Then we can move on, um so the question is: do we want to talk about this? uh Maybe something else we can talk about is the process that we want to follow as a group to Define what are the topics and is there any homework that we should be doing ahead of time? You know in the case, you know the last conversation we had I think it was useful having that um that issue to discuss the net plug-ins as a starting point for us to frame the conversation around I.

A

Don't know if we have something similar for the the topic here that Michael suggested.

A

um Should that be a prerequisite going forward? Do we need to? Is there a process we want to Define here as a group that hey in order to get a topic to discuss, get your problem statement written up in a way we can all review in advance and and then have some process for prioritizing the order that we talked about them.

B

I think that would definitely be helpful. I think um the easiest First Step, that would add value I, think is just having um a schedule where people can even like propose slots and then we'll, of course, have a list that we can work to prioritize against. But I think even just people on the invite being able to see what's projected into the future and where they'd want to be engaged I think would make a big difference.

A

Okay, yeah, that's an easy fix, so I'll a couple couple: improvements I'm going to make sure we're linking to these agendas in the uh in the calendar. Invite and we'll be at I'll add a sort of a calendar, rudimentary calendar on upcoming topics and and a way to sign up for those for future open slots.

A

um This time, I just did put a put a comment: give give an opportunity to thumbs up and thumbs down, and we see we had a few thumbs up. Let's say: First Step tab. What's tab, Ross.

D

You know that little button on the left next to the queue tab, oh yeah, so like you know, you know what Matthew is saying order complete. So, oh.

C

D

Complete is tab, tab, tab.

F

Wow yeah: that's what I tried I tried to tab. It doesn't do anything. So that's why I'm asking oh push it harder. Oh, oh sorry,.

E

I believe you have to install it. um There are scripts that have to go in a known location. So if you don't want to install it like I I hate installing anything but uh I think you can probably copy those scripts manually, uh I'm, not sure where the documentation for that is, but we can maybe find it and point it to you point you at it.

F

Well, that would be good because there's nothing in the instructions.

D

You see what's happened here, it's it's the little things that are the most important to us.

F

Absolutely I use tab for everything, so auto complete is awesome, but totally.

E

All right yeah we need to. uh We need to get the docs together for that and and um and get that somewhere yeah, we'll.

A

Take it, you guys are talking about leaked util now right, yeah,.

D

B

D

B

E

Yeah Cliff has the autocomplete as well I believe that made it in this release right. It did.

B

Yeah yeah cool.

A

Cool, so we've got some feedback already.

E

Yeah, the easy answer is I believe you have to install them. um If you don't install them, then we'll have to get you the manual instruction.

F

Well, I mean I just want to build copy the binaries over to the machine and yeah. That's what I've been doing since you know 2018. right.

E

Exactly and that's what what I would do as well, but you know if you actually do an install it, it puts in script somewhere. That makes all.

F

That stuff work right, which I don't because build machine and the Run machine are two different machines. Yeah.

E

Yeah, so we should get the manual inspections, because it's just a couple scripts you got to throw in a directory. It's not that big a deal yeah.

B

We'll we'll get that taken care of it won't happen.

E

Last week, obviously.

B

But within the next week yeah.

D

We don't care about pairing, we don't care about any optimize that we care about order. Complete demo I mean we've seen it. It's advertised show it to us.

A

All right back to State database trimming well.

C

Actually, sorry to sorry to interrupt before we go to that, I I wanted to ask um Kevin and and Stephen whether you think it's worthwhile here to discuss some of the optimizations that we've we've been discussing with regards to the P2P improvements.

B

Well, I I think we mostly discussed that last week. But if, if there's more things, I guess we did have a meeting since then that we could bring in here um yeah.

C

I think we got. We got more specific with like some of the things that we we might want to uh do for the next release, um and so I don't know. I don't want to derail by any means if we'd rather focus on state database trimming, but as we have an update and we've sort of talked around the P2P improvements a little bit, I thought I'd at least bring it up.

A

Maybe, let's settle on whether we need to have time for State database trimming today so I, don't I don't have a problem statement I'm. This is foreign to me: I'm, not a node operator, myself, I'm just I'm just here, organizing the meeting um is there anybody who has a good grasp of this problem that could Frame It Up to have a conversation on this or again do we need to wait uh another day for when we have Michael on the line to to frame that conversation for us problems.

F

Should be well understood, it takes too much RAM to run node us. So if you don't wax and the problem, how do we, how do we reduce it? What's the techniques? What's the plan, ideas, I.

D

Guess so maybe maybe it's mislabeled? It's not really the state So when you say State database to me I'm thinking about what's saved on the hard drive I suppose it is a replica of what's the memory but yeah.

F

Top priorities so.

A

um Yes, so say that again, how would you reframe this.

D

Well, well so either he was talking about that. It threshes the hard drive, because it's a replica of what's in Ram, or it was actually this to save some save some Ram.

F

D

F

Combination thereof there up.

D

Here, I I suspect it's just because it's the ram thing: saving Ram.

A

Yeah, so the problem is we're using too much RAM. Basically, is that or.

D

Yeah I think I think that's the problem. Yeah we used. We use way too much RAM, especially when uh everything's in memory we needed to be fuss but modern Ram.

B

So quickly on this topic, uh it is correct that this is an antelope Coalition priority.

B

um I can also say that they're actively working towards approving an RFP, that's related to research uh for some of these Ram limitations, um so I I think that we could definitely gather feedback today, but I also wanted to provide some awareness that I think that there's going to be a request for people to perform research to propose the solutions to these specific things as an opportunity to you know, contribute towards the coalition's goals.

B

um So I think your the feedback would, of course be helpful. I just wanted to provide that context that there's a set of requirements. That's been drafted around doing that research, specifically um across a few different strategies.

E

I think one question there: I have just as a um something to think about to you know from from you, guys's perspective is: what are you willing to trade in terms of performance for reduction in Stateside in memory size? um You know it. Is there a? Is there a percentage? Is there something that you would talk about uh there in terms of what what any particular change would be willing to um trade?

E

You know ten percent, fifty percent, you know what sort of performance would you be willing to give up for a smaller uh State, Ram size.

D

From my perspective, there's clearly a lot of stuff: that's in Ram that isn't regularly accessed and we get around that by you know maybe dumping some of it inside swap or um using the other mode that you guys have maybe there's a way to optimize, what's actually kept in Ram, some more smarts.

F

Yeah, when I'm in the or like the first step, could be like a more codified, the temp FS solution so uh like.

D

It's kind of there really with the other mode right well,.

F

Yeah the the database direct access mode on disk yeah, but yeah yeah. You know it uses all your the your iops here and kills your your nvme drive right where temp FS, you know, there's well I'll write a script to load. This move the files around then. Oh, it works, um which is a bit awkward to explain to somebody how to use that.

F

um Even though Ross wrote a nice article on how to do that, it's.

D

But the the temp FS strategy, uh like inside of Linux, the way that it uses the temp fs and the swap file seems to work quite well. It'd be nice. If we added another level of smarts on top of it, that um yeah like you're, saying that that it immediately went there not putting everything into resident memory that needed to be replicated.

F

Yeah so I'm not sure you know how do we determine what is needed and what is not needed?

F

Yeah right, because I guess the the the as transactions arrive to a node they get speculatively executed right away, which would then pull stuff into the cache so by hopefully by the time you got to produce a block uh it's already in Cache. However, if the transaction arrives, while block production is in progress, then I think the problem is it's going to get billed for CPU for loading, the stuff off the disk or wherever it's not inactive memory? Oh maybe Kevin can correct me.

F

Oh wow, okay right, so it probably needs some strategy to not count that load the stuff off disk into memory as part of the transaction toss. Otherwise, you're going to be randomly billing people for too much depending on whether it was in cash or not in cash, which is you know if we look at what we're trying to do in the Eos and Telos ecosystems. Right now is trying to give a nice average billing to people without all these spikes right.

F

So I don't want to introduce a feature that introduced some more weird CPU cost fights.

D

What are you thinking Kevin? Have you got something playing on your mind that you reckon no.

E

No but I mean somewhat implicit in that in your answer. There was I think was that you're not willing to trade any performance for it.

F

Well, that's the normal answer. Right I want all of it.

D

E

Mean the question still stands: are you willing to give up any performance for Less memory use and if the answer is no, then I mean you know that certainly changes I think what options are available.

D

Well, from my perspective, we really are trading some of that in because we we we're running a 128 gig um estate database inside 64, Meg, 64, gigs of RAM, using temp fs and letting that sort it out for us and it seems to kind of work.

E

So does that mean we don't need a solution? Just try.

D

To avoid that, well, what Matthew is saying is maybe yeah something yeah and is it doing? The right thing is anything that's sort of duplicated and do you have to maybe replicate everything and uh you know I, don't know you know from a from a fundamental low-level blockchain operational standpoint from where I know that you are quite very you understand the tested details of it is, you know, there's any way to optimize it. We've really taken a step where hey stuff that isn't regularly accessed we put in that on disk.

D

If we could make it, let me take it to the next level. Like okay, you know, we know this is definitely not going to be used much. We can immediately put that through to uh a tamper, fast folder, or something like that. You know, I, don't know.

G

But that's the thing is: how do you know? How do you really know it's not going to be regularly I know one.

D

F

En't, that's the thing right so that comes back to my comment about. We need to be able to load stuff from the not recent cash without causing a CPU billing issue for our user. So I guess maybe that's the requirement, so maybe a transaction, maybe transactions can't get in a block if they arrive, while the block is being produced, for example, so the transaction has to arrive before the block production cycle um and if it doesn't well, then you know it waits the next block, because.

D

How about there's always a warming cycle? It always yeah. It's it's kept on the side until it's requested once and then boom, then it gets kept there for a determinant amount of time and.

F

So so, we'll put out again so we're willing to trade off one block production cycle to delay a transaction. Maybe is potentially the trade-off we're talking about here. Why not? It seems like a reasonable like if you haven't accessed your account in a year and all of a sudden, you do a transaction if we delay You by one block or two blocks. Does that really matter nah, probably not not at all.

G

But that's also under the assumption that someone's not deliberately abusing this heuristic.

F

Of course, because they could create a million accounts and then cycle through them. Well,.

G

Since you know any contract can read any other contracts tables, uh you know it's someone can someone can basically start spamming reads across the entire database.

F

Then you get subjectively billed for using too many accounts.

E

Yeah right, even if you support the speculative execution you can you know you can rig your transaction to execute differently when it's actually put into a block right, so I mean we've. We see that already and you can do that with what a map she was talking about with reads right nominal case: you read your own tables and your attack. Vector. Is you read other tables, although maybe we shouldn't go into too much details on how to attack the network? You know.

D

Attacking Network 101.

F

Watch the Youtube, but yeah I mean that's the thing I mean all of these Solutions. Are you know? How does it help the normal case? And then how is somebody gonna? You know do weird stuff with it, because we've seen interesting activity for any yeah, weird stuff over the last four years.

D

So I suppose I suppose the problem statement is that uh we're using too much memory, especially with the Vizio chains, and it's just increasing on a day-to-day basis. More and more and more and more until eventually, you know I, don't know about Moore's law on on RAM. But you know it's getting hotter and harder to effectively be I'm, not pure everything running around, rather than some kind of temper solution.

F

Yeah, so so a couple of things here, yeah, is there an option to like make the temp FS solution more out of the box? uh I do remember, opening a issue for Block one way back when to try to do that. uh That's one possible option which doesn't require huge redesign. Maybe uh the other thing is maybe um what was I thinking now. I forgot.

A

uh Am I capturing this right, temp FS.

D

F

Tmp tmpfs one word: yeah.

D

One acronym.

F

uh Yeah I forget what I was gonna say: oh I'll, think about it again.

A

F

What was the other side.

A

Of this trade-off here you were talking about we're.

C

A

We're willing to trade off one block production cycle for.

F

For you know, loading data into RAM, so.

D

Yeah I like it, what do you call it in cash like when you catch something? It's like a warm or warming block.

D

D

I, don't know what you I, don't know what the correct way to say. But you know what I mean. You know what I mean Kevin like you know you just requested putting in cash and then it's available put it in the ramp and it's available.

F

Which then comes with some issues because it could be maybe abused all right so.

C

This is kind of a abstract and sort of ignorant thought I think maybe but like I, wonder if the uh some form of economic incentive and disincentive could apply to it to encourage contract developers to be explicit about what things should be held in ram versus you mean.

E

Like a disc versus Ram sort of thing that you specify yeah.

D

Two tears of memory.

F

Well, that would help it doesn't help with the accounts unless we're rewriting the account management system right, which is the big problem. Yeah.

F

um The other, the other thing that I I thought of like two minutes ago, which I've forgotten that I remember now is I, don't know. If is there a way to speed up uh like keep mode like when you start node with uh in Heap mode, which loads everything into RAM? It takes bloody forever, so yeah.

G

I mean it that's clear that clearly can be as fast as your disc can be, and it's not today, yeah so I mean that's a low hanging up holy well hanging fruit, optimization there.

A

D

A

um Heat mode faster because.

D

He brought it similar to Champion Affairs, especially if uh you just restrict how much RAM is in the the machine and and uh you've got a big swap file. Well,.

F

No well yeah, but it's it's worse than Tampa, Fest kind of because it loads about it, because it's slow, like it loads. Everything in Rams.

B

Blocks everything.

F

Out and it's like so.

B

F

Lot but it's really not ideal I.

D

Mean yeah I mean and Champion phase isn't super lightning fast, but um yeah I I use Tampere first, instead of heat because it just uh it was I like to have the memory management done from an OS level.

D

I didn't do any benchmarks against the team.

D

F

Yeah I don't know if there's any other suggestions. This is a hard problem.

C

Right so I think that the initial discussion or suggestion was State database trimming, which I think is the one thing we haven't actually talked about. It's like.

D

But that's that's what we think this. This is what he's talking about. I, don't think he's specifically talking about like trimming. The database Estates database is just what we run inside memory.

E

I see: okay, yeah, you don't really need it smaller for, like this size or something right. It's not like it's. You know it's large, no.

F

The size on disk is not a problem right, it's the size and round. That's a problem. Yeah.

E

And that last sentence that you have there, you actually mean make heat Mode start up faster, not uh yes, start up, yep and I just shut down as.

F

Well, with startups specifically correct: okay, sorry Matt, you were going to say something.

G

Well, I just think part of the problem here is that you know I think it's it's possible to imagine how, in the current design, even just memory usage can be reduced, but you know that's really just like even like a short or a maybe a midterm solution like if the idea is that you always want to be able to operate a network on the scale of you know, wax or something on 64 gigs Ram, then yeah I start wondering if you really need some more uh invasive changes to the protocol.

G

Like kind of what you're talking about with the the ram versus desk.

F

uh Yeah, potentially yeah, so maybe we need to bucket this into a short-term things. We can do in the next six months, nine months whatever and something, oh, let's think about long term, we're doing disk storage and what that means to contracts. And you know how do you migrate from an existing system too?.

D

You see what Pat just said they met and we we're not just trying to run wax, for instance, on the most venous nodes available, he's saying that we need about 256 gigs of RAM to run what's actually required in the state database.

G

That's a bit surprising: why do you need 256.

D

Well, that's, that's just how big it is. If you had, if you didn't have any kind of um sorry he said 128. so last time I looked, it was over 128 gigs of RAM. So as soon as you go low, then that you'll be doing some kind of swapping because it needed more, like the actual State database itself is already right up there on 128 gigs around or 128 gigs of data size. So it's just about you know. How far do you go to start trimming that off and swapping it out?

D

You know and it'll just keep on going and going and going as far as growing in size to maybe it'll get to a point where it's unsustainable.

D

You know ready right now, some of the high performance VP nodes they're, using like consumer-based CPUs that have got a trouble with um you know you can't necessarily put that amount of ram in a box. You know so it's that kind of stuff. We kind of look at a turning point where you know we need more RAM. Some platforms can't even take that kind of ram at this point. So it's a bit of a challenge.

D

There was a way for us to um manage that amount of physical RAM usage from the actual nodia side of things. That would be, it would be awesome.

F

Yeah or we could just get the hardware vendors to create really fast CPU cores, which supports lots of ram, maybe that's easier!

F

Dear Intel, please create us a platform like this yeah. You know because, like seriously acquiring Hardware is the is a big challenge because it needs to be. You know very fast core with a lot of RAM and how many use cases have this uh situation blockchain and uh gaming gaming, but gaming doesn't need you know, 512 gigs of RAM.

D

Sorry I'm talking about uh fast CPU cores first.

F

Cpu cores with high amounts of brown yeah.

D

So two things together: that's the problem right because the gaming is where it's at. So all the high performance, CPUs per call are gaming, but they don't need a huge amount of ram. That's right. So the platform is by default. Don't like suddenly support off a terabyte Ram the support, if you're, lucky 256 gig, but most of the time maximum of 128 gig.

F

So the only other thing that I've seen is like maybe like animators, need a lot of RAM and high CPU speed and they're doing that so but yeah. But this is not a normal use case for uh for vendor security, Hardware forms.

F

So it's been a challenge to uh to acquire such machines.

F

So yeah I mean if we, if we want to discuss you, know totally re-architecting core fundamentals of how node, EOS or leap or Antelope protocol works, uh great uh I'm, not sure I'm, the best person to do that.

A

Do we want to so I think that was a good um good thing you've mentioned there? Is there an opportunity to bucket these short-term long-term sounds like there's some low-hanging fruit here, I heard this one maybe is a little hanging fruit. We can put under short term to get that right.

D

I just had a look: wax is 86 gigs in use in the state TV.

F

Yeah, it's reduced over the last couple of weeks.

D

Months, it was over 128, it was over 128.

F

It's been reduced because they're, freeing up from the the random number generator they're, releasing Ram.

C

I think the temp FS one would would probably fit under short term right.

H

Enabling disabling conqueries also plays a role on the the overall amount of from conjunctional.

A

So that enabling and disabling.

H

Account queries.

A

Account queries account, queries yeah because.

H

They're, looking to interim or not so typically on a on a on a producer node, you will not enable the concrete on an API node. You will so the conjunction on an API not just a little higher, especially with wax where we have millions of accounts.

A

H

A

That is that a new configuration request right now. It's always.

H

Unable no it's already it's already there. It just plays a role uh when you when you, when you yeah, run your own notes depending on, if you enable or disable it um the wrong consumption, is it's higher laws.

F

Yeah, that was a feature request that was added in 1.8 2.0 I forget, based on the discussion uh to save history. Queries. Let's just have node us provide this.

E

Is not State Ram, that's just ram ram yeah.

D

I was looking at that yeah yeah, okay, there's no more RAM yeah.

A

So um would it be disabling or enabling would be the reduction or does it depend.

E

H

Already that's already available, it's already it's already there. It's already a feature. That's already there.

A

But right, okay! So, but is there a instruction to provide to know hey node operators, disable account queries you save on RAM, no.

D

You have to enable to enable it by default. Yeah you have to enable it by default, it's disabled, yeah,.

F

Yeah and and I'm not sure, there's anything to do there on this right.

D

I actually never noticed it, but.

F

Yeah but it's a great point but yeah: it definitely uses more RAM on the the API notes.

H

Then you know your sentence is wrong. It's it's disabling account. Queries saves around and.

B

H

Just increases around yes, it's disabled by default, so yeah also.

A

Just leaving it disabled, okay do.

E

You have a feel for how much that adds how much RAM.

F

I agree, don't know, I never checked.

E

uh Well, I wonder if that's something like for short term like if we just move that to disk um yeah, it would make it slower but I wonder if you care.

H

I would assume that if we have like a clever caching mechanism for for accounts that would work well, I mean how many of the accounts we have on wax are frequently accessed. It's probably just a small percentage, and this small percentage is exist again and again right right.

E

um So if you could give us an idea of how much RAM like that's using on wax um yeah I mean maybe short-term solution, there is to just move that to disk with a very small cash size, and you know, because that doesn't affect your um your BP performance in terms of you know like how much stuff's getting in a block. This is just an API, uh no sort of thing, so you might want to trade off ram for for performance.

H

A

So, what's that moving the account queries to disk.

E

Yeah yeah now granted we get a feel for how big it is now, if it's one gig well, then, obviously that's not going to make any difference, but you know, if that's a substantial part of the RAM on your API node is, is enabling that account query thing which it may be I I, just don't know what that looks like uh on on wax and you know like we could do that. We could evaluate how much that is and yeah look at alternatives.

D

Different amounts of peers and stuff doesn't that affect it as well? It.

E

Does yeah so I mean we might need a little bit more fine, okay,.

D

I'm, looking for the 32 gig sailing, I'm gonna find it.

D

Oh, let me just have a look: I mean we've got, we've obviously got it on our public. Apis and uh um blood producer doesn't have it. So let me see.

E

Yeah, it's probably not very good comparison, because a lot of stuff as well and.

B

Okay, well, that's gonna bring us back around to a relevant topic, which is just um how we actually qualify. These improvements and I think inputs on like how tangibly we can evaluate that, and even like early ideas around like controls across different configurations.

B

um Those perspectives are always valuable.

D

Okay, how about this given I'll public API, a public seed node, is using 94.4 gigs of RAM, and the state database is 86.8. Gigs of RAM Darius is probably using less than a gig, so maybe somewhere in there there's three gigs of RAM four gigs of RAM Maybe. It's not huge.

E

Yeah I wouldn't sound like working like it's worth it, but.

D

Yeah, it's not like this. Oh there's, the golden ticket here, yeah.

A

Foreign, could you repeat what.

B

A

I was like you're suggesting.

B

That um input on, like what the actual configuration should be to evaluate these changes, would be helpful uh from the perspective of some of these node operators.

B

um Even just comparing those two notes that were just mentioned, um you know it. It would be helpful to have perspective and input there.

B

E

Exercise for you guys in terms of like this understanding, this memory stuff is, you can use the new utility convert your state database to Json um I, don't think that made the 3-2 release, so that may just be in the main branch, but you can do that and then you know you can open that Json that very large Json file in an editor- and you can look around- you- can even kind of get a feel for like how much of it is is accounts by looking at the section, that's accounts and how much of its user table data by looking at those you can look at individual user tables actually to get a feel for how big each of those are so kind of an interesting exercise.

E

If you guys want to dig around and see like where your memory is actually being used. Now, of course, it's not exactly one-to-one, but certainly gives you a good kind of feel for it. Foreign.

D

If you edited anything in it,.

E

uh Yeah, you can do that. We don't I, don't think we have the feature in in Maine or anywhere yet to read in a snapshot uh or a Json format. uh Maybe we do I I forget where we have that functionality, but um maybe that's the main right now, but yeah. You can actually do that uh and read in the uh the the changes.

D

But if we had to go and change anything, would there be like a different hash on the file that it wouldn't enable it to communicate with its peer or something like that. Just other interest.

E

um I'd have to go look where we landed on on that I think we forced um a chain idea, ID different by default, but obviously it.

D

Was just out of interest, you know because I.

H

D

Saw it as yeah always sort of something that needed to be perfectly match its pit.

E

Well, yeah uh for consensus to to work you you have. You know your states have to match um now what consensus requires and what you can do by hacking up a node or two different things: foreign.

H

Tes are for roughly 14 million accounts.

D

But I mean I've also got like um monitoring, agents running and like I, don't know how to specifically break it out. I, don't know how I don't know how accurate this is literally thumb. Suck yeah like I mean four and five gigs is a huge difference in data right and I'm. Just this is just it's not 32 gigabytes, there's not so much stuff.

H

Just if someone comes back in the future reading these notes and asking for where this number comes from, it's for 40 million accounts and on I mean on Euros, it's much less, so it makes even.

D

uh It's wax yeah, smaller difference, yeah yeah, it's just wax, because this is the monolith. That's the one, we're always measure against at the moment.

A

Okay, well, we've got less than 10 minutes left on the uh on the calendar here today sounds like we've got some starting off points, at least any uh any other thoughts or ideas we want to discuss while we're all here.

A

And otherwise, if we want, we have five minutes left to maybe close the loop on that discussion. Brian and Steven you had about the peer-to-peer improvements. I, don't know if there's anything within five minutes, we you wanted to share.

C

Sure so there were I'll, give kind of the the breadth rather the depth first and and then, if there's anything, we want to go deeper on. It could be a topic for next time or something. um So there were three three major um sort of ideas for optimizations on the PDP side that we've sort of discussed.

C

um One was that we could. uh We could make lighter the the validations that occur for um for blocks on relay um right now, they're doing like a full, full validation, and we could just do header validations to confirm that the signatures from a trusted Source like the producer that should have produced that block that sort of thing and that could save um time and make that relay faster.

C

um Another suggestion was that we could um sort of codify if, if a block is full in terms of build uh CPU time, um immediately broadcasting that and starting on the next block, um so that you know you can kind of get ahead there and then the last uh of the suggestions was um sort of Auto configuring. Your your um I didn't actually take this note.

C

I believe it was peers, Auto configuring, your peers to optimize latency, um which I think today is something that BPS are doing manually based on, like the the rankings right, whose comes before and after them, um and so that could just be built in to LEAP as an option that you can turn on.

F

uh Well, I think there's two different parts to that last one right: there's optimized for the schedule who's before me, who's after me versus optimized for who's got the lowest latency foreign might be on the other side of the planet.

E

Yeah, so most of that was what we talked about in the meeting last week. um The kind of different idea there that I, don't think had come up at all last week in our discussion, was the the relaying block Faster by doing just header validation before uh without doing a pool, a block validation, so that gets. You know that that's targeting the idea that you know you've got to get those last couple of blocks from the VP before you to the next BP. uh That mechanism would get those blocks there very faster.

F

So how's that different than the existing trust producer or config option where you can say who to trust or that's the same thing right.

E

um It's not the same thing, the the trust um uh you know turns off the the signature validations on the the transactions within a block. It still does the full block evaluation, um except for wait now that you mentioned it.

E

um The Trusted producer may actually go ahead and relay the block quicker, but this would be do it for everybody, not just for trusted producers. So the idea is that you'd still validate uh the signature on the Block and validate the the the transaction Merkel route, so that you know the block, isn't um you know you're doing some set of value evaluation of block Integrity you're, just not evaluating all the you're not actually executing all the transactions within the blocks to modify your state as part of that before you relay the block.

F

So does this mean we can have nodes that don't have accurate State and just relay blocks.

E

We didn't talk about providing that sort of functionality. No, you would still evaluate the block to just it just changes when that node passes the block onto the the next to to its peers. It would still after it did that continue to apply and evaluate those blocks. It's just when it released this block to it's blocked that block to its peers is sooner than it currently is right.

F

So relay and then validate rather instead of validate then relay correct.

G

Yes, yes, but that had been useful, though, to have some sort of service that can light validate the chain in forward blocks along, like you were asking.

F

Well so last week we talked about well, maybe we could get rid of these blocks, only nodes as an option, but what if we just made the blocks only knows really fast and they didn't have any state right, because if you're just gonna forward blocks yeah the Merkel route is good far of the block, but never even just throw it away after right. So you don't actually have a state database. I don't know. Is that possible?

F

Maybe that's useful. Now we can relay the blocks really fast. They don't take a lot of RAM.

D

But you use them as a demarcation. That's what you're saying.

F

Well, I'm, saying, like my request last week to Kevin to think about or to the team, was like I have these nodes that just relay blocks so that we can get the blocks faster to their peers right and what I my suggestion was well, could we get figure out how to make it so no DS doesn't need those and can relay the blocks faster, which is what this suggestion is.

F

What, if we don't follow it, we just cross the luck form well, we could also have nodes that just pass blocks long and don't keep any state right. Then they're really cheap to run yeah right.

D

Because they don't need any.

F

Memory yeah, it's just relay box yeah right and maybe those nodes also deal with like loading. You know snapshots, for example, when we get to doing snapshots and pushing snapshots through the peer-to-peer Network you.

C

Can't get transactions here, but.

F

You can get blocks and and I don't know, snapshots.

D

And a like a like a piece yeah.

D

Yeah I, like it.

A

Cool another feature for the backlog, Perhaps Perhaps.

F

Yeah for for for people who are smarter than me to talk about things yeah, it makes any.

D

Sense, like spin up a notice light that only pushes blocks.

A

Cool well we're at time uh for next week anything here we want to carry on or again we have this suggestion from Matthew to talk about what statistics are being added to the Prometheus Explorer in 4.0 um is that is that I see some heads nodding sounds like maybe that's a good topic for next week. Is that a good framing of the conversation or is there some way to reframe? Is there a problem statement here? We want to kind of put out there for people to start thinking about between now and next week.

E

Well, I think, from our perspective, I would love to hear what those are uh if we're going to discuss them. So you know bring them to the next meeting like bring your list of what you'd like to have in Prometheus. To do so that we can gather those requirements. I mean that's kind of what we're looking for at the stages like. What do you want and then you know, maybe we can start prioritizing those or you know, get low hanging fruit first or you know, look the various ways of providing those.

F

Yeah, so the the problem statement is node. Eos is the black box. We have no idea what it's doing. Let's provide some statistics about what's going on and what is the right priority of what things are going to be added? What.

E

Do you want to see.

F

What do you want to see what's doable and then figure out the intersection between those I I have a long list, but they're, not oh I'm, sure a reasonably doable in a short time.

F

I I know my my list won't uh get full fully done, but let's start with something easy. You know yes, cool.

A

Sounds like a good topic sounds like we've got uh people in agreement here, no Alternatives being raised. So that's we'll uh pencil that in for next week, uh unless there's anything else, we can wrap things up. Thank you. Everybody thanks.

D

Everyone thanks guys bye.