Everything EOS EOS Leap Upgrade Group, 7 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: EOS Node Operator Roundtable - Dec 7th, 2022

Description

This week the EOS Node Operator Roundtable updates the community on software development and continues the technical discussion on Prometheus focusing on Node metrics.

00:00:00 Start
00:00:02 Recap and this week's topics
00:00:29 Software development
00:02:18 Prometheus Exporter technical discussion
00:06:48 Node Metrics
00:09:47 Unapplied transaction queue sizes
00:16:25 Subjective billing sizes
00:18:47 Scheduled transaction size
00:20:09 Blocks by Producer accounts
00:33:49 Transactions per block by producer
00:42:47 Replay Status
00:46:32 End

The EOS Network Foundation coordinates financial and non-financial support to encourage the growth and development of the EOS Network. We’re harnessing the power of decentralization to chart a coordinated future for the EOS Network as a force for positive global change.

// EOS Network Foundation //
https://www.eosnetwork.com
https://twitter.com/EosNFoundation
https://www.linkedin.com/company/eos-network-foundation
https://t.me/EOSNetworkFoundation
https://discord.gg/eos-network
info@eosnetwork.com
#EOS #Web3

A

All right welcome everyone. It's December 7th, we're uh 11 weeks into U.S Independence. Today we're continuing the conversation we started last week about Prometheus exporter data we'd like to see added.

A

um We can do a quick recap of what we discussed yesterday before we get into that. As we always do, I don't know if there's anything on the software side from Steven or anyone else on the app want to add yeah.

B

Just one small update is that I think this coming Monday is, is what we'll be targeting for the next CDT release, um so I think that'll be 3.1 and that's what we're targeting at this time, I think Dunes going to follow, probably within a week or so of that, but just trying to like wrap all of this up before we really get deep into um like the the holiday-ish season for people.

B

So that's what we're working towards right now um that I I feel pretty confident about CDT for Monday and then um I'll have a clear update on June I think next week. Okay,.

C

uh I have a question: is there any um plan for a leap? Three two one release with the code that uh makes uh blocks go faster. We.

B

Are working we.

C

Are discussion on that yesterday.

B

Yeah I I know that we've been working through some discussion about a potential patch release. uh I, don't think that we've addressed when we would uh perform a patch release, but I think that there are actually a couple of things that we've identified an opportunity for a 3.2.1.

B

um That kind of needs a little bit more discussion before I can give a clear timeline around that.

C

Okay but I guess meantime: I can uh Kevin patch the code to pull back PRN uh once it's finalized.

B

Yeah, what do we call that is that kevver, how do you, how do you version those captions.

B

But yeah, that's it from our end. uh Nothing uh too big.

A

Cool, thank you um all right, so recapping from the where we left off last week on Prometheus Explorer, uh some of the things we'd like to see ability to configure how deep you monitor minimize impact to node, while Gathering statistics running on a different thread. Everything that we're currently logging get info head block.

A

Last irresversible block Etc uh configs of the node EOS runtime is OC enabled the console is logging on the features uh block logs, full note or trim node in the future. Some alternative get info endpoint that formats the data as Json I think this was mostly coming from uh Michael I'm sure that we've got.

D

A little bit on that I I did talk to uh Chris who's, doing the uh Prometheus implementation right now and he mentions you know like the the configuration information. That's there that Prometheus has really kind of set up as a Time series type database. uh You know sort of reporting, and so things like what the config is um doesn't quite fit into that model very well.

E

um It does, but it doesn't you kind of got a hack from atheists.

C

Should be a new API, endpoint V1 producer get config or something which dumps the config back.

E

Config I mean that would that would work as well just an endpoint, but a lot of times in Prometheus you, you know Define it as like: OC OC enabled and it's a Boolean over one, and then you know whatever you deal with on or off or you can roll text and stuff in there. But I mean back to that. General concept: I I, love, I use Prometheus, but just some way of exposing.

C

It I think, as Jason would be better, would be more usable.

F

I yeah I remember last time, um I asked that question and then the point was made I think maybe by Ted that that there could be some value in um uh still logging in the current config into a Time series, because it gives you a history effectively. You can pinpoint like. Oh that's when that config changed. You know that sort of thing.

F

um So it's it's not it's not data. That's going to do this like in any kind of meaningful way, but on.

G

Then off then off first two years, then on for three years: things like that or six for years and years and then seven for years and.

E

Years you'd be surprised when you run 40 50 nodes and you're tracking version numbers and stuff like that. No I get I mean you're right, they're much more static, but you get a lot of them out there. You'll you'll actually see a lot more activity than you know, one or two a year, but yeah I mean it's definitely different than uh uh head block or lib or or whatever, but um anyway.

C

All right, let's move on, uh can we uh does that somebody have a list of what the existing PR implements in terms of metrics I saw. There was a PR the other day to implement some first ones, so maybe we can see so.

D

I'd say: there's there's hardly anything yet I mean it's more the framework at this point. There are a few things in there that um you know the the pr is kind of working with. But at this point it's it's I'd say it's kind of wide open in terms of uh of what what you're, what we're? Looking for.

C

C

If you had some that were already accepted, so we don't have to discuss them. So it's like. Oh that's, already done.

D

I mean there is a small set, but I mean it's at this point, so small I'm not sure it's worth um coming about it except you know. The goal is to kind of go through the ones that were listed in the original um um issue or wherever that's documented I forget now, where that's documented, um you know and see which of those are, are possible.

E

And trying to stick within the mentality of the bully or number based, Prometheus right, you're still trying to focus on using it as Prometheus versus just defining a info point.

C

Yeah, do we have the uh original list somewhere that we're working from in this release.

C

I mean I could probably yeah.

F

There's there's Matthew there's what you submitted, which uh I think it was you that submitted it, um that we could pull up for sure um yeah here it is um thanks. Stephen I think. Another thing that we talked about was the uh the distinction between node status and change status and that we might want to focus on one of the two first probably node status. Right is that with how people are thinking about it.

C

E

Did it great status is definitely changed.

C

Easier than change status right.

G

C

So we can definitely start with that, and this list that I created has both. You know you.

A

C

Looked at all the charts that I had and then just dumped it all in.

E

A list here so and that's basically, what I kind of did or charts that used to work that no longer worked because I had to it's so manual it just broke, but yeah. He has some of the local node information. You know like uh CPU usage per thread for node OS things like that. But you know mine are even config or or get info Style, which I rattled off several of them, but yeah.

C

So are we going through the list, then it keeps switching the screen, so we can't see it I'm back five notes.

E

So, obviously, that that's what Matthew monitor some of what Matthew monitors on his nodes, in addition to.

C

The channel yeah so I guess the request is to find the things that are the the node related stuff right from this yeah.

C

So like uh the the first one is the second one is the third one is.

F

C

Step is five: I guess are real related stuff, because you can get that right, specific to a node.

A

C

Yep right, the the number of forks by producer drop by anything by producer. Obviously, that's a network kind of.

E

Metric right right and those are some of the Black Box metrics we're struggling to see when chasing.

C

Around you know, I.

A

I'm gonna call this node metrics.

C

If the, if the first version just talks about node us itself, oh.

E

It's beautiful yep.

E

And I think those are stateful somewhere right, I mean that's. That's.

D

E

In its memory, so you should be able to kind of query and dump them I'm doing much.

F

Is the second one there unapplied transaction key sizes? Is that number of transactions or bytes or what? What are we talking about? Let me say: transactions yeah, okay,.

E

Right technically, there's both, but it's mostly the number yeah yeah.

C

Yeah but but yeah, so this is exactly the discussion that we were talking about earlier. Is that currently the software limits it in terms of number of bytes, but it measures it in terms of number of transactions, so.

D

If I want to limit it,.

C

What do I need to set the number of bytes to I, don't know because I don't know how many bytes the current thing takes. So uh you know I think enhancement requests coming. uh Can we limit the number of transactions in the queue by number of transactions?

C

uh Or can we measure this in terms of bytes together? It would be helpful.

E

Or potentially that's two endpoints one is transaction count, the other is megabytes or bytes. I guess if you.

D

Yeah exactly yeah both of those I think it would be interesting also to potentially limit it by the number by the time of the speculative transaction execution. So you do it in terms of like actual um CPU time.

C

Oh, if it's greater than x, then throw it out yeah yeah, because if it's more than whatever 330 milliseconds, then I'll never get a block anyway.

D

Well, I wasn't talking about individual I'm, just talking total right. You know if you've got three hours worth of of processing queued up in your in your unapplied transaction queue, or you really don't be able to get through that anytime soon. No, so you.

C

Know so so I, so actually this metric has three things: it's the number, the total amount of bytes and the total amount of CPU yeah.

E

Yeah specular CPU of that queue.

A

Is there a note there that I should be noting a number number total number of bytes.

D

Yeah this whole number yeah total number total number of bytes and total number of speculative execution time.

A

And that's specific to the unapplied transaction key sizes, yeah.

C

So we just we just made that one three we've turned one thing into three got it.

F

Is there a similar pattern for blacklisted or do some of those not apply for the blacklisted well.

C

Blacklisted might have like block listed by what, like this, is by a account like listed by key block listed by uh contracts, because there's settings for each of those.

F

So you're probably going to want a total right like and then all those.

C

Individuals, well it's one of those Prometheus metrics with like the little sub metrics in inside it. So you can split it into a subtitle. You can generate the total or the seat by type.

E

A lot of times, I, just I, know that all three would equal the total and and.

C

C

Yeah, let's tell you, but it's the way you define it. So you just say: blacklisted transactions and then inside out a little curly brace type equals whatever and then the number and then you can roll it up easily.

D

So my Prometheus knowledge is pretty thin here. um So did I hear that if you've got something that accumulates into a total there's no reason to export the total, because Prometheus can easily provide that for you. Yes,.

C

Okay yeah, as long as you define it as like a key with a type associated with it, but it's not like three individual keys, there's one key with a type okay.

D

Then it'll give you the total for that.

C

um Because it knows that those are related, so, okay, you can all right.

F

And for this one is, is, are all with all these be defined by count of transactions. Yeah.

C

I think that's just count. I, don't know if there's any.

E

Other decals or some of of.

C

Biases, it's just the running counselor right together!

C

Well, yeah we're not storing it. So there's no reason to report bytes really right. The the unapplied one is true.

G

E

That's all sitting in memory.

C

Somewhere right.

E

C

Up space right.

E

It's going down my network pipe. Apparently it's crushing the bandwidth usage.

F

And is that just just for my knowledge is that a metric that would always increase or is there yeah it's just a counter? It's.

C

E

Yeah, okay. So our counter, since the last reset yeah.

C

Since you started node us, it just goes up right yeah. We.

E

Can do this and- and you can say, get for the last hour or whatever. So the point is just two data points and get smart enough to figure out all the rest, so yeah yeah.

C

None of these need internal counters to keep track of averages or anything right. For me, this will handle a lot so.

F

um Whereas the unapplied transactions will go up and down based on queuing right, yeah.

C

Okay, so it's a gauge rather than a counter yeah.

A

Okay, are we clear on these five.

C

uh Scheduled transactions, do we need to talk about our subjective billing? The last two.

E

I think it was the same right. Well I mean we haven't, read: there's lots of scheduled transactions out there. Oh, are they scheduled anymore? How does it even work with the new well deferred December transactions right so yeah when you deprecate deferred? Well, then you.

C

Don't need it, but the third is.

E

C

F

Yeah, so what are we actually measuring with subjective billing sizes.

C

Subjective billing is the number of accounts that have subjected billing applied to them.

C

Yes, my just my description of these issues was uh Uh Kevin Lovell description. You know, Kevin could probably interpret it, but anybody else probably has a hard time understanding what.

E

I'm. Writing. Yeah. That's just the count! You don't really care about the keys.

C

Yes, the number of accounts that have uh so that the building is currently what the there's a chart for that I.

D

I think if we tried to report um subjective billing for each individual account, it would just be gigantic, so I don't agree. We can actually.

C

Do that I, don't think I don't know, because.

E

That's right: 70 000,.

C

Accounts or something on wax or something yeah.

D

So that's not doable otherwise yeah, uh so it's just got to be to count I. Think for for that one.

E

Would there be a I, don't know? Is there a way to like create not like create snapshot, but don't dump it to a text file or something just so you could see it if you wanted to if it's sitting there at the volume.

C

Yeah, you need an API call, V1 producer, get subjective accounts or something correct.

E

Yeah at the separate thread please, anyway,.

C

We added that in like 3.2 there's like get unapplied transactions, API call right that Returns the list. So there's no reason you shouldn't be able to do the same for subjective. Give me the list.

E

Oh yeah, yeah, I, guess I gotta, look at 3.2 and see what's new, but that's.

D

E

There yet right, yeah.

D

So that would be a GitHub issue, a new one. If we, if we want to do that, add a new endpoint for getting subjective uh billing, get a get subjective, build account info or something.

C

So I guess I'll create that one yeah.

E

Same spitting I mean.

A

Pretty good at that.

E

And so then yeah I mean ultimately the next chunks are by producer, which are chain change. We didn't get the scheduled.

F

Transaction size, yeah, scheduled.

C

Transactions, um do we need the bite, size and CPU like the unapplied ones, for this.

D

The schedule ones uh you uh you get billed for the transaction size, um you know in terms of ram I mean we could report it for sure I mean it. Could um you know total there's, maybe interesting, but you know people are actually paying for that ramp.

C

Yeah, so it's less of a worry in terms of abuse, Vector right, so just the total count is probably okay, unless the other stuff's easy.

F

And that's interesting, something I didn't know. So that's a um just sort of like a transient RAM usage. Would you submit a uh transaction before it gets applied? Is that.

D

Now these are the Deferred transactions um different than normal transactions, they're actually deprecated, but you know, obviously you can't really renew them. Remove.

G

D

But uh you can certainly discourage their use.

F

G

C

Yeah yeah so moving on anything, that's by producer, we're going to skip that for now. So.

D

By producers, for all those, you really mean by node right, not by no.

E

My account technically it's by producer.

C

E

Yeah well I mean it could be any number of nodes, hopefully not two nodes at the same time. So, okay.

C

So let me explain how the current system works so the way it works today is, you know: node us prints a lot of logs and then um you know, you'll see like a message like hey. There was a fork and if, if you see which producer is producing the block at the current time, then you can kind of blame the fork on that person or if you see that a bunch of unapply, this block is unapplied, unapplied, unapplied. Well, probably they're double producing right, so we can blame.

C

So it's it's trying to associate some level of blame or you know indication that who who's the problem, um so you can get. Oh it's me or somebody else. It's. The current log reading is not 100 accurate because the logs don't exactly tell you and you have to do some interpretation. So these are not the easiest thing to collect, because this is not something nodius tracks by itself. So this is. This is really a chain metric yeah.

F

Would it be appropriate to split this request into basically two parts and uh skip this Okay cool.

C

So yeah, so we can move this down to a future one right. That's what we've just said.

E

I'm sorry, one interesting thought on that is in the same concept of just exposing a data point. If it it's trackable, what what Matthew's trying to do is more detect.

E

It then try to figure out who easily to point the finger at or gently nuds in a telegram channel right, but really the key is to be able to detect hey, there's forking, going on on this yeah chain, which a simple counter exposure of that from nodos start time as a local statistic, you'd at least be able to Monitor and know that your node is forking because it may not even be the BP's fault. To be honest with you, I've put some potatoes out there that fork on their own. So.

C

So absolutely so, if we remove the by producer part out of all of these counting the number of forks counting, the number of unapplied blocks counting the number of drop blocks. You know these are all log every time. There's a log message that says one of those things increments counter by one should be doable.

D

Right so my question on like something like this, then is: do you in Prometheus, try to say, okay, a fork happened. uh The BP at this point in time should have been this, so I could report that. So that's one way you could do it or you could just report number forks and then a separate Prometheus entry being um who, uh when producer schedule, changes right um and just say you know it's one of those bullying things like okay, it's now it's his producer. Now it's his producer.

D

If you had both of those metrics and Prometheus, can you overlay them to see what you're looking for? Do you really need, like the the producer that the fork happened on? Does that make sense.

C

I I think I'm gonna have to go back and get you to say it again.

D

With Prometheus is like you know, you can graph things. You can see all this information, so I'm wondering if, like if all we exported was um the current producer.

C

Number four.

D

Then you would, in a Time series, you'd see when those Forks increase, so you know when it increased in Prometheus. If you had a separate Prometheus entry, which was just producer schedule that were that were updating all the time. So it's like you, know, zero one one. You know, as as producers come in and out, if you, if you plot those together, it seems like you could see um who's producing during the the high Fork time.

C

uh Sounds difficult.

E

C

E

Could you yes um I mean you could overlay two different metrics on the same time, series yeah so to say right, but I mean it's more and you at least I guess: I use clios for it, but we already get schedule and I mean I, know the current lineup and even the order of VPS somewhere.

C

Well, actually, that's another thing is get the current schedule get the number.

E

Of schedules- and that was going to say, is that not coming from an API I.

C

Guess I am using yeah. So currently that's what happens right. The logging tool gets the current schedule and then looks at the log and Compares. What's the related to the current schedule and how uh you know the the output matches which it it doesn't actually handle like when the schedule changes in the middle of a block production round, then it goes ah BP.

E

Has a problem box, you know, yeah I mean I'll, be honest in my head, really I mean the way it kind of works. Is you almost need like a reader and doing some of put it in some some relational data and being able to accurately go back and almost read the logs for us, but in aspect of a node monitor a local API, a a counter for all of those yeah? So it does address the alerting side of it.

C

Yeah so I'm quite comfortable with the the first version of this, and maybe the forever version that we never do the by producer part but yeah just count. The number of forks I think that's reasonable, because.

A

Yeah, a new request- that's not on here! That's.

C

Remove the by producer and then any metric that doesn't make sense without the by producer, then we've removed it from the list. So.

E

Right I was about to say most of them, so the requests are.

C

Yeah a fork counter every time, there's a fork count: one unapply blocked every time. There's an unemployed block count one every time, there's a drop block, uh Miss blocks. You can't do that. One yeah! You know the schedule um double production. You can't do that. One um block arrival time, maybe I- think that's probably doable by producer, because no no DS has all this info.

C

Maybe it would be hard to do it by average and last. uh That might be difficult, but certainly one.

F

Thing that these are still effectively chain status, information right.

C

F

And so one question I.

E

Added they are in there. No no I'll I'll.

C

Say because Forks is not really a.

E

C

F

Right, yeah, okay, so is it meaningful that it would be? These would be counters from the last time that you started up notice.

E

Right, these are all just kind of right: yeah, yeah, totally stateful from the startup really just to indicate eight went to nine that at least at this time that at least tells us and can trigger and then yeah you got to go, read a log to see who was producing whatever, but exactly also you can trigger a lot of this I mean the the forks not related to the chain, purely related to the node and its peering, and somebody else down. Yeah.

C

Especially when the node gets off into its own coma State producing on its own problem. So that's where Michael's coming from on this.

C

The number of forks on a node is different than the number of forks on. Another node tells you that that node has a problem. Yeah.

F

So right I've got Forks unapplied blocks and dropped blocks. Then we, we said, missed blocks of.

C

Missing producers or double production, yeah.

F

C

F

Okay does block arrival time make sense here.

C

That's really important one from a troubleshooting perspective. um You know you're trying to figure out why there's a fork um block arrival time uh very often figures into that it shouldn't.

D

Be here because.

C

Nobody us logs the I, received this block from this producer with this CPU and this net, and this whatever like there's a very long long line for each one of those. So keeping that should be um very doable.

E

C

So that's getting.

E

That more related to your holistic view of the data I mean that wouldn't be a counter from startup. How would you, how would you.

C

It would be the current value right, the last or of.

E

The last block.

C

The last block- I, don't know, are you yeah? No.

E

That's what I say: what are you reporting.

C

You could just do it as a um as a total and divide by the number of blocks. So if you count the number of blocks received from a producer and the total time it took to receive the produces, then you could divide those two values and get the average.

C

It's not as good as what I have today.

E

Would that be a stateful collection of the last full round of blocks and then right.

E

Really, oh I get that that's! Why I mean.

C

It needs to toll it up. You can't do it by round, so it just needs to Total it up. Every time it receives a block count the total amount of time, the average time it took to arrive. It makes 200 minus 200.

C

and then count the number of blocks. Then you can take the the arrival time divide by the number of blocks and you get the average amount of time that it took to receive blocks.

E

Is it helpful to have a whole.

C

Because the last block is very different than the rest, and the last block is the most important well.

E

And then going into a bigger broad range, why would you want every block since it's been started? That's an even.

C

Well, you're averaging it out over an hour or whatever, but.

E

It's 21 different people, I, don't I mean I got yeah, but many people that have great latencies in one. That's a potato.

G

C

Do it you do it by producer.

A

So everybody has summarize here number.

C

Two, the producer there's two counters total number of blocks received by producer and total amount of time it took to receive uh latency by producer, so you'd have like 21 above 21 of that, and then you could divide by that 20.. But Michael I totally agree with you. You just add them all up and forget the producer. Then it's totally useless that doesn't help at all.

E

But yeah, if you can I mean Brian, is that because that's not a simple counter of a.

C

Number that's a counter with a subtype.

E

Yeah I mean you're keeping by producer block received. This was all of the time. I mean I, guess the last hour's worth of time and you get an average back of it. I I, just I, don't I'm just trying to think of how that data is stored in memory to then be able to.

C

Give me back 54 bite, counselor.

C

E

C

Some of the block times are negative, and then you get a positive block timing.

E

C

It's so let's skip this one. For now it's a bit complicated. That's.

E

Why I was trying to rip my head around yeah.

A

So are these two here that Brian Post in the chat number blocks received by producer counter total block arrival time by producer counter? Is that a node yeah renewable by the note and worth worth like? But that's what we're talking about, calling that low hanging fruit.

C

And I'm not sure if it's useful, even if it was lovely, improved. Okay.

A

So not worth well that's what we're trying not worth noting. Okay,.

C

It needs more complication and.

E

Do it right, that's what version.

C

Doesn't give us.

E

Enough to make it worthwhile, maybe we try to say on this V1 we try to stick with simple. You know: startup based counters at least notify something's out there, and then maybe we can start to get into some more of the stateful and analytics of the last blocks by the producer whatever, but.

C

um So let's skip this one I agree: I think it's getting bit. The use case is getting yeah. Okay, diluted, then yeah exactly so. Let's skip this one uh number of transactions. We can probably just count the number of transactions we've received right, it doesn't have to be very produced, searches I mean so.

E

This would be: is that received or processed I guess validated.

C

Steve, well, that's good! So there's a couple like number of unique transactions after that gets deduplicated uh or the number of total I I. Don't know what counters would already be in node EOS I would assume I.

E

Guess there's an upcoming.

C

Counter right, then it gets deduplicated. Then you can count the number of unique transactions after so then we can see. If we compare those two counters, we can check how many duplicates we're getting right because we have like 100 peers. So we get every transaction 100 times. Maybe.

E

C

I guess you're.

E

Thinking about from.

E

I was thinking in my mind, even more meant from like the the validation side, because I mean there's only ultimately one set of blocks or one set of transactions. That's going into the block. I thought you meant transactions being processed.

C

I did when I wrote that, but then we were saying we said we're taking out the bike producer thing so that I.

E

Need to correct correct, that's and I'm I'm trying to think of how that morphed, but is it helpful.

C

I think it'd be super interesting, because I have no clue right now how many duplicate transactions I get.

E

Like what was that, with this Encounter of how many total transactions you get yeah.

C

I mean if we can get it by connection, then even better.

C

Right, the number of incoming transactions by the peer on the peer-to-peer or via the API right how many transactions were submitted to the API.

E

Then they'd have to open the blocks up, I mean no.

C

We're not looking at forget the blocks. This is just looking at the unapplied transactions as they're arriving on the peer-to-peer or the API right. So all these transactions are arriving.

E

On the spec, it's still on.

C

E

Speculative outside Zone yeah.

F

Presumably, though, under normal conditions, the reason you receive duplicates is because multiple peers are sending them to you right right, it's helpful! If we, if you did it by peer.

F

How would you account for total versus unique? Does that you get what I'm saying.

C

Well, the unique counter could not be, but pure yeah, okay would be by peer, and the unique would be by well after you, you duplicated. It then, which peer came from is irrelevant.

F

Yeah yeah, okay, so number of transactions received by peer and number of unique transactions received.

C

Yeah yeah and the peer could be a peer-to-peer or it could be an API.

C

So maybe those.

E

Are two different things: I'm curious, if that had like I know, Ross runs 200 peers or something on his public API. Knowing that that could be a pretty heavy stateful set I mean. Is that any issue?

E

It's fine when it's three Piers? What's it like when you're tracking you know, I mean these.

C

E

Apis, what's that going to do to that I I, don't know I I! It may just be 200 rows of no problem, but I.

C

Don't know 200 rows doesn't seem like that. I mean we don't have an infinite number of peer-to-peer connections right. True.

E

But mine are much lower than Ross's I.

C

Can tell you, yeah mine are in the 50.

F

C

Range and then when it gets too big, then I create another one. But you know.

D

C

Seem like that big.

E

uh uh Because but it's also going to be State I mean it's going to be since node launched so to say, I mean so. Every single IP that just gets peppered in well.

C

But I don't know that seems like.

E

A back door that could cause some well.

C

The the other thing that I will add to address your thing, because I agree is that at some point, if you didn't receive any data from that peer, then the metric needs to go away right. You need to expire, the metrics, otherwise you're your your document just gets huge.

D

Yeah I think you clear that out when you close the connection right exactly, it's only uh open connections that you're getting that information from.

E

You clear it out after when the connection every.

D

Connection close, you have to close, which means you might miss some information right near the close of the connection between your last metric call and when you close, you don't miss that amount of information, but I mean that's really, probably the the time to do it as opposed to weight some amount of time and clear it out later, um but yeah.

E

D

Sense like say I've,.

E

Got some weird peer-to-peer traffic that it's not a connection? It's like a peppering but yeah.

C

We have that all the time the low balancer sees people connect, disconnect correct.

E

But all the other things yeah, okay, possibly if it could possibly be a wrong chain that okay, but anyway, that's why I was like I know. Not all peer-to-peer is a steady connection, although normally it is um but yeah that that so.

C

There's another one number of connections on the wrong chain number of new connections number of currently connected right, so.

E

I mean I, guess I mean that's that being the.net or the net plug-in, but.

C

Yeah it's different than the net plug-in. It's the peer-to-peer plugin. So like how many connections do we have? How many are working, how many are defined that are working? How many to find that are not working right, because we have a peer-to-peer entry in our config file account look up the DNS name anymore,.

E

um You know, does it do error counts I mean that yeah I remember it seeing at least retry attempts so get.

C

Gets connections or whatever the API call is, uh has a whole bunch of stats. There.

E

Okay, that's a amazing how to look for it.

E

A

I missed a couple here: did I miss anything that I should write down, that I, missed I think there was a couple that Brian number of transactions received per peer I think there was a second one. There.

F

uh Number of unique transactions received.

C

And the per pure can be those two parts to it. There's the peer-to-peer connections on the API.

A

Number sorry, which one here.

C

uh Fourth from the bottom, it says, prepare huh slash, API.

F

So those would actually be like two separate types for that is.

E

F

C

Or you could have a special peer called API.

A

And these are all node metrics, so we've simplified the what would have been a chain metric and turned them into what can be done. Yeah.

C

We're trying to do we're trying to focus on node metrics right, so we don't have to do any weird.

E

Yeah basically turn theming into counters. Is that some way of knowing may not have all the detail related to it, but some if.

C

You're looking at all of them when it's good enough to get an alert, but then you can go look.

E

C

Logs that go through yeah.

D

Well, we are getting close to the uh top of the hour here. We.

A

Are hard to stop uh any final thoughts in the last two minutes? Did we get through the list? We.

E

Didn't get done quite see you at the bottom, but.

C

Yeah just quickly the ones at the bottom up time, CPUs is by said this face, use those are all uh node, yeah, node related, so I think those should be usable, Rockstar.

A

All the way down this space, not necessary cool.

C

Yeah roxyb and block vault have been since removed, so those are no longer relevant so but those are okay.

A

E

I put that one on there I like that, one that last one on that list replay status. Oh my God, some way or another.

A

In scope, no metricable, it's.

C

No metricable yeah.

E

I mean current replay block, maybe I don't know I, don't even I I assume, that's, maybe a mistake somewhere or queryable, maybe it'd be last latest Block in the block log would be what it's replayed up to right. That makes sense. Yeah.

D

Yeah current block number I think is maybe what we're looking.

E

For here right, because your good info shuts down during a replay I mean ultimately, if get info worked, I think with that block number yeah. So.

D

It's Prometheus it's going to be shut down during replay as well.

E

uh Fair enough cannot be on the get him. Can you turn the get info back on.

D

I mean we can, if you want to slow down your replay.

E

I mean trade-offs right well, get info is heavy too, as I understand anyway. I know we're out of time, but I. It sounds like maybe that's outside of this scope, because it would shut down the whole plug-in right. I.

D

Mean you can certainly get get you know get in for works, and you could report number and I. Think current block number is useful outside of anything right. That way, if you've got you know by looking at Prometheus, if you've got a node, that's way behind.

E

Right, correct, correct: we check head and live and stuff like that, but and that's why I say as long as it's there we use get info and and whatever or current block but yeah during a replay that is brutal because they replay for weeks yeah. The problem.

C

Is there's no metrics whatsoever in a replay right so.

E

C

Right, the only way to understand what's going on is read.

E

Logs so oh the replay finished up uh it just magically came online anyway, so I would say that was that the nice thing on there does it need to be in this scope, because it's not going to be in the Prometheus Plug-In or even get info right now. So maybe it's not in this scope- and we finish off, but don't forget that one put it in.

C

Yeah current block number.

A

E

C

Would say so this one we're calling out of school I would think so, but the current block number is in scope, but that was at the beginning.

E

Yeah, but not during a replay, yeah I'm not doing.

C

A replay you know, but yeah the get info was listed at the top, get.

E

Current correct, yeah I was about to say: that's, that's already there, assuming it's on so yeah, but.

F

I saw that and I was like no, no, that was the best Stage.

C

Cool anyway, I got a wrong turn on thanks.

A

Everyone, so are we have we exhausted this topic? Are we back back next week talking the same thing or uh resource model, I.

E

Think let them kind of run with it and try to work some stuff up in scope or whatever, but um maybe yeah. We talk about resource model, or at least that's a that's a holding point, but but yeah I think this is a good set to keep Brian and team busy right.

A

Cool yeah and then we're heading into the holidays, uh I'm thinking we like the last two weeks of December. Maybe we take a pause for Christmas time. So next week might be the the last call for the year and we pick it back again in the new year.

A

uh So we'll see what we can do to coordinate uh talk on resource model for next week and uh and then enjoy the holidays thanks Daniel thanks everybody, okay,.

C