Mina Protocol Node Working Group Community, 15 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Mina Community Node Working Group | Meeting 02.15.2022

Description

The weekly Node Working Group session serves as a platform for the community, ecosystem partners, and the Mina Foundation to interact.

This meeting is a time to discuss the latest core protocol releases and for members to propose suggestions, raise issues, and provide and get feedback.

If any community members have questions or feedback on the latest 1.3.0 beta 4 release, please visit the #mainnet-block-producers channel on the Mina Discord: https://bit.ly/MinaDiscord

Issues discussed can be found on GitHub: https://bit.ly/NWG-issues

Chapters
0:00 - 0:32 Preamble
0:33 - 1:09 Intro
1:10 - 6:44 O(1) Labs - Release updates / feedback
6:45 - 21:59 O(1) Labs - GitHub issues updates and overview

Mina Protocol – the world's lightest blockchain

A

All right, everyone welcome to the node working group session today, um we'll start off with a preamble and then we'll move on to our agenda.

A

The node working group preamble, the node working group sessions, are a time when members come together to discuss and collaborate with ecosystem partners and the amina foundation. Topics generally discussed include the latest releases, node operations and other technical matters related to the protocol. Although this is a discretion of the node working group, so hello, everyone welcome. My name is john robinson: I'm the community operations manager at the meena foundation and I'm here to support and facilitate these sessions. So please, let me know if there's anything I can do to help.

A

um If anyone has any questions, please use the button down the bottom and raise your hand and we can bring them up. uh If you add your questions to the agenda, it would be a very big help to regina who's, kindly offered to take notes for us today.

A

um So what we'll do I'll share my screen here? We'll get up the agenda and first off I'd like to introduce our ecosystem partner owen labs, um andrew.

B

Yes, hello. Everyone um I'd like to just give some updates about the 1.3.0 release, uh which is uh we're working on stabilizing now and, and we've been in the beta phase for uh a week and change a couple weeks. um There is a new 1.3.0 beta 4 uh that was just tagged and we'll have release notes coming very soon and an announcement to go with it.

B

um That includes a couple more optimizations that we discovered uh recently performance optimizations that we're very excited about, and we want to get included as well as a couple updates to the docker entry point script, just to bring it up in line with um what we're doing in debian uh debian packages and other things like that. um Yes- and uh I would love to hear any results that people have had with with beta 2 over the last.

B

uh However long it's been, uh there hasn't been a super wide release, but I think everything we've heard so far is great news. So we're we're looking to yeah, like I said, get, get an even better version of that out into your hands and test it and hopefully disabled. As soon as we can.

A

Thanks a lot for that andrew does anyone have any questions or feedback for andrew on the release? um I know I know it was really running smoothly for me and had pretty good uptime and it was running it on my uh server and also on uh noted home and that- and they were both really smooth, does anyone else have any feedback.

C

D

Only came across.

C

The transaction disappearing, but it's not concrete and I'm not sure what happened last book, so it just happened while the transactions will process like 1200 something transaction and it was never issued before. So we were very confident on the existing system and somehow paid to separate from the mempool.

B

um Okay, that's worth noting, I'm not we're, not sure I think in in our early investigations that uh we're not sure that this is new. It's just uh you know, potentially that you're observing it now or um that your situation is running into whatever the bug is or has been um because we have some older. You know logs and reports of other things on other versions doing the same, but at the same time I will say I I think, there's.

B

Yeah um uh yeah never mind like I said I, I think we're not really sure where that comes from, but we're investigating it further and looking for it, because I know that affects a lot of the larger staking polls and anybody who has you know, masses of transactions to send every every epoch.

C

Yeah, I think, since first the broadcasting feature uh came to the release. We never faced any issue for some reason. It was working very well, so that was first time happening. Sauce.

B

That's very useful feedback. Thank you.

A

So ivan has commented and chat here. He says a running will here, but have not seen much of a difference in comparison with alpha one in terms of stability, and he also added. uh What I have noticed is that the latest release is able to get more canonical block uh sorry canonical blocks into the chain, but this was already mentioned a couple of versions ago. So stability is the same for him, and the performance in terms of the block producer is better.

B

Yeah, that's good news and also, I think, part of the problem that we were running into with alpha 1 and that sort of era of releases 1.2.2 had some of the same problems is that we had bugs that were where the node was appearing stable, but was actually you know further behind the pack than then your block producer is able to be, and in order to be profitable in order to you know, successfully produce blocks or do anything else um so uh so yeah.

B

I think we've addressed quite a lot of those bugs now, and these versions are much much better it. Just you see a new block, basically two seconds. You can add it into your transition frontier and start using it, and then your block producer is in there for a much much better position to to just get their block out into the world and have it be correct.

A

Thanks andrew does anyone else have any feedback or questions about the new release.

A

All right, uh if that's the case, then we'll move along on the next item on our agenda and next up, we have uh prithvi uh our ecosystem partner, owen labs, uh technical product manager, ruthie.

E

Hey thanks for the intro john um regina, would you mind sharing the the uh the github project um yeah and then I can kind of talk about it as you take notes on the other tab or something.

A

Actually, I think, that's john john is the one sharing screen. Oh.

E

John is the one sharing okay, okay yeah. Would you like can.

B

You present the.

E

The project itself yeah instead of this document.

D

Okay, um thank you.

E

Right so uh this this is just a triage board.

E

uh Basically, my aim is to collect various issues that either the node working group or other community members have raised, or even things that the internal team has has flagged or are, are working on and use this as a medium to kind of communicate, the status of different issues um and where we are in resolving them, and so what I'll do right now is kind of just since this is going to be our first time discussing this uh through this get a project medium, I'm just going to go over the issues one by one, and let me know if you guys have any questions, and one thing that I would definitely suggest is that, even though some of them might be old um rather than discussing them at length here today, I would say to keep a single source of truth.

E

We also, you know, make sure that we discuss them in the issue itself, so that we have, you know a proper history for these things. um So I'll just go one by one. I think the first one that we see here has been raised a few days ago. I think, like five or six days ago in beta 1, that a node is being restarted. Quite often, we haven't started investigating this, yet so there's not really anything to communicate on this one.

E

Similarly, for 10164, where it says, node crashed at the same time that you know produced like a certain block.

E

This is an old bug that we believe is fixed, but we actually don't know what version that person is running, um and so we need. We need that information. We need what version they're running to really be sure about this sort of thing, and so I've added a comment in that ticket. Asking the author to you know give provide that information um and that's it for those two and so the next one is number 9417 uh generated blocks. Are not gossip well enough to appear on the network. This is a quite old issue.

E

It's raised by emory, and I think you know you guys might be familiar with it, but just to summarize it quickly the issues that blocks are not produced and gossip around the network in time they kind of have the same root cause. There's this finite window in which blocks have to be produced and they have to be submitted to the network, but the demon is kind of busy and other nodes in the network are also busy validating things and it just doesn't end up being propagated quickly enough.

E

The good news is, it is related to the issue that you see on the second column, which is 10108 that is related to prioritized block production, um and so we once that lands. We should see some improvements with with regards to this uh andrew mentioned at the very beginning of this call that there was also a lot of valid validation, optimization work that we were doing that has you know we'll likely go into the next beta release, so we can see some hope to see some improvements uh from that as well.

E

um One thing is: there's an item. That's on our roadmap for the networking team that we hope to address this in the long term. So main thing I want to stress for this issue is: it requires a series of fixes, there's no easy cure for this sort of thing, and there are items in the road map that you know we can communicate to you guys, but really it's dealing with, like prioritization of of you, know uh different different things.

E

um So yeah, that's that's! That's it for nine! Four one! Seven! Do you have any questions about that? One.

E

No okay, I'll move on um 70 20 uh out of memory error. So when I talk to the team, we believe this has been fixed in 1.2, um but we need to follow up with matthew who had done some initial investigation on this. So once I talked to him get an update from him with regards to this thing I can hopefully come back to it and then close it.

E

If we're not seeing this on the latest releases, and if we do see it, we can more than happy to reopen it again, but we believe you know we found the root cause and- and I fixed it- it's just something: that's old and open for a while.

E

Okay, so 8351 um single block producer creates multiple blocks in one slot. So when I talk to the team, the understanding was, we think this is already fixed, but it's also quite rare and we don't really have a great way to reproduce it. So the reason we think it's fixed and we're you know very certain that it's fixed is because we did the way.

E

Did the refactor the way that the block production loop um works, and we have like an out of process way of of doing that and where previously, it used to be an in-process thing. So, with the new logic um we believe you know, we won't hit the bug in the old logic where you know there was a job that was being canceled and you had to reschedule it.

E

So to summarize it it's quite hard to reproduce in rare the primary reason it's hard to repro is this: it's just a very rare race condition, that's caused by specific network activity um and it's just a low percent of chance, low percentage. Chance of you know hitting this sort of issue.

E

So if you are seeing this issue consistently still, we do need to prioritize it more highly, but we're kind of confident that, with the refactoring work that was done in the block reduction loop, that we might not see this uh very much if at all- and so like. I said, if you do see something with regard to this- make make sure to flag it in the comment or in this forum here and then we can put some more eyes on it.

C

uh For reproduction, it's just restarting a note on the lower right and shifted.

C

About the state of it, I'm not sure, because I think a lot of blog products, producers just split between various versions because of the version is not clear at all in general. Wait can.

E

You repeat that last part, I don't think I.

C

I'm understand, I think, out of block producer, just split between health of not versions because of the issues not being clear and some of them going backwards, etc.

C

So I think when there's a significant update with everyone going forward with their version, I think this can clear more clearly, maybe nice, because at the moment we are running like 1.1, 1.8, 2.2 and 2.3, or something like that. It's all around the place.

C

Like I remember seeing it recently, maybe two weeks ago, but it might be all of that version, I can't be sure right now.

E

Andrew, do you have any thoughts on this? Is this something that you yeah.

B

Yeah I mean, I think this is a big part of why we're trying to keep uh hold 1.3.0 to such a high standard is because we want to be able to say at the point when it's stable, that really everyone should be running it everywhere, and I think that we're really we're getting there and and it needs to be zero regressions. You know it needs to not have any.

B

It needs to not benefit anyone to go backwards into a previous release, and then we will get to the point where no one will do that until we get until we can make sure that every release is better than the previous one. Then we you know, uh um then you all have the your own prerogative to do that.

B

You know to to downgrade, but uh I think we're making releases now at the at the point already of these 1.3 betas and uh very soon stable, that that's the case that it just is better than 1.2, and it just is better than 1.1.8, and you are, you know economically incentivized to be upgrading to it, because you will have a better time and a better experience and then, when hopefully, more of the network will follow suit.

B

With that, um and also important thing to point out here is that when we do hard fork that will force everyone to upgrade and we will have even more changes coming along along with that that uh you know we'll we'll create a new sort of checkpoint in the world, where everyone must be running a version that that we're confident in um yeah. Ideally, we can get the vast majority of the network upgraded to 1.3 stable, like very soon after release, because I think it will. It will be deserving of that.

E

Cool thanks for that message, andrew, so I guess moving on to maybe if you can scroll down a little bit or no, no sorry, this is in the high priority column. This is uh twenty 1024. uh This was raised pretty recently. This is actually a concerning issue and we're investigating.

E

um We found a similar issue uh if you scroll all the way down um john on the needs, triage column, yeah node restarts after 30 minutes. This one was raised by george and we're seeing this um in 1.3 0 alpha 2, where the node was crashing steadily every 30 minutes. uh These two are concerning and require the active investigation. So, unfortunately, at this moment I don't really have anything to to update you with regards to a root cause analysis, but just know that it's on our radar and we are going to definitely look at it.

E

um If you can scroll up for me just a little bit john 9983, uh this one also needs investigation. uh It was, uh I guess, you know, 1.3 alpha. One bug, don't really have a rca for this one, just yet um 9874, uh which is the amino d min keeps crashing in bootstrap right. So it looks like uh this one was raised for a pretty old version 1.1.8 and it may not be as relevant as you know, there's been a significant number of changes and there was a lot of improvements in the 1.2 release.

E

So we'll see if if we see this in the in the new releases, if you guys see it as well, uh please flag it for us, but right now um we don't really have you know much confidence that this is going to be reproduced in the later releases. If you feel different. Definitely let us know, but with that in mind, since it's for a much older release, we'll probably going to move this to lower priority um and then lastly, oh there's one 9869, that's in the middle.

E

This is more about the rosetta, so I'm just going to skip this I'll. Probably remove this from the board. It's not directly related to to us it's kind of more directly related to the rosetta work. That's being done so going on to 8331. This is transactions removed from the men pool and I think it's kind of similar to what emory you were alluding to earlier. So the main thing to communicate about this issue is there are two things being talked about that are unrelated.

E

um The first is there's gaps of transaction on mainnet and there's a number of blocks that are back-to-back. That may contain you know not many transactions or zero transactions and the other one is that sometimes we evict things from the transaction pool mistakenly, I believe the second one is what emery alluded to. So we have investigated the second issue where things are removed from the transaction pool by mistake, um and there has been investigation work done on that, but it actually due to other competing priorities. It fell down our priority list.

E

So what I'm going to do is follow up with the status of that and because you mentioned this earlier in the call as well you're going to try to prioritize it more.

E

The first issue that I mentioned, which is kind of conflated in this and that github issue, is the large gap of transactions. So we believe that is related to a snark work, gossip, which is nine zero. Nine seven, the issue that's directly under that one. So when we're bootstrapping, we can't process snark work. um So even though it gets gossiped to us and added to our local cache there's a chance. We just ignore the message and you know nobody will gossip it to us again until it expires, and so in that bootstrapping sequence.

E

If a node doesn't have enough snark work, they can't produce a block with any transaction. So uh definitely 9097 is something we will actively prioritize to kind of address that issue, but as of today, you know we're not like actively working on it. So that's just the the status of how things are, um and you know I'll be coming back next week to kind of update you guys further.

E

But um if you have any questions right now, let me know- and like I mentioned, the best thing to do to keep a single source of truth would be to continue to discuss things in the github issues as well. So we can have like a you know, a really nice rich history in there and we'll definitely know when things have been addressed, but that's all I had to discuss. If you guys have any questions you know either I can take it or, if I don't know I'll defer to andrew.

A

Great thanks a lot for that birth. We really appreciate it. um Does anyone have any questions for prithvi on this or andrew.

A

All right great well, thanks for that, um for anyone watching the recording of this, um the recording will post on youtube, um we'll put a link to the uh github here and you guys can follow up there too. If you like wait, we still have about two three minutes left. uh Is there anything anyone like to discuss or talk about uh we're? We've got all the stuff on the agenda so far I'll change my screen over and we can have a look.

A

There were a couple uh comments in the chat there. Emory did you did you ask these already? Is that right.

C

Yeah but I asked you already yeah.

A

All right guys! Well, if there's, if there's nothing else, you'd like to talk about what we can do, uh we'll finish off the call now. um Thank you so much for your time. Ander. Thank you. Prithvi really appreciate that. um We'll hang out for another couple minutes here and if you'd like to stay around for a chat, that's fine!

A

So thanks a lot for coming everybody, and you have a great week out there.

D

Thank you. Everybody.