Bisq Developer, 15 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Bisq Developer Session 15.08.2019 - Seed Node Stability and Reliability

Description

See written agenda and minutes here:
https://github.com/bisq-network/events/issues/27

A

Here we go well welcome to the first try on on the DEF CON next generation, with a bit of a different schedule and appointment finding process I see we have we have one or two do we have to signal operators here now, I, don't think so. Well, I'm.

B

Yes, I'm running one of the notes now I just took it over from Devon a few hours ago. Okay,.

A

And there is someone else, ok, just joined in where you.

C

Know they're from the current you know the operators, nobody here with.

A

Ok, that doesn't matter, let's try it and we will see if we, if we keep this for my deaf course. First, the first point of the agenda is is well.

A

We had a feeling that people are a bit lost if they join the project. Now, because there is not much response sounds like there is not much response on github issues and so on, and so forth and I wanted to talk about why that is and how it may change in the in the near future.

A

First of all for the for the merging process and so on, we have three developers total it allowed to merge things into the main branch one is Christophe, Christophe is on vacation, the other one is sq and as far as we know, ask you is moving house or something like this just moment.

C

And maybe you at this name because we have not to christopher's Kristoff's group so that not confusing okay, it's mr. Bonta to join and maybe I invite him quickly. I'm, like maybe he lost at the time, I think he was expected to come as well.

A

C

A

He said he said he can't make it. No. That was Devin okay. Well, so we are three three people that are able to merge something, and the only one active is as if now is me. So there is quite a huge workload on me and progress is slow, monthly retired from his activities, but he recently jumped in to review for requests to to get over this.

A

Let's say: slow progress.

A

And last but not least, it's summertime. So it's vacation time people are outside and enjoying the Sun and not much is going on in the in the bisque world on the on the on the the the guys that worked on bisque for a longer time now so my project, my projection, I, think within two or three weeks we are fully up to speed. Again, people are coming back from vacation people getting involved with bisque again and the the slower movement we have.

A

We see now it's going away so I believe we we can make it, and that is basically the reason why why maybe some people thought there is not much going on and Biscay I don't know Manfred. Do you have any thoughts on on that? Can you maybe.

C

Make it a little bit it back a long long as a back run through history, also because yeah for new developers just joining it's kinda like a little bit a weird few, it's basically the West at the time what we have in bisque from the development resources site over the last two years, I think I mean just in back in April, we have been basically five, roughly five full-time developers.

C

Now we have a sink fluorine. It's the only kind of full-time developer, Christopher, today's a holiday on on single person on it's called in English Parenthood maternity.

A

C

Majority leaf and vsq has moved house and so on and needed more time for her for this private stuff. Oscar left the project when I was stepping back in in April or May, but maybe come back his considering to come back again and so yeah. It was a coincidence of a few bad developments that I was leaving that and Christopher was going to Judas fatality and x-cubed moving house and so on. So all the main developers had less time like, like, usually also Devon.

C

Usually he worked it all tomorrow, and that moment is a lot of stress in his full-time job and so on and for new developers has a little bit of that bad impression project. So that's also why I decided to jump in a little bit more and try to help out onboarding knowledge transfer and reviewing oh yeah.

B

As a newcomer right, I understand it's very important part of European culture to have a nice two-month long summer. Vacation is that correct? So you know I totally understand it's.

C

Not so long, usually as it's just erm, yeah, I, think four or five weeks, it's usually the normal stuff in Europe and basically.

A

Yeah, you don't have more more vacation than five weeks so.

C

A

C

I mean this term: anybody can do whatever they want there or no vacation. He can I mean it's good when he communicated well. Basically, it don't need to ask any anybody. The test, it's positive size, of course, but makes those sometimes a little more complicated, but they mean payment in surveys that developers has two months on vacation a year.

C

B

You're, coming from like an American or Asian perspective-taking like five weeks off of vacation, is kind of unheard of, but I understand it's totally normal in European. Think you in the summertime, so I can understand that, but I could see how some other newcomers who don't live in Europe might think. That's unusual, some kind of cultural difference. You know.

C

Yeah I mean this five weeks is usually all just clicked over the year. It's not so. You think we take five five weeks in one in one path or vacation, but in Pisgah said no.

A

I will not do actually work, sometimes.

A

Not just vacation yeah, but it's takes up if the first one is first made. F is on vacation for two weeks and the second one is the two weeks after and then there's a third one with some some overlap and then there is no one else left in the project, and that is the situation. I believe we find us in just now.

A

My guess is that in a few weeks it's inspector- let's say three or four full-time developers and things are speeding up again. That sounds great yeah and.

C

Luckily, it's yeah two already very active new developer at christopherkrum and which- and there are a few add, a new scene in slack like this battle of wizard and another UI developer and I- think a few more I thought I didn't have time yet to get in touch with samosas, but it seems that we were speaking up now on the left side and hopefully in one or two months we are much stronger. There I mean the colas basics.

C

Is you had the midterm gold basic eight that we're getting back to something like minimum five full-time developers in average, I mean full damage more than 30 hours, I believe.

A

That the current bottleneck is the guys that can can actually merge stuff that has been done before. There is a lot of stuff that has been done in the past weeks and nobody was there to emerge it so.

C

To be very careful to not much too quickly or reviewing to tweeze eaves, of course, because yeah, it's it's a big risk and with everything, especially when we add new dependences from security point of view, and so so we shouldn't make any compromises just because of discretion. Yes,.

A

Exactly and I believe we had the the the tradition of having at least two reviews, so one one reviews and gives his or something like this and the other one reviews they put a request and then merges it. And but if there is only one available for merging, there's only one review and so then pour etre stays open until the second one is going to review of merge it down. So yeah.

C

Really strong rule becoming the batteries, of course, when the to refuse I think it depends on the on the pure gas, when there are any curves as before, as carries more risk, I think it's really important that and also that it's refueled with with the aqua switch attached it that it's tested, also not only on small trivial stuff, can be merged with on Thursday talk about anything which is a little bit more complex need to be tested as well. If.

B

You watch the Bitcoin communities consensus on when they merge things into Bitcoin core. It's extremely controversial right I mean they treat their system as I. Don't know what the market cap of Bitcoin is now say: it's you know ten billion dollars or something but they'll often say like. Are you sure that this code is good enough to merge into a ten billion dollar system? And you know bisque might not be at the billions dollar yet, but it is, you know, measured in the thousands of Bitcoin per month, I understand of trading volume.

B

So, if we're going to you know, follow that same logic and say yeah every day, 50 Bitcoin are being transacted. Is this code good enough to you know and and however much money from a security standpoint? How many you know jar dependencies? Is everyone running on their computer? And you know these are essentially Bitcoin wallet apps as well. We can't forget and the Dow stay for the whole compensation everything. So you know there's a lot of market cap.

B

That's maybe not it's not as simple as Bitcoin, there's a lot more moving parts and it's a more organizational thing. So there's a lot more to lose. That's not so obvious, so we need to be just as cautious as the Bitcoin community when it comes to merging anything or adding any new dependencies. That's very dangerous! Yeah.

C

Yeah and especially we sit down I think we have to be a little bit more conscious and I've a little bit the feeling that this consciousness is not very well deployed in there in the developer and also cenote operator group that when, when we have continued when we continue to have such troubles with the Dow and at some point, it really breaks and it can break.

C

Of course, when we have a deep problem and wish you incorrect, vsq and people are selling this vsq and later we have to fix it, and this because q become invalid, it's it would be a terrible incident and it it's up. It's ugly, a physical right. The Vista will survive this. So all the well you all what everybody has around in bsq can go to zero, and we this up so and also not only they're going to Sarah.

C

It's like we've seen over the last month's trading volume on psq was going back because, of course, all this troubles. Many people couldn't really use it because yeah didn't it was not in sync and they didn't get the best cue blocks and all the troubles. What we had not so it has direct consequences on the market cap and on the trading possibilities of peace q. What everybody learns its basic, our money from the contributors and yeah.

C

We have to really be careful that we are that this is working smoothly smoothly and that we are not, and that's.

B

Something I'd actually really like to contribute to, or you know maybe work with Florian on the monitoring system is to get a very active monitoring system of just the overall state of the Dow and if there's any Forks or any you know block mismatches, we should get some red sirens alarm bells going off on slack and every, and we should look at that right away because, obviously not everybody upgrades their bisque client apps right away.

B

But if it's something like the seed nodes disagreeing on the state or some of the other issues, I saw over the past few days where I was setting up my new seed node, then I think that's actually a really high risk issue. I guess we can talk about that in the next agenda items, but if you had anything else under dev yeah.

C

If you want, I can jump in here a little bit to give the background what what what's causing this no.

A

I I don't believe that is necessary right now, I like to I like to focus I, believe that's a good point to check off the first point of the agenda and move on to the seat notes. If you are all okay with that yeah.

D

Can we quickly maybe cover some quick thoughts on attracting new developers as far as because I think about it in two parts and one letting the world know that we want new developers into onboarding? If there's anything, we can do there to make it more clear, more smoother process for new developers.

A

Yeah edit, your your topic down there I believe we should focus on the topic set for this def call and schedule another one for for other topics, except we have time today. Then we can of course talk about it, but I wanted to keep it focused so that we can saddle it it's about seed nodes and we talked about eight nodes and that's it and focus focus on the topic at hand. If that is okay for you. Oh that's,.

D

Fine I kind of thought of it as an extension of the first topic, but that's fine too yeah.

A

The first I originally did not plan to have this first topic in an agenda, but as stuff I watched stuff, I thought it would be good to at least loose a few words on this on this topic, so that people kind of settle down and say: okay, it's it's everything is or track, but it's just a little slow. So there it was the motivation of the first chain that point, but it is in the topic here and we will talk about it eventually.

B

A

Well then, seed nodes, I, don't know I I. Have this point sync here? Is everybody familiar? What is it? Node is what it does and what the risks are because other than not what the river, okay I believe, I prepared, very, very extensive graphic and the annotation I talk a bit about what is. It know the ascent, what it's there for and the issues we have with the seed nuts and maybe the issues they cause on the whole network.

A

So a signal is basically nothing else than a normal, day-to-day bisque client, with the difference it runs without without the GUI and but the the functionality is basically the same.

A

You start you start a seat out and deceit, node searches for other seat nodes, of course, like every client does- and let's say we have this deceit node one and it starts up and tries to reach signal to let's say seed, no 2 is already up at running, and so this is a gate. Preliminary data request something like this. So it gets the the status of the network before it's hidden, so is hidden services even published if the hidden service is published.

A

So the signal is reach every PI RS there is this updated it updated data request something like this and things are we try to get this unit up to speed on the current network status?

A

Every signal does this and that's basically it now. If a client starts up, it asks the seed nodes, because a few signals, at least these eight signals. We have our hard-coded and if a new client starts up it contacts, the seed nodes and de-seed nuts are basically dead for bootstrapping, let's say bootstrapping the network. They are not needed. If you already started your your client once successfully, because then your client knows other appears, and maybe it doesn't need the signals again.

A

However, the signals are among the first client connects to and start up because we believe that it's much quicker and reliable to get the peer-to-peer network status to the new client if you use the seed nodes. So with that, if you see it node misbehaves, let's say it's gone out of sync: it has not.

A

It has a lack of. It misses a number of, let's say mailbox messages and the client connects to the seed note that misses the mailbox messages. Then another the disc client will not receive the mailbox message. That says, for example, that someone has paid preceded with with the trade you had to someone. So if this message was missing or you don't receive the message, then there is yeah.

A

There is a betrayin to you, so the risk is if, if seed notes do not have the whole status of the network, there is inconvenience for the user and the arbitrator pan out. You want to say something: I think you just raised your hand Soviet, so those hard-coded I.

D

Understand the hard-coded seed notes right, but so the other ones they just if I would start a new seed note. It will connect to those existing ones and advertised.

A

As a Sydney, exactly so, the.

C

Node Christina does when there is only the hard-coded seed, not which are treated as seed notes. When you start acetyl, it's like the normal bass note, it's not used as a signal for other notes. There.

A

Because nobody knows about it: yeah.

C

So nobody will connect, I mean the seed notes are really only hardly required for the first time user because he doesn't know any other notes. Theoretically or practically. You can add on the program argument. Any other notes when you know one on your address and then you even don't need the season for that and when there are no sweet notes, you connect to any other here what you know already, but it's slower because they might not be on line and there may may have such a good connectivity and so on.

C

The main purpose, how we use the signals is for convenience and for performance at the end. So when you start in the end, we have only this eighth seed notes. We find it that code, which are used by every user every user. We start up connect to multiple of those eight seed notes, as in q3.

C

Usually so you have more redundancy when one is slow is response, you get it quicker from another one and yeah and another thing: I mean they're the worst case in in debt for that row, because the signal hasn't and by now we set out as he has an another important role. What Lauren has not talked yet about, but also for the classical case, which was already the case before the Dow.

C

The worst case is when a dispute mailbox message was when you have dispute with in in arbitration, and you receive some message from the arbitrator and because the seed note was or badly connected and either didn't get the message and you next time connecting to this seed node and you are not receiving the mailbox message, then you might in the worst case you might lose arbitration case because the arbitrator think you are not responding to him or whatever and you could lose basically the money, and maybe you are in the right position, but because the arbitrator saying you are not responding, he make the payment to the other guy and we had a few, not I, think there was never money really lost, but we had a few cases where the lead led to complications and to chaos in the dispute.

C

So yeah, that's basically they're the worst. What can happen? The others you having a trade when you don't get the message you end up in a in a dispute unnecessarily and but then yeah financial loss, usually so.

D

A

D

What's the point of starting this new season odd, but it's not hard code.

A

There's no point basically.

C

At the moment, I mean that's, maybe on our to-do list, but not really specified so far, but would be probably good at some point to make their deployment of cenotes more flex. So that's not hard-coded. It comes with a little, especially because the seed notes are now serving the Dow blocks or the blocks which contains bsq transactions, and that has a little bit of a similar like HPV in ever, in a way as a malicious seed.

C

Note who is sending you blocks where transactions are missing, can ya can trick you in a situation again like you can lead to a dispute or wherever somebody paid? You is queue, but the signal is not sending you the transaction, so you are not aware of it. When you are not running a full note, a full download, you can run your bisk application as a full download, but then you need Bitcoin core configured with RPC and everything and most people are not doing this. So most people are running bisque.

C

As a light note, as a light note and full note- is here only referring to the towel, not to the Bitcoin pod and all the signals are running at Stough all nodes and they are serving yeah. You make a get blocks request. Give me all the vsq blocks, what I'm missing and the seed note is sending you the blocks, which only contains the bsq transactions. It's. Basically, the signal gets the full block from the Bitcoin core which the two thousand transactions is filtering for psq transactions.

C

There might be two or three psq transaction one block and then the seed node is sending you the block with only those three transactions. The date is much less like a normal full block.

A

I like to jump in here just to get the non bow staff of the of the signal of the role of the signals out of the way- and you can you can see my screen- is this: yes, it's correct. Yes, we can see yeah.

A

This is this. Is the last seven days, let's say, of monitoring of the seat nodes, and we see we see. We have here that, for example, the filter messages, we have mail book, storage messages and we have offer payloads messages, and you see the different colors are the different seed notes.

A

What we, what we can see is that, for example, let me get rid of this point. We have. We have some some differences in the status of the of the seed nodes. There is, for example, the 7 to 3 seed. Node has 24 mailbox messages more or 25 more than the rm7 seed note, and that is.

A

That is a difference we, but we don't like to have, but of course it's a it's a dynamic Network and and it's what it is if there is, if there is a difference of two hundred five hundred something like this, then it would be a real issue, but I think we had this. We get control over this I believe what we can also see is, for example, this orange line. The dots the orange dots shows I believe this.

A

This it looks like the the daily restart of Manfred's branch of the seed nodes, because there is a difference of let's say: 18 blasts of mailbox message for mailbox messages, for example, and it's it's some some it's the 10th of August and then on the 11th of August, the same time that there is another jump in the in the lion and we have less difference.

A

Yes, the that is, that is the the issue we have with the seat, notes that they run out of sync, and we don't know exactly why that is, and that is an open issue. May.

C

Something old all those all those Network messages here they have a time to live, which is different long or like for the office it's only as in five minutes. So when offer alpha has to be refresh all the time to stay online when you losing network connection you're off its dead after five minutes, and it get removed everywhere from the network but office, it's very usual that they are not offers come and go when every time when you are starting up an application.

C

You're publishing your office when you shut down you're, removing the office and the state is not arriving everywhere at the same time. So that's very usual that it's not hundred percent the same for mailbox message. It's a very long time to live, I, think one or two months so, but they should be basically not so much different, like we have seen now and I think it's related to the problem, but we have, over the last five or six weeks with the seed nodes that yeah, especially as a few weeks back.

C

There have been some issues with the connectivity, or so there was one port open and when this port cut or when people were sending some garbage to this, what it has caused, basically a recent from Genesis, which takes in two or three hours and this time the signal is basically on 100% CPU and not fully available and I'll probably lose a lot of connections.

C

And it's very likely that in this in that time, the seed node has lost a mailbox message and I think that's the reason why we have now such a big 5th wisdom, inbox messages, which is not good and shouldn't be so high, but the mailbox messages they get removed from the user when the users receiving the mailbox message, he's removing it and send out the remove message. But again when the remove messages.

C

That's not so what that's just flooded in the network and when it, when it's too later right in, like at the moment, probably have some bad mailbox messages which have been removed already. But some seed not have not received the remove mess. So they still kept a mailbox message.

C

And now, when another seed note has removed it already and now, when he's requesting their mailbox message or the data from one of the signals who have more mailbox message, he get this again and this will be tangling another one or two months after one or two months it cleaned out because of the time to live. But in this time it's basically yeah. It doesn't work. Really it's just a little bit more data, which is basically not needed anymore. It doesn't cause any harm, but we stable seed nodes.

C

This piece should be quite low, I think not more like what I remember the past. It was usually not more like yeah mailbox, Meister, usually the same number of first window 300 off first and there roughly 300 mailbox messages and I think what I remember roughly. It was maybe a bit first, something like 5 or 10, but not a singular moment.

C

We have something like 20 or more but I think that gets cleaned out and it's not only not really a critical issue, but it has shown that the instability of seed nodes over the last weeks has SGI has had several consequences. Yes,.

A

We're the alert in the other graph is set to plus minus 20 and we have a total of let's see, 550 made book storage messages. So maybe that's! That's! Okay! Yeah! Yes, miss! Yes,.

B

I have a question so now that I'm running the jhg seed node. How can I receive notifications from your monitoring system in regards to the node that I'm looking after now, because I'd like to know if any of these parameters, you're monitoring, go outside the acceptable limits not only like in a critical state, but just if it's a little bit out of sync with the network. Just so I can kind of check on the node and see if everything's, okay,.

A

Well, you can't that's the simple and short answer we had tried it to to marae's. We can connect it to slack, but let me see if you, if you observe we have false positives. We had false positives in the in the past because of the stuff Manfred talked about. The city is having too much CPU load and so on took too long to respond and the monitors as well I didn't get anything.

A

I must assume that is offline, and then we had multiple alerts per day and we decided to notes root it to slack and first tweak a little on the on the monitor timeouts and so that we reduced the false positives, which is the progress I am on to and just I believe I can I can make it for to a reasonable, false positive rate. Let's say by the end of August.

A

So in two weeks or something like this, then I'm reasonably sure that spamming, slag with alerts will not hurt the whole thing so that people do not take the alert series anymore, but.

B

That is in percent other ring systems. My understanding of how they work is that if there is a false positive rate like this, where a node might just be slow for one request or something they can, you can define a certain number of slow requests that must happen in consecutively. So if the past 10 requests have all been slow, then issue some kind of alert or something like this might be just a simple way to filter out those false positives.

A

Yeah, the monetary system is not that powerful, let's say, but what we are doing is we wait, I, believe it's about 50 minutes or servers, 50 15 minutes before we check again and even though it is still not up, and we ask every 7 6 7 minutes for for our data and if the monitor is if the node is not up in the 15 minutes, we wait another 5 minutes, try again and only then. If then it will still not up, then an alarm is triggered.

A

The issue is has been that the nodes have been under heaviest load I, don't know why we don't know why, and it happens, it happens quite often, so basically they allow the alarms were correct, because the note, if it doesn't respond to the monitor it, doesn't respond to its client as well as oh, but they they caught it themselves and and healed themselves. So we knew there is an issue, but nothing. We yeah it's a difficult. It's a difficult decision to make yeah.

C

I think I mean, of course, we shouldn't spend too much, and so, but I think it would be good when we get soon to this point, even if it's a tune to too much and their operators need to look into it. Maybe they don't find anything, but then it's kind of like the play.

C

Okay, when it's when it's issue of the monitoring, they come back to you and you have to fix it when it's issue in the code and they come to the developers and the developer has to fix it when it's something else like this issues, what we had, what caused the problems yeah, it was mainly this open for what at something super easy to fix at the end, but it was not aware of this and this for what caused this some random calls to this, what cost a recent from Genesis and then to three hours.

C

The monitor was basically completely under heavy load and when we would have discovered this earlier when when the monitoring would have posted all the time messages on slack, it would have been better actually because it took us whatever two weeks until we really fix this or more and so I would I understand that, of course, it was not still reliable enough to yeah I think the earlier we get there, the better and even I think everybody involved should have basically take it very serious investigate when it can escalate it to two who can further help, and we need to get this soaked over the next one or two months really, because it's taking and we've seen now the troubles I mean that if you is not traded as much as before it's a result of this and it has financial consequences directly, not.

A

Just Lauren just activate the notifications: let's see how either yeah okay.

B

I'm, a quick question for flora and um and I've never used this I'm. Sorry what what is it called again: the savanna or fauna, yeah, very beautiful graphs and charts, and everything and I think this is correct me if I'm wrong meant to be used more of like an analytics platform or maybe for the actual monitoring system notifications. We might consider using something like Nagios or saying one of these more.

B

You know, and not necessarily enterprise but open source enterprise monitoring systems that I think might be better suited for the actual slack notifications, because it looks like you've done a really great job with the visualizations and actually getting the data from the individual seed nodes. But maybe Nagios would be better for doing the actual, like alerts, notification since I think they they spend a lot more time. You know, like you, said, filtering out the false positives. This kind of you know mobile app integration and things like that.

A

Yeah, basically, basically, the graph on a thing is just a different end and you can't create non beautiful graphs.

A

The main domain. The main issue is monitoring in general. In our case, how do we get the data and for now it's it's a it's a it's multiple, very small bisk nodes, basically with very limited limited functionality which to vary the network periodically and try to provide the data.

A

This data is then sent to the the graphite I. Think it's graphite. It's it's a it's kind of a database. In the background, a very data is stored. Quick Ravana does only the display.

A

Yes, I haven't, haven't looked at nine years to be here to be honest, but we have how it's called prometheus. We had. We had the thought we could use Prometheus, but in the end we decided that this approach would be the best, because the nodes are already another heavy stress and if he, if he carried them every second or so, it wouldn't be, would make stuff any better. So we more or less settled with this and yes I will take a look at it now.

A

Yes, let's see if we can use it without much, let's say overhead, but.

D

B

Yeah, let's work directly together on that after the call I have a lot of more experience, working with Nagios and it's modular and optimized for the notifications part. So, let's, let's talk after the call. Ok.

A

Ok, ok, so we have to seek node situation. What is it it is and what it does believe, yeah well, the dowel part is missing.

A

Basically, the dowel part is exactly as the other messages has it not distributes. But, of course it's it's tower state messages, and if they are out of change, they are critical. It's they're critical to the to the whole base network. So if they go out of saying it's more bad than if enough, a message got lost, but basically the concept behind is the same.

C

Yeah I think you forgot Sara for the normal this network. There are non critical data like the office or the meta box messages which already a little bit more critical but still non critical. But then there are critical data which are arbitrators or the filter. Messages are very arbitrators when you cannot try to and there is no arbitrator and when there are two arbitrators so that always all she don't have to exactly the same arbitrate. The number of arbitrators or I have.

A

This this graph here there's a separate graph for the the critical stuff and it's the alerts in tier traitors, and we see there is no know.

A

The difference has never been. That's it's pretty much. Taylor yeah.

C

And with the Dow I I don't want to interrupt you too much Jetta I'm wondering if I should give a little bit of background about this hash. It's because that may be not so clear what it means exactly the three different hashes. What we're using in the Dow monitoring yeah the Dow state is basically the main data structure which contains a state input Bitcoin blocks which contains bsq transactions.

C

Then the proposal pain, app European network data and blindfold unit, but there's a dose three. The sources are the main input to get to the dow state and the Doucet itself. It contains the whole result from the voting and all this stuff, so it's quite complex and quite a lot of different stuff in and that our state has to be basically the same at every blockers at every new block. The dow state changes just by the block hash of the new block.

C

It can be out of sync, because bitcoin can have a change split for a short time until the next rework and in this time it's well at case that one user received or the Bitcoin transaction from chain a and another user from chain B, and then they would have a different out state.

C

It's I have not seen this so far, and but it's a valid case can happen, but it should be resolved very quickly versus then, when you yeah, when you were when you, when you, when a rework is triggered on the Bitcoin side, you go back to the next snapshot, that we are making snapshots locally from every doubt, state every 20 blocks and you go back to a neck snap shot and recent again from there, which is very fast because only between 20 and 40 blocks. Then, and then you are on the same state again.

C

So the Browse. It is the most important that really has to be the same hash and we are creating a hash of the whole state. So when you have two same hash, you're exactly the same date, the fracture when you don't have to seem hash, it doesn't mean that there is a critical impact like we had a week ago.

C

What we discovered it was basically a small but which didn't have any consequences really, but it created a differentiation needed to be fixed, of course, but it's a very help very helpful way to discover any any any inconsistency after of the network and the other two are the proposal, hash and blind mode. The proposal dater only get kind like frozen after when the proposal phase is over, then all the proposals get created and- and they are not changing anymore for the whole cyclic.

C

So there's, basically only one block when a new hash is created for this proposals and they have to be the same amount of proposals, the same number of proposals and the same patch and the same is basically which the blind both so after the blind vote space we are collecting all the blind votes and creating the hash, and they should be the same in here. It can be I say it's a little bit less critical when there would be nodes which have not received all the proposals or whatever it's still yeah. It still can be.

C

Okay. The network has to deal with this because the peer-to-peer network paid to have this eventually consistency, and you cannot guarantee that every node has the same data but in the boating. But I don't want to get in this that yet so much we have built in some extra features and security to ensure that the vote result is based on the same state.

C

So let's say you have only received 19 proposals and 20 blind boats, and somebody else has 20 proposals and 21 votes and at the vote result, there's a maturity who has seen the 20 under 21. Then you were basically on the loser side and you have to get this missing data.

C

Otherwise, you are out of sync of the network, but before the voting is basically both are valid in a way only with the voting who is using the vote state when 80% of the votes take have one different few one special few of the peer-to-peer network data that is basically the winning few and then those users who don't have this peer-to-peer network later. They need to research and get the missing data from from the other nodes from the seed nodes.

B

Gosh and thank you for going into detail about the how the DAO state works. Maybe is there a flowchart or some other valid documentation on how this tunnel works.

C

Fortunately, not no time to write, let's.

B

Let's make one, because that's something I'm very interested to understand is how the dowel works. You know thank you.

C

Can maybe make extra session months about that? I mean there's one video where I explained also I posted in the dev channel this video series, where I did in a half year ago, so some videos about this different areas like the peer-to-peer network, but now the Bitcoin part and so on, which covers already quite a bit. But we can go on. We say we should follow up sessions about the table, because that now is see as a huge part and it's quite complex. Oh yeah,.

A

I agree: my father, I do have a question. If we let me zoom in on this this part of the of our history that we had some some believe it was. The last voting cycle we have different hashes is is if we, if we take the seed nodes out of the equation, does it all work.

C

A

There's no note, does the dog work.

C

You need you need a full node after when they would not be feed notes. You will also try to connect to any random node and when this is not the full node, he will not deliver. You have psq blocks, so you cannot use psq. You saw.

A

Plant applications are never full nodes, then, if the seed notes die, we don't have before node and sewed it out. I mean.

C

People can run locally, they're full note or when you know what I think this. So it's up it's not badly critical, but practically it's very critical of winter. The Tao is not working when the treatments are not working practically can.

B

We financially incentivize the ordinary users to run full down nodes in a trustless way.

A

Trustless is it bad.

A

B

You can get some you know discount on your trading fees by running a doubtful. Node I think this would be an excellent way to decentralize the network. I think.

C

It basically, the easiest would be when we extending the seed nodes. I mean it's a little bit distrusted. I said if, when you connect to any full node- and you get- and this is a malicious node and it's holding back some use queue transaction, you cannot lie because you are doing all the verification, but when you don't get it transaction yeah, you cannot verify it. It's similar like with HPV. You can only make whistled attacks, basically, which cannot cause big troubles, but can cause still disputes and so on.

C

So, but that's already a reason why we don't really want that random peers are delivering this service, so I think it's it's a trusted role in a way and when we see that we need more I think we should just shoot, extend this number of seedlings to 20 or whatever. What we need to think yeah, but I mean maybe over the time when we have had really under control way. We should extend the rice with something like 12 seed nodes, but yeah.

A

But but I think the idea is this is quite nice actually, but of course we had you have to make sure that we don't have to trust the peer that is running the full node and it is quite difficult, but the idea is good and if we have our signals under control, so there is no going out of sync or they resync themselves and so on. We may think about that in the future, but I think it's not a topic for the next year at least mm I can.

B

Imagine one possible attack might be to do some kind of coordinated DDoS attack against all the seed nodes, and then the attacker would run their own doubtful nodes with they're withholding attack. Something like this yeah.

C

B

C

I mean they're they're push with many possible attacks, but a sinker yeah. When somebody want to shut down bisque, there are easier ways, probably and I- think the text which really would lead to lost money are probably more critical like those with like we sold again, you cannot lose bsq by this. You can the worst case basically get a dispute, because you expect a transaction and we don't see it which we are.

C

Doesn't you can always look up in the psq, a block Explorer, and you see it there when this is not a text as well. That's a cool note that kind of the attack can be a fetus, but it cannot be. We say we sold the tech and I mean another thing: is it this problem is less critical when we we have to improve the requests. The way how we are requesting P is Q blocks at the moment. It's only requested from bonds node, and we still have troubles with this.

C

If anybody like I can explain a little bit more closely. What's the problem.

A

We have this on the agenda, for it's, a wire stuff goes out of, and maybe we can, we can include it in that one.

C

Yeah, but when we would request from several nodes, the PS q blocks that we sold attacked, which had would have less effect because when at least one on this node is giving you the protection you get, the transaction but to dude is really right. It's a little bit complex because there you have to look into the blocks. It's not only that you are counting the blocks and you see. Okay. I have passed block number five hundred thousand.

C

You also need to check if block thousand from seed, not a contains the same hash, basically or the same transactions like block 500,000 from node B, and that's at the moment not done, and so they get a little bit more effort to do it right and it's the question: if it's a reasonable, it's a it's a it's a basic weakness, but I think there are probably from priority other things which are more more relevant yeah.

A

Okay, so I believe we can move on to the next point in the agenda. So we now know what a signal. Oh, no, there's something missing. The issue itself is missing: let's go back to this mighty graph.

A

If we have, if we have the issue now, is that if a seat node is up and running, it has its its initial synchronization with the network, but then it only listens to the network and there is no synchronization between the signals going on after they have started. So if, for example, a client says well, I I, don't know there is a mailbox message made waiting for I want to client one. Let's say this is client one at can to a client. One wants to send.

A

I have started the payment message to client to and client 2 is offline, so the client one sends to the seed nodes. Well, there is a payload message containing well: I have started the stuff and I, don't I, don't I'm, not sure, but I think the message is not only sent to one seed. Not one. Can you.

C

There are you, send it.

A

To every client you know so every node you know.

C

Yeah, basically to nearly all your peers, usually or roughly eight peers and it gets flooded in and that even if you're not connected to reseed note, because after a while, usually you don't have this- he not anymore under your peers, but the seed not receive it from any idea. What he's gonna get you? Okay,.

A

And let's say: let's say it: three is I, don't know that there is some reason: seed, no three, don't it doesn't get so signal. Three will never get the message, maybe except signal two forwards, the message, but there is a chance that signal three, for example, doesn't get the message and if it stays that way, signal three doesn't have the message.

A

So if client two connects to signal three four five and six, he never gets the message, for example, and we do not have a synchronization wants to see, it notes are up and running, and that is maybe something we we have to discuss, B by or or even we have to solve. Yes, yes,.

B

Wouldn't a simple solution be to just have all the seed nodes connect to each other. So, for example, if there's only eight nodes, that would only be an additional seven connections per seed, node maximum.

C

Yeah so I mean the synchronization is basically escaped, get state the message, but you usually do only at startup and I. Think we just need basically do this get data message continuously every hour or every half an hour or whatever, so that the seed node like we had with this problem. With this open court, it could have been that the seed node 3 was doing the full recent from Genesis and the network. The resources were so occupied that he lost all the connections, all the other nodes.

C

So in this 2 or 3 hours he didn't get the mailbox messages and then he carried on going so without any restart or whatever. He would never get the mailbox messages because it just gets flooded in the network, and then it's done so. The flooding stops after 1 minute or whatever, and so when the seed node 3 would have periodically get date, the request he would have synced up again yeah. He was 3 hours.

C

He was basically in bad condition, but then, with the Netscape 8 that he would receive all the missing data and the problem would be less critical. Yeah.

A

That is the this. That is the issue. It's not that simple. Of course it's not December.

A

C

A

Well, I tried it I tried it half a year ago, I started half a year ago to do a continuous sync, with the update data request, something like this and then you get data and I tried it with a with a seed node and the seed note had double the messages every on a sidenote head, because it's different it's difficult to adjust the local status I. Don't believe that is that it is practical to just remove the let's start: let's talk yeah, we have we have this.

A

Let me get rid of these lines so that they can see or yes, close well tools. That is basically what what we have in progress. There is a daily restart. That is what Manfred's branch does and what the master doesn't, and yesterday I created this branch on the master we can use for for receipt notes this. This branch is, let me show you quickly, the last change. Well, there is, there is the the shortened interval and even if it, if it exceeds the chaton interval it's 24 hours.

A

Yes, then, the whole thing goes down and if everything else is configured correctly and the OS knows ok, it goes down because of failure. Then you can restart, and so we have average 24 hours to see casinos restarts.

A

Basically, what we are doing with with this approach is that a c-note loses all the status. It has and starts completely new. So there is no no history involved, and that is a solution. I would say, but not the best one. If you, if you try to sync it online, then you have to decide well, there Eaters are some some race condition situations where, for example, seed node 2 gets a remove data request.

A

Just it a tad before seed node 3 gets a remove something requests and while seed node 2 already had the remove data request, then ask Sigma is free for for a sink and signal. 3 has this data still because it hasn't received, remove data request and so signal. 2 has the data again against.

A

It already had removed it, but it got it again from seed node 3, and so you have this one prop of data that doesn't go away because you know 2 already received to remove request, and that is that is difficult to synchronize this whole stuff, so I tried it, and it's not that simple, yes, miss.

B

I I mean so this problem has existed for many times so, for example, starting from I think the 1980s PGP key servers tried something similar right like if you wanted to delete your PGP key from one key server. You could do that, but then other key servers would just sink and put your key right back on the all the key servers, and so what they realize is that oh yeah, actually it's impossible to delete something from the Internet. The only thing you can do is post a cryptographically signed revoke certificate.

B

That would, you know, append a new entry to the key and kind of tell everybody that yeah. This key is no longer used, but you could never actually delete that data and I. Think that's why, for example, the Bitcoin blockchain is also append only because there's no way to delete data like House Ardis ever work right and.

C

Yeah butts in terrific point, I mean one solution is the time to live as though state there have anyway expiry date. So when, when yeah like the office when they don't get updated, they're dead after five minutes, the mailbox message after one month so so and I think we could I mean we have a minute.

C

It's a little bit too complex to go into the details here, but we have to manage at the state this usually stored in some hash tables and hash of the data is used to see if the data fall right there and I think we need an additional data structure at the moment when you remove it, you just eat it from the hash table and then when it gets added again, you don't know that you have already removed it before and you don't want to get at it again.

C

So I think we need an additional hash table or kind of like you saw to remove and then what before you add it you check if this, if the state that was already removed by a remove message in the past when it was removed, you don't add it again, and then you avoid that you are filling up data again which had been removed. In fact, by direct.

B

What did that be vulnerable to some like security concerns like, for example, somebody could flood the network with these delete data requests and.

C

You have to you have to verify, so you only can remove the mailbox message can only be removed by the receiver with cryptographic signatures protected. Only when you have the yeah, when you can provide the signature which match the public key, it's allowed to remove it, and also all the state is encrypted and signed. So I think I mean their only real basic problem with everything in a peer-to-peer. If these offset that okay.

B

So I guess the obvious follow-up question to any of these DDoS attacks is that does this currently have any implemented method to automatically ban peers that are doing bad things like, for example, in Bitcoin? You see there's score based system, okay. So how does that work? Could you explain it very quickly? There.

A

Are there are multiple mechanisms in place? One, for example, is if you, if you get, if you get spammed with messages, the time between messages is too short you you either not receive the message or dump the message or eventually I think you block the host. For example, it's a very, very easy one and the most powerful one I think it's the filter messages.

A

If I'm, if I'm, correct, filter messages, there is a there is a developed developer, key hard-coded into the the Bisquick client and if some nodes, misbehaves you can a developer, can adjust by a by a filter message adjust the network to ignore this very node. So there is a whole span of things in a mechanism I.

C

Mean yeah filter message mainly used against came off like we had this charge fix em up then what.

A

You basically do, is you say well, don't talk to this node I.

C

Think it's actually not on the peer-to-peer network level. It's only for the trees.

A

Not okay, yeah.

C

I think I'm not sure if we have something on the Pew Network level as well, but I think not, but maybe a good point. Maybe it would be good to add this as well on the p2p, oh yeah, it's several things like when you get data which are too big like expected or different message types but not defined in bisque or whatever. All this would lead basically to disconnection.

C

So an attacker need to stay in all these limits, but still by staying in these limits he can he can details and create damage in its ongoing efforts, long term effort to get the peer-to-peer network more resilient. He I mean it's like limp. It connotes a Bitcoin Europe unit, but is vulnerable to DDoS attacks, but it costs money to do it and when you stop to it, the peer-to-peer network comes back. So it's kind yeah.

C

You can stop it for a while it's a kind of like a race with resources and so on, and but it's.

A

The same Gabe all over yeah.

C

A

Only I'm biscuit yeah.

C

But I mean it's for sure, a big area where we need to improve and lots to prove for sure, but basics protection is already in place.

A

Yeah, okay, so that is, that is the difference between the master branch and Manfred's branch. We have this daily restart to keep the notes synchronized. If.

C

Yes, something there's also a check for the memory, which is not hard coded with 1200 megabyte and I test it when I was running a signal that has the different settings, and that was basically working well for me, which usually triggered a restart once a day. With this memory, we.

A

Had- and it has been a long time ago, isn't it yeah.

C

I'm, not percent Jewish, there's still rise of memory, but probably not, hopefully not that it is memory leak. Was this fixed, it seems, but it's 24 hours is basically when it was not started before. By going over this memory limit, then it's 2010. It starts at least every 24 hour. Otherwise it starts when it goes over 1200 and beador. You are defining it by a program argument when you want to set the different yeah.

B

So, wouldn't they collect the analytics show if we have a memory leak on the graph? Yes, is it a spot.

A

But we do not collect the metrics as of yet so you are basically the first one who sent me a key to include in the in the monitoring. That's.

B

Kind of disappointing to hear yes, why is nobody else, monitoring their sheep notes? This is like critical infrastructure. It well.

A

First of all, is I have only last month of a few weeks ago, I have updated the seat note how to and in the process of doing this I created the infrastructure and the information how to do the collecti thing. So it's quite new and nobody has adopted it since, but of course,.

B

Kind of financial incentive or or some way to push the seed note operators to get this up and running as soon as possible. So.

A

There is another proposal, ongoing I, think manfred initiated it that we have that we can pay some some additional bounty or something if a cenote serves a level of quality. But we haven't come to a conclusion. Yet how to decide on. If this is.

B

More of a negative thing, like you know, even if they have to post a small bond like, like you know, a hundred bsq or something where I promise to always. You know, keep up to date with the current policies of running a seed node, and you know if, if people are worried about losing their small bond, they'll, probably do it right. We don't have to make it such like a huge. We should use that bond revoking mechanism, just just a threat of of losing her yeah.

C

I mean midterm, we should use the twenty thousand psq bond, but I think we are still not there that their infrastructure is of stable documentation, I think at the moment the cenote operators are a little bit in hanging in the air. They don't know exactly what to do and so, and we have to make it more clear like at this branch, for instance, it's not. Please run this branch. You have to run this branch point you have not.

C

You must not run master, because it's a risk when you are not sinking all the time and we really don't have better solution. So that's the best solution and every seed not operator have to run this, and when we are further with the reporting and and with this monitoring, everybody has to point. There's no discussion about this.

C

So when somebody's doing as the role of the thing is he's not operator, they had to do this, maybe not in one day, but at least in one week they have to install this when they cannot do it, for whatever reason we have to find other people who are taking the role so.

B

I would like to propose that we start enforcing this immediately, but with a lower amount in twenty thousand, even if it's only a hundred bsq or something you know it's better than zero, because it creates that enforcement mechanism to you know, get the seed node operators to start using collecti or run this a branch or they've. Had you know, I mean.

C

The psq pond when he got confiscated, it's also not a light decision, I mean. Basically, it should be only confiscated when it is a severe violation when it seed not operate it just doesn't carry literally malicious and that's absolutely, of course, I.

B

Mean we could just threatened to confiscate it right right now we have no power at all at.

C

The moment it was just, it's I mean it's, not the operators fault, it's the hold at the end developers and I mean it's it's mixed yeah. We just need to improve the overall situation and I think then we can be more strict, basically also and but I think a moment. I have a little bit the feeling that the operators are not and enough aware of the severity of the situation and of the role and of importance of the signals, and we have to make it more clear.

C

Like I said there is not optional run, did whatever you like. You have to run that when the developers have found out that's the best way to do it. Then the operators have to do this because I mean at the end the other developers know have them most complex and the most knowledge about it. Okay, something.

A

B

A different proposal: how about we just you know, vote no for the compensation request for the 200 bsq that they would normally get this month if they're, not running collecti or not running the correct branch or latest version, etc. Yes,.

A

Exactly why not we can? We can add that on and give them the basic or not maybe.

B

There should even be a role to audit the other seed, node operators I mean- and this could be done by the monitoring system.

B

This maybe should be done by the monitoring system automatically, but I don't know if there's a current way to get the get hash of the commit that was used to compile the disk binary or something like this- that we could maybe try and retrieve over the monitoring system, but we should be able to verify that they're running the correct version, the correct branch and obviously we should get all this over the monitoring protocol.

B

And if you don't, if you don't like qualify all these things, then you shouldn't get your compensation right, because you're not properly doing your job as the seed, not operator I mean I wear, maybe.

C

To talk a setup like which, like you suggested, then that's basically everybody just using the provided, lock and convenience and that's it I'm.

A

Just I'm just on the under about Jim thinking about getting idea and idea how they're held a biscuit, but it looks like in in terms of which software versions are run and and and that would be the way to get also a diversion of receipt. Note. So, but we don't have that here, but I mean we're not.

C

It's not what we.

A

Can do is just words about computation requests down if the if the collecti, for example, is not there. Let's start that's yeah instantly I.

C

Think, probably it's good to yet to try to get this everything done like we want to have it and then have a call with all the signals operators to make them clear what to do because I think at the moment it shows lack of communication and information. Nothing else.

B

D

B

This the seed node call right now and I'm the only seed node operator on the call I.

C

Mean we we announce it, we're a promise, Evan he's in Canada and has his be a paid job, so he has only very limited time slots which is possible for him and I think it was not clear to the others. I think it funny is not really following slack so closely or not for deaf channels, only the signal channel and so on so I think I'm. Just.

D

Going to so I just trying to an eye muscles, you see news operator, so there are two people I like yeah yeah,.

C

But I think, let's, let's try to make it as it kind like a priority for the next weeks. I don't know I mean it need to play together with all the documentation, what to do exactly with the set up with. Maybe it let the talk of containers already there and then have to point I. Think at the moment is just important that nobody's running master.

C

Everybody has to run this branch and I can send out the message to the three guys and that shouldn't be a big issue, but I think it's out it's not their lack of motivation. So I think it's just the lack of information and they like when these troubles have been a few weeks ago. They didn't really know what to do, and there was still a monitor, not reliable enough, that they can look up and see really what was going on.

C

And but we need to get to the point that when there is a problem, the seed not operate. That has to check, if you can fix it when not here, to escalate further and that there we are not losing a few weeks until it get fixed, because that's very dangerous, I suppose.

B

One of the big issues was collecti, wasn't working with the Java. That miscues is right, I guess open June.

A

I believe we have been the only one trying it and you you found itself, but we will. We will see if we can. Okay.

B

A

Progression so that we have, we can enforce it in the next weeks. Let's say, but not you wanted to say something a while back. Is this still? It's just you don't need for.

B

I in in regards to the collecti running, open, JDK, 10 or 11.

A

Bernard raised his hand while ago, but maybe he yeah.

D

I wanted to just that we don't have to introduce any new bonds. We just gonna reject the compensation requests, yeah.

A

Agreed, okay, so I believe we can. We can close the info part on the countermeasures that are in progress and, of course, I'm still working. The last point I'm still working on getting a online sync so that we eventually don't need a special branch for running as heat node anymore, and then we have some 20 minutes left and I believe it's time to decide on a strategy.

A

The draft strategy is yeah. Well, we we have to use this special branch until something else is available and I believe we have to find the cause for the state hashes the Dow status, to get out of sync. So, for example, this on there on the right top right here. Why did this happen? There is? It has been out of sync for two and a half hours approximately, and we have to find the reason why this is happening. So if something like this is happening, we have to discuss it. We have to feel the logs.

A

We have to find the Asia yes Mike yeah.

D

So I don't know if that was my issue from yesterday, but they resolve that by restarting the seed note. So if that's the five, no it's a bit.

A

Random one yeah.

D

But so I had the same. Icd and I had to restart the signal to resolve it. Yeah.

A

The issue is: if we, if we do it on a broader scale, we can't always restart it and.

D

A

Have to find the cause people why it is going out of string yeah this one. This one was yours, yes, so that is it basically, we need it and what what it happens mostly at least in there. We have the fourth cycle now behind us, but it happened mostly when the voting cycle ended. So maybe we can, we can find some. Maybe there is something wrong with the code simply wrong. Does it back I, don't know, but we have to find the cause why this is happening.

A

Do you guys have any other points we can? We can add to the strategy because we I.

C

Like are you running my brand? Try running master I am.

D

Really do branches, ok,.

C

And restart should happen every day at least I mean yeah.

D

I mean it's your branch from I think you or two weeks ago, when you said you have to switch to my branch.

C

Maybe maybe a best adopted now all right also to this new branch, because there have been important fix in the last week ago. We salute this release, we serve on 15 and when it runs all the versions and it wouldn't have to fix than it.

D

Yeah so I have to take it's a fairly recent other this, but to is which branch or to the floor.

C

To put a new one to the official to the pitch to the season, everybody should fix a switch to date that all.

A

At the same time, just just to see if it works, it should work if tested, they've started it up. But let's take some caution here: okay,.

D

So maybe able to it now and then you see how that goes.

C

When you had problems, do it do it now, because I seen here I mean the seed notes from Stefan I had still access and I updated them. They are working fine, so I'm not aware of any issues with his seed notes with.

A

A new branch, no.

C

We stir it's not updated and to me what was updated to my branch so and it's way to get the same codebase, it's only the resource file time it up to date, but that's not relevant when you are already changed.

A

Any other suggestions how we can streamline the strategy I'm, getting to see, notes more stable.

B

Obviously docker is got to be the best strategy, a long-term. If we can, you know solidify that docker configuration to where people just need to pull the latest release tag and rebuild the container and then also some automated test cases would be really good actually started. Working on that already I was looking at your biscuit honor code, but I think we should maybe have some test cases so that after like like for me, the biggest question are the biggest unknown.

B

Was okay, I think I'm running the bisque seed note, but is it working okay and I? Couldn't really? There was no like black or white pass or fail test case to determine if my seed notice operating in the you know within the acceptable parameters. I know you have this really cool monitoring system for the production nodes, but maybe we could make a second monitoring system, for you know, testing or development notes.

B

You know just to maybe staging is the correct word when I was staging my node before I put it in production, I had no clue. You know Devon and I were kind of spinning up our own client node. So we're pointing to it- and he saw some issues in the logs and I- saw some issues in the logs which we eventually concluded was you know it's now been merged into that new seed, no temporary fix I think that kind of those issues were resolved.

B

But if we had some automated test cases, I think that would have you know. Those test cases should have failed and said: there's zero, there's zero data being returned for this get blocks request or something you know more binary like that. So I think is part strategy. We should aim to move to docker and we should aim to have some test cases.

D

So I don't know if has been discussed before, but I already also have talker installations and I have a PR that I can push to the signal. So maybe, if that's of interest I can spend some time on doing that. So it's just my my talker configuration a it's, not a lot of work to do it there.

D

So that's one part and the other part is Bernard- did some integration testing for the API in which he spins up all the necessary Togo containers and the some integration tests? So we something similar could also be done for the cenotes.

A

Yes, I believe the testing stuff is is a whole deaf, core multiplet ofcourse actually itself and we should yeah I agree. We should have some green red light. If you have your seat not up and running, and then you get, yes is okay or no, it doesn't work, but we are just starting in getting biscuit tested. So there is not really an infrastructure we can build on, but.

D

Just to look at the word that Bernard did for for the api's that that's one way of doing integration, testing, yeah.

C

One from development side I think that's one hype for you even more hyper.

C

You probably like this cross-linking, because when the signals are running stable, we didn't have this issues in the past when it was most of the time it was running quite well, but now, with the Dow yeah I still see from my own app that, like often I don't get the beers cue blocks or when my computer is in sleep mode and I come back, then I don't have the latest blocks on I'm missing five blocks and it doesn't catch up anymore, so I have to restart so the Dow code for requesting the bsq blocks is not yeah.

C

That's missing! Some more resilient work need to be improved to be more resilient and more stable and careful and and to deal with this edge cases like yeah. When, for whatever reason you lose connection so that you're catching up later, when you have connection again and you of just have done this gap of blocks and you cannot use it out and when you're number of the century and it's a little bit difficult, catching really anything deeper into tau.

C

It's not about consensus or whatever it's basically about requesting the data and getting when you get the blocks. You're passing the blocks, but it has he already to complexity that when you would receive multiple data sources from multiple signals, you get the blocks and you are starting the parsing. You must not interrupt the parsing and so on. So you need to manage it that only after the parsing is done. You will check if you have a diff in the blocks, that one season has maybe send you more blocks, and then you apply it again.

C

It has some complex. It is not trivial, but it's also not so complex. You don't need to understand the full doubt to do this when any developer feels that you want to work on this I can assist and can help a little bit.

A

C

A

Some specifics on where's the problem: what is the government and yeah.

A

A

Believe that getting the monitoring up and running for every seat node, we can learn a lot. If we have a seat note that is going out of sync, for which reason ever we don't know, and we can then cross-reference it with I, don't know CPU usage or Java heap usage or something like this. We can maybe pinpoint the issue much faster, so I will I would push the monitoring and collecti stuff.

B

I couldn't agree with you more honestly, I'm kind of surprised that we don't have any monitoring in place up until now, because it's kind of like you're operating the seed nodes blind. You know, I'll, give you a good example. I once was doing some consulting for a client and they didn't have a monitoring system and I implemented.

B

One and our our monitoring system immediately noticed that they had 21 failed disks in their servers raid arrays, you know, there's probably no servers yeah and they did not even realize that all these servers were about to crash after they lost the second. So not it's, it's actually a lot of cases. The monitoring system is not just a green or green light or a red light. With some critical notifications there's also a lot of yellow lights. That's saying, oh yeah, your disc is 90% full.

B

You might want to increase the size of your disc before it gets to a hundred percent full and actually something crashes. Because of that, and so the monitoring system is is like everything that's I. You know it's like trying to fly an airplane without having any of the speedometer or any instruments display.

A

A

Okay, any other suggestions how we can improve.

D

Saw remark on colic on collecti to not like it open, JDK 10 I am running my signals since forever. I think on 11, so so 11 should be fine. I.

A

Believe Christoph storm did some some work on the code base and is at least he says that his grandson JDK 11 without any issues he he didn't, observe any issues. So maybe my monitor is running on JDK 11th as well, and it's the biscuit so.

B

Okay, so for my server I'm running disc, on open JDK, 10 and I'm running collecti on open, JDK, 11 and the way I configured, it was I'm not using docker. Currently, but OS is actually Ubuntu. Does not support open, JDK, 10 I understand it's actually past the end of life, so I just did the OS official package installer for open JDK, 11, set that to defaults, and then in my system, D script to start bisque I'm, just setting Java home to open JDK 10, and that works very nicely as.

C

We have I mean the main reason why we have not went up is because Oracle has not provided a Java package which is required for the binary for qtk, 11 and I'm, not sure if it's now there, but it was at least in the past. The reason why we couldn't update and forward it doesn't has any tropical shirts on the origin. So we are, depending on this I, think we have another win.

A

For really yeah.

C

For peeling the binaries for us yeah, but when you're building it you're shipping, their JDK with the binary so I think it's a little bit risky. When we developers are basically using JDK 11 and we have shipped a version which critically 10, because there might be some small, whatever issues. But we will never deny have to cupboard. But.

D

Country, first to the key 11 or 12 to make chili kitten compatible. I.

C

Don't know I have not looked into this, but I mean party for.

D

Souter but.

C

Say you are shipping JDK 11 with 2010 Java package, sure the.

A

Stuff words to compile it for for Java, so.

C

D

So and then only to release also a manager has to actually have a dedicated, can use, 11 or 12, but just to make it source compatible with it and I mean.

C

It's still a different I think I mean probably it doesn't break anything, but it's not the same, and it would be a little bit careful with this as long as not absolutely must reason and you need some- they need it for some some some things like they're collect your. So they are. Maybe we can find a compromise here, but that we are just moving to that developers are using to ATK 11 or so even with compiling back to JDK 10. It's a different set up basically like what we are shipping and I. Think that.

A

Experience we we had pick at University. We had the issue that you can. You can compile for a jdk smaller than the older than the current one, but it's not the same. We work today in the crypto part of all things, and it made a difference. It made a huge difference.

A

It would not maybe it's it's no issue at all for a HelloWorld application, but I wouldn't risk it for such a yeah. I was.

B

Just supposed to say something similar if you're running an unsupported or untested, maybe is the accurate term or if you want to be brutally honest, you're testing, open, JDK 11 in production, on your seed node and while that might work perfectly fine I think we should test it on a staging setup and with get like the full monitoring run, the full suite of test cases before we run any seed nodes in production on these open, JDK 11.

B

Just on the off chance that, like you, said the cryptography or or some performance timing differences, it could be a very subtle like threading, who knows what, but we should probably have all the seed notes running the same Java the same dependencies, the same everything as much as possible to avoid the DAO state, because you know as earlier in this call, we were just educated. How critical it is to not to prevent the DAO state from getting out of sync and the Java mismatch could theoretically cause something like that. We got to be really careful.

B

C

Unfortunately, or a good up completely the new policy with the ocean, so I really have to complain a little bit about this, because we lost probably half a year of developer work with all these issues when we update it from JDK 8 up to 210 at the end, and it was just a pain in the ass in all areas because it was immediate, they deployed it as a kind of like production-ready in version which has a super short lifetime and at the end, it's a it's a nightly.

C

It's a it's a beta test, so alpha testing. And what are you doing now that they're shipping three-decade 12 already as kind of like yeah? That's not a long-term version and they don't have the tools ready like the java package is, from my opinion, completely professional. What they're doing I mean we couldn't?

C

We should get to the next long long-term version, which is treating it well for sync, but that will take probably another year until they have shipped the packager or we have find another solution for this and what yeah, if was basically to live, we submit the version and now they are declaring it dead. Already, it's just a pain in the ass and with Cherokee eleven, it's not much better and more, not much worse and as sue said yeah. When we did this update, we didn't see all the troubles in the beginning, of course.

C

Otherwise we would never have done it, but yeah you're starting and then you cannot go back because you have already worked a lot on changes and then you think yeah here's, the problem. We can fix this and here's a new problem. We fix it so we're moving too much in the next in the next trouble with JDK 11 I would be very cautious, I mean it doesn't provide us any anything. Important, there's no feature what we need or whatever and well.

B

The security updates are the primary concern of using a and.

A

B

That's no longer it's past its end of life, right, yeah, yeah, sure.

C

Of course I mean would be the best, but really.

B

The whole time.

C

They're in the new version, how many factors I mean it Oracle's, American company, so it fixed.

C

D

C

I would like to use soft, which is older than five or ten years, because in this time there were probably less factors like in today's software yeah.

A

Of course back, then everything was better, except for the future. The futures, not panic, I.

C

Mean there's probably more security holes, but probably less vectors, yeah.

A

Yeah well, okay, so we have about two minutes left I believe we can wrap the the discussion up. I will compile some some meeting minutes and compile some next steps. What we agreed on and the whole talk is, if I, don't it up, I have records the talk and I believe we have a christiev to get it on to YouTube and well thanks all for joining and I believe here we can end it now, except there is something really important. Someone of you would like to to note on.

C

Just a quick question when it's roughly planned the next call because think they're the thing with the new developers, how we are reaching out to new developers and how we are dealing now with the quite a few new developers is somehow we shouldn't wait, one or two weeks think the truth. Either we have a follow up in the next days or we are maybe doing now, or maybe somebody who is still time just stay and- and we are talking a little bit about this.

A

Yes, so did the we agreed yesterday before yesterday, I, don't know we agreed on doing it on a short-term and- and we tried it today- only announced a new two days before and we have a list of topics. So there is no no reason why we should wait two weeks so I believe I'm I'm available again on weekend, so maybe Saturday Sunday Monday, something like this will be I, will shadow the desk next deaf course or belief. Vr green.

A

Okay, so thanks for training, guys, happy coding and let's make some progress and not just talking about it yeah. Thank you guys.