GitLab Gitaly group, 25 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-02-25: High availability Gitaly demo

Description

Walk though of setup documentation.

Learn more: https://docs.gitlab.com/ee/administration/gitaly/praefect.html

A

B

Shame ice cream.

C

Okay well, I've got my ssh working I, know my service this time and I've done the just installed omnibus from nightly on all of them. The other thing I learned the other day was that we don't have nightly builds for Debian. I went through this whole process realized. We only have them for a bunch.

C

But I think I can follow the docs now. So this is the from the merge request that I'm working on with Doc's fixes and there's some lint errors which I need to fix. The faults can be merged.

C

This is an updated diagram trying to explain where we're headed.

C

So if you want to review like it sort of matches the styling of the main ha1, but trying to highlight the the different components, a bit more clearly than the current diagram, but will get me to write on that. Alright preparation, I have one prefix when I disease. If we.

A

C

I've got a working kit, lab instance. I've done configure.

A

I've provisioned an SQL server, I.

C

Have one prefetch node 3d nodes? I have the IP addresses and all of them.

A

These are the secrets that we're going to be creating. I.

C

Just can use a default. Alright first step is Postgres.

C

Don't really use the application perfect. Okay, I will need the prefect node in the Postgres node. Okay, so I changed this in the documentation, because previously we had this sort of like weird requirement.

C

It's like make sure that you have a server or like a computer with the psql tool installed and it was sort of like a dead end, but I think we should just instruct users to like SSH into a box that omnibuses has already installed psql on, because then we don't need to like have people to install local dependencies as part of this process to create the user. So I said I changes to read basically SSH and the prefect node, so I've done that.

C

So this one down here is the prefect node root. Okay can write also I made the server address the last argument, so it's easy to copy and paste because it Alijah always like arrowing back to the first argument.

A

B

Sql you're so efficient love that.

C

This isn't my first rodeo done. We did all this work with geo, so going through the docs is just like I. Remember all the things that we like it took us like weeks and weeks and weeks to fine-tune, and so it's that's why it's coming faster.

C

All right, password, okay, the password is very securely prefect, SQL password for anyone watching on YouTube. This node will be deleted before you see it all right template one I found this confusing because I don't know that template one is a magic table that apparently or database that or always exists. But apparently this is a trick. So this succeeds I think we should probably add some explanatory note like what is going on here, because otherwise it's just sort of magic and likewise template wanna. Think it's a bit confusing.

D

Every rhino's template one well.

C

This guy didn't know.

C

Yeah, there's all these different magic tricks that I know it's nice to learn what magic is going on when you're reading Doc's all right, we're going to create the user I did notice. This is kind of a different process to the Geo Doc's, because they connect in and then run like a slightly different command for creating the user and then creating the database and doing some older thing. So I don't know enough about strengths and weaknesses.

C

With the approach we use versus the approach geo is using, but it's interesting that the Doc's have different methodologies for essentially creating the user and creating the database so might be worth standardizing. Yeah.

D

It's good I see what you're doing here so you're giving create permissions. You may not want to have prefect to have create database permissions, so you could argue that yeah.

C

B

D

Why we probably do it probably doesn't do that because they want they probably assume every super users creating the database and then pre-fit gets to deal with just adding tables and such, but not like creating some random database yeah.

C

Because I think you probably generally would want to have as few accounts as possible with create yeah missions drop-frame. Exactly so. I can probably just update this to match the geo docs, and we can try that out next time, but for now I'm going to follow the instructions.

C

C

This is another like nitpick I. Don't know that I, like the /q, like copying and pasting like running and then immediately quitting like I, feel like either. We should explain how to exit or like ask them to do that, because it's a little weird to like start a process and then like execute a command and suddenly it exits. I don't know if it's still exits if the first command, but it feels a little strange to me anyway.

C

Reconnect okay,.

E

A

Be in touch it password.

A

C

No, so we're back into template, one and network rate you take advice.

E

C

So I think that section is done. Can someone make a note that there's no sort of like conclusion, summary note sort of saying, like you've now configured the database to serve ends prefix next section to complete this section? You will need configure Postgres server. Yes, we just did that. We need to know the server address and the password okay, so we're still on the prefect node from the last step still is root, so one thing I also did was break up.

C

The docs word I went through last week when I did the recording by myself it was just a huge wall of config for gitlab are being. It was like copy and paste this and update five locations, and then you had to read all the comments in the blocks like above and below different chunks and I found that kind of hard to read and like work out.

C

What was going on so I I broke this up into like multiple steps which are like they're all adding config to the gitlab are be, but it's like clear, a sort of explanatory like we're making this change to do this and we're making another change. It's the same file to do this other thing happy to hear feedback. If people think that's a bad change.

A

C

Right so we're disabling everything and enabling the prefix.

C

I also discovered the set paste like fixes all my paste problems.

C

So that was having so many paste problems.

C

Alright, that's that configure prefect to listen on network interfaces by editing her.

C

Configure a strong token for prefix by editing. This will be needed by clients outside the cluster. Like your lab, shell.

A

To communicate it pretty much, it's.

C

A very very strong token: it's long.

C

Configure prefect to connect to the Postgres database you'll need to replace that.

A

As I address to be address and password password.

B

C

Do like the fact that in the docks we weren't surrounding things that need to be replaced with Curly's, and so everything is a word in vim. So you could just change the word, so you don't need to know all the advanced word motions if you using vim to replace these placeholders say I, don't know if anyone actually thought of that when they were doing it but I enjoyed it.

C

All right, if you do not want to use SSL disable this I think this comment is wrong. I think, okay, know. That comment is right. It's just confusing that it's like uncommon, this line. If you want to disable something.

D

C

Why would you not want to use SSL for your database, I mean if we probably need to document it somewhere, but yeah.

E

C

Made maybe it should just be like that should be like a PS at the bottom, so like people who are cowboys, and then we spend more time documenting the SSL like how to use more advanced SSL configurations which are more likely to be how to do it all.

C

We could have a section at the bottom for, like SSL configuration and link to that I, don't know I, don't need to change any of those settings, configure the prefect cluster to connect to each Gili node in the cluster, so I one thing I did based on last time we spoke. All together was talking about this configuration here and trying to explain more of what's going on and I did this in the get early nodes as well and to explain what each thing is doing and what needs to.

A

A

Not let's grab this and.

C

Stop placing some IP addresses I.

C

Really dislike the way I'm zoom.

C

Zoom, like has this green border around the screen, which always inadvertently like covers something.

C

Basically, I can't tell which mode of your moment is my complaint.

B

C

Complaints are not worthy of some notes.

C

This does feel easier to surf than Gio, probably because there's not like the whole back wasn't for slipping between each nodes. It feels quite linear the the setup instructions I.

D

Should hope so yeah there's a lot more components: okay,.

C

And we'll just leave the token, because we're going to use that very secure token save the changes and reconfigure.

C

C

Did notice in the docs when you view them on docstoc gitlab, there's a nice little copy and paste thing next is blocks, but it means that whenever you've got one of these commands, it actually requires you're doing a substitution inside of it. You have no opportunity to do the substitution because you paste it and it immediately executes.

C

Right, it did make me wonder like if there are things where substitution is needed. If we and they're going to like actually execute like they're run in the shell, maybe we should not try to avoid those substitutions where possible, to prevent people like executing commands. Unexpectedly, all right verify that we can reach.

C

We can psql all right job done next step, so.

B

When you copy it, it actually inserts a new line at the end. There's not what you're saying yeah.

C

It's just when you go to like this Doc's interface here and you copy this right. But let me run this one.

C

Like the one we just did here, so if I just click copy and then I run here and then just go, I'm just doing command, V I noticed that executor I did not press the Enter key. So I noticed this in the context of these commands up here, where you made it into the server address, because I like copied it and then I tried to paste it in and then like backspace over it and like it was already executed.

C

So I mean it's fine because it's just a failed connection to a server because it's like not an IP address, but it was kind of like. Oh that's, weird, like it's executing things for me. Yeah.

D

The very least it should does not put the new line in there's that you have to hit Enter.

C

Yeah, maybe that should be like changed to the actual docks yeah.

D

I think so good someone.

C

Maybe they don't just.

D

Automatically generates the sequence of things for you, based on your thing, but still.

C

Well, that'd be cool like a little input form and you, like you, put in your tokens and IP addresses at the top, and it just doesn't I. Don't.

D

Know how is it secure, stuff copying stuff into your? You know clipboard and all but yeah.

C

D

Away, no, it's nerds.

C

Playing each of these steps on each get early node, we need the prefect node configured done. We need three or more says: we've get eleven still to be configured as giving grades. You're not run other services on these every goodly server and assigns the priests every server configuration is the same as this except storage names are exposed to prefect, not get lab secret, total perfect.

C

Yeah, so this was the thing I was mentioned before the configuration of all get early nodes in pre fit in the prefect cluster can be identical because we rely on Prefecture out operations correctly.

C

Particular attention should be shown to the auth token and the storage names and the data ders must match the storage names and the prefect virtual storages. This was set in the previous section, all right.

B

C

Very important.

A

All right figure get away, don't listen! All Network addresses.

C

My plan is to configure this once and then copy and paste it to all the nodes. Italy.

A

Auth token.

B

Just so much mission once again.

A

Configure the key lime shell.

C

Yeah, okay: this was another thing that I noticed is that we set the get web shell token in the configuration, and then we manually set it in the get lab application server and that's interesting because that is different to what geo dos geo copies the secrets JSON file from the primary to all the other nodes. So they share the same secrets.

C

We should probably be consistent in how we do that, because it's a bit weird that it might be setting the secrets, JSON file and then we're overriding it, maybe in the get lab RB file on the application server and then we're copying that around, even though we could be using secrets file, so it's just done, it seems like we should use the same approach that geo does, but I don't know. If there's a specific reason, we did it. This I don't know.

C

Maybe this was just like what seemed like the laziest fastest way to do it. Yeah.

D

Probably because geo is trying to copy you know a file with you know: 20 secrets in it. This is one secret one password so so.

C

If I go to this primary right, I, don't know when secrets live. Do you know where the location of this is Stan was.

D

A prefect secret resistor this.

C

Is the this is the gitlab secrets, there's like a gitlab shell secret somewhere I thought.

D

That was a file name that gets written now, like you said it in the config, and then it writes it to a file afterwards. That.

C

Might be the case you're.

D

Right, so that's probably why those instructions are put in this password because then omnibus will generate the files you need from that make sense all right.

A

I think we need to change the server address.

A

What else do we need to configure.

C

Configure the storage locations yeah, so this is again that thing these can be identical.

C

A

Actually, then,.

C

We got a copy and paste this.

B

Just asking over well.

C

We're gonna be lazy and just rely on it.

C

Actually, that's on my clipboard now.

C

And I'm really fat fingering everything today, I.

B

Think this is another reason why you might want to use console right to just propagate all the key values to all those ghibli nodes. Instead of having to manually configure.

C

Hey excellent, technically copy and.

A

D

Well, even if you use console, you still have to get that information. There's some bootstrap, it's somehow all right, yeah.

B

You just have to configure console correctly, first right, pointing it like prefect or the right size.

E

C

B

Less information that can go long.

D

All right tank.

C

Completely staff each gear late night, after all, girly nodes are configured. You can then run the prefect configuration checker to verify prefect connect.

A

All giggly servers.

B

C

Connection is still stable. There.

C

Well, so I was really confused. The first time I saw this because this looks like a lot of error output, but on closer inspection, I think it's same success three times in a row once for each node, but it's not obvious without like carefully inspecting this. So look.

B

It looks like the output change, because you have all these info statements there. That's not related to this. Is it.

D

B

Not from this get labs to.

E

B

Command right, okay, so.

B

That might be a regression because I wasn't in there the last time we demoed I, don't think we had one like weird error message that we had an action item for, but now it looks like there's a tonne more things and they're like it looks like the logging configuration is different than the last time we did this yeah.

C

So I, so this is been a regression in since last Wednesday or Thursday when I did this demo. So it's a change. It's probably like from the last between, like one and two weeks ago,.

B

I'll make note of it thanks.

C

Alright, that we're making progress alright get lab, get a lot of times.

C

This is nice. I can give myself a bit more real estate.

C

Alright gotta edit this old.

A

Familiar your file.

C

B

C

Dated is I can.

A

Just put that at the top I think.

C

Okay need the IP address of preface.

B

C

Yeah we better set our super-secret shell token.

C

No, we can configure the reconfigure that.

A

C

Next verify everything's working.

C

Should that be running as public niche, oh I,.

B

Think I think he's referring to how we're using external IPS right, yeah I.

C

Could very well be using in general, I pasted and I, probably should have been using them the whole time yeah.

B

The GCP is doing that magic for us, so it doesn't matter yeah.

C

Google is kind of magic.

C

C

The other thing I just realized is I did reconfigure. It then I never set a password so that this whole instance is waiting for someone to basically claim the admin account off.

C

It's a super secure well.

B

C

Someone can take me to it, you could be admin. This get eleven instances, yours free to a good home.

C

C

Now update the repository storage settings to make the new cluster the default and then create a project to make sure it works.

C

There is yeah there's like some caching thing. I noticed last time, like the first time, I load this page, it would only show default, then I'd, reload, it and prefect would show up yeah.

D

I think there's like a one minute delay, but should its be actually no she's showing that cache version there so I'm, not sure, what's kind of oh might be, uh might be the local cache like.

C

Oh cache on my machine no.

D

I'm just sticking out the application settings work, there's caching.

C

D

The on the process and all the database, and it might be loading yeah, look at that.

C

Can someone make a note of this? We should flag into the docks and have any issue about this, because it's it's just an eagle that makes it confusing to an administrator. What's going on alright, so prefect is going to be the storage location for repositories.

B

C

Hey that looks promising. I noticed this as well. I, don't think this is an issue with prefect I think this is an issue with me being lazy and not setting the main name and if anything is possibly related to changes that the source code team made about like the app like downloading things asynchronously. So it hits the API and it uses the configured server name. For that, so I think the fix is to actually change that the config here.

C

To have the external URL be the IP address rather than a domain, it doesn't exist.

C

We'll see if that fixes things but yeah the repository did get.

A

C

It did get created, we can see the initial commit, so um just make sure this thing comes back up.

C

C

Yeah I don't know what still there might be some sort of cash all right, they're, not critical to this demo congressionally, you have configured a highly available prefect cluster and made it the default storage for new git repositories. Okay,.

D

Really highly available at this point well,.

C

You know we should put it. Maybe we couldn't asterisk. Oh man.

C

All right, so what should we do? This thing now have.

D

You checked that it's actually replicated.

C

No I think that's what we should do now. Yes, Kai, so the storage locations is the sharding mechanism. So it's possible today to configure multiple shards, so they wouldn't have any extra availability and it would be round-robin between any of the selected storage locations. So if you select both of them, get lab will just choose randomly which one to select so git lab. Comm has 50 different storage locations, one storage, location per shard and so prefix shot. Look just look like a normal giddily shard yeah. Let's check if things are being replicated.

C

How do I do that? There's no real debug male instructions for how to me to work.

E

C

E

If you want to try failover, that's one way to see if is replicated so if you bring down a primary and then refresh the page.

C

Okay, so in the configuration on Prefect, let me just open it: there is a hard-coded primary should that be there for automatic failover, or should that not be there anymore? Well,.

E

For now it's very naive, so it's on startup it treats the hard-coded value in the config as a primary and then like. If you bring down a primary, then it'll fail over to one of the other secondaries and then the config doesn't match what the applications be. Okay,.

C

So if I do get lab, CTL stop here.

C

The primaries down.

C

And if I reload, the page we shouldn't get an error, I should just work.

C

Right now that kind of thinks propagated and we can view the file, so something else must have taken over presumably.

E

Yeah, so it's probably filled over to give me two, so if you tell those blog that she's pumped you see some activity there, okay.

C

And I do that by doing get labs, CTL tile or something I, don't yeah.

E

Yeah tailing it! Well, you probably want to do that on big good lead box. We can do any prefix marks to.

C

You tell me, which is the best place, to talk.

E

Yeah, do it on Gilly, okay,.

C

E

C

A

A little ways Italy.

E

And I'll just give you the Gilly logs.

A

D

So how do we tell you know, can the user we give feedback to the user that that request was serviced by Italy? It could be nice to put in performance bar I mean, like all these queries actually came from yeah.

E

But that doesn't mean we have to do I think I. Think there's.

C

Not shows with that yeah is there one about the performance bar specifically, because that will help me and mortals much more quickly determine what is going on yeah.

E

I'll write a note about that: yeah.

C

And that'll probably speed up demos because you'll be less missing around command line.

C

So I can't say the raid request coming through for this, like if I click on that.

E

D

I'm going to Gilly.

E

3 yeah should there's three of them. Ok,.

D

C

Really going for the high availability gold standard, you can't have quorum with two nodes, like that's. That's a dangerous business.

D

Standing there.

C

That's shutting down that that's easy.

E

Sure that repository got traded on precinct.

C

Well, I mean wasn't meant to be the default storage. Let me check it's.

D

A good point in me: I wonder if it's still I took maybe it's another well.

C

Maybe I didn't click. The save-button could also be you to error. I'm.

D

Guessing it's probably, you can actually look at the project and see where it's coulda could the admin projects there yeah and click on that hey, hey it's on default! Alright, so we need to fix that make it less dependent on time again. So.

A

D

Won't you want to bring them one again or is that is that would.

C

It just that's, that's a good idea.

C

And I can that.

D

Was the Thera nose fell over demo? Look, it worked, hey turn off.

C

D

How long does it take to detect that things back up right away ever.

D

E

That's a good point because it should have filled over sewed. Oh.

C

Wow, what just happened there.

D

If you go the left come there.

D

I, wonder, is this proof? Is it able to talk to pretty fake I'm wondering it feel like it just isn't working.

A

What happened? Oh oh.

D

Gosh, okay, so is that just a transient thing, or is it actually that.

E

Looks like it can't talk to get Elyse all.

D

Right, what is it like permanently done, or is it just.

C

Should I stop and then restart? Please.

D

Just try to do another project I want to see if it's stuck in this state or just recovers.

D

Okay, so recover. So maybe what happens when you brought down the first one? It it got into a state where did detect that it was down and then.

D

C

I looked like it recovered, which is good. Oh.

A

Yeah I mean see some very healthy-looking checksums zero, zero, zero, zero, zero I, don't know if that's intended. Oh.

C

It's empty, there's, nothing in it, even though I said initialize it with a readme. They do this.

D

C

Maybe I didn't click on this one.

D

If I saw you doing this one, maybe on the second one.

D

Your internal API is enough.

C

Maybe that's what's going on, Oh is.

E

The when it gillies configure the reach back to the IKEA letter, I thought.

C

I did configure that who's.

D

That on three notes, right, yeah.

B

A

C

The URL, oh, it's HTTPS, whoops.

C

Classic rookie air wasn't.

D

There an instruction to write, though they like to run a check, doesn't.

C

Check that evidence, like exact.

D

There is a get web shell check.

D

C

Someone make them not that that's an easy mistake to make that will make this whole thing not work.

D

What happen you get three knows you got to do it three times.

C

Three times the opportunities for a typo.

A

Let's stop tailing this again.

C

Yeah there it is primaries going crazy stuffs happening on the secondaries nice.

D

All right that.

A

C

Positive, oh, this thing's have checksums that look like actual checksums I.

D

C

Because there was no actual content in the repo, so there's nothing to check sir all right nice. No, let's stop one of them.

A

D

Good, we know everything: okay, I mean.

C

It's just get away running on that one.

D

All right so should we see something on the left, it says: hey, this thing has gone away or is it just.

E

In the precinct logs, there should be something this is I switched over, but.

D

Would it always see it if we tried to do as an operation.

E

Now it is something things in the background: it thinks prefect things, goodly health checks and you know, but actually I'm, not sure if I like that.

D

C

They load the puzzle, receive it.

D

C

Can you make a note to improve the visibility of the health check, logging yeah.

E

C

C

It's down down.

D

Count, so it's still yes, this is transit. Failure. I would hope that this would at least cause the pre-victory really down. If you.

C

D

It isn't doesn't recover still. Oh.

E

um Actually, does it config value I, just remembered it otherwise scale over is not enabled.

C

Bonus feature just for you.

E

C

E

Definitely don't want it to be turn on process. There's.

B

Only works with only one.

E

Prefect, so that's.

D

E

Not it's not on me, but though.

D

We can have so you can character, config, Tom, all right, yeah.

D

So James quit could have this because you're not gonna.

E

B

D

Of us and then go into /var.

D

E

Right, protect reign supreme.

D

On things, a config tunnel, yeah.

E

Yeah and then it's a failover, it's a top level today and failover underscore enabled equals true. So.

C

D

Like the cheat codes in games under.

C

Store underscore one enabled.

E

Equals true yep.

C

D

Do we want a everything you want.

C

D

Giddily one on the upper I just.

E

D

It yeah I mean hopefully.

C

We'll we haven't actually done any rice.

D

C

D

Okay, so Rick well.

C

D

C

Bread from somewhere, oh and we're seeing health checks. Maybe this is why the hell trick wasn't running when the magic features not enabled all right. So you're look we're getting the transient failures from probably get early. One doesn't.

A

Actually include.

C

The name of the node: can you make a note of that John, it's just showing.

E

You the address, is not a generic name.

C

So if we edit this, we.

D

Do want to bring up one again just to I am.

B

D

Unless you want to go down like this T note,.

D

Alright, so hopefully that hello check gets quiets down right yeah does it? Does it indicate the transition? No, it just goes quiet, yeah.

C

Yeah, we can say health check came through.

D

Yeah, it proved you nice to see you know no days online, but I guess that's yeah.

E

C

D

We have no idea which one's the primary right this.

C

D

One, as you can tell just by.

A

You yeah and we can see the replication job to get early three and to get early. One I think.

D

All the checksums look good, look like replication finished, okay, yeah, so I would say, bring down number two and see see. Can you still reload him.

C

I'll, stop killing everything. If you stand, okay,.

D

Every time every.

C

Time, I just kill everything you have consumed all right. Reload, hey like that incredible. This is amazing. It's highly.

D

Available right.

C

B

D

Edited I assume you also work yeah, but then now you're in this.

D

So three looks like it got a lot of activity.

D

Still a state, so if I turn on to does it know to rethink or what happened.

E

He turn on to the replication. Job should retry how.

D

Long does it sit and wait to retry or just until it's actually no.

E

It's well right now it just tries three times and then.

D

E

Puts into a failed state so right by now, it probably is recoverable yeah.

C

Sorry, if I stop all the other nodes, we should say time travel backwards on the thing, because hardly it will fire back together. Will he be that date? A lot.

D

All right going back in time.

C

Is it too fast? Maybe maybe it's all like? Maybe it's like number two is irrecoverable, so it's just no go.

D

Okay, yeah. So what second time.

C

But it's like recovering like you're killing nodes and like it's, it's finding some state to continuing like limping on in so in in the sense of eventual consistency. It's doing what we'd expect yeah.

C

C

I'd know if there's anything else we should and like so I guess if we bring them all up again and then we right, we should say yeah.

D

Yeah everything's break. Aren't you breaking things then? Because, basically, if you write number two you'll have the wrong history and then one.

B

Of them, you won't be distinct.

D

And this is why eventually consistent, it could be a really bad thing, because you can't reconcile that.

C

Alright, let's watch what happens when we try and.

C

Make Italy two awful things: alright, so I'm gonna edit this.

D

Alright number two got it and then number one and three I would expect them to to just not be able to just think that right. You should see some failure. I would think I.

C

Think replication just plows through isn't that right, John, it just says yeah.

E

Well, it does a fetch, so I'm, not sure that would.

D

Be fit would fail if the mess those branches are out of sync right, like you're not doing.

A

On roads, right.

D

A

I can't actually see.

D

Actually seen an attempt to replicate, but.

D

There's no replication replication.

A

Finished storage trying.

D

A

D

Also, it only did on one. So maybe if you look up scroll for one.

A

Yeah hang on did.

D

D

A

D

Something something in there fetch internal remote.

A

D

Looked at that project, what state would it actually be? There's a yes.

A

It does replicate repository fiction. Gentleman does replicate repository.

C

Fetch would probably fail right, like the family.

D

Expected bits failed, but it's possible. It is kind of a force over right and then it just takes whatever so.

C

If we shut down giddily to and then see it, failover it'll be interesting to see what.

A

C

C

It's transient failure, so it's.

D

Not going to three is it oh,.

C

D

Did I went to three three three Deborah got it. So that's what we expect. We didn't see the replication on threes now.

B

D

Well, right now, all three nodes are in different sticky time: okay,.

C

So I think, what's surprising to me: is that number three never recovered and like I guess it was? It was marked in some permanently failed state. So, even though he did the rise that right never got triggered like a force. Update like I would have expected that it would have been consistent in that every no.

D

We didn't even do we didn't even see a propagate to three I. Think I, don't think the replication even ran yeah I was gonna say. Can you stop three and let's see what's on one yeah.

C

That's what I want to do.

D

And if we were right, I think one should just be bites. The dust I.

C

Think I reckon we should put that song. I reckon the other one is the the crying emoji.

D

So it didn't it wasn't able to resync right good. Now you got, I mean. Basically, this is the.

C

A

D

Where do you repair all three noes have different master timelines.

C

C

In in theory, what we should be tracking is like if a node becomes inconsistent, so like so the third like they're all consistent right. It is a starting state. If, for some reason, like number like three becomes inconsistent like it should be that the next right would fail, or we should detect that with the health check or the checksum, and then that one should be marked as like meeting recovery, which is like forced over right from the current, and because it's in that state, it should be prohibited from being promoted to the primary.

C

When we do the failover I.

C

Guess the this light gets to one of the questions about a leader election with console, which is something that's been discussed because I think the way console would do.

C

It would be simply looking at whole of node like up down, and so we wouldn't have that control to sort of like have fine-grained repository level management, and so we, the risk of these kind of situations, is higher and where you just have like a whole node go down because console besides it's not accessible and then just promote some node which, because it's in a real world scenario, you are likely to have like some repositories on every node which it can consistent. It's not like right. This is gonna, be like a mixture. That's.

D

Right, that's why that's this? Why I'm scared, eventually consistency, because reconciling these problems is much harder right. You save you save complexity on the replication part, but then the the reconciliation part is actually really hard and I would say almost impossible because got so many different projects. Yeah.

C

C

Yeah, it's a problem that, like we knew, was there I think that the question will be John. I, think you you do. Zj spoke to you about investigating the right rest, RPC, yeah,.

E

C

So I think I can look at.

E

C

Hopefully, in the next couple of weeks, we'll have some more insight into and what approaches we can take there.

D

Right and then we are dependent, even in that case you have to imagine like two out of the three get the right, but one of them does not. Then you immediately have to say that one's broken and then you know resync it with one of the good ones right yeah, but re-syncing is tricky because now you're like we have to know which, like we all want to resync the entire note, if we can help it right, yeah.

C

Yeah I mean it really should should be at a pear repository basically I mean maybe it's possible to lean on console to like take a node out of rotation if it is inaccessible and deal with it like removing nodes and taking them in and out.

C

But maybe we need to like at a repository level, still have some sort of housekeeping to sort of keep track of this, and so there's like two levels of housekeeping, because we're always gonna need like recovery mechanisms for like when something doesn't reach quorum and that that's I guess kind of what we've got. But the layer. We're missing is like that state tracking, where we sort of keep hold of that state around like who he's out of core right.

C

Sorry I got a drop I'm late to my hole, but this is very useful. Thanks for joining.