GitLab Geo Group, 9 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Geo: Discussing potential improvements to Geo updates

Description

Toon Claes (Backend Developer for Geo) and Fabian Zimmer (Product Manager for Geo) discuss potential improvements when updating Geo. More details are available here: https://gitlab.com/groups/gitlab-org/-/epics/1450

A

Alright, alright.

B

We're online the.

A

Online, so that did you have a chance to look at the upgrade epic and specifically the one issue that I opened up last Friday or so yeah. It's, oh, my! It's.

B

Somewhere in my tab, somehow my browser I know that.

A

So go ahead, I would say like okay, so like I, actually so I I, think I, wouldn't say: I made a mistake, but I I think if you go to investigate the geo update process in a nonage, a deployment.

A

Okay, there, it is so I actually think that's wrong. That's the first thing I wanted to confirm with you I think. The steps that I actually followed were the steps that are required for a high availability upgrade, and that just happened, because I I essentially just try to follow the like flow of the documentation so like we can go through this, but I think this is actually the H, a installation and the I also I tested it again because essentially I like when I went through this coached by this is really odd.

A

You know, like all of those steps, a simple sort of primary secondary setup and then I also performed, like I, basically tore down my my instances and upgraded again and I followed like essentially only the like updating the act steps, and it also worked right so either. You know like some of the things in the instructions are superfluous or I actually follow tha instructions, and they just also work when you have a simple installation right, which makes sense, because it's essentially isn't just more stopping and starting.

A

So we can okay, so so the first thing I think that we can do, and that is maybe non-technical, is I. Think we can improve the documentation a little bit for for how to operate. You I think you already suggested removing, or at least moving to a separate page, all of the like 15 different steps of you know: old versions, I think that really Trotters the the thing that makes it really messy. It makes it really messy so I think actually perversions for more than the old yeah.

A

You know, I think we shouldn't meet the information, but it should be. You know if you are on a really old version of kitchenette here you know: are some extra instructions not like a long list of things on the same page? Okay, so but I think that can be done. I think when I, when I read through the so the first question I sort of had is when you like, go to sort of Docs, not get let of calm, and you just type in geo update.

A

You actually like end up in general, update steps on the Geo page, which is great because I think that is the correct end point for for users and then, when you follow these, like three steps, essentially I think there is a little bit of phrasing to do, because if, for example, says login to each node, primary and secondary nodes and I think that's not necessary, because you have to follow things in order, you have to first login to the primary and then you have to log in to essentially like sequentially, each each secondary and so I'm being a little bit nitpicky here.

A

But if you are like a large customer and you have eight eight instances or an 880, you know it's you just open a terminal windows and so I think we can say. Actually this first infraction is kind of superfluous, because in the update get leveling number two all of those things I explained again so I think we can just delete that.

A

You know what I mean yeah.

B

I was looking at it right now and then.

A

If you go to update gitlab, this is this is the first thing I think that that confused me so I just went from there to the omnibus packages, but I think my my research so far is limited to you know the Omnibus, not the doctor installations or other things, so that's good, something that each to look at because I'm not sure how different that is. But if you go to on like the omnibus installation, instructions or update and get laughs installed with the omnibus get lab package here, you have like essentially in the documentation.

A

You have top-level points on hey, you know. The the first thing you want to do is like updating using the official repositories right and that's just essentially like up get update and install well, that's super, straightforward and I think you know for essentially a regular installation. That is all you need to do right.

A

So if you, if you are non AJ when you just need to like up get update and then essentially after that, follow the other two instructions potentially which are like restarting the tracking database and I'm, not even sure, if you do that or updating the tracking tracking database on the secondary nodes and testing so I think that's the password for a normal installation.

A

Does that make sense.

B

I would even argue that update tracking databases and needed as a manual step yeah.

A

I mean then we can just remove that as well and I think that's good and then I would also argue, and this is this is a question for you, because I don't know what the would be correct or a black D. Let's say the preferred way is: if you know you have a regular normal get 11 sedation with Jia were enabled, and you are updating now. Is there ever a scenario where you don't read like you? Don't want to test.

A

You know that your geo installation is working after the update like I would say you always would want to do that. Yeah.

B

I guess so what do you? What do we mean by test like verified with rake tasks, getting up to your check or good.

A

So there I think yeah the there's the pseudo gitlab break, get left. You check the rake task, yeah all right. Let.

B

A

To check it's a simple check and like I would say there is no harm in doing it right. You want to know that things are okay right. So my question.

B

Is simple channel sanity check it's just like a command that.

A

Runs for two. Secondly, so can we, for example, run that automatically like? Essentially, if you run the update, update and then the apt-get install and and give it all under this end like this, it detects hey. You know this is the deal installation like he's on a primary or on a secondary during that process of updating at the end, you know when you're done, why not just run the gate lab? Do your check I.

B

Guess it would be possible I'm, not sure if it's something you want to do because, like the omnibus package, is quite opinionated on how or what what is done automatically and whatnot so yeah I mean that's a cost with them.

A

B

We can do that if we want to I mean.

A

That my point is that at this point you know as a user. You need to remember that you need to do this and it's a manual step and I like I. Personally am this is also a question for you: I, don't see why you wouldn't want to do it automatically?

A

Well, because it is just a simple check and it gives you a a you know: I've succeeded installing, and if you, if you write a little note saying after you know, you update gitlab, you know we will automatically check if your geo installation is working right and otherwise you will see some kind of of information in a screen that says hey. This is under t yeah.

B

I'm not sure how Caesar Menace traders would yeah forces the output of the of the updates. Are you know that that's maybe like that's.

A

Sort of my first question all right: we can do it's possible, not pretty sure about that, but I think it's a good idea.

B

Yeah, that's broad consideration right, yeah, yeah,.

A

Okay, cool okay. So if I go back to my updating gitlab installed with only this kit, maybe a show my share. My screen and.

A

You see my screen. Yes,.

B

A

So we've done this, you know and I think this is enough for the regular update anything other than potentially like alternating or like check at the end. But I think that's that's fine, but you know this is where I got confused. So let's say you have an H a deployment right or zero downtime updates.

A

So all of these items here on the right are top-level items right, so you have a zero downtime update section, a single deployment, multi node geo deployment, so I honestly I, don't even fully understand what the difference is between a single deployment here and the like what is described here with the official packages, but that's just the way it goes beyond geo. What's beyond geo right, so I don't really know, but what I can see is like okay, if I go to zero downtime updates right.

A

It tells me I need to also follow like this specific section here for geo, which is the geo deployment sec. Sorry sure you think that, like the.

B

Heading levels are warm, yes, that's the first time. I saw this like mismatch and documentation, and then it would make sense, actually yeah.

A

And then, if you go to a multi, node AJ it actually and like here, I, don't know right but like this is these are the each page, a instructions and then at the same level you know again after that is the geo deployment instructions, yeah I think you know these are the instructions for an hae upgrade, so, let's maybe I think Douglas knows right so I said for an Ag or or for zero. Downtime I think it's both but I'm, not sure. But it's there's definitely a notion here that if you.

A

Really follow these instructions in order right, single, like, depending on your deployment type right, so multi, node geo, multi, node, h, a and not quite sure, because, like this is this is not my first question like clearly. You can have a multi node h, a deployment and also use kyo right so which ones are actually the the right instructions. I think it's those but I think that's worth having.

B

But what does what does the multi node update the.

A

Multi node update.

A

Basically, it makes a distinction between a deploy, node and a and all other nodes where you have to like create this here, and this is we.

B

Were that I don't like to skip Auto reconfigure, I taught me throughout that I don't know so you have a deploying node and.

A

Then you have all other nodes, I.

A

Think this is also a little bit confusing to me because here in H, a that has a difference between a deploy, node and a and all other nodes right, which makes sense.

A

But we have a primary node and secondary nodes right, so is it like I would assume that in a age, a setup with a with a geo deployment, the deploy node must be the primary node right because it says you you have to essentially dedicate you need to make like designate a deploy, node and then I think you have to do things like on the deploy node first right and that's similar to like the things that you need to do with with geo right. But it's unclear.

B

Yeah I agree, if looking at it like this I, don't think like the geode appointment instructions or a fully hae rare, because it doesn't talk about like on the other notes, from the primary, so yeah yeah.

A

So I, don't I, don't know it's.

B

A bit like they tweet chill as a type of H a deployment. Yes, it feels legit. So some sort also is because the secondary is also Postgres replica and that's also something that exists in nature setup. Exactly.

A

But I think I think that is something that we can work on, because I think like just from like essentially writing down like saying like okay, you know you can have a single like a geo deployment. You know where you have a primary and secondary said: it's not aj, follow those instructions and then you can say like if you have an h, a deployment right, and you also utilize to your notes. This is what you need to do. I think that's! It's mainly I. Everything I think is here.

A

I think that's just a for most like revamping the documentation that needs a bit yeah.

B

And to be also that I also think that maybe because, like below yesterday, starting discussion about getting a provision around and working using it with you- and that was my feeling I had in the past already with particular provision or day didn't want to like try to document all types of configurations, because they're so varied and you can do H a in several different ways.

B

So, instead of like try, the document and the right one try to automate it and use that tool and make that's a product make that into a product that customers can use and I think that's why they also wanted to build CHEO into it. So it's it's uniform for everything. How you, how you install kit, lab I, think.

A

That's a I mean this is sort of on the list as something next in a way like the intent love to you is obviously much more involved than this, but I thought this is sort of a good starting point for us to start thinking about these things and make some smaller changes. I.

B

Agree: yeah, ok,.

A

Cool but I have more questions and I hope. That's! Okay. You may be a little bit over time.

A

Okay, so you know, like the combination of like h, a deployment with you and you don't and that's okay. I think that's something that we need to need to think about right.

A

So the next thing is that like- and this is like- where I'm hoping you can help me a little bit I, don't fully understand what these settings in gitlab Rd actually do right. So, for example, we have this insured gitlab Auto migrate, equals false, but is set in get get lab dot Rd and every instruction that I I, see in in here I think has some.

A

You know like some iteration of this setting being being considered. Somehow right and I was just interested in knowing what it actually does because I don't know.

B

There are two types in there, but yeah you have to line with like get logged will, so you have to set a negative dot. Rb and step forward is like the skip a post deployed in migration. Yes, so those are both like migrations and then like update the data schema to the newest version. They need to be an.

A

Ria, what what this is basically saying, don't upgrade or don't migrate. My my database automatically during the installation is that correct, yeah, okay and so.

B

It it's related to the zero downtime and now I have to think, because I'm not completely sure.

B

Like the the migration, the normal migrations are usually fast and like the post-deployment migrations, they run in the background when, when, when the new kill app is already running but I'm not sure how that applies to zero, downtime and I, just think, like all all migrations, even if they're like, before or after of post-deployment migrations, that they should not take the database down or lock up on things like that. Okay and then yep the post-deployment, migrations or migrations that are not needed for get lapto.

B

And so, if you have to like, add a column and I need that column. In your code, you have to make sure that it exists before you get laps running I.

A

See, okay and so we're just skipping things here. Essentially the post-deployment migrations in the beginning, well, I'm.

B

Not sure how this works at the moment me neither.

A

So I think this is sort of a general like comment from me because I I don't know, but I have no inside the way I felt I just type these things in and I have movie you like idea what they do, but that's, maybe okay.

A

Like this thing here, I had a like on number two, so I understand this and I think it's maybe fine to do this manually in this configuration file.

A

We live with that. This thing here is not even still relevant because that feels, but this is just like my feeling right now like creating an empty fire with a specific name to trigger something I, don't know you know that feels feels not very nice. You know. Do I have to clean this up later on right. Well, that's weird, because I recently saw some virtue.

B

Case or somewhere in the best I recall, you send that to me so I'm, not even sure if we agree with that, but there's also in the AJ step. So something is weird here. We have to have to dig into that.

B

But yet the the reconfigure command, it's it's. It does so many things it's hard to like, say what it does and what it not owes. It's really like updating the whole configuration and problem for. We also do the auto migrates, while while doing that, so it's like the magic command that fixes all things but yeah.

A

B

What you sit there.

A

Okay, but I think this is maybe something to follow up, because if we don't need this, and we should just eliminate that step but I'm also little bit weary because, like this is also a common test, like you know, I'm sitting I'm potentially changing this from true to false right during the installation, but at the end nobody tells me that I have to like set it to you again or like what the what the correct state office is and and the same with this file right.

A

You you created and I'm, not sure if it's equal eat it again, all right so I just don't know. Yeah.

B

That was my biggest worries also like why. Why are into reverting that change in dinner party I? Think the Skip Auto reconfigure file would be delayed automatically after one run yeah. That would make sense to do it like this. Then you just created ones, run one command and then afterwards, everything's back to normal, but with the change in get la partie other home lose baggage, does never edit the get up RB itself, so you have to turn it back at some point. I guess.

A

B

A

Question is like, but maybe you know- and this is also maybe something I look up in the installation of gitlab for geo. Maybe the default state here after a geo installation is that this is set to false right, and so this is more of a hey. You know just make sure it is right. I could I. Look that up.

A

You know like so when you install like it like with Geo. You know like. Maybe there is an instruction and I it like I. Just don't know what, if the top of my head, that you need to like set, get like rails autumn, I try to false, and then you know that's fine.

A

But I don't know, that's gonna be something to quickly look up.

B

Yeah, because if that would be the case, if you're running key, oh, we can detect automatically if kyo turn this flag off yeah exactly.

A

Let me have a quick look.

B

It's not I'm just wondering if you do a manual kit labrie to figure if that flag set a falsehood to the migrations or not I assume not, but it sounds weird to me. Yeah.

B

Trying to look into the defaults, and so.

A

Actually, here it's just like disabled temporarily and we enable it so there's something we do I think during to get the Geo insulation right. Yes, configure the primary server so I think here like at the end, it is set to two again all right, so it's it's! It's flipped from like its pitfalls and then I think the end state is yeah.

B

Because there's also like the or the other issue, I tweaked I talked up some time like we should have a command or some some thing that we can temporarily disable it. While we do the Chu installation, yes, then it will make sense mode, yeah,.

A

And maybe that makes sense here as well right because again anyways, if we, if we get a little bit deeper so.

A

These things here so this I, like I, obviously understand so at least. We are now just updating and installing get lat Enterprise pas, but then you have to run for 5, 6, 7, &, 8 right, and it actually says to check things here again manually right. So that's also a little bit of duplication.

A

So much my question is: is there like some potential and that this is like anyway, you were like I, don't know what the best way is like these are so sequential, but they're like for like four or five many steps right after each other right do do folks in ingot lab, sometimes like bundle those things into the rake task or combine them right, because you're not really doing anything here other than like executing those things after another, and my like one, one idea, but it may be really bad at here- is why I need your technical assessment?

A

Is you know if, if there are sort of like you, if there are like steps to take right after after installing get let Enterprise in order to sort of like get geo up and running? But could we like add a rake task?

A

You know that or like something else that essentially says get lab geo update or get lab guh a update or something reconfigure or whatever just does it right and then all of this sort of swings together to install be like or update the the omnibus package which I think everybody understands, and then we can say like okay and depending on your your setup, you know if you are in an age, 80 fondant.

A

You know like execute this, you know or if you, if you are not, you know you're already done right and then I think the the errors that happen are potentially happening in here, but they could still be service to users right. It just says from from typing, in all of those things manually,.

B

Yeah, yes, I told you I also not sure why they have to be run separately. Yeah because.

A

You know like this is I think like I just talked about it that may be sort of the like most minimal iteration we're not even changing. You know how its solid we're just minimizing like what people actually need to type in, because I think to me. That was maybe at least my my frustration. You know if you have to do this, you have to do it like in the secondary, which is very, very similar.

A

You have to do it like as many times as we have secondaries right so essentially, you're you're, like putting like a user in a copy and paste loop for 50 times, side and I. Think then things can can easily like be mistaken. I'm just worried, like let's say, like you've, done it three times right and then all of a sudden, your copy pasting, number files and not number four.

A

It looks so similar right that you may not oh yeah, it's very rare from and then that's very error-prone, and so especially when you're dealing with updating geo I think would be nicer. If you, you know like, if you eliminate that I.

B

Totally agree and that's something we can like: if that would be possible, we can. We can even do that without singing sound anything to the omnibus package, because that's totally in the gitlab source code, then you get a real source code. I.

A

Think that you know that was my initial idea. This has been essentially what what we could do is have a task that says you know like pseudo, get Lautrec get lab geo update, but and then you know or like something like something that we think makes sense and that break task and essentially just execute those things I, don't know. How is my great task?

A

Sorry, like I, had a quick look at the source code, but he could even so the instructions here from four to eight for the primary are like very, very similar to four to eight in the secondary but they're, not exactly the same right and I I, don't know if the rake task would also be able to do sort of detect am I on the primary or am I on a secondary, because then you don't even need to like have a a different command. It just says essentially.

B

True, it doesn't so it's possible yeah, it would be great at every like update on dependent of primary or secondary, would be the same. Yeah totally agree.

A

You know that's um and if we like yeah, that's I I, like that, you agree and I'm happy about that.

B

I'm not completely sure about step seven but arrest seems pretty obvious obvious to me. I, never even like when you talk about spoke about adding the guillotine check to the installation update process, and we also don't need to do that. Yes, that's true man that makes more sense to just like put it there and yeah.

A

No and I think what what then would essentially happen is you would run Zulu capelet break it, love Geo, update right and it does all it stays and then, at the end it runs a get litleo check, but and essentially the output after that to use the respite happy days, everything is working and that could be the same on both the primary and the secondary. Even though, and that makes me sad, I already open an issue, I think get lebryk G you expect is, is not available on the primary.

A

A

You know so I think that maybe something on the way that we can sort of add as well I. Think me.

B

When there wasn't it wasn't this issue about it yet I.

A

Don't know quick check.

A

You know make it let breaks that is available on primary note. That I opened that a few days ago also Faramir to see that it's you know it's the kid lap namespace, but whatever yeah that's another thing. It's like the I think the namespace should be changed, but it's like on a on a primary that says, like this command is only available on a secondary and I. Think.

A

The information that it produces is pretty useful on a primary as well and.

B

So it will be more briefed and compared to on a secondary, but still would show like some information. What was up, that's primary note and I.

A

Think actually like it could also display some different information, for example, how many times how many secondaries are active, because the primary knows about that right. So that could be an interesting way of saying you know. I am the primary you know like these are like this is my status and I know of these, for example, of these files, secondaries, all of which I can meet you at the moment. You know, that's, that's a that's! It.

B

I only only worth worried that we would like fill the screen up and lots of screens no yeah I mean it should be valid. Me yeah, but you know you're showing the gate like the G of status and the deal check and we yeah you have to make sure that they don't miss information. That's the I.

A

B

Is this, the Geo state is something they need to run, I'm, not sure I.

A

Don't not really I think they.

B

Can optionally but I.

A

Mean I think the the Geo tip that do you check should essentially like I think this is also a semantic like difference right. What is the difference between checking at the status, but it's like why? Why do we have two things and they contain slightly different information? I understood the the do check to be essentially a a I am healthy or not right, where the status gives you more information about. You know the syncing and you know like what like projects are available and things like that, so they have slightly different purposes.

A

B

Basically, does geo status is the text version of the admin panel of geo? Yes, yeah.

A

Well, that's a good way of describing it so I, don't I! Don't really think that you need to verify that you a status yeah.

B

I would toom the admin would go to the admin panel anyway, after updating yeah.

B

A

Yeah and that like but I, think ultimately I would argue that like, for example, the solution that we discussed, you know it would solve like a fair chunk of the like issues as and I think it would make it way less error-prone, because people don't have to like copy-paste different commands, and they would also make it more automatic.

A

You know, as and people don't need to remember, to run the check or it will just happen and I think that that me it may be a relatively boring solution in a way you know, but it may it may it may bring us like a lot closer already.

B

Quickly, checking if there's how big the overlap would be with an H, a setup or winners. Single single deployment. I need to learn more about the zero downtime deployment Chris. Yes,.

A

Because I, like I, think actually like none of those steps are needed at all. If you, if you run an on AJ setup, that's what I, like I've, had success. Upgrading zero testing only only following these instructions, yet.

B

That's it yeah, but you you probably would not have the zero downtime. No.

A

No, no, no, like I I'm, saying that the sorry I've dabbled I think for a non zero downtime long age, a deployment. You know we are already at a state where you can just execute like the apt-get update, yeah and I get and we're happy right, I! Think! That's fine! So.

B

I also didn't to the migrations on the secondary right, I think so, step 4 from the No, the step, 3, sorry yeah.

A

I documented this we have a look.

B

Well, I I would I will assume so because I've seen it exists in the Omnibus code should be running it. So I am not sure why it's still in the documents. Yes, no one just deleted it. I forgot.

A

Yeah, no here, that's exactly what I did I just followed. The updating using the official repository destructions, which is literally just up, get update up get installed, and that was it was possible for me to upgrade from 11.11 to a 12-point or 3 on both the like right, like on primary and secondary and.

A

Yeah that just worked I ignored all other steps. I.

A

Can do it again with the ends of a notebook? That's relatively quick.

B

Yeah I guess we have. We have any few on that so I guess whoever will be work on that can can do that chicken yeah.

B

We're just looking and I think the the Skip Auto reconfigure is permanent. When you create it file, it stays there.

A

Which feels dangerous.

B

B

Depok rate the cricket Omar, it's renamed I- think it used to be skipped out of migrations, and now it's skip all the reconfigure.

B

For you, sir, will you skip utterly configure file to skip automatic migrations and reconfigures during upgrade be skipped it's in the get up, twelve specific changes.

B

Removal support of skip out of migrations file. Let me find a link of that.

B

So when you go to the updating, kidnapped and omnibus package guide and then there are instructions about kidnapped, 12 and then it's over here, oh.

A

B

Good process will no longer take skip bottom equations file into considerations.

A

Okay, but we still need to use the Skip Auto. We considered.

B

Now you know you need to do the Skip Auto reconfigure and you need to change your kids up or be tools. Have the auto migrations to false.

B

Because I assume you every do you always want Auto.

B

B

B

But if I look at it again, I see that the other great has to stay to false, and you just do the pseudo Gator break evening. My grade manually.

B

But why'd the environment variable skip post-employment migrations. That also feels like feels two different ways to do things: mm-hmm.

A

I think that's I think this is generally like. My impression is what, if you make it look at this you have here, you have I mean if you're really nitpicky.

A

You know you have like four different ways of modifying your environment: you're, changing your configuration file, but it left Rd, but you are creating in key files to like indicate some kind of change in behavior right and then you are setting environmental variables commands and you used like rake tasks to actually like do some other things, so it's like they're there for at least four different sort of concepts here. You know that that are like happening all. At the same time, and at least to me, you know like as a slightly naive user I.

A

Don't fully appreciate. You know the like like why. So, for example, you why-why-why create a file right and why not such an environment variable right or why is the Skip post-deployment equals true, a variable like this right? It's like. Why is the notice skip post-deployment migrations file, the same as the like? It is like the gift. Let's give it all to me: Connecticut right, I, don't know.

B

Yeah I have to be I'm, not sure how it's all pseudo noise- oh, that's, not true, I was thinking it's when we like combine these commands together in wrong rake tasks. You have to be careful that at some version things an omnibus do not change that. The kid lab wheels code is not aware of cost and then our wake toss would break if they do something they aren't aware. We are automating.

A

A

But I mean I think the the first iteration that we're proposing is, we would shrink the setup so number one setting the gate learn: rails Auto migrate to false. That's still something that needs to happen. That I think. Unless we have a better idea, we will probably have to do that and then number two. This like skip auto, run like creating this file skip Auto reconfigure is kind of in the same ballpark or that it's not it doesn't seem very geo-specific right like we can't really do anything about that either easy I.

A

Think number three is fine. You know. That's just the update package I actually really like that, like when I installed, yet let the first time and I just brand like the omnibus package that sounds pretty 8 smooth, it's 750 megabytes, but hey.

B

Yeah, it's pretty cool because I'm running a good lab somewhere else and they're like app cat is doing updates automatically and so the new kid love versions like all medal year. There's how did it get her I wasn't aware of its of thematic updates and.

A

It just works wow, it's really cool so and then like essentially so these three things they, they would say the same, but then from 4 to 8. You know the suggestion is to combine all of those in to a Hitler break task when which would be, for example, Zulu gitlab break it let geo update, and that would be the same for a primary and secondary, and that would essentially reduce the amount of manual intervention spiral.

A

50% already, yes, which I think would be really like nice thing, because it would skip all of these like, like a little bit more menu little tasks without messing with whatever other things.

B

Well, we have to dig deep, we'll have to look into it more in detail. Yes,.

B

Because there I saw somewhere else, the other was in the same issue. They suggested to spine to create a new kit, lab control command or something like that and that maybe can try to like wires while, yet, why is there light I get to control zero downtime time upgrade or something like that?.

B

Okay, yeah I'm still not sure what makes an update similar downtime. That's a good question understand that better. We might.

A

Be closer to the solution yeah, but this this thing I think we agree that if you like, the geo deployment, instructions are part of a zero downtime update. Yes,.

B

Adding and adding levels warm and I'm pretty sure. Yes,.

A

And I think that's a.

A

And I think this is actually like.

A

If you about it, it's it's really better like interesting from sort of a human perspective, because I mean like we read so many things and you like I, expect like documents to be organized hierarchically right and if they're, not, you know like it's really confusing. You know it's like sequential I, guess, like sequential, is, like you, don't have that just okay.

A

Cool so I would say: next steps are like we I think we are like I feel we are in a position where we can create a few more issues on this mm-hmm yeah I've tried to take some short.

B

Notes which point you want to tackle on and our document perfect and.

A

I think then we can. We can potentially continue the discussion in those issues also asynchronously, and so my goal, like I, think we could if, for example, by the end of the week, we have them like these issues in a in a state where we both understand them. You know because we don't need to like implement right now, but once we start working on 12.2 I think I think then that needs to be actionable. Potentially by. Like another engineer, that's you yes, yeah cool.

A

I enjoyed that yeah me too. Let me stop recording.