GitLab Geo Group, 17 Oct 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Geo HA upgrade from 12.1.9 to 12.2.8

Description

In this video we're performing an upgrade from GitLab 12.1.9 to 12.2.8 on a Geo HA setup. We're evaluating the new documentation, which is not merged yet.

Issue: https://gitlab.com/gitlab-org/gitlab/issues/32435
MR under review: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/3577

A

Computer so welcome everybody we're going to try a second attempt for the 12 points to upgrade for the GHA cluster. Let me share my screen and then we can get started.

A

Okay, if you should see now my screen, where I have the secondary and the primary note open on the Geo admin page, so they're, both healthy I, didn't verify this. Let me quickly check.

A

Your project I have one project that is being mirrored and it should be updated, so it'll keep was updated eight hours ago is that same on the primary.

A

Updated 22 minutes ago, according to Dan, was the last time I checked for our newer grades, but the last committed from 10 hours ago. So this seems fine, so we're good to go so I have the documentation. Here, it's update. It's the version from the merge request. We were working on at mount or still not merged yet so we're gonna follow that I have my terminal windows open for each machine so on the left side or the primary machines and on the right side or a secondary with one more in the Postgres tracking database.

A

So let's get going multi node h8 deployment with Geo.

A

Ok update the geo primum of the new deployment on all nodes, create an empty file, skip ultimately configure. One thing to note: maybe I have normally the skip. Postgres upgrade is present. I, don't know where the dork is that again, cue the to the beat ok double check. These files still exist.

A

A

So this file should already exist.

A

Yes, so I should leave all remain there on my machine because I didn't delete them and I hope. No one else did sorry sure we're only doing the kid labelled grade and not the Postgres are great in the same run.

A

A

We configure should also be there. That's all notes.

A

Now may be something to note, but all notes means all primary notes, as you I guess: you're redoing, the yes.

B

It says I think the the heading is like the included in the gia was secondary, multi node deployment yeah well, the same.

A

Is in the secondary section but yeah it might be worthy to explicitly say, do.

B

All the primary notes how much air we I think we had that in, but then we changed it at some point, because people said that there's a lot of repetition, okay, but I- think we could make a on all nodes in brackets on the primary cluster, similar to what we have yeah like I'll. Just make a note: okay,.

A

So this is I will be using app or as a deploy. Node and all our nodes need to have this line in the gate lab party and now the question is he already go and do this manually or they would try to be smart again, I, don't trust myself again.

A

So I think in many cases it was all in there but to be sure, that's all the instructions.

A

A

This one is already good.

A

Okay, so we do to them all other nodes. I have done that we configure Oh, which version are we updating to.

A

It's still 12 to 8.

A

Aha, the command.

B

ha ha ha Cool, oh that's, new, I, think.

A

Ok back to the instructions, so we first to deploy. Note yes, so apt-get update.

A

And then this one.

A

Okay, now it has starred package page little, while yeah I.

B

Spoke with a friend of mine yesterday, when he you know, has to work on Windows at the moment and every time I see package managers I'm so happy that they exist. Yes, it makes everything so much easier. Don't.

A

If I was it's called chocolate or home in them, or something like that or one or two.

B

If you also retrieve a beverage of choice within a drinkable temperature range, Wow,.

A

That's a very good point, all right! Well, it's empty now, but I didn't check the box yet where's my issue to make sure I did so retrieve the beverage check box. Well, it's empty already, but we did hit record and we're now working on the great steps.

A

Thank you for your minor Fabien. No.

B

Problem I just looked at the issue itself and chuckled a little bit. Yes, it is a really good.

A

One I have some sparking water here. Yeah me too. Actually.

A

Still unpacking some after the trade front, one.

C

B

Window manager did you use I three? Yes,.

A

Cool yeah I'm not very, and you waited, but it gets the job done. Yeah.

B

I use an awesome for a long time that was pretty to be good.

A

That from from succulence right, maybe yeah I have like a really simple webpage and stuff I, really like I, really like the style they're using like no hassle just to the point yes Haley really enjoy that. So next step is to the reconfigure it skipping the post appointment, my creations: let's go I'm keeping the keynotes admin page open to see. If anything changes there.

A

Yeah, it took me quite a while to compile a set of these windows together. Yeah.

B

No, it is it's a lot of manual things you need to get in line.

A

Yeah, you cannot automate it, but probably not worth the effort and answer. Xkdc comic.

A

So these are the migrations, I hope you're. Only the normal migrations I assume so.

A

A

Request 500.

A

So how's the primary note feeling.

A

Now last time I was taking if I could see like info in the load balancer, which of the two app notes or I was healthy of their vote for but I couldn't find this in in the Google Cloud console. So it should be possible. I, don't know where.

A

Because I would assume like if we're doing the upgrade on the or 102 still should remain healthy.

B

Some of us, what does this say? Yeah, that's interesting. It's.

A

A good question so so before we started to reconfigure.

A

It did to a PostgreSQL version check if I'm not mistaken, up, checking post, critical, executables well, I'm, not sure the site's check subversion, maybe just presence of it.

B

So we are, you can look at the post based version in the admin interface. Oh.

C

A

C

If you go to general, not in the deal all right, yeah.

B

A

B

Is what it's supposed to be right, yeah, what we expected.

A

Let me first call to the locks to see what what was happening check submissions so I, usually just check for the green lines.

A

This is some configuration upgrading, also the dead, lap, yellow, okay, I think that's fine version revision, SSL Certificates greatest configuration, that's being restarted. That should be fine, I guess.

A

It's kill to who the block warned and then it was Chris Cole always charge.

A

So it did expected post quiz Club restored I'm, not sure why I.

B

Mean it shouldn't be started right because we skipped the upgrade and music.

C

Post-Deployment.

B

Migrations right, yeah.

C

A

Lot of migrations.

A

Period cash well- and this is something that I've noticed already- that sometimes unicorn and sidekick is already we start, but we have not really sure about that.

A

Unicorn configurations to have more workers, that's something I've seen in the release, post notes or in the notes for the upgrading to talking to so that's correct, but that shouldn't change anything version. We're course I.

A

So there's nothing besides the PostgreSQL.

A

We start notice. That is weird I guess I mean.

B

I would say we should probably make a note that.

B

This thing so that the complaint is the version the attendee maybe just copied this morning.

B

So that we know that that deception happening.

A

We're an issue or.

B

Yeah I would I would just put it in the.

A

B

Yep, so what so shall we we started now or play? Well, we starting imposes downtime correct.

C

B

I would say no.

A

It's hot now I'm thinking about it, because if I check kidnap both stars that it's even running Postgres, it's an application note it shouldn't be running for us. We have Postgres over here. Yes,.

A

So that might be like a configuration issue. I.

A

Cannot remember, maybe it's so we have the application wall here and that's that's according to documentation, but I would expect that an application wall would not her yeah. So we have to double check that, but I'm not gonna, restart port. Well, wouldn't matter anyway, because it's not connecting to this PostgreSQL anyway, so I so.

C

C

B

Mean it's a question to ask right why that is maybe for for the distribution, all right, I would say we can leave it as it is right now right, because we.

A

Yeah I gotta leave it this disease and do the upgrade on the other notes. So we have all the other nodes and we're upgrading the package here.

C

B

It's a really interesting process process actually to me just understanding what is happening here.

A

Okay, so did we intentionally remove these steps from this one, or is it just I think.

B

We did I thought they were supposed to go at the end.

A

Well, when we, when we did to the apt, install I, get installed, it did we start unicorn and sidekick, but I'm not sure, that's always the case mm-hm, but if that would not be the case, I think we would better like restart them on the deploy node first and then do the upgrades on the other nodes. One.

B

Second, because I think there was a long thread on this.

A

Yeah I was not able to keep up with all happy Irish Christa.

B

I'm not sure I.

B

Mean there's a lot of discussion right I am would have to I.

B

Thought that was the discussion to say like you, you should do the like hub bits at the end sort of after everything else is done.

A

Well, I wouldn't do them synchronously on boat application notes because intend, and one cannot take over all the other one as we started so yeah.

B

I think that I mean that is true right for.

B

Well, I think this is this is similar, though, to the instructions on on the zero downtime upgrades for woody node H a deployments long Gio.

B

They also do like deployment installation skip autumn aggregation install on everything not to deploy node reconfigure and then then do this. So I think this is pretty similar.

A

Well, yeah I, agree, I'm, not sure.

B

It was right thing to do, but yeah no, but that's that's a valid comment. When would you put it I.

A

Would put it after we configure all the nodes that is running in the course of sidekick.

A

Maybe we can take into this later, but at the moment my notes aren't healthy. So maybe we should not continue before we do me before yeah or sure about it. So this node this is nice, so you think the mercerized about this happening. It was commercial ready. So that's cool yeah.

A

Okay, the version mishmash is there, there's no problem but I'm, not sure why we have a 500 here.

B

So are they? Are they updating the nodes at the moment,.

A

In the nodes- or they have like download the packages, but they don't know don't have it reconfigure it yet sort of not renown yet. But we artists and help is take before I start. That I should have made made sure that the healthy state would have happened before this.

A

You know I, it's.

C

A

Think that this is a problem, it's with the don't balancing.

C

A

Yeah I refreshed page, the behavior is different.

A

So that's the thing with the load balancing, so the assets are not always that's what you get. Assets are like certain one or the other, and then the version with the assets does not match like the front, like the code order, the rails code with it, and then it gets stuff like this. Okay.

B

But yeah I think this is a maybe a general problem for it: zero downtime upgrades then right and how are the load balancing works.

A

What I have been doing is using the API to connect to notes and see like what the status is from. The idea actually gives the same thing for API key for.

A

A

A

Resulting oh you.

B

Have a number okay.

A

Yeah turn sometimes mean this is a internal server error. So that's what we're getting you see. This is probably served from the other application. Note, mm-hmm and I think we could even like worded cold again, so we have app on Apple.

A

201 is healthy and have to is not.

B

An Apple one is the one that we already upgraded right, yeah and reconfigured, where Zeppo to is the one that is currently upgraded.

C

But not be considered yet so.

A

A

I hope you would see the arrow here only we can first trigger it.

A

Production Jason, oh yeah, so the load balancer is checking your readiness. Note. That's what you see here here. We have a back trace of what is happening. Oh no, our redness nodes also feels I. Think.

A

Huge back-trace, this is from grapes, orders from the API. So when we reach it, we should be able to see what's happening.

A

This is another error. I think I reported an issue somewhere.

A

What I cannot recall the issue exactly again.

A

Look at the par yeah.

B

I think either recorded in the issue and just say like it is this error I think there is an issue and we can do it later.

A

I think I even could like check out.

A

So I'm now trying to log in on a stone, the Apple to know directly to see like what's the state of it, and so it hasn't.

B

Load the assets yeah.

A

It's expecting a different lot because there was a hash in the name of the assets and that mismatches and then it cannot find them. So probably when you mean network inspect.

A

For or for these are style sheets.

A

So yeah I'm really curious how like would normal AJ installation City with this.

B

I have a suspicion exactly in the same way as we are now, because we haven't done anything yet specifically different from a normal AJ upgrade.

C

A

Let's skip this now and continue with the process yep next up, so I did to disk command and I can reconfigure them now, all of them right, yep,.

B

On all other nodes, nobody called.

A

Again, someone complaining a pause, Chris girl. That's we deal with that already. This one is the poorest world note and yeah I'm, not feeling too happy about it. But let me first look at the latest note.

B

To see what you I think this is reported. This is a known issue with redness I. Think. Let me quickly have a look.

B

Like I'm bad at the moment, finding my my issues, yeah I, know you're feeling you can have those days when you're.

A

Usually, your browser is helping you a lot, but when Sunday's day or it isn't yeah.

B

I guess I've seen this before for sure.

A

Who are making a note an issue already, then we're gonna start the conversation from there.

B

Yeah but I I believe that the only thing that you need to do is.

B

Is to actually restart or like weave on the configure I.

A

Sure didn't already some looking for something like.

B

Give me a second because I.

B

It's reddit that is no earth right.

B

Yeah I found it so you see it's the error.

B

Rate aborted commander, an OAuth authentication required.

D

B

I think you can just run reconfigure again and then it will work.

A

That's it complete yeah.

A

And it's on the only cash command. Clearly, cash.

C

Okay, well, let's treat it an issue: yeah I.

A

Meantime I was looking at post quest well and then I was taking like version that is installed, is 9, 6 14, which I think is strained, because I've seen 9 6 11 over here. Yes,.

B

A

Yeah, it seems like the the skip. Postgres version does not skip the minor yeah, but it skips the version upgrade from 9 to 10, but not from 9 6 11 to 9 for 9 640, right I.

B

Would say that is unexpected to me.

A

Well, the difference is the the database format for for doing this. This upgrade is not changing. So it's it's. There are less steps to take to do this upgrade compared to from 996 to 10 sure but yeah. It still might call start third time. I guess yeah.

B

I mean I think it's probably fine to continue running this right, as is, but you will need to restart eventually right and that will cause downtime.

A

Yeah, that's true and.

A

I'm not sure I want to say something: Scott.

A

Yeah it or not was taking. This upgrade was done for a security reason, so I wonder what or advice about it. It should be.

B

Well, I think I mean the thing. Is that I think the advice is that some upgrades, potentially even including security upgrades, will imposed on time right, but that should be. That should be known right as I know.

B

It's kind of unavoidable right, but you could say that hey.

B

If you do this, you will need to do it. You know at the time that is convenient for you, because many of our enterprise customers can't just.

A

Okay, so it keep it running and just continue with the process. I think so. Yeah, it's fine! So as far as I know we're ready with the primary stuff now or at least for now, let.

B

Me switch screens. So what about the rerunning of how unicornia these two things.

A

Yeah I, don't think they're necessary because they already did. We start so I see here: analogs, unicorn, sidekicks, already started and just as a check. So now the this endpoint works again. Mm-Hmm.

B

That's a good thing, so are these instructions superfluous? Can we get rid of them or do we need to ya.

A

Detracted any issue somewhere because four deployments and get lots of counter doing a bunch of checks whether they need to do help or not, so it check-in and a post quest and reconfigure locks. If this line is in the logs and they're checking, also I think they're, comparing like process ID from the unicorn they had before they did. We configure an accident and see if it changed and if it didn't, then they still do the hop from from the ansible playbooks.

B

Okay, so it's essentially safer to keep it in, even though at this point we don't need it.

A

Yeah but that's the discussion we had a little bit earlier about like we don't want them to do this step on all application machines at the same time, because that then it would be a problem so.

B

Maybe we should add a note saying like hot reload, unicorn or psychic services on all nodes, but do it sequentially right? Otherwise you will get downtime.

A

Now I'm thinking about it, but that's probably a bigger problem. It like the whole AJ stuff. So when we saw like we totally did have that upgrade the first application note. We saw that the second note wasn't really healthy and yeah I guess our health checks should be smart to take that up. But if you then have the load balancer not sending any traffic to the application.

A

To the second application note, we have a problem when we would do the hub on the primary so yeah, but that's the discussion you already had with Camilla I, think yeah I would say like.

B

It's probably okay to leave it. That's you know, and then there are wider problems to discuss it to to to discuss. Yeah I was rather than Valerie, I didn't know. You joined yeah.

A

I also saw participants free a little bit ago.

A

Okay, if everybody is good with this I gonna continue with doing the upgrade on the secondary multi node deployment. This is a great section, I think double check, so.

A

Skip all reconfigure all.

A

A

Plato is this really correct? I'll write a note setting through secondary falls and all non-deployed Newton's is strictly it's not strictly required.

C

Useful to have this here, I, don't know we can remove it if well,.

A

I think this this should read like a happy part to do this and you're always safe.

B

Yeah, that's there, so it's like that. We can maybe make a comment like internally but yeah. It's.

A

Probably probably like a result of a discussion like well, is it.

B

Necessary is not necessary, but if I think I like the idea of removing it, because it's already long and I mean this is not really helping. Anybody make a decision right, it's like if it's not strictly required, but we still want it to be done and have to do it in a way. And then, when somebody really wants to know, we say like well, it's not strictly required, but you know.

A

Okay, so let me so all our notes. We have to add this line to give up, give that power or be.

A

B

This is already here. Why don't you just kept this into the into the fire.

A

hmm Why why would it be, then I can see if it's already there.

A

Whether it can be like a little salty.

A

Well, we could like write the instructions to do it with cat. Well, I'm, not sure everybody will be.

B

Happy with that, no I think if I think it's better to state what needs to happen rather than give the implementation on how to do it. If people will do is like fight why's, that, pending something I'm.

A

So this all all these have like the grills autumn, a Great Falls but I think they. It is implied by the Geo secondary wall, but yeah. That's something for this for the setup dachshund. Whenever we're doing those.

B

On f02, could you just show me the music again yeah, okay, I.

A

Can run reconfigured to make sure it's supply, but pretty sure is already are we? We did.

A

So, on the deploy note, let's do an app get, updates and I need this command again and I read.

B

The setting up of gitlab.

A

Yeah, who knows what that does.

A

Okay, what's the status of our notes.

A

Maybe I should.

A

Run this on secondary.

A

A

Now, probably again hopping between the two application notes.

A

huh Interesting I guess the adjacent requests for this data for filling this page was from Dion sent to the automatic machine and then.

A

Yeah: okay: let's not focus on that, because I know that that that's the thing.

A

Deploy note we did this and I will do the reconfigure while we're skipping the particular titrations yep.

A

So this might run some Marv Grayson's on a tracking database, but only the normal migrations and know if.

B

That succeeds, which just did I, think what we will observe is in the next upgrade phase. The Redis node will sail again in at the reconfigure stage. Oh.

A

Let me know what, with an issue is already.

C

A

Looks like about the same as what is happening on the primary node. Next, we do the upgrade on all other nodes.

A

Where's my update lazy.

A

To be honest, I think you could run these two steps on all machines together. Yes, I think you can.

B

If you really wanted to.

C

A

Yeah I, don't think I would advise that because it would make holdbacks harder.

B

But I think it's safer to do it sequentially, but I think if at least yeah you know done it I've seen it once done in parallel and it was fine, but.

A

Well, if you wouldn't have had too much discussion on what was happening, then it still would be easily possible to do within our I guess. Yeah yeah I think that's fine I mean.

B

Let's not praise the day before this is extra time. Yeah! Sorry, that's true! I.

B

Think the I mean the happy path. Is you know we just follow through those steps and everything is healthy and if you didn't discuss anything like the the runtime itself is certainly under an hour. If you have the right set up, which is great, I think what I would like to to arrive here is.

B

It is a documentation that we believe, if you follow it, you know, is very solid right and then we can talk about improvements and automations and and all of those things yeah totally agree, because I mean the fact that you have one two three four five six windows open in parallel right is, from a usability perspective, a little bit daunting right.

A

Yeah, it's very easy to make mistakes so I'm not sure how what the best way to advise this would be.

A

Harder than all.

B

The meeting is yes, it should be automated or you always invite a lucky charm like Valerie, who can just be on the call and then magically everything works.

A

Is that Valerie superpower, yes.

B

It is cool, there's a German word for it. It's called the fourth favorite effect. Essentially, something breaks only when you show it to other people, and so Valerie superpower is to counter the four fury effect.

A

Okay, let me run reconfigure on these machines. While we keep talking I.

B

Predict lettuce, failure.

A

Little bit spooky to do it. Do it do it normally configure all the other machines also yeah.

A

Yeah, we really need to get rid of this, but I'm looking to enter that issue yeah. Hopefully we could put it in the github RB and then yes, it makes a lot of sense.

A

Yes, this stuff Redis fields.

A

Okay, so let me double check unicorn and psychic did restart and for some reason, also the jewel or curse it restored. That's the same here, so we didn't explicitly.

A

Shut it down, I, guess and then it's it's when it's running hardest, intentional Aria rights for your running the lock cursor on both app nodes in this configuration. So that's correct.

A

Yeah awesome yeah.

A

It's it's weird because, like you'd like to see here, giddily only giggling notice, also how to it automatically so I would expect. Like kidnap reconfigure knows when to help things. Yeah I mean that makes sense in a way so why these instructions are still here, I'm, not sure, but we've seen these I've seen this one so that cool so I think we could skip it for now and move on with the post-deployment migrations and checks on the deploy node on the primary cluster.

A

So we have to go back to this window and take this one. Okay, give me.

A

Your partition, yeah I, was the lone e-minor.

D

A

Yeah, nothing wrong with I want to make sure like that.

A

I was expecting it to be completely healthy. Now.

A

Why is it still reporting the old Frisian.

C

It's a good question.

A

Syd cashed, this one is good I.

B

Think maybe the future.

A

Oh yeah, if there's time out, there's like the caching things still happening, so it might take you a minute or five.

B

I think I've never seen this error message here, but it is actually completely correct.

A

The database version does not match the latest migration.

B

What are we advise our customers to do with this message? Well, if you follow the instructions, this is after the DB migrations on the primary. If you run the exactly that command on the secondary yeah I think it's.

C

Oh yeah, that's.

A

True yeah I was expecting, it would be like with a but that's a DW. That's not a thing, and now the status has not been added for the past five minutes.

A

B

Will go away eventually and I know it's like I've had that when I upgrade like single, deep Long's I was just like this. Is this bad? Is this good? Why is this happening? Okay, it's just fine! Okay,.

A

So I'll go back and continue with the post migrations.

A

Yeah, it's really hard, couldn't not single user, that the things are unhealthy, but P I, don't like a shrew catch. The cat, I guess like.

B

Something in between not healthy, not unhealthy, but something yeah. It's like I think that the user interfaces essentially says things are problematic, but they're not really problematic, because we are in the middle of an upgrade right and I. Think that's that's, maybe also potential improvement to say, like hey, you are you're upgrading at the moment. You know like this is not the. This is not unnormal yeah.

A

But I wonder like I can't imagine when, when when the admin administrator is doing the upgrade, they want healthy, healthy notes at some points to his process to make sure that whether they have to continue or fix it fix the state before they can in.

B

A way the UI is not a correct reflection of the state that the system is in.

A

Yeah and well I think one thing: maybe that to note is the the caching I'm not mistaken. We remove that already in this release or the previous one, maybe also but you're, not in the one. We were installing right here, so that might help also guess maybe.

D

We need to have some service mode or something where we handle those messages differently than in here.

B

A

Potentially yeah, we can think about it.

A

Okay, what's next verify so all deployed note, we do.

A

A

This indentation seems to be wrong here.

A

A

Yes, he labeled yeah, we've spoken about. We have an issue about this.

B

A

We want this I think this is yeah.

A

And all the rest yeah our rest looks good. So now we move back to the secondary nodes and do the same thing but with the tracking database. So.

A

We only check that on to deploy node. Is that correct, because that's a little bit confusing for me I would expect we do this on all machines, but that's.

B

Not how I read witness I am I, am an agreement with you. I read it as you just check it on the deploy node. Oh.

A

I I think it would be interesting to run it on all nodes anyway, it does not hurt. It's just nice to see like everything is fine.

A

So meantime, also, the post deployed in migrations on the tracking database did happen and create.

B

Table well I think we now found out why why we only run it from the app surface.

A

Yeah right I think it's just a configuration issue. Well well, yeah. It doesn't make sense, surrender Sonic, Italy. Note, that's another thing to note.

B

What I think, but you know, there's maybe some cleanup to do.

B

Kidnap about what well, if I, see this now I'm, like the exceptions everywhere that I don't understand right mm-hmm, but actually it's fine. So if you run this on the given B note, it should either not work and say like hey. This is not available on a get to you. Node or you know, actually produce a different output, because this I would say this. This would result in a support. Ticket right people would say I rent this I got all of these exceptions.

B

What's happening.

A

Yeah, this is really weird like this is not relevant for a gatling know to do the authorized cookies, command and stuff like that. I.

A

I'm pretty fine for now to just document that, yes, they shouldn't run it like that. Maybe she should run it on on all application notes. Yes,.

A

That's still interesting, I haven't seen this one before, but I just consider it a hiccup I'm not really blocking for this moment, so the documentation makes sense. This point that it does not do this on all notes which yes, one thing for the deploy note: how are people selecting a deploy? Note I think it can be any node yeah. That's that's not entirely true, because you cannot do it from the giddily note, because you don't have a database connection, for example, so.

B

Should we just say she was an application note because I've never seen anybody else choose anything other than an application owner.

B

Yeah, this is not entirely correct, so should we just say it can be any application load.

A

Well, I was thinking of that's that that's correct, I'm, not sure. That's the best choice.

A

Like maybe, if you like, have a set like a sidekick note that I think that would also be possible, and if you do that, then you would maybe not break the application notes like we have seen with app 1 and the app to that one of the code was broken, but yeah.

A

That is just what what what stated here is not entirely correct. Okay, I would yeah, I'd, probably safe, to say an app count. Yeah. Let me make a no.

B

A

That's the same as just a non zero deployment pick a note. It can be any note should this is also not correct. Okay, that's why we have it.

C

B

Think what we should do is like advice for a reasonable default right. It's a like sure. You can do it on a sidekick note or whatever, but just choose an application or if that is like, let's say, o, be an obvious choice.

B

A

Yeah, we don't have it at this moment, but I know Stan note somewhere that when you want to the migration should also need to run that on a machine that has direct access to the Postgres database. Of course, if you're using a load balancer for Postgres, like you like, or a red line repair yeah daddy, then that.

B

Could also break and what node has direct access to phosphorus.

A

Yeah, that's the best Postgres itself, maybe.

C

But that doesn't make sense so yeah.

A

After they get a little bit, yeah.

A

We can do that later. Yeah I was looking at like the post-deployment migrations for geo and I'm really surprised that, like create table that this happened in a post-employment migration. I really doubt that.

A

A

The upgrade like we did, it did not do like the normal migrations for the tracking database. For some reason you see if I could find that migration I.

B

Have to admit to you, you lost me now because I I can't answer that at all.

A

Yeah, so this this migration at the container repository registry is a normal migration for the tracking database and normal relations normally should happen if during the upgrade.

A

So if you fall back to the documentation, when you do this, it skips the posted program- migrations, but not normal migrations or shouldn't I. Didn't do that or is there a miss configuration here.

A

Nothing particularly in here or that states that used it should keep migration sap.

A

So very curious, where that goes from.

A

Second, dairy wow: that's too late, now to check if it was heightened before so. We have its reports.

A

Container is there a container status in this endpoint yep? It is.

A

Okay: let's make a note in home.

A

They took migrate on the secondary note. Now we can run like the Jewish tech command. I'm not gonna, do these forms because they will fail anyway,.

A

Okay, what do we have here? Yes, yes, yes, and now we have a foreign data records out of date. What I did we didn't expect? Yes,.

C

It's fusion home this.

B

A

B

Is wrong and we've written.

A

Other than that, everything is fine, except for the laughs DW that we like put explicitly not, and these documentation.

B

Are no I think the question was if we wanted to fix it for grammatically, but I think we won't be able to do that at least for the first iteration yeah and so I think we will just have to put it in after Tod, be migrated and essentially migrate. The phone data refer tables so.

A

Now I have to look up to troubleshoot the shooting.

A

This one, so we can do that right, yes and then again will we have will have a caching issue that says it stays out outdated for a little while.

A

But other than that I think we're up and running I waited succeed. Oh wow.

B

I think you, it needs to be run on the on a specific node.

B

A

Not allow library.

B

So should we run it on the tracking, yes, I believe so.

A

So this note did not load the Postgres extension or how should have she see this.

A

That also failed.

B

hmm On what note I thought I had I had notes on this I can't quite remember.

A

It could be related to the post crash that should have been updated or restarted.

B

It shouldn't be, but I'm just wondering where.

B

Let me just try.

B

I'm not sure where, where this needs to be run,.

D

D

Believe it should should be run on a application, not because it is.

D

Can you scroll up the message on the application? Not if I don't see the actual error message? Oh okay,.

B

Can you I think I have a note, can you we run the command on the application node? So it's just sweet run it yes,.

A

Is it fail again it's so loading built.

A

Same error: okay, I.

B

Think the note that I found.

B

I think Douglas restarted it on the deploy node and that worked I think.

B

But I'm not just deploy note exactly.

C

B

I think that's the only note I have of this.

A

Now, I really have and like enhanced errors, like a version mismatch.

A

Because of the nine six fourteen, ninety six eleven.

A

A

Tracking database, what version visits.

A

I'm not sure how we can resolve this yeah.

B

Me neither I mean.

A

So this is the same executable. Okay,.

A

So shall I follow the instructions and maybe try to restart PostgreSQL. On this deploy note I.

B

I think that's.

A

It shouldn't be shouldn't matter, but I out of options now.

A

Maybe we should do it on the Postgres note itself, I think that's yeah the thing! That's the problem, so it's so! This fourth class note is still running 611, 96 11, but the library for happy w was updated and it's I think it loads it during run time and then it's there's a version which match match between the Postgres version that is running and a post quest and the library version that is installed. Yeah.

A

Or should do it or not tracking, oh right, Wow.

B

Sorry I think it needs to be run on the application node from my OneNote that I have from a call with Douglas yep.

A

You mean there the refresh table command, yes, yeah, that's true, but I was wondering whether I need to restore it to Postgres node or the tracking database and I think it should be the dragon database, because it's that one, that's loading in the data from the main database and it made database doesn't really know aw. Okay, but let me run it first without doing the tracking database. If it fails again, let's switch towards the tracking database and if Stan fails then I don't know.

B

But does this impose downtime.

B

Okay fill again.

A

Well, a little bit I guess: I'm, not sure how fast approach Chris no twist towards.

D

Yeah I think it depends because it depends on what would see tables. We have.

A

Yeah, but follow that we start of poor screen should be fairly quick. Isn't it.

D

Sure it is I mean before the very uncommon DB refresh for in tables. We can have some downtime on secondary no, but it shouldn't be like very critical, because in most cases we didn't change things too much right on.

B

A

B

So it was the mismatch of the of the running Postgres with installed library version. But now do we are in the situation with the secondary is using a different database version compared to the primary different like.

A

Micro version or patch version- um oh yes, okay,.

A

Well, it's like the day. Today's format should be compatible between those four regional it shouldn't matter, but.

B

I think then, like I, think there are two things to follow up here right. We we have no documentation on this behavior and skipping. The Postgres version actually also like the skipping. The Postgres version update does not actually skip upgrading the Postgres version, which is why we are now in the situation where, essentially, we are requiring a restart of the tracking database, and that is not something that should happen right. I would argue.

A

B

Because I think, if you know if we are saying hey actually between twelve point, one and 12 point to you know, we are upgrading the Postgres database anyways, then that upgrade will impose a little bit of downtime and we should let people know about it. Even though it is not nature right.

A

Yeah, but that's also again, I think that applies to any age, a salad even on here. Yes,.

A

What we mentioned like when we were doing the upgrade on primary, then people need to restart progress sooner or later, but they have to do it at some point. Yes, if you have that figured out time, we can look into and to the rest of it.

A

Okay, yeah I'm, doing the the healthy state to be just outdated, I.

A

Would assume the note is healthy.

A

Did I do that? No.

C

You didn't actually.

A

You mean this one yeah, but also that one returns, the cached cached value, unfortunately I'll at least for the foreign leader rappers.

A

See or notice it does not appear to be replicating database. From a primary note, oh.

A

That's interesting.

A

So did we break the database replication because we upgrade that really started to Postgres server.

A

huh If that's correct and.

A

My previous statement about the versions are compatible. It's not true.

A

So shall we restart the Postgres on the primary note? Also, yes,.

A

A

Wish there was an easier way to see or on a faster way to see the replication lag.

A

This is our downtime where Herrera spoke about it, so how long does it take before to restart of Postgres.

B

Well, I think it's for me: it's not so much about how long does it take it's about? We we make this happens, it happens right and we make statements that say we do those things and it doesn't look like they are. They are true for that specific update, yeah I.

A

Hope they we connect automatically.

A

Everything is broken yeah. Probably this note also does not update.

A

Is not recovering our.

A

Opening connection.

C

A

A

Yet so this page or it originated from the load balancer, so it also detects that boat application notes are not healthy at this point, which is already okay, that it does that, and we know the load, balancer is working and the health checking one I.

A

Don't know why it's not recovering.

A

No implicit conversion from you know into string, I think.

B

We saw that it's at some point in the car before hand, and then it was fine or I. Don't.

C

I, don't recall any more.

A

Yeah I didn't notice. It.

B

So then we have to restart get lap here or yeah.

A

I would expect so I'm out of options over rice.

A

A

So, when I would check this rhyme.

A

Yeah, but it takes a little while takes maybe two minutes before unicorn has started correctly.

A

Database is still lagging behind I.

A

Have no clue what's happening at this point, I.

A

Am Not sure in three or maybe just better to end the call here and look into it a sink.

A

The Postgres version is now are just a secondary now it yeah. Finally, no it's unhealthy, so we cannot look into that. Oh.

B

Sorry, I'm, babbling and I was muted, I wasn't looking some I didn't see it yeah no I, yeah I, think I, agree, I mean the only thing we can probably do is restart everything on the secondary node or the primary node and see if that resolves it, but other than that. I think we need to do a sync.

A

Yeah I'm really bit worried that also like database fake blood vacation is broken and it's it's not good. I.

B

Think we need to follow up as soon as we can on your team, because I don't understand holy cow. This is happening.

B

Yeah I agree: that's all the time: the Collier okay, thanks stone for all the effort and yeah. Let's follow up on the individual issues.