GitLab Geo Group, 30 Oct 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Upgrade Geo HA installation from 12.2.8 to the latest 12.3 package

Description

The Geo team performs a Geo HA upgrade as described in https://gitlab.com/gitlab-org/gitlab/issues/32437

A

Like welcome everyone, this is a Geo upgrade mo. Today we are going to upgrade a Geo H installation from 12 point to eight to the latest 12 point we package, which is twelve point three five, so I've gone through this checklist here, we're recording a meeting I, don't believe there are any version, specific changes that we have to be aware of just double check. Nope.

A

The perfect post was here, version is at ten point nine we upgraded that a couple days ago. So that's good I, don't believe, there's any upgrade warnings specific to to this either and we can quickly look at the health of the pre upgrade cluster, so we load we're currently at twelve point two. Eight, the primary utility worth point two eight here on the secondary: it's also healthy if I'm not mistaken. The database version here is ten point nine and on the secondary.

A

It's also ten point: nine, so I think we we are good in that regard. I have retrieved a beverage of choice within the drinkable temperature range, so my I have my tea and.

A

We are recording okay, so because we form the upgrade steps crafting the latest documentation, so we recently upgraded added new documentation for this scenario here so I'm going to follow those steps and perform them all in a row and I'm joined by Gabriel, who has to rescue me in case I have any questions? Okay, so first things, first I just want to make sure that we have this file in every on every node, so in the primary, let's quickly check. If this is present.

A

So I set this up in such a way. These are the two application notes: a post cos mode red is Italy on the primary and then here on the right side is the secondary in the same order. It makes it a little bit easier to follow. So, let's just see yep.

B

Yep, that's also there.

A

Yep yep and that's also theirs. Okay, so we have. We have this another thing that I quickly wanted to check.

A

Actually is my if we are disabling the Postgres upgrades so I just check if this file is there as well.

B

A

We go. Oh, this seems to be interesting. Interesting scaling model for I turn one second, when.

C

You try to try.

A

To use Maximizer, it will interesting here. I will.

A

That's much better.

A

Okay, so let's just check yeah, that's also there.

A

And reduce the.

B

France I see a little bit.

A

Now do the same? Actually it just over here.

A

A

Okay, so that's all good, so we want to see any automatic post press upgrade and I, don't think, there's also a newer version than 10.9 available, but I'll just make sure that that's the case and that this is here. Hey Douglas I thought it's a bit too early for you.

A

Okay, so this is present, we are not upgrading the database so I'm on the primary.

A

Let's look on all other nodes, not the deploy node, so I have to designate a deploy node and for me I'm just going to do the app server, 0, 1, so I know I'll check that this is not present in gate lab RB on everything other than the or that this is present in everything other than the deploy node. So.

B

A

Ok, so here that's ok! This is the deploy node, so we could then either Auto migrate is set to false. So that's correct.

A

Auto migrate is false, so that's also good.

A

Again, false good.

A

And this is also false, okay, so this is true and now we're going to reconfigure everything just to make sure that we, okay.

A

A

Okay, so one one thing that is a little bit peculiar is that it says here: the version of the running PostgreSQL service is different than what is installed, but this is the app 0/2 server and it doesn't actually say this on on any other node. So I'm, not 100% sure why that is. Do you have any idea.

A

We're running Postgres 10.9.

C

No, maybe we have had a minor version of great as well.

A

No I don't think there is a new version other than 10.9. So if.

C

You do like PG I, think version of what does it return.

C

I think I have to go like opti, it lab, embedded, Bing and then BG.

C

Pps Kelso kiss kill.

C

A

So I think, if we compare this to this one here.

A

A

Okay, so I think what is happening is we. We upgraded the Postgres version to ten point. Nine, but I don't think we restarted the app servers and.

C

The app service is still relying on the old version, so we just upgraded the binaries on the nodes that has those groups I do not so he notes that doesn't have those grace. Yes, so so with that, this is interesting. I think the current code, dozen.

C

Doesn't have any way to check that, so there is no way to perhaps trigger that are not known. That I am aware of that's. That would say. Yes, this should upgrade the binaries to the new version.

A

So the first question here is.

A

The Postgres server here right, I think we, like tonin I, only upgraded Postgres on on the actual, like server where Posterous was actually running, because me we thought that was the right thing to do.

A

Do we believe we actually have to upgrade post curse on the application service as well, because it is not use. Is that right.

C

Thus, we don't have the server all the applications, but we rely on the client PE application to do some queries on the main main one to figure out a few things. So.

A

We actually also need to upgrade the post based versions here. I.

C

Believe, yes, if we are getting error messages, then it means that some code in homnibus is relying shelling out to the post chrisfite okay, to recover some data from the database. Yeah. Okay,.

A

A

Let me pull this up, so these were the step stone and I followed roughly, but only on the on the Postgres field and tracking database services. So I suggest we just run Zulu. So good luck, control, post press upgrade on the app services. Well, yeah! Let's try as a prerequisite, for you know the actual upgrade.

A

A

Yeah, so it definitely upgrades the post quest version on the App servers. Well, it says here to 10.9.

A

I'll do this in parallel on the other application server as well.

A

A

A

Okay, oh it's finished. They say the replication has been shut down. So let's see if that is actually true.

A

Go to the admin area here.

A

Okay, it looks like both the primary and the secondary are still healthy, so I reckon you know. Let me just add a project.

A

You, okay, if we go to the secondary.

A

Okay, I think the replication is still working, so the message is not 100% correct, because it's actually only the application service, not actually the replication on the Postgres service.

A

Okay, I think. The only question then is to me we should probably do the same thing on the the secondary as well, because it is probably exactly the same yeah.

A

So I think we need to upgrade Postgres on the secondary too so that that actually matches any objections.

A

Okay, go ahead.

A

A

Okay, so on the secondary, it actually complains that there is no omnibus managed post press on this instance, which is correct, so this is a little bit different.

A

A

C

This this is interesting because, if we're not using a managed instance, we still want to upgrade the binaries. Yes.

A

So I think what this kind of says is on the primary I can easily upgrade post Chris on the application server, but on the secondary. This is not support, supported by by thomna bus at the moment, in the configuration that we're using. So that may be a problem right that we like I'll, make a note.

A

But I would I would suggest, given that the nodes are healthy.

C

Yeah we could try to continue.

A

It yeah I think we should try and continue. Let me just.

A

Yeah and I think the okay, that's all fine okay, so that was a quick detour, but we will continue with the the actual upgrade okay, so we've run reconfigure on. Essentially everything we've checked that this is here so now we can actually update and then install the package that we are that were interested in oops just past my there we go so.

A

And this is on the deploy node so on my first applications over here and it says, I should update and then I update to a specific version of Gitlin. So I wrote that down somewhere.

A

So we're going to install a 12-point 3.5, which is the latest version and I'm doing this on the deploy node.

A

Alright, it looks like this is complete. So, following these instructions, I'm going to skip the post-deployment migrations.

A

It's going to run the package update, there's no harm in doing this now.

A

Okay, so this worked, so we did this and now on all other nodes, we need to install the right version of get lab, so I'm just going to copy this here and I'm, going to run this on everything else and.

A

That should be the next step.

A

Oh I'm, actually like muted I'm babbling by myself, so the authorized crow fauna is. It takes a little while I recall that, but.

B

This is all okay, this is okay.

B

That looks good too.

A

Okay, next step deploy node up, get update, so f 0 1, the the deploy note again. So let's update this.

A

B

B

Let's grab a quick package.

A

This needs to be the same package version as on the primary. Obviously, so we will do this.

A

A

This is just the packages, so it just needs to be done. Saves us a little bit of time.

A

Waiting for this, let's peek ahead a little bit skip post-deployment migrations again and then, after that, upgrade the packages on all other nodes. Reconfigure, reload everything, including the Geo locator.

A

A

Okay, that worked, and now we do this.

B

A

Super hey that worked, and now we just need to update the packages here and install them on all other nodes, so I'm just going to copy this command.

A

Just to make sure we did this, this okay.

A

B

All right, the first two are already finished.

B

The red is known as well.

B

Let's just be patient.

A

So, okay, so now, let's reconfigure.

A

Italy finished here: this is the Redis issue again, so we have to do it once more.

B

This looks good.

B

A

Okay, all of those worked so off to the next step on the application servers again, we start amico in sidekick and gee willikers.

B

A

And Gabriel tablets, the lock cursor, also runs on the application surface that correct.

C

It can run anywhere I.

C

Don't know what what's the current recommendation that.

A

B

A

That was this: let's just check into our administrator interface and see what what is happening here.

A

Okay, so we are recognizing that we updated twelve point three five. This is good, but we're still not healthy the foreign data reference out of whack but I think in general you're, not looking too bad.

A

So on the prompt in the secondary, so the web interface is already working, so we'll sort those out. Okay, let's look at the upgrade instructions. Okay, we finished everything on the secondary now post-deployment migrations and checks so deploy a node on the primary cluster. We run this so I have to go back and app server 1 on the primary key be migrated. So these are the post deployment migrations we skipped earlier. So I'll do this.

B

Followed by a check.

A

Okay, that did a bunch of equations. Let's check what the Geo statuses.

A

Okay available and able not configured yes, yes, checking. You finished that works pretty good. Okay, deploy node on the secondary cluster post deployment database migrations, so the deploy node on the secondary. This is Epsilon.

B

A

That didn't do anything but fine. Now we need to refresh the foreign tables. This was the area we saw in the interface.

B

That worked, let's check Joe.

B

Okay, so let's see.

A

Okay, this is interesting, so it says the foreign dental records Kimura is not up to date, but I believe that this is cashed, and this is a known issue. So this may be wrong. What.

C

What does their web interface says? Yeah.

A

Exactly so, let's see no.

A

It says the same.

A

One second, please.

A

A

Sorry I needed to take something: let's just try this again.

A

Yes, as I thought, this was cached, and so it wasn't quite right. I think you look at the web interface yeah sales, you know, so this is actually something that we're working on for 12.5 and to make this more reliable.

A

So that's maybe something to take note of and then mmm updating, Geo status.

A

Okay, we're healthy repository synched to no replication lag. It looks.

B

A

Good, let's go and add some some things to my test project just to.

A

Make it if you.

A

Let's just submit it.

A

And now, let's see if this actually gets synced, the secondary is also reporting healthy. So if I now go to my projects and the.

B

A

Project here there we go Atlas and Gabriel. You are awesome. You can close this. What I can I'm one of the secondary another problem, another time? Okay. So if we recap I believe at this moment we have successfully upgraded the Geo h-a installation from twelve point, two point, eight to the latest twelve point B, which is twelve point three point: five everything is healthy. We had a little bit of database shenanigans in the beginning, which was the leftover from the upgrade to ten point nine, but that was actually fine and.

B

That concludes the upgrade demo, so I'm going to stop the recording thanks.