GitLab Geo Group, 31 Jan 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Upgrade Geo HA installation from 12.5.7 to 12.6.6

Description

The Geo team performs a Geo HA upgrade as described in https://docs.gitlab.com/omnibus/update/README.html#multi-node--ha-deployment-with-geo

A

Okay, so we are going to attempt an upgrade of a Geo H, a setup from it's currently 12.5 points. Seven I believe yeah to the latest twelve point six package and we're going to follow the instructions that are listed here.

A

Let me get to the right point here and just a note: I have a pipeline running here. That's just running tests against the cluster throughout the upgrade we'll see how that progresses as we do the upgrade so starting with the first. It says we need to nominate primary deploy, node and a secondary deploy nodes, the primary one we the first app node here at the top okay and we'll get to the secondary node. When we get to the second step, um here's first updating the Geo primary create an empty file, so we didn't upgrade yesterday.

A

This file should still be there. Let me make sure.

A

A

Okay, so on all other nodes, excluding the primary deploy node ensure that get lab rails Auto migrate is set to false. Let's do that. I said all of them.

A

Except the primary node.

A

Okay, that's a false.

A

B

A

A

Like do you know the damage or classified yeah.

C

So hit forward slash and then you can type string there.

C

And then, if you want to find the next instance of it hit enter and then you can hit n, oh okay, so that was the only one I.

A

Was I guess it was? It was actually commented out.

C

Yeah from the template.

A

Sorry cut out: that's.

C

Also researched.

C

Games Treves for some that will attempt to a.

C

Did you catch that.

A

No but I think help sorry, it's that it's my sump up, the internet connection is not very good. I.

A

Think I can just add it here at the bottom that sound.

C

A

Think it's comment it up for.

A

B

It's false: okay,.

A

Just check what it is here, true: okay, that's the deploy note. So it's okay, um so on all of their notes, excluding the primary 2.0 ensure the notes are running the latest code, so I'm going to run reconfigure.

A

Okay, so the next step is only for the primary giddily node and it says to update the get lab package so I have it here. The right version copy it. So this is the good ol e node.

A

In the meantime, since the next step for all the other nodes is to update I'm just going to run that to save some time.

A

A

Okay, so it's finished now I'm going to run reconfigure on the good Olli node.

A

A

Okay, so the next step is on the deploy node, which is this one here. It says, update or install new version.

A

B

A

No, never read this copy this on the deploy, node.

A

Okay, so this I'll mark down this has been known to be a little flaky, but I'm gonna mark down when it failed. So this was when we were running the Skip post-deployment migrations on the deploy. Node make a note of that.

A

Okay, so the next step is on all of the notes, excluding the primary node we're going to install the new get lab package, stay updated one. So I'll run that on the second up.

B

A

We've already run it on the giddily node so over to the Redis node and the Postgres node.

C

There there is a thing that we're not where I suppose we're not doing truly zero zero downtime yeah, because the primary deploy node is one of the nodes that is serving requests. So, if you are using that node currently like in the web UI when you like uploading attachments and stuff, then that could fail before the migrations are run or finish. Okay,.

A

Yeah cut some other feeling here, oh okay, so this is during step, one they can hook here that went all the other non deploy notes were updating.

A

Let's see what's going on.

C

C

It's not really zero downtime.

C

It's kind of hard to say because I like we know that there is one way in which we're not doing zero downtime so, but maybe for next time we need to remove the primary the deploy node from the load balancer before updating it. Mm-Hmm.

A

Okay, well I have this video itself. I'll know exactly when this yeah.

C

A

Take a note of that later, but it looks like all of the nodes are finished. Updating, so just keep going. The next step is to run you configure, so that would be on here.

A

We've already done, giddily, it's got Redis.

A

You know what I'm gonna see, what happens? They kick off. Another one see how things go.

A

You can at least get some information for the rest of the upgrade most probably yep. Okay, I'm gonna start another one.

A

Let me start the other one. First.

A

Don't you know what I have to stop this.

A

Smokes sweet is still running just take, no okay, I actually have to.

A

Make sure that that step runs otherwise Sophie there might be some conflicts, they might start deleting things from the new pipeline, so I'm just like okay.

A

Well. In the meantime, let's continue so that was, we ran reconfigure on all of the non-deployed looks like they're all done so for notes that run, you know coroner sidekick. So that's the app notes, hot reload, unicorn sidekick. You.

C

Know I noticed during the reconfigure it said.

B

C

Restarted unicorn and maybe sidekick as well.

A

So, oh so maybe we don't have to do that. Step. Yeah.

C

But it's also kind of weird, because we hop unicorn, because we don't want to hear yeah. We we hop unicorn in our instructions, because you don't want to interrupt things that are still running well, but reconfigure did a full restart of it. So that could be another problem for a zero downtime.

C

A

C

We have a document that for this or should I just like post comments and the issue can.

A

Post comments in the issue: okay, okay, I'm still gonna run it cuz. That's how it instructions read right now.

A

A

Okay and those are all the steps for the primary note, so let's see what's going on, okay, it's this!

A

This is the admin.

A

Ok, so this is updated. This should not be we're. Gonna do that right now,.

A

A

Okay, so create an empty file, should already be there from the last upgrade, but check.

A

In this, for all notes, including the deploy note, the deploy note is going to be the first app server.

A

Okay: okay, excluding the deploy node, which is the first one, insured geo secondary bottom migrate and set to false.

A

Out of migrate, it's false okay,.

A

Let's just check what, if it's here.

A

A

It didn't say to do anything explicitly for geo secondary Auto, migrate on the deploy, node and there's nothing like explicitly set.

A

Do I need to explicitly set it, not any instructions for.

C

Get library auto migrate instead of geo second Arianna migrate,.

A

Sorry, can you say that again, I was checking for this one, but you're saying about they get lab rails Auto migrate, oh yeah,.

C

I was just asking: if that's what you're asking but yeah we do, we do need geo secondary Auto, migrate, false right, oh.

C

You're saying on the deployment I'm.

A

A deployment yeah there was no explicit, detect them step for checking it, but I just wanted to see what it was and it's not there. It's not explicitly like set.

C

Right, could you scroll down a little bit on the instructions where we update the deploy, node sure.

C

C

So we expect it to run the migrations.

C

On reconfigured.

C

But when you ins, but when you update it reconfigures right, anyways I, think it's okay, I think.

A

Is? Okay? Yes, don't we chat this yesterday too? Okay, let's go back, make sure you don't skip anything. That was this step and then so. Excluding this that point out, I'll run reconfigure. Do that now.

C

And I'm a little confused because I.

C

Don't I'm not sure why we would update all the non deploy nodes? Oh no, we didn't. We didn't update them. Yet. Okay,.

A

No we're just checking that setting okay.

A

Okay, so now in the giddily only note we're going to do the update and install me here and I'll run update on the other ones, and then you turn.

B

B

A

Can start another one.

A

Just make sure.

A

Okay, so we were on updating, Gili node and it's finished. So let's reconfigure.

A

Okay: okay, now on the deploy, node we're going to update or install git web.

A

This one up here.

A

A

The majority of these tests are running against the primary node, but some of these well, the Geo ones, will always involve replication to the secondary. mmm So we'll see how those run.

A

Hey so now here, when this command.

A

Yeah, well, although this is for the primary so.

A

Okay, because that was a deploy node and now we're going to update the rest of the notes.

B

So that's the second.

A

Half node we've already done get Ally.

A

A

A

Okay looks like well done um so when we configure.

A

We've done that one.

A

Okay and then we start these on the app notes, even though we mentioned that might already be like it will be already part of the reconfigure.

A

Okay, okay, so that was the end of step two now we're going to run post-deployment migrations and checks, so we're going to go back to the primary deploy, node and run this. Let me go back.

A

Okay, verify Geo configuration.

A

Okay, you see Miss before machine clock. I have.

C

Yeah I have yeah, try it again.

A

C

That but I have never seen it be a problem.

A

Okay, so I know um interesting mm-hmm, so this step here I think this doesn't even be corrected to be all on all deploy nodes, excluding the primary deploy node, as opposed to, as opposed to all notes, we're gonna, run yeah.

C

It's just yes,.

A

It's already a barge that.

C

Did you just say on the second very good point.

A

I'm gonna run this on the secondary to point.

A

A

So the next two steps are to wait for the primary migrations finish and the migrations to replicate your defined data replication like 4-h not listed on admin area geo. This these weights help ensure that the FTW tables are up to date. So here whoops.

A

This is updated still saying.

C

Zero yeah in practice, especially for our demos. This should be near zero.

A

C

So yeah it should be okay,.

A

Okay, so we the table should be updated by now or what happened yesterday's we saw this is zero, so, okay, I guess we've waited where we don't have to wait. Yes,.

C

A

Let's see, the next step is to run reconfigure, leave the.

C

Only thing is like if.

C

Post-Deployment migrations on the primary have.

C

Well, yeah, you can. You can see because, like in that earlier step, where you ran the post-appointment migrations yeah, you saw them finish mm-hmm, so it that really should be done. So, okay,.

A

Let's see so we're gonna run this on the secondary, deploy node this one.

A

Okay, oMG Oh check.

C

While you're waiting for that, could you do a find in the output of the reconfigure, just above that for refresh.

C

Nothing, huh what about FTW or foreign.

C

mmm I see our check says we're out of date.

A

Yes, this is I.

C

Don't remember what the okay so in in reconfigure.

C

It skips running foreign refresh foreign data, you know whatever it is it skips that if, if it you know if, if it looks like it's up to date,.

C

So we can go up and I and find where it skipped it if it did skip it now, apparently I, don't remember what that's you know looks like.

C

C

Why don't we run the gitlab geo check again and then just kind of like look up in the output for before whether it skipped doing the refresh.

A

See sheriff I was actually just like searching all of the know.

C

Oh, look what oh, it's happy now or I know it's not.

B

C

Yeah, can we can we if it's.

C

Maybe you could copy, could you copy the output of the reconfigure and paste it into you.

C

Into the issue that were doing, this upgrade on mm-hmm might be relevant to.

A

Okay saved it in a text file, so I'll post that um in meantime hold on.

A

Okay, shall I render refresh burn tables rake tasks.

C

So what I would like to do before we do, that is run reconfigure, one more time and.

B

C

C

Okay yeah: let's.

C

Let's run the refresh, then.

A

Okay, no okay, thank.

C

C

Well, we got there, but we had downtime. It seems like and.

A

C

Refresh thing is still unresolved: I'll take a look at that.

A

I'm gonna stop the recording.