GitLab Geo Group, 28 Oct 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Geo PostgreSQL Upgrade 2019-10-28

Description

Geo PostgreSQL Upgrade 2019-10-28

A

Okay, thirty minutes late here we go, we want to upgrade Postgres or qha stands installation from nine six. Fourteen to ten nine.

A

You wrote a bunch of things down fabian about how you expect it. We would have to do this up the weight yeah.

B

But it may be completely wrong right, so maybe he would quickly talk about it's. My my thoughts were in the GAO cluster I. Don't believe we have a distributor, Postgres database right, so there's only one post waste node. Is that correct at the moment? Yes yeah so I, don't think a lot of the instructions for upgrading a get let H a cluster for the database apply I did nothing, and so, if I look I looked at the Geo high availability or the Geo upgrade notes.

B

And I thought what we, what would likely need to happen, because that has happened also for the minor post twist upgrade is on actually not so true, but I think we have to upgrade. We first have to find the replications, lock name fused on the primary.

B

Then we have to upgrade post Chris manually on the post person node in the post press H a in the term primary we'll see what happens but I anticipate that should upgrade post close to ten point. Nine following that I think we have to upgrade the secondary Postgres and also the tracking database, because they need to be all running the same post, best version, I think but I'm not sure on the tracking database stuff.

B

They do. You know so.

A

That's from that's.

B

A

And five, four or five Oh.

B

C

B

And four actually.

A

Well, the order I would expect her shouldn't. We replicate the database before we do the upgrade of the tracking I'm, not sure.

B

So this is why I'm not sure, probably.

C

B

Best best to wait with the tracking database after we've fixed all of the replication I think mm-hmm.

A

Okay, that makes sense um so when you're upgrading an H, a cluster, let's consider like the primary in the secondary together, be an H a cluster. Unfortunately, this documentation is using a rep manager you're not using that, but.

B

If you go down to, if you scroll down I, think there should be the upgrading a tu instance and so I I have essentially it adopted this for an H, a situation without upgrading gitlab right, yeah.

D

A

B

A

For this pony to PostgreSQL yeah, so if we would start with checking our application slot, there is a secondary underscore and now yep. So we can get started with running PG upgrades on.

B

A

Primary that that would be logical first step anyway, I think.

B

So, okay and see what what actually happens.

A

So I'm secondary the.

B

Upgrading post goes to ten point: nine.

A

Mm-Hmm, okay, oh yeah, sure.

A

These are the the dots of doom yeah.

A

It's funny how they UNAM neighbors, have different ways of help. Make people be cautious, like the replication you have to type and replicate with this. You should call hit control-c if you're, not laughs, if you're not agree, but here we go.

A

My take of all because it has to like migrate the data to do new, post quest and format, but.

B

Then again, there isn't that much data in there right, yeah.

A

Yeah, because last time when we restart the replication was pretty fast, so you know I agree it's not that big and.

B

It's also running reconsider already so I, don't think we have to be configure anything manually, yeah.

A

It's doing that right now, so so we were doing I'm, okay,.

A

So the primary is down now, which is expected mm-hmm exactly.

A

Okay, oh, that was pretty quick.

B

And you go to be yeah yeah.

A

I was about to okay. Well, that's not bad! That's not bad! Let me quickly see what it has been doing. I'm always curious to look in the logs of the reconfigure kit. Lepke is not that important, I, guess or related I think.

B

It's also fast because the omnibus already ships host with ten point nine. So it's not really installing anything new I think it's just you know like migrating and rewiring yeah.

A

And that not always has to happen, because when you do an upgrade between two Postgres versions, you need to have food versions to like migrate data from one format than the other. Otherwise it won't work so.

A

This is the Postgres called I guess this.

C

A

Commented outlines, okay, sure, I guess we have to trust them on this yeah.

B

I mean I'd say: let's just do it and then, if everything works, okay.

A

I can't continue so um that was the first step to it on the primary so.

B

The secondary should not be unhealthy right if we look at the.

A

So we're now on the secondary and if we go.

A

Why is there no from menu for Geo beard? This.

B

Is probably because of the.

B

Okay yeah, so this also makes sense the replication broke, because you have different post rest versions.

A

Okay, yeah its replication of low protein, that's true single formation sort of replication! hmm Well, maybe that's a cached value or something I'm, not sure what we are. The difference from these two come from what know.

B

Like like, we have stopped replicating like.

A

Yeah yeah, oh great, did that break replication. So that's that's true! Okay! So let's switch to this side and do this one. Okay,.

B

Okay, like they definitely give you a lot of time to think about this, where the dots are. You know.

A

Do not really have a clear indication that your server will be down for a while now.

B

I think that's actually maybe sometime add feedback to say here should be a banner saying this will take your server offline, yeah.

D

B

D

B

Also SEC detected a geo secondary node somewhere in there, so.

A

A

So secondary J.

A

This is a different error.

A

Yeah, let's just wait for a bit configure to complete.

A

Okay, not okay, I can imagine this is expected, but I'm not sure. Okay,.

A

Although there's something going on here.

A

A

Yeah I'm just was wondering whether it was would be logical that, like the migrating of the data to the new format on a replica, would work or is it all you're supposed to restore to replication from scratch. I'm not sure I.

B

Think it will I mean you definitely need to restart it right, but maybe not from I. Don't know, okay, so what broke here.

A

hmm Bg upgrades.

A

You can find this.

A

Detected the Geo secondary note, so this function failed.

A

Secondly, I know it's hesitated race, replica.

A

Aha, it was trying to also upgrade the geo tracking database, but.

B

That's not there because it's on a different node.

D

A

So it's it's not checking that, so it's not built for a chase. Are we hitting something? That's not built for AJ, because tracking and the main database are on different machines and should I guess this node should know that it is not running the tracking database. Oh yeah, that's pretty clear if.

E

If the main database in the tracking database are also on separate nodes may be. That also explains why some of the performance for the joins is so bad and why there varies for different customers that are running it.

C

E

They're dealing with a latency across the nodes, that's.

A

True, didn't I'm surprised if we didn't ever think about that before, but that's a good point, so yeah I'm not sure which line is failing at the moment, but I guess it's yeah. It's relating to the tracking database. So can already able to see if the data is the main database is migrated and if it is running and what should we do now so get lap 12 status, I guess the database? No, it's not running.

A

Can we start it?

A

Okay, yeah? It's it's not running. Let's make sense because it has.

C

A

Database, let's try to start it. Oh not everything.

A

Normally, when the machine will be properly configured, only this process would be able to start I guess, but setting up a jason is hard.

A

Is there something in the logs or more interesting.

A

Relation features does not exist at character, 30.

D

A

Would expect this as a version mismatch error.

A

So dis versions running and that.

A

Stand out ten, nine, okay.

A

Postgres database.

D

Did you see this message, I think.

A

Checking cluster from each.

A

This is the post teo Postgres database, so normally the regular database.

D

A

D

B

So, are you saying that the normal post post database has actually succeeded yeah.

A

I see this line, not this one yeah upgrading it you I'm missing something here upgrade has completed.

A

Please verify everything is working.

B

Okay, no I think it does two things in a row right. It first upgrades the main database mm-hmm.

C

B

It attempts to also upgrade the tracking database, which is now not working, because the tracking database is on the secondary nodes. Yeah.

A

Can I find this line in search? Oh.

A

That's this one.

A

Shall we run reconfigure yeah.

B

That helps and then we we should attempt to also update the tracking database on the tracking database node right and in parallel. No, not a parallel like ah and just wondering.

B

To me, this sounds like a bug right, as in yep.

A

It should not attempt to upgrade it when it's not yeah.

B

A

This machine, yes.

B

And there is a configuration that actually tells specifically. This is not on this machine.

C

A

Again, this is taking a while was it just on the primary that it was taking well also on this machine. I can't, remember, I, think this machine.

A

I guess if we did, we configure I I.

A

Would restart birth control.

A

Still the same issue.

A

So as well is it related to the tracking database not being upgraded or like.

B

It doesn't actually do anything yet because we also have to be instantiated in replication. No.

A

No, the tracking database so I'm just wondering whether it's rails, trying to do foreign data queries between the two databases and their stuff grows wrong. Yeah, maybe yeah.

C

A

It's hard to see from the query where, because I feature gauge not something that we would directly do for.

B

Like, let's say the procedure that was attempted here, who was updated main database and then update the tracking database mm-hmm.

D

B

That failed, because it's an installation but what? If we now actually try to update the tracking database and then after that, try to be instantiated, PD, replication, okay,.

A

I would offer I agree so I'm doing it on the track database. Now.

A

There was a checking the logs here and so from real. There was all this background. This is a sidekick, I, guess, yeah and sidekick and define table.

A

And where's this code triggered from, is it possible to see I? Don't think so? What is happening here.

C

A

Currently installed PostgreSQL omnibus instance found. So that's because we disabled the main database.

A

So we have disabled.

A

A

So that's not a bug, so these these pieces of code, they should be wrapped around like if this part enabled- and is this part, enabled yeah so I'll I'll show us just fake it and enable it for now yeah.

B

Just right I mean we are going to record this any of them anyway. It's right and just see what happens.

A

It would be hard to pass the code to skip it or or to you have to move the test and just fake it for now. I guess.

A

How's, your tea or your outdoor.

E

But he's very good: I am.

B

A

Must be cold by now, our.

A

Rachael has a really large cup of tea. That makes a difference.

E

That big, a cup just looks bigger because it's close to the.

B

That takes a long time the graph found our stuff.

A

I'm not even sure what it does. It will be block I. Guess it also like migration, because kuo-fan also stuffs store stuff in a database, and maybe maybe it's related to that I. Don't know I mean.

A

Yeah it shouldn't be much data so and I I cannot imagine why it takes so long. So it's become figured. Let's try the upgrade again.

A

D

B

Can go wrong. It's let's put it this way. Joffrey has survived a few things by now,.

A

So I hope this will succeed and thus like the boat database, because if it's only does does the main database, which we don't use, then we're stuck.

B

It looks every message free so far.

A

Okay, so now it's doing a reconfigure again and I think it has been doing it for about three basis: yay. Let's wait for this again.

A

Okay database upgrade is complete.

B

Every time I see RM months of cringe little bit.

A

High, it's removing a deploy page.

A

Please verify everything is working and run the following. If so, that's why it was also messing with missing a step to do this for the tracking database. So that's one other thing to note and I'm hoping we're up. No we're not up again, no.

B

But we still have to.

B

Reinstate we initiate the application, correct, yeah.

A

But I would expect like the data that is, there still is Fallot yeah.

A

It has been trying to migrate the data, so it should be able to. You know it should be Fallot, otherwise it won't make sense to even try to make it migrated.

A

So it still leaves weird.

A

Still same errors about like data's.

A

Was I'm wondering whether the the tracking database first Chris was he started? Oh.

A

It's even down.

A

A

Wonder if you have similar errors about this, because it's sidekick, why why why psychic have trouble with.

A

We checked in the primary if it's like the front end is working but I'm, not sure if sidekick is also working on a primary.

A

And failing to start, can we see what's happening why it cannot start.

A

Data in most post quest 96, so.

B

Do we have to feel.

A

That would be fatal, oh well, that would be data loss because then I would lose.

C

A

E

A

So we try to restore it but didn't work.

A

Today them my creation, oh damn, my logs are gone.

A

What is our unix times, I? Guess.

A

Ok, so here are three reconfigures, not sure why that we, but I guess one for each database.

A

Yeah, but this won't print info about, like these lines, won't see where, where steps are going but okay.

B

Let me just copy you an issue.

D

Okay, you can tell if I could I have a great short-term memory.

B

C

B

I think we just confirmed this problem.

B

And didn't the customer resolve this I think this is still in development.

A

Yeah, but we must have found a way to resolve the issue for the customer. I. Don't.

B

Know maybe scroll down.

A

Will fall out when you.

D

A

A

The detected was that we did you see this line on there.

A

B

I think we saw that on the definitely on the secondary, but that.

A

Secondary it should have the the roll cue secondary line, I think and if I'm not seeing that.

A

Walls include your secondary.

A

Like this I think so,.

E

B

I think there needs to be a space between roles. The first really oh,.

A

D

What am I crazy.

A

A

Guess you have a problem now, because if we still want to migrate data- and it already has done that for them like the main database, it won't do anything for this tracking database. Maybe yeah.

A

B

Don't think this is actually I.

B

Would have to follow up with in and Gustin here what they actually did. Okay,.

A

A

Yeah, that's what I was expecting.

B

So we essentially we we are on ten point nine, but the data was not migrated on the tracking database. Mm-Hmm.

B

And that means we have to backs it again or.

A

B

What because the tracking database.

A

Yeah we we need to run the upgrade command manually. I think like do the data migration step. Ok,.

D

A

Add command under category okay. It is but I'm not sure. If I'm able to.

A

Shall we consume consider.

B

Yeah I mean we're trying out here right. It's I think as long as we document what we do I'm totally happy with.

A

Okay, I'm just thinking if we should like revert the conflict change. First then do the revert of the PG upgrade then add the line to get a Barbie again and then run it again or wouldn't it matter.

E

I say question again.

A

Yeah well, we're gonna try to run the revert, so there's a for PG upgrade an omnibus and but we just added the line and get up our beat. So it sets the Geo secondary role, but I'm gonna disable. This line again to revert, then add line again and then again do the migrate, because you've noticed the taste part. What did not run because the roles did not include cue secondary and that might have caused a problem with the data migration, because there was a miss configuration which, with our tracking database.

B

It's so good that we do this and I mean it's not fun, but I think it's really important to actually do it.

A

Yeah it'll gives us less of surprises with customers. I.

A

Really like to know what is this but I kept it's a ruby block, yeah. When you run Ruby, you often need a little hole, replication of the whole Rails application and that usually takes a while okay. So we have reverted to get the Part B change. We're now gonna try to revert command.

A

What was it again, this one get clap.

A

And this probably will just plays back the data there 49.62 data there that we could relate if everything was okay and then start the old version again, I guess get that post-class PostgreSQL would be running now and also key or PostgreSQL is running. So that's good I'm happy. We have a stable state again.

A

Are we actually recording.

B

Are we not yes, we are, we are not.

D

Very yeah I cannot see her at top. Don't try to touch the laptop. We know how that ends before I hit.

E

Record, it's recording on my my logo. Okay,.

A

No, that's fine.

A

So this is the reconfigure and I'm hoping we can do the PDA up great now.

A

A

Then we wait again well.

B

But we could revert right, so that's at least something yeah.

A

Yeah, that is a nice thing that we are. We have this in place, so.

A

Maybe that's what customer also did revert it and then wait until they have a solution here. I think that maybe.

D

A

This is a quite different error that error.

B

A

We might run into it now, but we haven't seen it yet. It makes sense that the Miss configuration Gator Barbie was causing this.

B

And Groff Anna again.

A

I'm pretty sure it's authorized by now, yeah.

A

Okay, this is the line. I was looking for.

B

And we continue to authorize.

D

A

Okay, kidnap and pull stars come on. Please be.

C

A

All right, it's running, so what does the front end say still broken. Okay,.

B

What's next, we ins reinstate the replication correct. Are you sure.

D

I'm always wrote on I'm. Obviously that's right.

A

Yeah I'm just I, want to make sure that you're not skipping some steps.

A

D

A

Can do get lap.

A

Know I always began to connect to post quest, can do it here. I also write.

A

That won't work, maybe test.

B

Nothing. Yes, sorry, yes, get lab. Psql.

A

It does that exist, but I need to provide passwords and stuff. Maybe I know this takes a while, but usually this works just I'll. Try it ok.

B

Let me quickly summarize so: I don't I, don't forget where we are so we have actually upgraded to 10.9 successfully on the secondary Postgres we think and on the tracking database 10. Is that correct, yeah.

A

I think so: ok.

A

Ok, how do I see all haha databases seems to be empty? Ok, you were right, I guess we have to restart replication well, shall we do that yep.

A

So this come no, not that command this command.

A

And this machine is this: one.

A

Now I have to provide the host I smell, toast.

A

A

I'm surprised this restart certification again, I'm not really curious if it does copy overall data, because that sounds inefficient to me.

A

20 seconds select it restart and I'm hope it would be up now, but it's not.

C

A

And I have to provide all the.

A

A

So that's correct: okay,.

A

We have primary and secondary, so why aren't we up and running now.

D

D

Oh, we have that right before yeah.

A

And that was related to the replication seems to be overwriting the configuration that should that did we configure right. So if we yes do we configure again that would overwrite the configuration again correctly. Yes,.

B

So you actually have to reconfigure after you, after you restart the replication yeah. So that's a missing step in there. I agree.

A

Did we okay, that's also in here.

A

Does the primary know anything about the current.

A

Replication status like database replication, it should be okay, I guess.

A

hmm It's feared that the primary did not refresh its own status in five minutes.

A

Or I agreed for the secondary would but primarily I'm, not very sure about that is the secondary back now yeah.

B

A

B

It's just like starting.

C

A

Does not exist.

E

I'm sorry to interrupt I'm going to drop off the call, because I need to prepare for the next one, but I know that I'm recording this on my local, so I'm going to leave the recording running and will you just pay me win when the call stops and I'll make sure that it gets uploaded.

A

Okay, fine I mean I hope that works all right thanks. Thank you.

A

A

So spike it I think psychics trying to report to ratification, lack to the primary note and that's why it's using that function. Okay, I'm, not sure why that whoa under course not cursor, is also.

A

Not sure white.

A

There's nothing obvious in there wide application notes are not working.

B

Maybe you have to restart the application note as well just.

A

Given that, but you have a bit, but we didn't on primary- that would be weird but yeah.

B

Yeah I, don't I, don't know.

A

Well, the question you could question whether it's related to the foreign data wrappers not being updated yet yeah.

A

We have to run that on the tracking elevator I forget.

B

Every time I think it's on the observer, but we will find out I think it is on the observer. Yeah.

A

Yeah, you might be right because the pregnant lady bass doesn't have the connection information about of the main database.

D

A

Would be surprised if that would fix it, but.

B

Can you just do a geo check here in the I.

B

Didn't include that there.

A

B

Fine okay I mean.

A

When I'm see when we see this page means the readiness note is okay, because the load balancer says sending requests to get lap and not stopping them at the load, balancer itself. Okay, so there's there are parts of Wales working, but not everything. Okay, but I'm, not sure where, where it's broken at the moment, I was hoping to see something from the log here.

A

A

Well, maybe we did to restart the replication so that yeah, when we did that last time that we restored application- oh I,.

B

Don't think that's in the in the instructions, for it does zero downtime upgrades I.

A

Can I'm seeing this error about.

B

Let me quickly.

A

hmm I think this line of code is being triggered by by the banner. We saw last time right, your your your replication is nine days behind okay and that command seems to be like well incompatible with the version of of PostgreSQL we're using so I wonder rare. Is it triggered from.

B

So it said, I saw one thing that we did have to restart was a unicorn sidekick and the Geo, not a cursor, so the forces. Sorry. Where did you see that in that in the zero downtime instructions right essentially after the reconfigure is done.

B

And before like we do the foreign tables right and everything we.

B

Yeah we hub unicorn and sidekick in the zero lucky, the updated multi-node zero downtime instruction.

A

Yeah yeah this this seems like like something that is cached in memory use this function instead of the other one, because it depends on the version.

A

So it's using the last o'clock, so you see, if location well, to be using the last wall receive well then I can we start one of both F F nodes and then we can see so how to unicorn and probably also restart, Yolo curse or because it's running on this machine also and then I have to let wait a little one while before this app note is fully restored, it not sure if you can access it directly.

A

That's going to the load balancer again or shall I do in parallel. Also, the second, the second application note yeah just.

B

This, at the same one I mean yeah.

A

Doesn't think lots of difference.

A

Where a sidekick running, also in this machine, aren't I think.

B

D

B

And if you go to the eternal now sorry, if we go to the admin.

A

Yeah there's not much here yet.

A

B

A

B

Can create a new project or something just see that.

D

B

And the last thing I'll also be interested in is, if.

B

The database version on the secondary now displays 10.9 correctly. Oh.

A

Yeah I'll check that in an instant, so hello from need, not forum, PG, 10, PG, 9 top 9.

A

So from this is the secondary, ok,.

A

It's reporting 10 9, that's also so on. The secondary comments should be there pretty sure that is.

A

D

A

And now we're gonna add a reaction and then things will no that won't work anyway, yeah, okay. So what were the steps yeah? We can do this I.

B

Mean I: don't need to do it now right, but we can, because we migrated the data I.

A

Would expect right on the secondary I'd like to know if, like the project, informational, everything is still their.

A

Project, so it's healthy cool primary is still not healthy okay, so this data is still there because it speaks all so I didn't delete that information. I wanted to make sure it wasn't.

A

That's awesome, I, guess, I, think we're I think.

B

We did it right.

A

Sort of yeah yeah we did it.

B

Okay, so if I have to recap this in the instructions that I wrote at some point you have to after this, you have to reconfigure so.

A

We did sorry, we do all databases upgrade first, yes, all right yeah, then we do the replication and then we do that we configure after they replicate yes and then we will restart Unicorn sidekick lock-on.

B

Af servers yeah and then we also upgrade the foreign data tables. Oh.

D

B

So the one thing that failed here was on the tracking database, where we had a miss configuration, correct, yeah, I think so, but also didn't you have to sake yeah.

A

We had two problems exactly so we couldn't do the PGA upgrade on our geo. Postgres only note we have to have like a combined note, yes and error.

B

Doubt I mean actually so what it tried to do. Is it upgraded the post press? The main post was and then immediately attempts to do the geo tracking database as well, and that then fails right which is expected because it is ignoring like it is. Essentially it expects everything to run on the same node right now and then, when you go to the post rest, the tracking database and you just execute the PG upgrade it says hey.

B

This is not configured to have a an omnibus Postgres, so you then have to actually add this to the gate lab RB file, even though that's not really true, then the upgrade works and you're done so there's essentially two small problems here that.

B

That should be fixed.

A

Yeah well, wasn't there just one miss configuration and one thing that I.

A

Think we would ever would have set the rolls to you a secondary wall correctly. Then we would only have had one problem: okay, if we would, if we would do like if we would have used the steps afterwards correctly, we were hoping to get a note. I began sooner, but it seems that doing the PG a great on the secondary database required to restore application again.

A

And that was that piece was already in documentation or not yeah.

B

I mean there is no real documentation for this right. This is why I wrote this up right here. This is this piece. This is the piece right, and that is the assumes a single node installation really right, because it doesn't really tell you where, like you, should execute those commands. Mm-Hmm.

A

B

So I essentially made it up.

A

Yeah, that's true, but yeah. It.

B

Was pretty good, so I think this is maybe a follow-up issue here is we should we should add the section here saying like if you are on, if you are running geo in an H, a installation with a single post course database? This is how to do this. All right. That's sort of a follow up item.

B

But we should, we should talk to Ian and the distribution team. Oh yeah,.

A

B

A

Yep I think it's overall, not a bad at all. No, you have to do a few tricks. What you'll be able to upgrade it fairly easy yeah, I think in boom yeah this. This replicates step I'm, not sure about. Yet why replication restarts from credit scratch or it seems to be.

B

Yeah I mean that's, that's some follow-up questions, but I think what we like in general, like these instructions, were already pretty close to what you needed to do anything.

D

B

We're now in ten point nine and we had no data loss, I think and I mean we had plenty of downtime. But it's it's possible.

A

Because geo is a sort of an H, a setup could should we because that, if that would be a problem for like if you run multi node Postgres on like an anon geo set up also, you will also have to run this step.

A

Around I was wondering if this would not have if you would be using replication manager, repetition I have no idea I, just don't I, don't know I.

B

Have no technical knowledge for this know.

A

Any tournament yeah.

B

Okay, I would say pretty good.

A

That's four out of five stars: four.

B

Out of five stars and I think now we are we also at ten point. Nine, which means like my hope, is that we at twelve point two with ten point: nine, which is where 10.9 was actually introduced and now going from 12 point two to twelve point: three using the zero downtime instructions. I hope should be relatively straightforward and then going from twelve point three to twelve point, four as well and I.

B

Think then we are actually up to date and we brought the cluster all the way from eleven eleven to twelve and four, which I think is quite the journey.

A

Well, shall we ask Rachel to stop the recording, then yep I.

A

Hope that would succeed as I'm not sure what she's doing.

B

Cool I have another meeting now so I need to I need to drop off. Oh she's,.

A

Still here, okay or who is yes, you're still here, okay,.

A

Awesome thanks huh fabien cool.

B

Thanks dawn, we did it. Yeah take care yeah.

D

D