GitLab Geo Group, 21 Jan 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Geo planned failover demo on a single node

Description

The Geo team performs a planned failover using a simple single node Geo setup (not a high-availability setup).

GitLab version: 12.7.0-pre

https://gitlab.com/gitlab-org/gitlab/issues/35735
https://docs.gitlab.com/ee/administration/geo/disaster_recovery/planned_failover.html

A

Hello, everyone: this is a demonstration of how to perform a planned failover using a single node installation of Gio and Valerie is going to walk us through it. So please do.

B

35 M, so we had two healthy nodes. One of them is primary and secondary they're, both both healthy. We have one project there and we are going to follow our manual disaster recovery for point failover.

B

So most of these steps here are very specific to your installation. So but let's just go through it yeah. We need to be aware that not all the data is replicated so would.

A

You like this to.

B

Me, oh sorry, I forgot, no problem.

B

C

B

I need to enable it in security settings okay, but my mark doesn't ever with okay, okay,.

B

Okay, can you see my screen or.

B

Can you human yeah.

A

We came here, we can hear you and we can also see your screen. Okay,.

B

So some data have to be synced manually and also.

B

Some preflight checks; okay, we don't use object storage now, but we need to do to read these special notes about objects. Storage, if you, if we use one, also what's important for us now, yeah, let's check the health of our notes.

B

B

So I checked before this call, so they are healthy.

A

But I think is missing, I hope so.

C

B

Yeah we okay, and also we have one more task. Good luck, give your check.

C

C

Yeah, so we can see you.

B

Yeah, it's okay! So, let's see what's next.

B

Yeah, we need to check with it's up to date.

B

Yeah we 100% synced here.

C

B

But if I integrate okay, we don't need to do this right now.

C

Yes, yet, let's.

C

Put this message mean every messages.

B

Yeah, so this step is very specific for your installation, so we will just stop the primary node and that's it I think easy enough for presentation. Purpose.

B

Yeah, it's not production notes, so we don't have to make sure that our queue is empty. So it's, but we just need to take a look admin area monitoring. Let's do that.

C

B

It's here, so we can see that we have zero busy jobs and zero and good jobs. So we are good here. We can just follow next step and the next step is actually to promote our secondary node. So.

B

First of all, we need to stop our primary node.

B

So we just commit a disaster here, because our family not is.

B

So we made sure that all the data is replicated to secondary early. We disabled primary, not here yeah. We need to do some more steps here.

C

The city gives less and.

B

Now we just need to promote secondary node, so we remove.

B

One configuration parameter in github: are we.

B

So we commit it out special parameter that says that it's actually a secondary node, so we don't need this anymore and finally, the last step is actually promote. The node.

B

So we get some a warning here. Yes,.

B

B

B

It takes something like two minutes.

B

And one more thing the note will be down like for 30 seconds or so.

B

Maybe more, but we videos you know.

B

And during these process.

B

Yeah I already said that, okay, so it's the final step which was recently broken and we fixed it in.

B

B

Yeah, so we we got successful message and, let's see the state of second, you know yes, so we need to wait like 30 seconds.

B

This happens because there is no connection to DB yet because promotion of always is helping and also maybe we need to show some more.

B

More wonderful message, but it's hard because we don't know for sure this is actually promotion now, so it would be hard, but we will need to discuss a bit.

B

Ok, I think it's already more than 30 seconds.

B

Okay, so they notice up now we we don't have now. This read only message, so we already read. So it is already a good sign and.

B

Okay, I think that was a cache but interesting. We need to take care of that too.

B

Okay, so we can see that this is a primary node. It's healthy.

B

Let's check that with special comment, because it's more correct.

B

It's a good-luck check. We need to give up geojo, okay,.

B

B

Yeah, so we can already see that its primary.

B

It's healthy and Davidson Bank good.

A

Thanks very that I.

A

B

A

Maybe 15 minutes.

B

Yeah in in Rio production it would be a lot longer because we actually have it data and we actually need to verify it. But again, it's not that important now, because we already have verification. We developed disaster recovery mechanism before we had verification. So now it's less important. But what, if you replicate some content but is not covered by verification? So we need to do that manually, and so so it will take longer. I mean yeah, but it depends.

A

Alright I'm going to stop the recording one. Second thanks: everyone for listening.