GitLab Repeatable Database Creation, 1 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Repeatable DB Creation Demo (2021-09-01)

Description

https://docs.google.com/document/d/1jGZow42nbmSOuK0TDv9WUHt9jkpUHDrt7acNL0R9DWs/edit#heading=h.iijg60y5n9qj

A

Okay, let's get started. This is the repeatable db creation demo.

A

I will share my screen.

A

So, first just a quick review of the blockers on the get side of things. It's really just the networking options that are the biggest that's the biggest blocker right now.

A

We either probably need to spend some time looking into this ourselves or think of other approaches, because this is something that we're going to need to deploy into our own vpc instead of the default ppc.

A

I don't think there's any complication here, so maybe we can get this done soon. On the omnibus side, uh there are a few blockers. um One is the bg bouncer one which we've worked around for now. The other two are related to setting custom service names in console, which is something we're going to need, but are actively being worked on. So I don't, I don't foresee any problems there number two.

A

uh These are just like the big items that we have left uh the first one is prometheus server and still thinking about how we're gonna do this. Now that we have the environment machines uh like we have a framework for deploying machines that will be for the environment and not the shard, we can maybe deploy kubernetes cluster and install prometheus on it and use console discovery. I'm just hoping that all that will work well, uh so we'll give that a try uh meet a bbc networking and we need to figure out logging.

A

Those are the big three really a and c are the big ones b should be pretty easy.

A

We had a discussion uh with the sharding team about how we can support them. I created a new epic, that's linked to the main epic. um What we're gonna do! What I'm thinking right now is that we'll just do we'll. Just have a separate environment for them to have as a sandbox, with two petroni clusters installed and we'll deploy an omnibus vm into the project.

A

And for demo, I just wanted to create um create database shards with iobs.

A

And I thought I would get started on this before the meeting because it takes a bit of time to provision.

A

So if we take a look at this, is my environment, the alpha environment, uh things that look a little bit different. Now we have the machines that are common to all shards or that sit above the shards, and then we have the machines per shard.

A

One thing I noticed is that we can't destroy environments right now because of the tcplb.

A

This is due to us defaulting to prevent, destroy on the load balancers in that terraform module so um and that can't be print can't be parameterized. Unfortunately, still that issue is open, so I guess I'll just need to work around this for now, so I wasn't able to destroy the environment.

A

uh I think then this should already be configured. Let me just check out the gcp console.

A

So we have four internal load balancers after we configure everything.

A

A

Yeah, this all looks good, so we have four internal load: balancers two database shards each shard has the sidekick group and the non-sidekick group for pg bouncer, and that looks good. I don't think there's anything we're missing here.

A

Cool is there anything else that someone here would like to show or talk about before we end the meeting.

B

Can we try connecting through the ilp just make sure those things are connected properly.

A

So that would be a good thing to do so, just like.

A

Just start a database session through the iob yeah, okay.

A

I'll share my screen in a second. Let me.

A

So how about we do this from one of the? We can't do it from the bash gin, because the bastion doesn't have omnibus installed. um You could do it either from the console, postgres or pgbouncer I'll, just I'll just do it from the postgres code. So I'll do it from like this guy.

A

I'll share my screen in a second.

A

A

So let's go to the registry: pg bouncer, regional.

A

Which is going to be that address.

B

um The skill, sorry, uh you can just go for psql directly without needing to close your own universe.

A

Okay and we'll do something like like this correct and then capital p. I think for port.

B

I think it has a small p but capital: u for the user.

A

And then um user, uh like we haven't, done anything with these databases or did uh because I don't think like uh use or user's automatically created by omnibus.

B

Yeah, it creates gitlab, but for some reason it doesn't say password for it, but yeah you'll be prompted for a password, but you won't be able to go ahead to get ins inside unless there has.

C

Been a password set.

B

C

A

And like production, maybe.

B

Hq all right, no.

C

No was that on disclaimer.

B

A

Okay, um so we don't think we have a password set for gitlab right.

B

I don't think so.

A

But maybe I can like I'm on the database. I'm not sure if this is the primary.

A

B

Recovery right, bg underscore is in recovery.

A

A

It's this right, pg is.

A

Yes, something like this and.

D

I think it's just select using recovery.

B

Or you can just show the list of processes, you know it could be easier. This way.

D

Another problem that you're, having is that you have a missing semicolon on the previous line, so you're executing pds and recovery.

D

A

A

Isn't there like also there's a gitlab ctl grant for this right, like it's like gitlab, cto um pg, something attorney.

B

Yeah but tony members should show it as well.

A

Attorney members like this yeah.

A

All right and we are um 56 all right- we're the leader all right, that's great! um So now.

A

Well, I can just do this.

A

So maybe we can set up a password for gitlab sure does.

B

A

This off the top of their head, or should I yeah.

B

It's a yes alter roll, uh then the name of the uh I guess within double quotes.

B

Yes and followed by passwords and the password itself in other codes.

B

Like that, I guess so: no, okay, uh it's in chimney double quotes to single quotes. I always fall for the filter. It's single quotes. Yes,.

B

A

Okay, oh yeah, I think, like everything, is single quotes.

A

B

A

Password for user postgres.

B

Or maybe you don't need single quotes for gitlab, but yeah.

A

A

Come on, yes, it worked super.

A

Okay, um that was a good demo. uh Does anyone have anything else before we end.

E

um I know the gp network stuff is is high on the list and it's higher on our list as well. It currently is set.

C

E

1.3 but um we're working at pace, and if we can carry the time for 1.2, we can look at it, but um if you guys wanted to contribute we're more than welcome more than happy to to to help with that um as I've kind of called in the issues, we've got design kind of locked down with the aws vpc setup. We've done so, if you're looking for essentially a copy-paste job onto gcp, with, obviously the modifications required for gcp's little differences.

E

um So I'm happy to take contributions. If not, we will try to get this as a priority, we're just finishing up some project car stuff and then I know obviously trying to balance between different projects out there. I know this is like the top thing for you, so I think it it's very high analyst so I'll be able to tackle it as soon as possible for myself as well. If there's no contributions, sure.

A

um And while you're here grant as well as alejandro, I I just submitted an mr about the nat, um and the problem we have right now is that um when we added that support, we sort of assumed there would be one nat per gcp project, but because we're deploying multiple reference architectures in a single gcp project.

A

We we really only want one that so I submitted an mr to optionally disable creating the net still disabling public ids, but you know disable public ips, but disabled in that, so we can create it outside of get because I don't think you know. What I don't know like is is deploying multiple reference architectures within a single gcp project, something that we want to really worry about. uh That's um when we have multiple see in order to do that.

A

Really you need to have networking support in dpc, because you really would probably have one reference architecture per vpc. So maybe we can think about that again when we tackle that.

E

The uh certainly the.

E

In terms of best practice, um what we say to people is, you should be doing it for excuse me project, but um we know in some cases some people do want to deploy different architectures to the same projects and there's various considerations in there to say vpcs, most importantly, resource quotas, which each project has but uh there's no strict requirement. So um I did see the armor open. I know you've tried alejandro to look at first when you're ready for me to have a look.

E

Give me a shout and I'll uh had a quick look and everything looks fine to me um but yeah. We we're happy to continue support to support deploying mobile architectures to one project for people that need to do it, but um in terms of best practice I'd say you should be doing a project.

A

Yeah like this is really it's not really that we're deploying multiple gitlabs per project. It's that we're deploying multiple petroni clusters per project right. So this is like where, since get doesn't explicitly support sharding yet um like having multiple petroni clusters associated with one gitlab instance,.

A

What we end up doing is deploying the reference architecture multiple times, so I guess this is something to think about, as we start like adding. First order support for sharding to omnibus and then later get how that's going to work.

E

Yeah I mean yeah sharding, that's a big, a big old thing to tackle that we know is coming and we will tackle it when it comes into omnibus, because obviously the last I heard about that project was still very early. This is where I'll go those early days on actually deciding how team do sharding and gitlab itself, because obviously that's a very complicated heavy thing to get done so yeah. We, we that's the, but what you describing this kind of the reason why we still support deploying on the same project in gcp.

E

This is only really only a gcp consideration, of course, in aws and azure and other club fighters. This isn't even nothing.

E

There is other forms of encapsulation, but uh generally most most accounts in, like aws, usually play in the same pool, so we're happy to support multiple projects, uh and once we get the network stuff in that, that will further that, I suppose. So um I guess that's that's where I come from the get standpoint.

A

Yeah: okay, great.

E

Yeah, we'll try and get in and if you guys want to tackle it earlier, you're more than welcome, um but if not, we will try and get an asap for you and we know so we hear you loud and clear. We know it's top priority, we're just. uh I just try to get it through all cool.

A

All right thanks grant thanks.

E

Everyone, thanks all cheers.