GitLab Repeatable Database Creation, 23 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sync on Consul configuration for DB Provisioning

Description

https://docs.google.com/document/d/1-lQj7JuiD8UcRZwrU6HCZBaF4208ImbL68PKewVvorQ/edit

A

Okay, so I wanted just to sync up with you real quick on the discussion about the console cluster. um Maybe there were a few misunderstandings or some bad assumptions on my part. Initially, I was thinking we were going to have one well, I wasn't really sure, but I was thinking we would have one console cluster for shard and um well. This is this is how we have it set up now, but thinking about this more and more, it sounds like it's probably not the best approach.

A

I think you agree with that, like you should probably just have a single console cluster. Otherwise we're going to have to have multiple console agents joining multiple clusters to do uh the discovery right.

B

A

So, given that we have a single console cluster, um then we need to decide whether it's going to be well. I guess it's going to be the main console cluster, that is, that we're already using for production and staging.

A

um Then it's a matter of making sure that we don't have duplicate service names, and this is where I think it gets a little bit tricky um number two on the agenda. I just wanted to like enumerate all the service names we have now and what we're gonna need to do. um I think for petroni.

A

Well, let's skip that for now, but for the exporters, I'm I'm struggling a bit because it feels like we given the work you did for pg bouncer, you install your own or you installed a console outside of omnibus right for pg bouncer.

B

You know I use console, uh that's provided by omnibus.

A

And when you register services- okay, then I misunderstood. So when you register services against that console, um how do you are you like dropping service files in the console config.d directory, or are you doing something else.

B

Yeah yeah just json files.

A

And does that like cause problems because omnibus wants to manage that directory and we're dropping files there, or is that you think that's safe to do.

B

I I don't think it. It manages the whole directory, it just measures the individual files inside it, but.

A

I could double.

B

A

I see, and this is why you had logic- to kind of remove the old service.

B

So yeah yeah is is independent of whether we're using our own console or our or.

A

This one because.

B

uh Since we can have multiple number of pg bouncers, we should have an ability to clean up afterwards or.

A

Otherwise it should they will just linger around and could cause yeah. So I guess we're kind of.

A

I guess my my concern here is that um what happens when we do a gitlab, ctr, reconfigure and.

A

I guess it won't create it, won't it won't. uh It won't delete or cause any problems if, if it tries to register a service by using that directory, because there's uh there's no like common services, I'm just like I'm trying to I'm trying to think about how this is going to work, especially with exporters, because um exporters also need to have uh namespace by chardonnay so and unfortunately like once, you have console enabled it creates the the server.

A

So maybe maybe the answer here is that we need to make it so that the service names are configurable by omnibus. Would that do it? Do you understand the problem here or not?.

B

Yeah uh yeah um uh yeah, having the same name for exporters, could uh could could be conflicting across charts and short names. But if I understand it correctly, it's that we only care about it, because we want promises to script from the correct exporter and.

A

B

So can't we utilize tag names like we just register servers uh as the services for exporters easily, but we just add a tag name that is uh unique to this chart. So we just add a bunch of tag, names and users.

A

B

uh Can filter from this because we can provide tags or not meta attributes, so we don't really need to fiddle with the service names or have unique ones at their chart.

A

Yeah, I think we could probably do that basically create a tag for every like a tag that corresponds to the shark yeah. um Okay, I think like. We would still then need to make an omnibus change for that right, because omnibus doesn't configure tags when it registers the console service.

B

Yeah, uh okay, I I don't yeah it's just this uh yeah.

A

B

But that's an easy change. You can integrate into only one second.

A

Yeah yeah that should be fairly uh straight uh forward.

A

I guess I guess where what I'm struggling with is like having a consistent approach to using shard- and um I was thinking about this earlier and taking a look at like we. We have this service called postgres sqlha and that uses tags, but it uses tags for master and replica.

A

We don't use this currently in production, like we don't use tags, we have two different service names.

B

Yeah we actually use tags, except that these are different ones like for right now, in production uh we explicitly say: yeah uh use tag master if, if a certain.

A

B

Is passing and that replica of another? It's just passing and the same thing is, is done here, but this service is registered by console itself. It's just a coincidence that it's using the same tags that we're using, but this one is not done by only was itself just passes, a configuration to between that register, console services and.

A

B

Up with posterior sha.

A

Right so so right, so I was looking at this and you can set like this patrony scope and that will change the name of postgres sqlhe. So we could create, like um you know like I was thinking, patrony scope.

A

You know equals postgresql, ci right, correct and that will create the service postgresql ci with two tags master and replica, um but I think then, then we're kind of establishing a convention where we have the shard in the service team.

A

So I kind of wonder whether we should do the same for the exporters too. You know.

B

Yeah, it's just a matter of how how big that change! We want to do it. uh Yeah yeah.

A

I don't know like I'm just I just was thinking to be consistent, but.

B

Yeah uh yeah. Another thing is that the consoles seem to prove uh to offer like multiple wave, multiple ways of uh dns discovery. So we just don't uh don't need to stick to uh to the whole uh tag: dot, service, dot, uh the console and so on. Maybe for for the exporters we need to because we are constrained about what promises can can search with.

A

B

Only tags and not meet attributes, but for for postgres, uh it's flexible because it's from the aerial side and we can maybe utilize something more sophisticated. I was thinking. Maybe we can look into the prepared queries this is offered by console, uh but this could be maybe complete things. But it's just that. We have two problems, because we we uh look up for two different reasons: one first trails, which is mainly zebusters hay and the other one for the exporters, which is kind of different thing.

A

A

Yeah, I'm not sure um so. Could you just paint for me like or give me a very high-level view so when, when assuming assuming, we have multiple shards and they're all registering to a single console cluster, what we need is to be able to connect to the iob so.

A

So um so we'll have like four, each shard and there'll be a db host which will equals the, which will be the ilb endpoint and then we'll have db load balancing for each shard, and that is going to be a record. A console record so it'll be like um do.

B

You think yeah.

A

Like something like this, like c c, I.

B

uh Z, load balancing is connected to the uh read only so uh it will.

A

B

Ci dash db replica dot, uh server.

A

B

Something like that.

A

But it'll be it'll, be I don't know if you can see my screen, but like yeah it'll be yeah it'll, be this right like you'll? Have the the tag will be replica yes or no and.

B

Then no this one is uh this service is pointing to petroleum itself, uh but for db load battery. We want to point to the eg bouncers in front of them. We're going to have it's going to be different.

A

Okay, so what will this be.

B

It will be something like ci uh db, dash replica, ci dot, dot services console.

A

Something like this, and this will give it- and this will give you all of the pg bouncers across all of the replicas correct.

A

Is this what we register now or do we need to make a change to our pg bounce servo.

B

Right now in staging product or production, it's just db dash replicas of services.

A

Yeah, so we just need to add, like a shard name to the service right, yeah and- and you don't think that we should make this a tag. Then right, you don't think you. We should do something like this.

B

Now we could uh yeah, we can. uh We.

A

Can definitely but.

B

There was no need for it uh in our current setup, so we just went directly with a changing c uh the service name, but uh for our use here we can definitely prefix it with attack.

A

Yeah I mean I'm not sure, what's going to be like better, I don't think it really matters that much.

A

um Maybe maybe it would be better just to put it in the service theme. I don't know um okay, so um I'm still like I'm really like you, you're, not you're, not anticipating any problems with dropping these service definitions into the omnibus console. I'm just worried that it's gonna, like you know I can talk to the distribution team to say what to see what they think another option could be is that we maybe add an omnibus configuration to point to a different directory for console so that it's maintained outside of omnibus.

B

A

B

What I can see here, and, and only it it, creates the config directory as just a normal directory. It's not a managed directory. So.

A

B

But yeah we can make sure from the yeah with the uh uh distribution team.

A

And it won't and it won't get like blown away or anything with a package upgrade right.

B

B

um Shouldn't because, um well I I I'm not really sure into how the the package extracts and do it.

A

B

I guess all of stuff should be done within the shift configuration so given the shift conversion I'm seeing right now, which just creates it directly. It doesn't remove anything.

A

Okay, let's, let's just open up an issue or have a conversation with distribution uh to see what they think and we'll go from there um when we configure multiple pg bouncers, so we're using the omnibus console- and so that's number three number four.

A

Yeah, so this one we don't need because we're not disabling it um for number number four. Can we? um How do I make it so that I have multiple pg bouncers on my patrony node right now,.

B

That should be the default right now.

A

Isn't the default really.

B

Yeah, if for the opposite, chris uh roll.

A

B

A

Okay, I don't know why I wasn't seeing that I'll check again.

A

For all sandbox environments, yeah.

A

Okay, I'll double check, uh cool um so and number five was the last point, so we haven't really settled yet how this is going to work, but um I mean currently, I have two database projects for staging and production and I thought we would just peer vpc, pier that in itself may change like we may decide just to deploy directly into the production or staging projects. What is your opinion on that? Do you think it's any there's any value in having a separate project for all this database stuff.

A

I think there's value in testing and validating it all works, but maybe, when we get close to going to production, we should just move to the actual staging and production projects themselves.

B

B

uh I I can't really see: uh can you elaborate on the testing part like how it's gonna be useful in testing.

A

um What I was thinking was is like we deploy what I'm, what I'm really worried about is what happened um when we were building out the registry databases and staging where someone like we, we overlooked the service name. So then, suddenly, like um the new databases, were showing up in the replica list, and um what my fear is is that if we start deploying to the production project and start registering to the production console server, that will have the same sort of mistakes.

A

So what I, what I was thinking was is like we have a separate project which really is completely isolated from prod. We deploy the petroni cluster, we deploy a separate console service, we register all the services and then we can kind of validate make sure like okay, all the services look good, then we can just like point it. We can. We can peer the project and point it to the existing console service and like register everything there, but um maybe that's being overly cautious. I don't know uh you kind of see what I'm saying.

B

Yeah I do and yeah I understand it was a database. I guess it helps to be extra, cautious.

A

B

A bit instant about it, because peering uh introduces all set of problems like overlapping ideas and so on. You need to be extra careful with uh with the ip allocations and, what's that.

A

B

And, as you mentioned, the dns and all that comes with it.

A

Yeah, I tend to agree like I think. Maybe we could do we could decide later. If it's, um I think, I think it actually doesn't matter too much, because we probably will create a dns entry anyway for the internal lb. I think we do already correct.

B

A

So it's not like um and for vpc purine. I have like made sure that these new projects are not overlapping, so um we'll we'll have that option. If we want to take it, um I guess if we decide not to do it that way, then we'll just deploy right into the production project.

A

What I really what I really wanted to avoid was as we're validating this like deploying into the production project, especially with terraform and ansible. uh You know it's like. I think I feel much more comfortable. Having like this separate project, that's completely isolated with permissions. You know, um I guess we can. We can. We can kind of validate it in a separate project and then just point it to get live production too, like there's, no reason why we can't do that.

A

Okay, um all right, so I think that so I think the main thing is we need to.

A

um I want to start an issue to see if it's okay, if we use con omnibus console for service definitions, and I want to drop an issue to see if we can override the service names for the exporters or whether we can customize tags.

A

I think those are the two things sound right, yeah, cool man, all right. That's all! I got.

B

A

Thanks, thank you. Thank you.

B