GitLab Repeatable Database Creation, 8 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Repeatable DB creation Demo (2021-09-08)

Description

https://docs.google.com/document/d/1jGZow42nbmSOuK0TDv9WUHt9jkpUHDrt7acNL0R9DWs/edit# (internal)

A

Okay, this is the repeatable database creation demo. I have the first item on the agenda share my screen.

A

So uh party one blockers, uh there have been a few that have been closed since the last time uh we met or the last demo um the networking options. This is still a big blocker, I'm suggesting as like for a smaller iteration. What we can do is just pass in the subnet instead of letting the gcp reference architecture module, create the subnet in the network, and the reason for this is our needs are unique in that we are calling the reference architecture module multiple times once per shard in a project.

A

So, um given that all the shards probably are going to share the same vpc, I think then we probably want one vpc for all shards, so we'll probably just want to pass it in.

A

The custom console services directory that's already in review and then the multiple pg bouncer instances that might not get done. I don't know it's not scheduled it's not assigned, but we have a work around for doing our own thing and then number two is just what we're working on now.

A

For the demo, I thought we could add a new replica to the existing to an existing cluster. I'm going to go ahead and do.

B

A

So share my terminal.

A

So the first thing to do this: we're going to need to increase the node count of the reference architecture, postgres cluster.

A

And that would be in modules, cluster main.

A

So this is var dot. Postgres node count. This comes from the environments, so what I really need to do is set it for a single shard.

A

And so it needs to be set here, but this is set by uh jsonnet. So I think what I'll have to do is um set it in dot, config.json it and here's the terraform and I've already kind of like forgotten the name. So those modules cluster.

A

So postgres node count.

A

And just for this test, I'm gonna increase it to three. Since the default is two, this is going to set it for all shards in all environments, which is not really what we want. So I think, if, in the future, what we'll probably need to do is make this like part of the shard configuration and but for now I'll just do this?

A

The purposes of the demo create a new branch.

A

This one, like I say, update everything. So now we have postgres node count set to three.

A

I think that should do it.

C

C

A

I guess I didn't push this properly.

A

Oh there, it is.

A

That's interesting: the main source branch doesn't exist.

B

And this will has sometimes happened. Randomly.

A

B

Yeah, I I remember it occurring like a year ago.

B

Let me just keep pulling my repositories your difference, yeah. There are sure.

A

I mean it doesn't matter since they're not merging this anyway, but.

A

Okay, so let's run this on my environment, see what happens.

A

A

There's the node.

A

Looks good, I think.

B

So your question, I'm assuming, is already running.

A

Yeah everything is already pre. I had it configured previously, so we had a working petroni cluster with two nodes.

B

A

A

So let's go ahead and try to configure it! I'm just going to do this on the ci chart.

A

C

A

So far it looks good.

C

B

So, in order in order to be able to configure different values, you said that you we just need to add this to the start: configuration right.

A

Yeah, I think I think, we'll have to figure out how that will work, but um I assume what we'll do is, we'll just add a replica account to the shards.json lib file and that will allow us to.

A

um I guess you need to configure both on environment and shard name dimension right, so you'll have to do both, but.

B

A

Yeah, we can figure it out.

C

C

A

Alejandra one thing that kind of bugs me about get. I opened up an issue for this is the inconsistent host naming that we have now, because we have some servers that have a prefix of the environment name and then other servers that have the prefix of the shard name. So we have, for example, ci postpress2, and we also have um you know sb alpha.

C

A

2., maybe maybe we should, I don't know, I think the problem is we're trying to use like this prefix and node type thing for multiple types like of servers, and it's just not ends up being very inconsistent.

A

Maybe we can override the names of the hosts, at least so that looks nicer.

A

I think we probably want to like prefix all of the hosts, with uh the name of the environment.

B

Yeah yeah, I was gonna, say like we could just.

B

Like the idea would be like sp alpha, ci console one, something like that right yeah. This is.

C

An error, it's just.

A

I think it colors standard error in red, so I'm not. I think I see oh.

B

Yeah here we go.

A

Yeah interesting failed to open.

A

Tcp connection to local os, this is console.

B

A

But console should be okay, so, let's, let's hop on the host and we'll see what's going on. um In fact, I can do it.

A

Bci post press 3.

A

So console is running. Interestingly.

B

It looks blank to me this screen.

A

Oh really: try to reshare.

A

You see now: okay, yep.

A

Okay, so it doesn't.

B

Probably fail before configuring attorney properly.

A

Yeah, I assume so.

A

Ansible's kind of going right through this without a problem, I guess it doesn't fail hard on a reconfigure failure.

A

uh So that could be a possible issue, but let me I'll issue: another reconfigure see what happens.

A

hmm Interesting.

B

A

Wasn't ready at that moment possibly um I'll, save the reconfigure output, so we can take a look.

B

um It's also interesting that the host ip- I guess that's, not super important, but if you see the host ap for posters, one and two are yeah.

A

B

Other one is in it's uh a bit different.

A

Yeah, but this is the uh the default subnet because we don't have and like that's a fairly large range. So it's probably just we've added a lot of hosts and enough time has passed. um We haven't decided, I'm not sure like whether we're going to use one subnet for everything or different subnets for different clusters. You know like we have to decide that at some point.

B

Yeah I mean in in production, we use different subnets for people yeah, but.

A

I'm not sure if there's really any value in it right.

B

A

um So I'll save this uh console or this reconfigure output and we'll take a look but based on, like.

A

So maybe we need to check the console log to see what happened here.

B

Worm pending console restart.

A

It said reload successful, sending restart to postgres exporter.

B

It's interesting, I said I said: okay, the demon was reloaded, but maybe you need to restart the console agent.

B

So maybe that's what was missing.

A

Yeah but console was running.

A

B

And also, if you needed a restart, why did it work now? Because you didn't.

A

Yeah exactly yeah, so this was at 13 53, 13, 53, 37.

A

B

I mean, I guess it might.

A

Have created a.

B

Restaurant and the galapagos reconfigure restarted it the second time you you run it.

A

A

I think it's worth like trying again to see if this is repeatable and.

A

Cool all right, I'm gonna end the call, so I can hop on the incident and uh I'll talk to you later.