GitLab Repeatable Database Creation, 21 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Repeatable Database Creation Demo (2021-07-21)

Description

https://docs.google.com/document/d/1jGZow42nbmSOuK0TDv9WUHt9jkpUHDrt7acNL0R9DWs/edit#

A

Okay, let's get started. This is another repeatable db creation, demo and uh I'll go. I have the first item on the discussion. I just wanted to give everyone a quick update on: what's happened since the last demo. Just read it if you have any questions, put them below, I'm not going to go through it item by item, because I want to jump right to the demo for the demo.

A

What we're going to be doing is creating a new database cluster in staging, and this will just be the infrastructure, not uh the configuration we haven't gotten to the part where we're configuring the boxes yet, and this will bring up um everything that all the infrastructure. Well, we think that we need right now: minus the ilb for the patronage cluster that so that includes the petroni nodes, the pg bouncer and the console server.

A

I'll start by sharing my.

B

A

So this is the db provisioning repository inside of this repository. There is a terraform environments directory. You can see that there's there are four folders here. One of them is common for just common terraform resources for all the environments. Then you have production, staging and sandbox we're going to create this in staging first one I'm going to demo is creating creating the infrastructure locally and then we'll show how it works by applying it to ci.

A

So this is set up to use, gitlab managed, terraform state in order to use terraforms or gitlab terraform state. You have to uh do a special terraform init, which points terraform at the gitlab instance. So I have a helper script here. It's nothing fancy it's just this, which basically sets up the backend config.

A

With your token and since we're using the ops instance opseculab.net, you have to have this environment variable gitlab, token ops. So when you're doing this locally, that's the first thing you have to do is do a terraform emit and then you just give it the environment name, which is stage.

A

And you can see here that it's using the gitlab terraform state for.

A

A

A

Okay, so the state's initialized. Now, um if I do a terraform plan right now, it'll be clean, because this is already run in ci, as is to create a new cluster. You have to create a new terraform file, um so I'm just going to call this charge.tf and I'm going to use the sandbox as an example.

A

A

So I'm going to call this jarv cluster I'll, give it a name jar test, one and that's all you need. So what this does is this pulls in a module called cluster. That's also checked into this project, which then references the get module which provisions all of the infrastructure. So if I look at this cluster name, there isn't much to it now, but you can see that it uses the um it uses the get module for the reference architecture for gcp, and I have three console nodes: three postgres nodes and one pg bouncer.

C

A

What will happen is, after I apply this. uh Those uh those resources will be created, um the idea here being that uh eventually, like as we build out each cluster. This will, like you, know, we'll, probably pull in what we need or what we can from get and anything else. We can just add this or you know, push it upstream to get. I'm gonna have to do another um emit, I think, because or maybe just to get, but it should be fast.

A

Okay, so now I can do a terraform plan.

A

And that's it so this you know, creates all the infrastructure. um So this is pretty simple, but what's cool is to see this happen in ci? So what I'm going to do is create a branch.

A

I'm gonna check in my cluster.

A

And then we'll flip to.

A

A

So the way the way we have this um configured is that uh we have the code checked in on gitlab.com and the pipeline's run on ops notice that, like these images, are rebuilding I'm just going to cancel them, because we don't want new images, and so now I need to go over to ops.

A

Which is where this code is.

B

A

Yeah, this is also for some reason, my my rule, for when to rebuild the images is broken, so it's trying to rebuild them, even though, when it shouldn't, but normally like, if you update any of the requirements uh like the version of terraform or the version of ansible it'll rebuild the image. I also I put this in the notes. I initially I had one image like one big image that contained both ansible and terraform, but it's huge. It was like 600 megabytes and it was really slowing down the pipeline.

A

So I split it into two one for terraform one for ensible the terraform one's only like 60 megabytes. It just makes things go a lot faster, so you can see the stages here. We have the image stage which like builds the base image, then you have prepare which does all the initialization and then you have um the build. I use the the gitlab managed terraform wrapper script. That's part. I built that into the docker image which sets it up to use like the terraform state on ops.

A

um So as soon as this completes, we should see in the plan.

A

You should see the same results and the idea here is that you know you look at the plan and then you merge it to master and then on master. It will run the.

A

A

A

While we're waiting on that, um I can show you a nice cool feature about this, is I created the sandbox environment and the sandbox environment? Just has one cluster defined and it's a child pipeline trigger. So you hit the play button here and what it does is it creates a child pipeline on your branch, which does the initialization the plan and the apply, and it destroys it at the end. So what I'm envisioning here is, as we like, add more features to this.

A

As we add ansible configuration, we can do the init, the plan, the apply and the configure run some inspect tests or run the ansible to do some validation testing and then bring it down. The downside of this is that there's only one sandbox environment, so people have could potentially step on each other. So that's you know, that's not too nice, but um I think we'll see how useful it is. So this will do basically the same kind of thing except it does it in the sandbox environment.

A

It will actually build the uh build, build the infrastructure and then and then tear it down.

A

So we see that the staging plan completed, which will be like the same thing I saw locally, which is creating all the petroni servers, creating the pg, bouncer and console nodes.

D

Sheriff about the sandbox quick question: um this is in a separate project, right yeah. If.

C

It's in a separate project, then you can do concurrency of one for that uh specific runner, so you can actually queue up all of these things. Now the problem is the problem. Is that you know if you have five engineers work on this and trigger five pipelines, all five of them are gonna expect immediate response, but at the same time you can make sure uh that things are gonna be sequentially executed.

A

Yeah, that's a good idea.

C

B

C

Idea I, like it yeah. I think it could even be like a more complex rule, not to be a manual but like uh like create some rules when it should be automatic. But this is a great.

A

Step so this time um we'll also apply in a sandbox environment, the vms, I'm hoping that this succeeds, I'm not sure if it will or not we'll see.

E

Do you think that might run the race of being a very long process to just create stuff configure stuff yeah.

A

Especially like you know going through the configuration. This is why I was made it a manual job thinking that we would occasionally do it. Maybe like obviously like, if you're adding a new database chart, then maybe you wouldn't do it right, because but if you're changing your, if you're changing your ansible configuration or terraform configuration, then you probably would do it, because you actually need to confirm that you know new infrastructure is being brought up right. That's kind of the idea. For now, um but yeah hey the terraform run completed successfully.

A

um I was really like hoping using terraform managed to state on gitlab that I'd get the nice mr widget. Unfortunately, because we commit code to gitlab.com and our mrs on gitlab.com. We don't have that benefit, so uh maybe we can figure out something else I was thinking like woodhouse could maybe post back to the mr with the terraform summary, but we do lose out on a little bit because of that after the applies uh done for the sandbox environment, then it just does a terraform destroy at the.

A

F

Hey john yep, um I have a question about the sandbox, I'm not sure if I missed the explanation earlier um where the traffic state will be stored for this particular environment.

A

It's configured or it's it's stored in an object, storage bucket, that is in the ops environment uh because it uses gitlab, managed terraform state, which is a feature. So um when you configure terraform you're telling it to use the gitlab instance and the gitlab instance is configured to use, object, storage. So it's a single bucket, that's indexed like there's a the object is like there's an object per or there's like a directory per project and in there there's a key for each environment, so um this would be using the sandbox key.

A

Does that answer your question.

F

Yes, um I was just thinking about the case where multiple uh engineers are working uh in the sandbox.

A

Yeah, so that's a problem and like what marin was saying earlier, like maybe having a runner with concurrency of one, would help with that. um We do. It does use locking you can see here like um this is the state file. So it's like you know using the ops instance, and there is a lock to prevent possible problems, but you could definitely see all sorts of badness like with you know. You have two pipelines running simultaneously.

A

Both of them hit the manual job to apply the sandbox and um like the environment's getting destroyed, and someone else is trying to apply it. It wouldn't be. It wouldn't be good, so we need to. We will need to figure out how that will work.

A

Cool and that's the end of the demo I'll go back to the document. If there are any questions, no no one has questions.

F

Thank you for the explanation.

E

um I was thinking about something, so you showed that there's a like an intermediate mode, there's a from your java environment. You pulled a module and then what.

B

Are the references.

E

So in that one, you have a set number of console nodes and there's a number of pg bouncer nodes. um Can those be over greeted if we want different numbers on staging and on production, for example,.

A

Yeah, that's, that's the I um that's the idea, so um I guess what I was for the for the setup of this. I was thinking.

A

um Let me look at the sandbox since I checked that in um so this is like this is the module you, you would define and um yeah and you don't really set any options by default. But if you go into the terraform module cluster.

A

You can see like you can you can set? Well, no sorry, my mistake, like yeah: these are hardcoded now, but we could make these variables. Okay,.

E

Because yeah, that was nice.

A

E

A

Yeah, so it's the idea is to like use sensible defaults here.

A

I imagine that this this module will start to get more complicated, as we like, basically it'll try to leverage get as much as possible, but for things where we can't use get we'll just define our own. You know resources here um and if you didn't see the note on get like for now, I just I'm just baking get into the image itself and um when you run things locally, you have to create the same link and I'm still working or still trying to think about how this is going to work for ansible.

A

um But I think that'll that'll come next.

E

A

Any other questions.

G

When you're talking about the uh the doc image being massive, was that because you're building ansible from source.

A

Yeah, I think that's the reason why I think if I used a pre-built um image that had ansible it'd be a bit slimmer or maybe I can figure out a way to slim it down it's building, so I'm building python for one, because it's using asdf as like the source of truth for the python version. So it's doing an asdf install which builds python from source and then it does a dip, a pip install of whatever ansible version you have specified in your pippin file. Maybe there's a better way to do that.

G

uh Well, yeah, I I've had the same problems with specifically radius, asdf and and tariff and ansible um for ansible's piece on alpine. There is a prebuilt package, it's not always the latest and greatest, but it's always a little bit behind at least and then it seems to work, but obviously.

F

It's still dependent.

G

We're still dependent on our package and on the repo and that's a lot easier that path, that's what we do in our pipelines, but um yeah all the other paths to try and get aspects of the dock image are painful and they are quite heavy. uh Unfortunately, um for asdf, um it's the same situation but worse, because there isn't a package for it on people's out there um yeah in the end, I just actually circumvent it by.

G

um If you look at my pipelines, all I do is I read the tools versions file for the version number with regex, which I don't like, but that it.

C

G

And I literally just download the terraform package and because it's just a binary photoreform- and I just put it into bin well.

A

Why wouldn't you just use, um install terraform with asdf, which does the same thing in your docker image, for if.

G

You have to yeah, but then you'd have to install sdf, which brings in python and all that all that.

A

No, no, no it doesn't it doesn't it doesn't bring in um yeah I mean you can take a look at the docker file that I linked to and see what you think for just terraform. It's only 60 megabytes, so it's pretty uh it's pretty small and um it installs asdf.

A

It installs the python installs terraform with asdf, and it runs like I, I run everything as the asdf user, uh but then, but then for python. It's a mess like for deployer for deployer. What we do is we use the we use the python image, which is uh pretty small and then I installed, and then I installed ansible into it and that's a bit more reasonable. I think it's like 200 megabytes or so, but um yeah probably we'll need to figure that out.

G

I have experienced a spirit of that as well, but even ansible needs some native extensions to be built by.

A

Yeah, so it is.

G

A

Yeah, you probably went to the same thing I went through like yeah. You have to install like a whole bunch of um alpine packages, yeah it even like cargo and rust.

G

A

G

And then these like open, ssl and stuff- and I get it just- it gets painful quickly. Yeah.

F

E

G

I will be going to be looking if you have time, I'm going to be looking at doing an official docker image for get in the next release. um That will I'm not so obviously the I'm not surprised build time, I'm I'm bored about uh image size, but the idea would be a.

E

G

Image but um I'll look it I'll. Look at I'll be looking into it more at the time, um but it is a bit confusing that there isn't really that no one's really correct answer in an image and a nice there's no.

E

G

Which is the, which is the weirdness red hat, really should be making official one that would that make life easier, but yeah I'll have like.

A

Yeah it it might be worth splitting out, although I think with ghetto you kind of need, both in one image. I'm thinking like there'll be a terraform stage and then that ansible stage so having like this very slim, 60 megabyte image, makes the pipeline run much faster because it doesn't need to pull down a massive image.

G

Yeah I mean the finalists initially just have the a one image for to rule them all. But um if that is looking like it's gonna be big, uh then absolutely I'll. We'll look at doing official, terraform and ansible variants, but um I don't have it as one but we'll see how we get on.

A

Cool um great, um if there are no other questions, um point number three is just like what we're doing next point number four um and you don't need to speak up here, but I'd love to get some more help with this. So if you have bandwidth- and you want to help me think about how we're going to do the ansible config and I created issues for doing the cookbook audit to get a sense of like what's happening.

A

Next is a lot there's a lot of work, that's independent of whether we choose to use ansible or our existing cookbooks like setting up these environments, setting up the bpcs, creating the infrastructure in terraform, we're kind of getting to the point now where we need to make a decision on whether we're going to convert our base, recipes and cookbooks to ansible or whether we're going to use the existing cookbooks and that's going to be a fairly like deep dive into all the recipes that run on the infrastructure and seeing how much we need to carry forward.

A

So it's going to be a bit of a grind, I think, but it sounds fun to anyone who wants to join me. I'd be happy, for you know an extra pair of eyes. That's it. If there are no other questions, I think we can end the call thanks. Everyone.

F