GitLab Support Team, 15 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Automatic Gitlab Reference Architectures in Support-Resources (Using GET)

Description

Pairing session where we demo how we can provision Gitlab Reference Architectures automatically (by using GET as a module) from Support-Resources

Support-resources: https://gitlab.com/gitlab-com/support/support-resources

Gitlab Environment Toolkit - https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit

1K Reference Architecture in GET - https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/quality/-/blob/main/1k/terraform/environment.tf

Gitlab Reference Architectures - https://docs.gitlab.com/ee/administration/reference_architectures/

A

Okay, um so thanks john for joining me today, uh we're gonna try to demo um the support resources. Get implementation uh basically create a reference. Architecture from support, resources and john doesn't have any experience with that. So hopefully this will go well and it will be easy and we will run through any uh problems that others might encounter.

B

Okay, thanks cool.

A

Yeah thanks for joining me, um do you mind sharing your screen, so we can start looking at the project and example. I think okay.

A

Okay, I see it, I think cool. So, let's go to support resources. I like that you have almost as many tabs open as I do.

C

Apologies, I did a lot of reading about ci cd uh yeah. I took a ticket yesterday, yeah and.

A

Always hey jason.

B

I think I have your.

A

Yep, it looks like it so we're going to start from the example uh which is in the examples folder uh in the repo, the second one yeah and then the very first one is get refs.tf.

A

So we're going to grab that and create a new branch with the content of this file. This file um and um yeah actually we're gonna, create them, because if you're gonna do it here, I don't think it's gonna run so we're just gonna uh create a new uh john l uh tf file in the root yeah, a new branch.

C

B

Let's just not follow the pattern.

A

You already, okay, let's uh try this formatting.

B

But let's try it again.

A

I think you need to use dashes, maybe underscored on working branches.

B

Let's see, maybe I have these branches already created is weird.

B

Let me see the branches.

B

A

Yeah, just something.

A

A

I don't know you can put caps anywho, so uh we're gonna need to add a well first of all the license file. We're gonna need a license file uh that we're gonna reference in the tf file. The terraform file yeah just upload that to the root, or I mean just what whatever we can. We can then.

B

Send my documents license.

A

Yeah just upload the license folder or whatever.

B

To here, yep create a new folder, or would you just upload the license.

A

Whatever I mean, I, I typically do a folder, but if you want, you can put it in the root.

A

And now again, upload file.

A

And do the the license name.

A

Cool, okay and now in the root: let's do john dot tf and copy paste the stuff in examples.

A

B

Here so let me go to examples.

A

Yeah we're just gonna copy yeah, we're gonna copy the content and we're gonna paste it in your uh john.tf file. That's gonna be in the root of the repository.

B

A

B

So I've copied it here's my branch. So where do you want me to put it.

A

uh Just do a new file: okay,.

A

And name it whatever.tf.

A

uh John yeah.

A

That's tf yeah and then just copy paste that yeah and don't submit just yet don't commit. We need to remove the the slash asterix in the beginning and at the end, because that's basically commenting out the whole thing.

A

Okay, um yeah and then we have to have. We have to put your ssh key in there.

A

All right: do you have another uh branch where you have that already or.

B

You can yeah stored in your projects. Yeah. Where is your ssh key store? Is it stored somewhere here.

A

I think you, you have a branch where you already have like a tf file. Have you have you used support resources before.

B

A

Okay, so on your other branch, you should have your ssh key, just like in this case, so you've probably defined that already somewhere.

A

Yeah just go to branches and search for your branch. I think that that'd be faster, so there yeah and then do john little or.

A

C

A

Little branch gitlab runner- let's try that and there is the uh there's a gitlab instance runner tf file, the the one before last yeah and there's another ssh key so copy that.

B

A

All right looks good, uh just yeah delete that first part: uh okay, cool anything else. um Let's see well email, but that's not really relevant. That's just for for your uh your ssl, so I mean you can leave that in or we're not using ssl, either way so yeah and then labels, that's fine, yeah, the rest and one other thing that you need to replace is whatever it says.

A

Vlades uh just do uh john, but do ctrl f, so we can do it all over the place, yeah and then push push the like the sign that says uh bigger than just before, uh just in front of vlades. There's there's a! uh I don't see your cursor! No, no! I mean in the search box, oh uh yeah there and then yeah just do john or yeah and then replace all the other one and yeah. It should have replaced all of them.

A

Okay looks good, let's see uh just a second yeah that one you can. You can take that out, uh because that yeah and then put another one at the end at the very end, because there's multiple configurations as examples, so you don't need all of them. You just do the first, the very first one yeah, that's it and let's go up again. Hopefully that's the 1k yeah 1k. It says 1k in the in the title: okay, okay! So at this point we should be. We should be erased here.

B

So where's the specified one k. Sorry.

A

B

uh Is it what was the 1k uh where's? It specified.

A

So the 1k is basically you have to pull that if you scroll down a little bit, there's a link there in in gray.

A

So if you follow that, that's actually an example of a 1k architecture and there's actually more examples like that for 2k, 3k, 4k, 4, 5, 10k, etc. So those are under the the get project.

B

What does it represent? What does the case.

A

uh It's one thousand users, uh it's from from our uh reference architecture, uh uh um setups.

B

A

Okay, so see, there's 10k 1k, whatever terraform.

A

A

And this is it, the only difference is that it says somewhere uh towards the middle actually lower from the middle. It says h, a proxy external blah, blah blah, that's actually missing in yeah, just yeah that one that line you don't need to add that that's that's just something that uh support resources uh takes care of automatically the way get works is you're supposed to create an uh ip address to actually provide to your um external uh aha proxy.

A

But that's that's done uh in the background in support resources. Okay. So at this point uh and if you, if you want to do like a 10k reference architecture which is 2, 000 users or 5 or 10, 000 users uh there's other examples in there. So you you just need to add more more uh nodes, so elastics elastic node count would be three, for example, for a 10k uh nfs would still be one but rails nodes would probably be.

A

uh You would see some other other types of nodes like prefect and a lot of yes.

B

So the clarify for the name month terms this each one of these is a get lab node. This one here is for elastic this. One here is for nfs exactly.

A

The first variable it's the number of nodes, so it's just going to be the one elastic uh search, node uh and then elastic machine type. That's just the the type of machine in gcp. So it's going to be an n1 standard, four, which is something like uh two cpus with 16 gigs of ram and some other stuff.

A

So that's that's just to know the the spec that you're using yeah and, for example, between the 10k and the 50k, there's no difference in the number of servers and like the the layout, but the machine types are more, are beefier in the 50k. So yeah, that's that's where the difference is there.

B

Do these run on uh tcp or yeah.

A

C

A

um Okay, so at this point you can click commit.

A

And that's it: we are supposed to be seeing a if you go to cicd on the left or click the well you're, not seeing it yet the the thing the pipeline, but yeah. It's.

A

Running and it's going to take about 20 minutes, 20, something I I mean we we've cut that down to from like 50. it used to be. It used to run very slowly slowly, but let's say that, for example, in in a 10k architecture, which is as high as I went uh in testing because it gets really expensive. After a certain point, um it spins up about 40 something servers. So you can imagine that that amounts to some some costs uh so yeah and uh in this particular yeah go ahead.

B

And so how do we once it spins up? I understand I profess the url and the password. How do we then run it down so.

A

You just you just comment out everything, so you just comment out and then submit, and then it will de-provision everything yeah, but obviously uh your your uh your entire uh setup will will go down where it will cease to exist. So uh don't don't count on the persistence of data or.

B

Yeah anything there cost obviously there's a cost associated with yeah.

A

So one of the things that I wanted to point out is uh so for a 10k, but okay, uh some something's changed since that uh benchmark. uh We actually use less servers now by by five or something, but they actually run run a test and uh having a 10k architecture running in gcp for one month adds up to something like seven thousand uh six hundred uh dollars before discounts and all that stuff that gitlab gets. But anyway, that's that's a lot of money right.

A

It's it's a whole lot of money to just have something lying around so yeah. We have to be careful to not forget about our resources and that's why we have frugal times and all the other good things, but still can.

B

You elaborate a little bit about that. uh First.

A

So frugal times uh it's actually in the readme, we actually have some time. So if you want to go to the to the readme, uh just click repository.

A

um And if you scroll down, I think the second yeah yeah go, keep going, uh see frugal program resources, so those are uh some labels that you add to your instances uh to basically have them shut down when you're, not working uh so by default. I think uh your resource, if you don't add uh the the labels, you will.

B

Do I have these audits frequent times.

A

Yeah, we can actually see those. It says: frugal hours equals uh non-available, minus 19., none of it not available means you don't have a starting hour. So whenever it will get shut down, it will not be restarted automatically.

A

So your your resources will be running until your uh seven o'clock and after that they will be automatically shut down and never restarted again. Unless you do it manually, you can instead of the n a thing you can put there, nine o nine and it will actually restart it every morning at nine o'clock and then shut it down at seven in the evening. So.

B

Just just to clarify that what does the n a represent.

A

So that means you, you are not specifying uh starting time. So that means that it will not start uh automatically okay, so you need so. Let's say: let's say you spin up something now, uh if you leave it like this, it will shut down at your uh seven o'clock, so 1900 hours, okay, yeah yeah, and it will not uh start again. You need to start it manually, so you actually have to go into the gcp console.

A

Look for all your resources with the prefix that we've set and um and then basically start them up to use them again. Okay, if you, if you don't like that, if you want to have it waiting for you at nine o'clock in the morning, uh you can set 09 there and at or whatever 07 or um whatever time you need, and you can, you can have it start up automatically.

A

So it will start every morning at nine o'clock, let's say and then be shut down at seven and then uh the next day, again at nine and then again shut down it at seven and it will not be um up over the weekend, so it that doesn't apply to week uh weekend. It will be shut down, but you can obviously start it uh automatic uh manually. Sorry and there's also an option. There's also an option uh that says frugal skip. So the the label is frugal.

A

Minus skip equals true, and that means don't ever shut this down, because this is important. I need it, but you actually have to go in there and add the label. If you don't add a label, it will just uh shut it down at. I believe it's seven or six in the afternoon uh in the afternoon of so the time time zone is of the uh region, gcp region, that you've chosen. So for you, it's europe west, four, which I think is in the netherlands.

A

So it would be uh uh six o'clock, netherlands time it will be shut down and then restarted at nine, and that's that's the default behavior. So if you don't add anything, your resources will be shut down at six and restart it at nine.

B

Okay, that makes sense uh yeah thanks for explaining it. Basically, we did not er. Oh.

A

Yeah yeah you're right, uh it's probably gonna fail. So that's that's a that's a good thing to to point out um yeah. We need to add the the proper uh path.

A

So it's licensed not licenses and then the name so cancel the last s.

A

Yeah it's still running, but I think it will fail at the ansible part where it's not going to like the.

B

So- and this is sensitive yet.

A

Yep actually, the path- I don't think it's case sensitive but so and it's gitlab.licensedmind gitlab.gitlab minus license.

A

I hope that's right, you should have copied it, but yeah. I think it's good, um and one thing I would I would do now is. I would actually change the name so uh copy the the name of the module so john little ref arc, 01 1k, so that whole thing- and I want to replace that with uh like the same but o2 just to change the name, so it it gets, respawned, basically yep, replace all and then commit. I think you you already canceled the pipeline right.

B

Yeah, it did okay cool and we have our okay, I'm going to put that n a to 15 or size set.

A

uh Be careful not to you know not to hit that while we're in the demo, because you will see your your resources being shut down from underneath you so.

B

Okay, that was sleeping for this essence. Okay, so that's run and.

A

Yeah, let me actually check on mine to see.

A

Yeah, no, it's good, it's it's running and the other one is cancelled. So that's that's good. um Maybe we should also go through some of the some of the options in the module because there's there's some stuff in there, not just the the resource types. There's also some options that you have when you're you're speeding that up. So uh let's see the uh jlittle machine.tf file again yeah, so uh yeah just scroll a little bit. That's it um so see uh where it says: prefix, that's that's!

A

Actually the name of the of your resource stack, so that's gonna be be if you're looking for your resources in the gcp console you're going to be looking for that, um then the project you're passing the the support resources, gcp project. I think that's also.

A

I mean that can be missing because we are uh defaulting to to the val to the support resources uh project, but if you're using it somewhere else, you would need to pass that the labels uh we've discussed so frugal times frugal hours and it also has a label with your username and support team and some some other things, and you can add more if you'd like the zone is where your resources are located license path. Obviously, we've just changed.

A

uh We should have done this before we actually hit commit, but anyway the first time um the gitlab version you can specify it or you can. You can actually leave that blank and you will get the latest um or actually have it missing. Comment comment that out um ansible ssh, key and gcp credentials that just needs to be there. You don't need to touch that. So that's it. I mean that's, that's all the the things that you you can put in um and then the rest of the stuff.

A

So uh that's in between those line, those hash lines. That's basically taken out of uh get and just put here from um to actually define the the reference architecture, and then we define the the source of the of the module and that's it and then the the last. The last part is the output that we need uh just so terraform displays uh nicely what you've created and, like your your password and your ips and dns entries and all the other good stuff um yeah if you'd like. Actually let me look yeah.

A

uh While this is spinning up, I think it's going to be a few more minutes like maybe 15 or so I can share, and I can show you um what my previous run looks like that's a 10k, so it's a little bit more involved, but just just to see like the output and all the other stuff.

A

um Let's see here, okay,.

A

Okay, uh are you seeing my screen yep? Okay?

A

So that's your pipeline. That's running now and I ran one just before I mean it was running while we were still speaking so that amounted to 37 minutes, not bad. I've seen it complete in like 32, but that's as low as it gets um and yeah I mean this. Is the output? uh Don't don't be too uh scared about uh about it? It's it's easy to follow along. Actually you don't you don't need to see all of it. You can just go to the end and then there's like your stuff. uh Well, I've.

A

I've spun up a few more things as well. But what you need to see is this, so this is the dns entry for your.

A

Okay and then you have your gitlab password, which is automatically generated.

B

It's always root: yep yep.

A

And that's it I mean you're in and if we go to to see it's probably going to be yeah 13 11 2, because that's what I've put in there and there's there's a license as well.

A

It's 100 user's license because I've I've added a licenses. uh I believe that if you don't, I actually know I'm certain that if you don't add a license, it will fail because it it needs it get requires it um and yeah here. There's it's basically uh a description of what has been generated. You can go through this or not. uh The useful stuff is at the end, but yeah so for a 10k uh architecture.

A

So that's how I know it's like there's three console nodes um and then there's three elastic nodes and then italy, three digitally nodes and then a gitlab nfs node.

A

So four gitly rails nodes, um an h proxy, an external hd proxy and then an internal hg proxy.

A

Three pg bouncers, which would be these three right here: three postgres nodes.

A

Three prefects.

A

One uh prefect postgres so the db for for prefect um three redis cache nodes.

A

Three red is persistent nodes and four side kicks and I think that's it. But that's that's a lot right and you actually want to visualize this.

A

It's you can go here to reference, architectures and yeah. This is it so this is what you're spinning up and the object storage that go with it. So four side kicks three consoles. Whatever I just mentioned before: okay yeah, you have a question.

B

Just the object storage, which can you go back to the object, storage, node.

C

B

Is the object, storage node also, where is lot.

A

That's in gcp as well.

B

Okay and automatically it's s https. Is there an option? No.

A

It's http get, doesn't support uh https, yet so that's a that's! A limitation on gets part. So each it's http. If you, if you look.

B

No, not try not get uh it's. The url I mean is the url.

A

Yeah, it's http, it's not https, uh see the it's okay.

B

Yeah, okay, so then, for what happens for I know, for example, in container registries, you need https. Is there a way around this for or is it always going to be http? I.

A

Mean you can do stuff afterwards, so after you've installed gitlab and uh like the whole stack, you can actually go in there and do the https related stuff, but that's considered a manual step at this point, um not sure about the plans. uh I can ask about that uh in the next get sync, but yeah I mean that's, that's a limitation as far as I know for for right now.

A

B

Yeah yeah, I was setting it up recently and I knew I had to have https, which was a little bit of a pm. Yeah.

A

B

A

So your pipeline is still running it's going to be a while still, but I mean I can see already that uh it's it's started to do the ansible part. So I know because I mean this is the output that we compile and create and like this is the the ansible configuration files and then see it says, running playbook, that's actually uh ansible starting to run after it's been configured.

A

uh One thing that I can show you, which is pretty cool.

A

Let me actually grab the link for the console.

B

Oh, that's interesting. It actually provides console link to gcp.

A

Okay, so if I take your prefix, this is your prefix, the one that you've said we were discussing before. If you go here still loading, that's fine.

A

Okay, so if you go here, you can actually look for that or the labels, but this is easier well without the dots at the end.

A

And there's your resources right. This has been generated uh in comparison. I can also show you mine, which is 10k. Yours is only 1k, so you saw it's only one elastic one rails, one um nfs and so on, and so on, actually not that one and so on and so forth. But I think it looks.

A

Let me just grab the prefix.

A

10K looks like this.

A

See so way more stuff, it's 37 37 uh servers resources which have uh firewalls and uh ip addresses and a lot of stuff attached to them. So that's why I mean we. We kind of I mean we're free to use this, but we have to be mindful of the the resources that we're spending up, um so I mean yeah we can. We can go for example, and check out this console uh box and we'll just see the labels that we've set and, like the the all the tags that we're using in the firewalls um see.

A

So I have uh frugal skip set to true, because that's that's how I set it in my in my uh here my code, but yours will be not available 19, so it will be shut down.

A

The another label is your your branch, the branch that you are spinning that up from so you can actually go into the the uh the project and look that up and see. um What's up with that that branch, if you find some resources that you didn't know, you had and then yeah a bunch of things that are also related to get and then owner project and user.

B

A

B

Actually share my desk top one second just to get to the same position as you sorry can you repeat, can I share my desktop, uh so I can get to the same screen that you are.

A

Okay for sure, I'm also going to scroll down to see the tags.

A

Okay, here they are so these are the network tags which we use to set up firewall rules, and I mean I can show you the firewall rules as well and, like the whole, even the state.

A

Let me know if you need me to show you anything.

B

Well, actually, I find that it's okay, I find it.

A

So it's vm instances and then you just search for your prefix and then there should be a list of things.

B

A

B

Let me just check.

A

Yeah, do you want to share yeah? Okay, I'm going to start my sharing.

B

A

Yes sure and then you, if you look for your prefix here, yep.

A

I think it's the refwork too now.

B

A

Yeah yeah yeah, so.

B

Start so that I can see that there's one two three four.

A

Yeah, the o1 ones are from from the pipeline that you cancelled and those will get cleaned out at some point uh from this pipeline. So you don't need to worry about those.

B

Okay, and at the moment I can see that there's one two three.

C

B

How many would be expecting five.

A

um You can just do ref arc o2, so jlittle referral ii, and it will just show you the those that I mean with the current prefix.

A

B

Yeah five yeah, that's! Okay! Well, that's everything from from my perspective, vlad thanks for taking the time to to demonstrate that's very useful and if you could send me the recording for future reference, because I'm like a cool fish and I'll forget.

C

B

A

Now the only thing would be uh for the the pipeline to complete. I think it's gonna be a like five more minutes. I don't know, do you have to jump or uh yeah.

B

I have to go okay, yeah I'll. If I have any problems, I'll I'll contact you, but that's I'm just looking at the pipeline. It's still technically still still still creating. So actually.

A

I can I can hang on the line and until it uh it finishes, uh even if you have to drop- and I mean for the sake of those following along uh just just to show them the pipeline completed and what the output looks like.

B

Okay sounds good and I'll I'll I'll stay on and if you're still here when I come back I'll rejoin.

A

Cool, but I I think it's gonna be just like five more minutes, so, okay, no problem cheers thanks. Thanks for joining.

A

Okay, so I guess I can also explain what's going on here, so I've said uh the the playbooks has uh have started they're actually running in parallel. There's there's a way to do that in uh in get when running answerable, you can run them sequentially or in parallel. um The sequential run is more verbose. Shows you more information like what's going on what operations are are going on? uh This is just showing you that it's still creating the grading and yeah. It just says when it's finished and successful.

A

um At the end, though you, if your pipeline fails for, for whatever reason uh in the artifacts, you will have logs for uh the ansible run and like a bunch of logs for terraform as well. uh Mostly the issues that that arise are from ansible. uh Terraform is pretty light in the sense that it's done in like six minutes and then the rest of like 15 to 30 minutes is ansible running a bunch of commands.

A

So yeah we see now postgres playbook, finished and gitlab rails. So I think this is the this step before the last one. So one more after this, the post upgrade.

A

A

Interest, another thing that would be interesting to see is the configuration files that were generated, uh so these would be the ones that get is also referencing uh when saying to so create a var that yamo file in the documentation.

A

So this is the file, that's actually been run and you will find that in the in the artifacts as well and there's the where is it it's executing.

A

10 seconds.

A

Yeah so uh inventory cb ammo, and this is the content of that and again you will find those in the in the artifacts.

A

So essentially, if you're uh trying to start up using get, you can use this as a as a baseline as a reference point to kind of see what it it's supposed to be configured as and the structure of the files um and then do your own manually uh do like the terraform run and then the answer will run and create that that structure, because I know a lot of people struggle with that in the beginning.

A

One thing that I I should also point out: uh I don't know if we can still see that yeah we can, we can still see it. We are now using a custom image.

A

This is the name.

A

Okay, uh we used to use um like a alpine image and then install a bunch of things that we needed during the runtime. Now we're creating to to save time and to to make it more efficient. We actually create an image that we deploy, that has all the stuff that we need, like python, ansible um terraform we're actually installing separately during the runtime, because we want to actually be able to set the the version from outside, uh but like gcloud uh that we are, you know using for authentication and stuff.

A

That's also pre-installed in this image. So you we don't have to wait for it to install during the runtime, so that cuts the time.

A

Okay, let's see where we are, I think we're supposed to be about done.

A

Nope, just a little bit left, maybe two or three minutes, and then I just want to look uh really briefly at the artifacts, since I've mentioned it and the structure in there, okay, so uh post configure that will take about one minute so and then we'll see the output and the artifacts structure.

A

Another thing that I might want to mention is that, if you're using an older version of terraform- let's say 12 4 and you have uh instances that are spun up on there and you want to upgrade to the to use the version of terraform that actually allows you to to do um this kind of stuff.

A

Like reference architectures, um you will need to upgrade your your version, so that means changing the version from 12 4 to first 13.7, uh just change the the version run it once it will update the state file, nothing will get created. It will just take a a short while and then go to 14 4, which is the minimum for um for reference architectures for for the get integration, and then we see where we've completed the run successfully.

A

All playbooks finished after 90 minutes.

A

And yeah we see that we have our resources here.

A

Just like I went previously with with john. What I wanted to look at is this stuff, so you will have the lab environment, toolkit and tf state.

A

If you go in here, you'll also have the logs to look at if something fails or whatever.

A

If you want to see the structure, you can go back to well, you can download this whole directory and go to ansible, and you will have the entire um structure that you need to actually run ansible manually. So you can download this and then go in your console to this location and then run ansible playbook, whatever minus uh all that whole thing from from um from the get instructions and yeah you'll see in environments, support resources, inventories you'll see the vars.tml file inventory files which uh I've highlighted in the pipeline as well.

A

So these are the ones that you have to compile to actually be able to run it and obviously the environment file, which is in terraform somewhere one more star.

A

Form um environments- well not here modules.

A

Gitlab rough arc.

A

A

I'm gonna edit bookmarks in chrome,.

A

A

The right ones.

C

And you go to the phone.

A

A

Yeah, so this is the one that you need to create.

A

Yeah you can find that in the in the artifacts, so yeah, that's it. If you have any questions, uh please feel free to contact me and I will set up parents session or answer your questions. Thanks, bye.