GitLab GitLab and AWS, 16 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Provisioning Ready-To-Run GitLab for 50,000 Users in 14 Clicks and a Long Lunch (Or Any Size)

Description

The AWS Quick Start for GitLab Cloud Native Hybrid on EKS is in DEVELOPER PREVIEW.

Table of Contents:
00:00 - Provisioning Ready-to-Run GitLab for 50,000 Users in 14 Clicks and a Long Lunch AWS Quick Start for GitLab Cloud Native Hybrid on EKS
01:01 - AWS Quick Start for GitLab Cloud Native Hybrid on EKS
02:33 - Completing the QSG Form Using Deploy Now
11:09 - Checking on the Stack
12:26 - Delegating DNS During Provisioning
14:33 - Time to Log In!
17:34 - Kubernetes Admin Via SSM to the Bastion
19:55 - Resources

Helpful Background to Consume First
- Explainer Video: Overview GitLab AWS Implementation Patterns (13mins): https://bit.ly/3lpGwpZ
- Explainer Video: Overview GitLab AWS Quick Start for GitLab Cloud Native Hybrid on EKS (9mins)
https://bit.ly/3lrv6C0

Resources:
- Helpful Background above
- GitLab AWS Implementation Patterns for GitLab Cloud Native Hybrid: https://bit.ly/3xhyJww
- AWS Quick Start for GitLab Cloud Native Hybrid on EKS: https://bit.ly/3lej9j4

Playlists:
- Learning to Provision the AWS Quick Start for GitLab on EKS (https://youtube.com/playlist?list=PL05JrBw4t0Koi8VBnoVhmj_MstnbJjGw5)
- GitLab and AWS (https://youtube.com/playlist?list=PL05JrBw4t0Ko30Bkf8bAvR-8E441Fy2G9)
- GitLab Alliances Tech (https://youtube.com/playlist?list=PL05JrBw4t0KqaWCrU4avIY6TQveKTDMso)
- GitLab Reference Architecture, Performance Testing, Scaling and HA (https://youtube.com/playlist?list=PL05JrBw4t0KqgVugqCld2fUvrBulanchq)
- GitLab and Kubernetes (https://youtube.com/playlist?list=PL05JrBw4t0Kqp16k3nHTMdrWdATrcDZ5C)
- DarwinTheorizes at GitLab (https://youtube.com/playlist?list=PL05JrBw4t0KoH74CcAtP0Ze7JXjX7zT6g)

A

Hey everyone: it's darwin here, I'm a staff solutions architect and get labs alliances team. Today I want to take you through provisioning, ready to run gitlab for 50 000 users in 14 clicks in a long lunch. So let's take a look at how we'll unfold this for you we'll provision a completely operational gitlab cloud native hybrid instance, on aws eks.

A

You can do this for any reference architecture size, but we're going to do 50k just to show that we can do the whole enchilada so to speak. You fill out a brief form and wait about two hours in order to accomplish this.

A

Previously, a couple videos have been recorded in order to help you prep, for this so explain something about our reference architectures, as well as how the aws quick starts and the implementation patterns that I'll be showing. You work all together to help you with this, so we won't be going through the basic information that was in these two brief videos. I advise that you take a look at them. Maybe even pause the video now and go back and go through them.

A

The aws quick start is a co-developed partnership with git lab and aws, and it has been qualified by both companies as being h.a and also being as well architected by both company standards and then we'll also be talking a little bit about how implementation patterns and their bill of materials can plug right into this.

A

So let's take a look at the actual quick start. A quick start has a certain structure that we talked about in the previous video and if we go down to the section deployment steps, this is where it gets practical and starts telling us exactly what to do.

A

We can also launch the actual quick start form right here, so this is actually a cloud formation form, but it is in a special, quick launch or quick, create format. So there's only one form you hit, go and away you go, and this particular form has a lot of information on it.

A

If we go back to the quick start and go down here, it tells you you know: here's the things that you need to fill out. It does not tell you the values, however, that you would need to use, and so you would have to read through the reference, architectures and kind of calculate the information about what you should deploy in order to deploy this appropriately.

A

So in order to try to ease that calculation, we also have the implementation patterns and so for implementation patterns. We give you a bill of materials, and we explained this in a previous video, so I'm just going to zoom right down to the bill of materials for the 50 000 k. So you can take the information on this form and fill it right into the cloud formation form, but we're going to actually take an even further shortcut, and that is we've provided these deploy now links.

A

So I'm going to hit deploy now here the deploy now link primes, several values in the cloud formation form, mainly the ones that have to do with what instances types to pick and what numbers of instances to pick for each item. So you can see here, we've it's given us a more informative, stack name now. The first thing you want to do is select your region, because, if you don't select it now, selecting it later causes the form to reset it'll.

A

Keep my values that I have brought in on the inbound url, but anything that you set manually will be cleared and that can be a pain to have start over. So, as I go down here, I'm going to need to pick my availability zones and I'm going to pick several zones here.

A

I'm going to pick the number that matches the quick start number down here of three. So that's three clicks.

A

We do not need to play with the subnet information and that's because for the deploy nows, I have used the quick start, template that actually launches us into the create a new vpc template. There's one for creating within an existing vpc, in which case you'd have to extract your subnet information here. But we won't need to do that and because the underlying eks quickstart comes with a bastion host, we can. We can configure this so that we will be able to have secure access to our cluster without exposing its endpoint.

A

I'm going to need to provide an external access slider here it does not actually have to be a valid one, and this is because we're going to use ssm to access it. So I'm just going to put in a fake one here.

A

And then I will also go in and select my bash and hose type. Now, normally, a t2 micro will be fine, but I want to do performance testing later, so I'm going to do an m4 x large so that I can drive performance testing fast enough for the instance.

A

So that is, I think, we're up to five clicks and if we go down here, we can see that the eks public access endpoint is disabled and I'm going to leave it. That way. We also see here that we've selected the c5 4x large. Now when I launched this template, I it said one quarter scaled.

A

So, whenever you're setting up large infrastructure, if you have scaling involved and you try to set up your full scaling right away, if that scaling fails, usually the entire stack fails, and that can be pretty painful if you have a large scaled stack. So in this case it's setting seven of 28 of the totals instances that we need, and then we can scale it later, which we'll do also. We've picked the c5 for x large. So let's say you want to test the 50k setup and you're like boy. Where should I do it?

A

A quick cheat on this is to use the spot instance advisor. So with the spot instance advisor. It generally tells us, what's the best spot instance pick, but it turns out that that also indicates the general availability of that instance type in that zone overall. So if I look here, I loaded up us east and I search for a c5, forex large, and I can see here that its termination rate is less than five percent.

A

But let's say I said, hey you know what I want to try out us west oregon and we go and search the same item, and here we have five to ten percent. So that means there's more demand for this instance type even for on demand when we're not doing spot, and I've checked out some other regions that I typically use for testing and they're, even higher 10 to 15 percent. So that's a good hack to know where to deploy your experiment.

A

It also helps you with sizing, so I'm doing 28 c5 4x larges, uh potentially to make my scaling a little smoother. So I'm not inheriting a massive instance when I need one more pod, but also you could tune this to be twice as large of instances and then go back to the spot advisor and check out if there's a better region, to deploy your test in I'm going to keep on going down here. So I think we're still at five clicks.

A

If I'm counting right, I'm going to install grafana so I'll change that to yes I'll leave my aurora database the same it's already set up my aurora instance size. The database admin password. I'm going to set it's important to know that this is in a developer preview. So there's a bunch of little things that need to be handled before this is ready for production. Hard launch. One of them is this database will, in the future, be auto, generated and placed in secrets manager like most of the other secrets in this stack, the other, the other.

A

There are other little details of things that need to be sort of fixed up before we're ready for a hard launch, so I'm going to actually just type in a password here.

A

So that's six clicks and then um caching has been set correctly. Number of replicas also the cache instance type, the dns name. We can't set that for you, so I'm going to do q s g for quick, start guide, 50k, dot, dev ops for the win- and this is a domain that I have under route, 53 management and it's going to create a sub domain qsg50k and then I'll have my git lab instance.

A

Name on the front of that, I'm going to create a route 53 hosted zone and I'm going to create a ssl certificate using acm. If you have your dns somewhere else or you host your own certificates, you could change these if you're just doing an experiment. This can show you having a completely ready instance, all within the automation, I'm going to enable an smtp outbound service. I don't need to do any of the other details.

A

If I'm creating a new one and then we leave the helm chart as it is, there has been a regression discovered with aws, classic load balancers and get labs 14.0 distribution, it's being fixed for 14.1. It's going to be a point release of 14.1. So if you just want to see this in action, you might stay with 1312 or wait until we're at 14, 1 or 14 2..

A

It also gives you links there on how to look up the get lab helm chart version you have to. If you're going to change the version you have to select the helm, chart version and the appropriate matching git lab version off of the web page. That's cited there, so that should put us at seven clicks.

A

uh We have all of these things for the git repository storage, which is the gitly cluster already preset. So we shouldn't have to set anything here and as we go down, that's it so 8 9, so we're actually under 10 clicks, create stack. Okay, so I failed to provide my ssh key pair for the actual bastion hosts. This is just a selection, so you have to have a pre-created key pair, so be sure to create that ahead of time before you get in here or if you recreate it, you'll have to reload the form.

A

So that puts us at 10 clicks. If we had gotten it correct the first time now the stack is launching, and what I'm going to do is let this run and come back to it periodically to see how it's doing the aws quick starts are a set of interdependent nested stacks of cloud formation. One of the interesting attributes is that it's also open source and there's dependencies, so the gitlab stack is actually depending on the eks quickstart. So as as it's improved, it already has spot support.

A

It already has all kinds of different, interesting support for native eks functionality. We can inherit that functionality and as it's improved and bug fixed and optimized, we can inherit all of that. So while the stack structure will end up being fairly complicated, it's got the benefit of a kind of a dependency hierarchy within it. So we'll let that run. If, if all things look good in about two hours, we'll have our get lab 50k instance set up and ready to go all right. Just a little check in here.

A

You can see that a lot more confirmation stacks have been created over here on the left. It can get kind of messy, so you can de deep select view nested so that it only shows you. The root stack just wanted to also show you how to traverse the stack to find out kind of where it's at what's going on and if it was to fail how you would kind of follow a path in order to find the root failure.

A

So, whenever you're, looking at the root stack, you can hit resources and in this case we're going to follow, create in progress to find out what's currently being created. But if the stack had failed, if it said create failed, we could do the exact same process for following down to the to the bottom. So in resources, I see that this one's creating progress. So I click it. It's going to load that particular stack in a new window. I'm going to go to resources again, I see this one's creating progress.

A

I click it and then hit resources again and I can see in this case the resource is not created, so I'm now at the bottom is actually performing this right now. So that's a way to drill down on either where the progress is or where a failure is so we'll check back on this in a little bit.

A

If you are doing the following things, you are creating a subdomain you're asking aws to manage your acm certificate. Then you need to do the dns sub domain delegation.

A

While the stack is running, if all of your dns is in route 53, then you'd follow the procedure that I'm going to show you, and this procedure is also in the current instructions.

A

If you're on a non-aws, dns or your dns is in a completely different account, then you would need to do this procedure manually anyway, we're going to go into dns. While the stack is running and we'll see that it has created a new subdomain qsg50k and we need to delegate the dns of the that subdomain in the root domain.

A

So we're going to go into this domain and at the time the stack is running it has this situation, you'll see that there's an acm validation record and that is actually what's causing the entire stack to be in a pending state for up to an hour after about an hour of operating, it goes into pending state for up to an hour. Is that it's waiting to find that record, but it can't find it until we do this delegation, so I have qsg 50k, I'm going to go into the route by the way.

A

You'll probably see a private domain for your sub domain as well, and you always want to be looking at and editing the public records in all cases and hit create record q s g 50k, select, name server here.

A

Paste in here leave everything else as a default, and then, once this record is created after two three minutes, your stack will start moving forward. So keep in mind about 50 or so minutes into the stack. The acm certificate will go into a pending validation state, which you could look in and find in the acm console or just wait until you see this public subdomain appear and that might even before validation is pending. So you can do it at any time.

A

You don't have to wait till the validation is pending as soon as the domain appears, you can go in and do it all right, our stack is completed. Building we can take a look at how long it took so down. Here is the beginning.

A

848 is when it started, and 10 38 is when it finished so 10 minutes shy of two hours. We now need to scale it up so that it's ready for our full on testing and we'll dive into that in just a minute. Let's log on to make sure that everything seems at least a basic logon is working.

A

If we go to our parameters, we'll find our domain name and I'm just going to put this into a new window so https and we're going to do git lab dot and then the rest and our front end comes up and we are running with ssl.

A

I am now going to go into secrets manager so get the quickstart puts the initial root password in the secrets manager of.

A

Of amazon in the same region that you're operating in uh notice to one of the things I want to call, it was the stack name that we had is everywhere. So this keeps the stack, unique and yet easy to identify all the things that have been created by that same stack. Both things like the secrets manager here, but also things like all the stack names and resource names throughout amazon. So it's much easier to understand.

A

You know what was created and if you clean up what will be cleaned up so I'm going to go to initial root password and I'm going to go down to reveal secret value. I will be changing this before this. Video is published so not a big deal and then I'm going to show you that you, this is uh base64 encoded, so I'm going to have to do a base 64..

A

First of all, echo the value to base 64 dash d for decode and copy it to bb pb copy, because I'm on a mac, so that'll put it on my clipboard and I'm going to do root paste in and see what happens boom we're in so we're. Looking good uh also, you might notice here in secrets manager that there is a grafana password so like, since we deployed grafana. Now we have an initial grafana password there as well, and I could do the same test to get into grafana.

A

I also just want to show you here that those login instructions are actually in the quick start itself. So we talked about the section here that talks about the what you set in the ui.

A

But then, as you get towards the bottom, it tells you how to get into the parameter store um and be able to get that password out right. Here's, the base64 decoding and then also tells you how to access kubernetes. So I'm going to start into this bit right here, so we will go into our aws stack again and we'll go.

A

A

Instances and the instances the bastion host will be named as such and so that'll give us our first clue so right here we see we have the bastion host and I'm gonna say, connect and I'm gonna use ssm session manager.

A

I'm not going to dive in right now to how many benefits there are to doing this versus logging in with a regular ssh client, but basically I don't have. I don't have to have any ports exposed. If I don't want to so connect here and there's a couple details, so this bastion host has the kubernetes utilities installed, helm and kube control. It's also got. I am permissions to the cluster, so you don't have to worry about having the right permissions. The the bastion host has them automatically.

A

So you want to control who can get to the bastion host, which you can actually do through ssm and ssm, also does very thorough logging of who accessed the bastion host and what they did on it.

A

So then, I'm now going to switch users, and once I do I'm going to use fully pathed commands for kube control, you could create your own links. If you want- and I'm just going to do a get pods to see, if I'm connecting correctly- and you can see- I am connecting correctly now in order to scale this stack to 50k, I'm going to reference you to the performance testing video, which is the next video in this series.

A

It will allow you to see how to scale up an instance in preparation for performance testing when that first part is done in order to scale up. It takes less than two minutes. If you, if I wasn't explaining it as we went along, so some total will be able to get to the full 50k in two hours, so that's a wrap and getting gitlab set up in about 14 clicks and about a long lunch two hours.

A

You should note that we would follow the identical procedure, no matter what size of gitlab you're setting up and it would take about the same amount of time as well.

A

So here's some of the resource links for the section that we just went through these should also be found in the video description down below. But we have the explainer video about aws implementation patterns at git, lab that help you to quickly get uh things set up, and then we have an explainer video also about the quick start itself that precedes this one. So you kind of get a better idea of what it is and how it relates.

A

Aws implementation patterns for gitlab cloud native hybrid, which is what we were the document we were using as we went through here in deciding what to set up and then finally, the aws quick start for get lab cloud native, hybrid on eks, the actual launch page for launching it. If you don't want to use those deploy now links that are a part of the implementation pattern.

A

If you're going to be setting up, git lab cloud native hybrid in a scaled aws environment feel free to reach out to us at gitlab, and we can give you a hand thanks. A lot for your attention.