GitLab Solutions Architecture, 7 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Automating Deployments to ECS with GitLab CI/CD

Description

Contents:
0:48 - Steps overview
1:26 - Adding the CI/CD template to deploy to ECS
2:58 - Getting the project image URI
3:38 - Creating an ECS task definition
4:51 - [For private projects] Giving AWS access to the project's container registry
9:30 - Creating a service from the project's latest image
11:55 - Creating a GitLab service account in AWS
13:37 - Pointing GitLab to the ECS cluster and service
16:51 - Testing automatic deployments to ECS
18:18 - Enabling automatic deployments to a review app
22:01 - Closing summary

Resources:
Deploy to Amazon ECS: https://docs.gitlab.com/ee/ci/cloud_deployment/ecs/deploy_to_aws_ecs.html

A

In this video we're going to take a look at how we can leverage gitlab's, CI CD pipelines to automatically deploy all your code changes to ECS, whether that's in a production app or in a review app or both. At the end of this tutorial.

A

What you'll have is a web app project where we'll leverage gitlab's pipeline templates to automatically build an image out of your project and then push it to your ECS cluster, either in production or review, depending on whether it's on a marine Branch or in a review or feature branch on the Amazon side of things. We'll have a cluster with a couple different Services one for your Production service and then one review service.

A

At a high level, what we'll be doing today is leveraging a git lab template that allows us to automatically deploy to ECS and we'll set up our ECS cluster to be able to listen to code changes made in gitlab, we'll use an access token to facilitate communication between gitlab and ECS and we'll set our CI CD variables.

A

So that gitlab knows where to point to so, let's get started now for the purposes of this tutorial I'm using the Ruby on Rails template that gitlab provides, as it provides a pretty simple and straightforward web app for me to work with what I'm going to do now is I'm going to go over into the CI CD pipelines editor and we're going to replace all the spoilerplate text with a very simple template. So I'm going to include a template called AWS, slash, deploy, ECS dot, gitlab.ci dot, and you should see that it says pipelines.

A

Syntax is correct just to make sure that I added this correctly I'm going to go over to the validate tab and validate my pipeline.

A

I can now go back to the edit tab, add a meaningful, commit message and then commit my changes, then just to make sure that things work correctly I'm going to go into the pipelines and take a look at this pipeline that just ran.

A

We can see here that we have two jobs: a build job and a production, ECS job, and this job is going to fail here, because we haven't really configured anything on the ECS side of things for this project just yet. But let's check this build job just to see. What's going on, this is going to build an image for US, based on the code that I have in my repo here and once that's done. I'll have a container image in our package and Registries continue to registry to be able to use within ECS.

A

So I'll wait for this job. To finish,.

A

All right so my build job has now finished, and so now, if I go into my packages and registries and look at the container registry, we'll see that an image was just created, so I'll go into the Ruby template project master and I'll, see that we have two tags or our latest and the version that was just published.

A

And so now we have our image URI that we can use so I'm going to copy this latest image path here as we'll need it in the next section where we work in the AWS console for ECS, so in AWS navigate into your ECS clusters and instead of actually doing anything in the cluster itself, what we're going to do is create a task definition where we specify the image URI that we just copied so going to test definitions and I'll create a new test definition.

A

I'll, give it a name or Ruby demo task and for the container details, we're gonna, I'm, gonna call the container web and I'll paste my image URI here, the one that I just copied and you'll need to specify your container port in my case, for the project that I'm using the port, that's being used is 5000., but in your case that may be 80 or 8 000 or something similar I'll, go click next and feel free to change any of these settings, but I'll leave them as is next and finally create so I now have my task definition where we're specifying which image we want to use for any services that are running this task foreign now.

A

One thing that we'll need to do if your gitlab project is set to private is we'll need to create an access token, so that your ECS cluster can access these images.

A

If your gitlab project is public, then this these images should be public as well, but if these images are not accessible publicly, what we can do is go into settings access tokens and we'll create a new access token here, I'll call it ECS cluster token optionally. You can set an expiration date and I'll set the role as reporter, since we want it to be able to read the registry.

A

That's the scope that I'm going to select for this, then I'll create my project access token I'll copy that token and now to use this token in AWS. What we're going to use is Secrets manager, so I'm going to open, Secrets manager in a new tab here and we're going to store a new Secret.

A

This will be an other type of secret and we're going to add two key value pairs. Here. One is going to be my username for gitlab and then the other one is going to be password and the value is going to be that access token that I just created in gitlab once we have that I can click next, give this a name so I'll call it demo.

A

Ruby axis click next again and finally store and I'll refresh this and I should see now the secret name that I've just created clicking into that I can now see the Arn for the secret and I'll need to copy this, to be able to use this secret elsewhere within AWS, so I'll copy that Arn there and now we'll go back to our task that we just created in ECS and we're going to create a new revision for this.

A

Where we'll specify the secret errand that we just made so I'll create a new revision with Json this time and Below, where we specify the image or the task I'm going to create a new section, a new key value pair here and the key will be Repository credentials and the value will actually be another section. So put some curly braces there and inside those curly braces, we'll add a key value pair.

A

You will be credentials parameter.

A

And the value in quotes will be the secret area that we just copied once we have that and I'll add a comma right there, we're now good to go and I'll click create, and we should now see that a new version of our task definition has just been created this time with the secret Arn, which points to our gitlab access token. That way, when it's calling this image it's able to access it now.

A

One more thing we'll need to do here before we're ready to use this task within a service in our UCS cluster is make sure that this task execution rule has permissions to access that secret manager, Secret.

A

So I'll click on this ECS task, execution role and that'll. Take me to the AWS IAM console and for this role, you'll see that I have a policy added for Secrets manager read write, but you might not have this in your case. So if this is not here, you'll need to go into add permissions, attach policies and then we'll attach our policy also I'll search for secrets manage your read, write and I won't see here because I've already added this policy to my role, but you'll see it here.

A

You'll check the box and then add permissions once you've done that you should then see Secrets manager read write listed under your ECS task execution rule with that. We're now good to go to use our task definition in creating an ECS cluster service.

A

Now that we have our test definition made, what we can do is now proceed to creating our service within an ECS cluster.

A

So I'll go into my clusters and I'll select a cluster where I want my app to be deployed and I'm going to create a new service.

A

For this service I'm going to specify the family as that task that I just made the revision will be set to 2, because that's the one that specifies my secrets, manager, Arn I'll, give the service a name I'll call it Ruby service.

A

I'm going to turn on service connect.

A

And then click create give it a refresh.

A

And now that my service has been created, I'll click into it, and we can see here that I have one deployment. That's currently in progress, so I'll wait for this to be completed and now that my deployment is completed, I can go on to the next step, which is making this service publicly accessible.

A

To do that, I'm going to go into the networking tab and we're going to go into our security group here for this service.

A

For the security group I'm going to edit our inbound rules in my case, I already have it here, but if you don't you'll edit your inbound rules and add a rule specifying custom TCP and the port on which your web app is running again, in my case, that's 5000.

A

and we'll set it to be accessible from anywhere ipv4.

A

Once you add that you can, then click save rules, and you should see it in your inbound rules with that. Our service is now publicly accessible, so we can go into the configuration and tasks Tab and take a look at the task that is currently running in the service.

A

I'll click on that task name- and this will show me in the task Details page, the public IP address- or this task I'm going to copy this public IP address and in a new tab, I'll paste that IP address and I'll specify Port 5000 and once it loads I can see. Now that my app has been deployed successfully.

A

Now, we've successfully deployed our application one time from gitlab to ECS, but the idea is to automate this with every code change in gitlab.

A

To do that, we will need to create a user in AWS that gitlab will act as when creating new deployments so I head over to my AWS IAM console.

A

And I'm going to go into the users, menu, I'll click, add users and I'll, create a new user I'll call it Ruby, deploy user.

A

Click next and we're going to select, attach policies directly in here, I'll search for Amazon ECS underscore full access and I'll select that policy and click. Next now, I'll click create user and I'll go into my user. That I just created I'll click that and we're going to go over to Security credentials.

A

In here, I'm going to scroll down and click create access, key.

A

And we're going to select third party Service, as we want gitlab to be able to access this user using this access key, we'll select this box to understand that we want to proceed anyway and click next and create access key.

A

Now that I have my access Keys, it's time to now save these on the gitlab side of things in our CI CD variables to do that. I'm going to copy my access key and head over to my project, I'll go into the cicd settings and we'll expand the variables section, I'm going to add a variable here, and this variable will be called AWS access. Key ID, you can see. Gitlab here actually suggests it for me and I'll paste the value that I just copied.

A

In this case, I'm going to not protect this variable as I want this to be accessible or unprotected branches as well, and if you want to use review, apps you'll want to uncheck this as well, and we can click, add variable next, we're going to add our AWS secret access. Key and that'll, be the secret access key here, I'll paste that there uncheck protected variable, because we want this to run again on unprotected branches as well.

A

But I will mask this, so it doesn't show up in jaw, blocks click, add variable and then we'll add the next one which will be AWS default region.

A

So you'll specify your default region which in my case, is Us West, one and again I'll uncheck, protect variable here, add variable, and now that we've specified the access key and Secret. We need to add some variables to also tell gitlab which cluster which service and which test definition to deploy. Using so I'll, add a variable here called CI AWS ECS cluster, and this will be the name of my ECS cluster, which we called Ruby dash cluster.

A

I'll uncheck protected variable here as well. Look at variable the next one we'll do is CI AWS ECS service, and this will be Ruby Dash service.

A

I'll leave protect variable, checked and for the environment scope I'm going to set this as production.

A

Click add variable, and this is because we only want our default or protected branches to be deploying to this service or our unprotected branches and our review feature branches we'll be creating a separate instance of this variable with a separate value or review service, and the next variable we're going to add here is CI AWS, ECS task definition, and the value here will be Ruby demo, Dash task and again I'm going to uncheck protect variable here.

A

So now you should have your CI CD variables that look like this. We have six of them and we can now verify that gitlab is now able to talk to ECS by making a code change so I'll go into my files here and I'll make a code change.

A

I'll edit, this file to say rails production, app, commit changes and that should kick off a pipeline.

A

I go into my pipeline again: I'll see that I have my build and production job and this time since we've configured everything for the ECS side of things. This production job should succeed.

A

Alright, so my pipeline has now run successfully, and so, if I go into my ECS cluster and take a look at the service and get that latest IP address by going to the configuration and tasks Tab and clicking on this new task that has just been run, I'll copy this public IP address and go to that IP address and the port that my app is running on and I'll now see that my code changes that I made in the gitlab side of things have automatically been deployed in ECS without me having to write any sort of pipeline beyond the template that we included.

A

Now, if you wanted to extend this to your feature branches and your review deployments, we can do that as well, using the template that we've already included to do that, we'll go into ECS and within our cluster, we're going to create a new service.

A

I'll use the same task definition as earlier I'll give it a different name: I'll call it Ruby service review, I'll turn on service connect again.

A

And click create and I'll wait for the service to pop up and again I'll wait for this deployment to complete okay and now that my deployment is complete.

A

I can now do the same thing to get the IP address for this going into configuration and tasks clicking on the task that was created for this and then copying that public IP address and I'll go to that IP address and that same port that my app is running on and now we'll see. It says the same thing, because we haven't really made any changes to our review branches yet, but you will see that a review service has successfully now deployed now.

A

One more thing we'll need to do on the gitlab side of things is we'll have to go back into our CI CD variables.

A

We're going to expand this variable section here and we're going to add a another variable with the same name: CI AWS ECS service, and this time we're going to specify our review service that we just made so I'll click add variable and set that same key, and this time it'll be called review.

A

Ruby service review, I'll, set the environment scope to review; slash asterisk, create a wildcard for that and I'll unprotect this variable so that unprotected branches can deploy to this environment, scope, I'm going to click, add variable, and now, let's test this out, I'll go back to that same file. I edited earlier.

A

And I'll make a change, I'll, replace production with review and this time I'm going to do it on a different branch.

A

And I'll start a new merge request with this change, go ahead and commit changes.

A

And I'll create my merge request as well.

A

And now, when the pipeline runs or this merger Quest we'll see that automatically using the template that we included gitlab is including a review job and a stop review job to clean up the review environment and it'll spin up a review app using the changes that we just made without impacting production.

A

So I'll wait for this pipeline to finish and now that my pipeline has finished running I'm going to go back into my ECS cluster. Take a look at my Ruby cluster Ruby service review.

A

Get the IP address there from the latest task.

A

And if we go to that IP address and the port the map is running on, I can now see that the code change that I made on my review. Branch was automatically deployed to this review service without impacting the production app in the process.

A

So, just like that, we have now leveraged gitlab cicd templates to automatically deploy our code changes in gitlab over to AWS ECS. All we had to do was configure our CI CD variables to point to our ECS cluster. It is set up the right access token in case that our project is private and we had to set up some task definitions for our services in ECS and with that I'm now good to go for automated code changes being deployed into production and review.