GitLab Solutions Architecture, 18 Sep 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Deploying to AWS Elastic Beanstalk with GitLab CI/CD

Description

Learn how to use GitLab CI/CD, review apps, container registry, and environments to deploy a secure Docker application to AWS Elastic Beanstalk

By Sam Morris

A

Howdy, my name is Sam I, Am, A, Solutions architect at git lab and today I'm going to show you a Docker application that is hosted on AWS elasticbeanstalk, that is built, tested and deployed through gitlab CI CD.

A

So if you look at my screen here, we'll see that I do have an environment called production and this is actually linking to my production application or my currently running production application. So if I open this up, I can see my latest deployed production app. So this is my very basic app and this is essentially what's running in our my main branch and is based on my latest Docker image that I built in my CI CD pipeline. So this is the end State and we're going to take a look at how we got to here.

A

So, let's start by taking a look at my pipeline.

A

So if I go to my pipeline, editor I can see what my CI CD pipeline looks like, so we're going to go through kind of what this is and then dive into the code behind it as well and kind of the the inner workings of the logic. So we start by building our Docker image based on our application code that I have, in this case it's a flask app written in Python with some HTML and CSS as well. Then we move into the review stage, so this review stage does a few different things.

A

It essentially creates an application in AWS elastic Beanstalk, as well as an environment for the application. It does this if there isn't an application and corresponding environment that already exists, which, in the case of a production application, we wouldn't be creating a new application and environment. We would just be creating a version of the application and deploying it to the same environment that exists. The only time we're creating a new application and environment is when we open a merge, request to essentially add a change, and we want to create a new review app.

A

The review app is where you can see your live changes on essentially a staging website, so we do our provisioning so again, either creating a new application and environment entirely, based on a new Branch being created for our merge request or creating application version and deploying to the existing environment.

A

uh We also want to check for environment Readiness, so in this case, uh I have it check for if there is a dynamic URL that or if there's a URL that has been assigned to the environment that we're deploying to for a review app. So this only this review stage only runs in the case of emerge. Requests right.

A

um Essentially, it is checking if the environment is ready, so there's a difference between successfully greeting the environment and for it to have you know: okay, Health ready to go and have a URL that is assigned to it so that we can deploy our changes to the URL that is created at this stage. So we checked for the existence of a URL here and then deploy review.

A

This is essentially where we actually deploy our review app in gitlab, so we can manage our review environments back in our environments, page that we started at and we can. You know, click into there and it'll automatically redirect us to the URL in elastic Beanstalk the deploy stage. We have a stop review job which essentially Force deletes the application and environment of the branch that we're working out of so that review app.

A

That was created, it'll be deleted in gitlab, but it also deletes the resources in AWS to do resource cleanup, so you're not consuming additional. You know ec2 instances and and such and then we have a deploy elastic Beanstalk. This deploy job only runs in the case of merging to my main branch, or my default Branch essentially deploys to that production, URL that stays static, um unchanging and your uh your main application URL. Finally, we have a dash stage that runs.

A

This will either run against our review app, um so it runs against the staging environment and produces any results that you would see on your production application. So you can run a full active attack as opposed to a passive scan in the case of running against production. I have a scan execution policy running dust so that it runs a passive scan on active scan against our production running, live application so that we don't have a chance of accidentally taking down our production application right.

A

So this Das runs either on a review app on a merger Quest or it will run on our production app in a main branch update.

A

So this is the pipeline logic. Let's take a look at the actual code for it. So again we have our stages. Ordered order is important in our stages. This is our Docker build uh job, so essentially, it is obviously using the darker image and it authenticates US, based on our username and password into our CI registry.

A

So what this is essentially doing is it's deciding? What branch are we on uh and a associate associating a tag with our image? If we're on the default Branch, our tag will be latest and if we are on a any other branch that is not our default Branch, it will be. The tag will be a commit slug.

A

So we assign our tag based on what branch we're on right and then we we essentially build and push our image to our container registry, which we can see in the deploy tab right here. If we click into container registry, we can actually see all the containers that we are all the images that we've built. So this is the name of the app.

A

I can click into this and I can see all of my images.

A

Including latest, which was published one week ago, so this is essentially the entire container registry, where we're pushing our images to, and we also make sure that this only runs if there is a Docker file that exists.

A

Otherwise, we will not get very far trying to build our image moving forward to provisioning the application. So essentially, I have some scripts that I wrote here um you can. This is just because there's some reused logic here, so you have to make these scripts runnable they're. All you know bash scripts, including an AWS authentication script, which all you have to do is you know, create your access key in AWS associated with a variable in your project. You could also make a group level one if you wanted to, but it authenticates into AWS.

A

You can use the elastic Beanstalk CLI, as well as the S3 um CLI.

A

So if we dive into what this script is doing, we can take a look at the exact logic here. So again we have our authentication script, which we dive into first.

A

This is reused, basically in all the jobs that interface with AWS. Whatever we want to set our access key, our secret access key and our region, then we provision our application. So essentially, what this is doing is again. It is checking if an application exists. If the application does not exist, we create an application in elastic Beanstalk um and then we also for every case. We always want to create an application version.

A

So if the application already existed, we would just go straight to creating the application version based on the label that we assign it, and this label is going to show up inelastic Beanstalk. So you can filter through and you know see exactly which version of the application that you're running right similar deal. We check if an environment exists based on our review environment name if it does not exist, we create an environment based on the application version that we created above in all cases.

A

Otherwise we see the environment does exist and we're just logging that to our logs and then we update the environment with the application version. So that's what the provisioning is doing here.

A

Similar deal, we are also doing update Source bundle. We actually run this first, my apologies, but essentially all this is doing. Is it's diving into our file in S3 and it's replacing the tag so in this case, like I, said before our tag for our review app um image or our merge request image or our Branch image? Anything? That's not our production image is going to have the CI commit ref slug as the tag right.

A

So we already built our image and now we're replacing our default tag latest with the commit ref slug and also providing the review S3 key, so we're creating a new S3 file right um and we're placing it in our S3 bucket as well. So this S3 bucket stays the same.

A

We're just creating a new file um called whatever is the content server review F3 key and the only difference is we're just updating the tag so we're pointing to the correct tag in our container registry and we're not always pointing to the latest, because we do want to deploy the changes that are in the merger Quest that were updated on the branch.

A

And this runs on everything except main again. You wouldn't necessarily do any of this uh for your main branch, because your uh environment already exists, your application already exists Etc and you wouldn't want to update the tag from latest because you want to use the default, which is the latest tag, so there's no need to update your Source bundle or provision an application at this point. uh This is just for your review stage and it only runs on branches except for Main.

A

So again, this environment, ready job does require the app to be provisioned, of course, because you need to make sure that you know your environment is stood up and running. So essentially, this is running. The script verify environment ready. So let's jump into that script.

A

And it's checking for the existence of the endpoint URL for the environment that was provisioned and then providing an exit code, so this job will fail if there is no URL associated with the environment, which is the case for if it is not provisioned. Yet it usually has a bit of lag in time before this URL is created, so I do have a retry. We only allow for retries up to two times, so this will rerun if that job fails. Until there is a URL.

A

Finally, deploy review so essentially, we need environment ready to have succeeded. That means that a URL does exist, for our environment again runs in the review stage, and what this is doing is it's grabbing, that Dynamic environment URL, and it is putting it into this deploy.env file, and this is what is controlling the review app essentially in sna, it's setting the variable Dynamic environment URL to HTTP colon slash, because this is the beginning of the URL and the actual URL that was um gathered in the previous step.

A

At this point, we also said a dash URL I do have a dash scan that runs against this Dynamic URL, so I do want to Port that into the environment url.txt. This is essentially saying: I want to scan this URL instead of my dast website, URL or whatever set to the dash website variable right so typically, I have that set to my production one, but in this case I want to override it by adding this Dynamic URL to the environment.

A

Url.Txt file, adding my artifacts the deploy.env file- and this is when I actually create my review environment as well. So, in this case, I call review, slash and then the commit ref name and I set my URL. Obviously to that thing that we said earlier right here: the dynamic environment, URL and then on. Stop I'll run the stop review.

A

So if I could click stop review app, then it'll trigger this job, the stop review job, it's a manual action and essentially what this is doing is it's authenticating in AWS and it's deleting the application, we're looking at the review, app's name and then we're going to terminate the environment by force as well. So it's going to delete the application and the actual running environment on stop.

A

So that's the case of the review app specifically for production. This deploys to our production, elastic Beanstalk.

A

So in this case the environment is called production always and the URL is the production URL, which I hard coded in here. You know it's like your google.com, your example.com. It's your main website, URL right, just defined in a variable called prod. Url I want this to be a manual job, so it is not triggered automatically. You can have automatic deployments and not have a when manual, but in this case, I want it to be a manual job. Just for extra security. Only running on the main branch same deal authentication.

A

We want to create an application version, it's the same logic as the script in terms or the same actions of the script we looked at before. But in this case we don't need to check if there's an application that exists or an environment that exists, we just want to create my version using my source bundle. That is, you know the S3 bucket that everything is stored in and the prod S3 key, so that in that main file, that has the latest tag in there.

A

We're going to reference that here and then we're going to update our environment with the uh application name, environment, name version label that we referenced before, and this will update our production environment. In this case the environment called production I, have it as a protected environment so that you know we can't deploy automatically to it. I also have it set so that uh the person who approves the deployment can also trigger the deployment. Just as an additional separation of Duty guardrail.

A

Continuing forward I do want to run dast, so I do have my include template dust or my Dash job. That is why the dash job exists is because of this line. Right here include template, dast gillapse, yeah yaml, and these are all my variables that I specify. So you know my production. Application name is Docker example. The environment name is Docker sample environment. This is this scenario, is unchanging, I have a prod URL. That also is unchanging. In this case, I have the prod label and the prod key and I have.

A

My Das website is the same as the prod URL right, so this gets overridden in the review app job. That's creating the review app. It's also setting a different dast website URL to scan all right, so this is kind of the logic for the variables controlling. Essentially, you know what file we want to point to Etc.

A

This is the entire gitlab ciml file. Let's see it in action, so I want to look at a merger Quest in this case, I added one line to my file. I changed my first post to hello, Solutions, architects,.

A

My merge request pipeline passed and nothing happened in terms of the scan result policy, so it could go here, but what happened in my pipeline right right? It ran Docker build so it created my container. You can see the output from that here, so it built the image.

A

And it provides the digest upon pushing then all my security scans run. This is managing a compliance pipeline external from the project, so we didn't see any definition of this because we didn't Define anything in the test stage. This is all managed externally, because I have a compliance framework associated with the project, the review stage, so in this case provision application ran.

A

So you can see the logs of it.

A

In this case we saw the application did not exist, because this is a new merger Quest, so we're creating the application and we called it. Docker example: hello, Solutions, Arc, so we're going to do is we're going to go into AWS now and in this case it created this application right here.

A

So this is automatically generated, and you can see here you can click into the version. So essentially every time I commit to the merge request. I would have a new version and that version label would be the commit Shaw from what I committed to the merge request.

A

Similarly, it has a source file. So this is what happened when we overrode the file in S3, we had a file called Docker run aws.json. That is our like main file, specifying we want to pull from the latest image. What we did here was we actually created a new file and we overrode latest with hello, Solutions Arc.

A

So this is pointing to the container in our container registry, which we can take a look at right here.

A

A

Next page alphabetical, hello, Solutions works. This is the. um This is the image that we created uh it has logs on when we did it in the Digest. So this is the image that we're we base our changes on and you can see it's in our container registry, all right, cool.

A

So what that did was it overrode, essentially just latest after this colon, so that we're pointing to the correct image.

A

And we deployed that to an environment, so that's our application version going back to our application. We see the health is okay and we can go into the actual environment name. We call it the branch name in this case solo, hello, Solutions, Arc, you can see our health is okay and from here we could click on the domain, and we can see it. We can see Hello Solutions Architects. This is our review app right and we could click into this from AWS.

A

But if you want to click into it from gitlab, you can actually do so because we generated a review app for it. So if I go into deploy review, this is when we deployed our review application because we checked the URL existed after you know, retry or not- and essentially we see at the top here this job deployed to review hello, Solutions Arc. So if I click on that I'm going to see my review app and I can view the deployment and it links directly to that URL that I clicked to from my AWS console.

A

So that is a very handy thing. It deployed it. It pulled the image from our container registry and deployed a new version of the application to this review, app environment and, if I created another commit to my merge request, it would do the same thing. It would build the image push it into our container registry.

A

It would see that the application exists, create a new version using that Source different source file that was created and that source file is suffixed, I suppose, with the commit Shaw of my commit and It'll point to the correct image and deploy that to the environment that exists right, it's a new version of the application, same environment and then our review app will point to that environment regardless with the new changes.

A

So this is again directly from git lab. We click view deployment and then we can see the deployment.

A

You could also access the same thing from the deploy or sorry from the operate section. You can click into environments and you can see in review. We have two that are open right now, because I have two open: merge, requests with running environments that I haven't stopped yet and again. I could I could open the live environment from here all right, great and then, after that happens, we do have Das run.

A

So again we don't have anything in deploy except stop review because we're not deploying to production at this time, but we do have our Dash stage and the Das runs against the review app environment.

A

So it's running against this URL, not my production URL right from the beginning, and it can give us some some Das results in here as well. So in this case we can click into dast and we can see the dash run.

A

A

So again, if I clicked stop, it would delete the application and terminate that environment by force. If I clicked merge it would deploy to production um by building, it would build the image correctly. It would see that we're on the main branch now it would build whatever my code changes were in this and create a new image based off of it or create update the latest tag with the changes, and it would deploy that to production which can be managed in your environments.

A

Once again in operate environments, you can access your production environment from here as well.

A

So that is a brief overview of how to use gitlab cicd to build a Docker Image store it in our container registry and create a review application based off of the changes made in a merge request deployed out to AWS elastic Beanstalk and you can preview. The changes live on your review, app run, Das against it Etc, but also host your production application on elastic Beanstalk as well. Thanks for watching.