GitLab GitOps, 12 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DevOps + GitOps + AWS with GitLab Demo

Description

This is a high-level overview of doing DevOps and GitOps on AWS. This demo was delivered on June 12, 2020.

A

Hi I'd like to take you on what a typical flow would be like for a developer using gitlab that will be using eks as a deployment environment for the first part of this demonstration. We'll also make a quick reference to our recent integration with ECS. I would like to then show you our behind the scenes on how everything works and integrates together and finally, I'd like to spend a few minutes showing you how to use, get lab to manage infrastructures code. Specifically what is known as get-ups.

A

We have a lot of material, so let's get started so as a developer. I track my work via boards. Here you can see my three boards open to do on doing and let's, let's say that: there's an application already in production, and it's right here. This is a web app, as you can see, the you says spring is here, and the background is green and let's assume that someone would like to make that more customizable and and for our friends are at AWS.

A

So there's a request that will be created by someone it could be created from the issues screen right here or simply they can just come to my to do a board and enter customized web app planning page and create and then submit issue.

A

This will create the issue under my to do board notice that the label has been assigned automatically and I will see this obviously up here on my board and then I'm going to start working on it. So I'll go ahead and assign it to myself and then I'll move it to the to my doing board.

A

Now this point I can actually open the issue and double click on it.

A

There you go, and the issue is what the problem is, and the MRR or merge request is the center of changes of the work that I'm doing and the place where stakeholders collaborate on the resolution of the issue, so here I'm going to say, create, merge request.

A

This will also create a branch and then, since I'm, starting to work on this I will go ahead and open web ID.

A

And in here I'm going to go ahead and navigate to where the.

A

Source file is, and let's just say, I want to change to make it more a customized version for our friends at AWS I'm, going to change the background, color and I'm gonna, say AWS and then also I know that there's a unit test in in this project that will test.

A

The color of the screen and the string that was being returned. Actually it's gonna test what's been returned, it's a call to that web web app. So I have to make the changes here. So I'm gonna commit and I'm going to commit to the to the branch.

A

A

Now, if we go back to the amar.

A

Now notice, here than the in the mr, you can actually click on changes and see side-by-side the changes that took place in the files of that project that I apply it in this case.

A

This is how the stakeholders on this, mr, can collaborate down to the source code line changes by providing comments. So, for example, you know if somebody would like to add a comment to this change. They can actually make a comment right here and say you know. It looks good to me or something like that for now, I'm not gonna, put are coming in there, but you know you can you?

A

Can collaborate all the way down to a single source source line, and- and this is good- because you know, through this collaboration, what you get is you get better better code quality? It also speeds up by the development of features and also increases developer productivity.

A

Alright, so let's go to the.

A

To this next screen, which is the pipeline, so a pipeline has been started as a result of committing the changes to the newly created branch, since this pipeline will take a few minutes to complete instead of waiting, I already have the same pipeline already completed. So, let's go see it.

A

A

All right so here is the pipeline has already been run all the way to the end, and this pipeline is part of our Auto DevOps pipeline, which comes out of the box and is based on lessons learned and best practices for continuous integration and continuous delivery.

A

These pipelines can jump-start your development work, saving you time and shortening your developing sree development lifecycle. As you can see, they also shift work left as much as possible so that you can identify and fix problems and preempt issues from occurring in production. However, if these pipelines all don't quite fit your needs, you can create your own pipeline or modify an existing one.

A

This specific pipeline includes our the review app, which is right here, and this spins up its own environment in this case in the eks cluster, so that updates can be validated by the stakeholders of this MRR. So, let's briefly now go over each of the jobs include it in this pipeline will start with build a stage and also the job which creates a build of the application using an existing docker file. The resulting docker file image is pushed to the container registry and tagged with a commit, sha or tag.

A

Then in the test stage you have a few jobs here in general test, analyzes your project to detect a language and framework that is written in or implemented in, and then it runs to the appropriate tests for your application. It will also use the tests you already have in your application. In this case, it will run the single unit test that I showed you earlier and if there are, no tests is up to you to add them.

A

So let's go down to each of the jobs here: code quality, it runs a static analysis and other code checks on the current code. It creates a report which is then uploaded as an artifact that you can later download and check out.

A

Container scanning is a vulnerability, static analysis for containers to check for potential security security issues on docker images, gymnasium maven dependency scanning runs an analysis on the project dependencies and check for potential security issues, license scanning searches, the project dependencies for their license and check them for compliance in your project secrets asks scans the content of the repository to find API keys and other information that should not be there. Another area related to secrets is the ability to mask and protect variables particular area.

A

Bowls gitlab provides this capability via our CI CD variable settings, which I will show you in a minute spot bugs sassed, which is also part of our static application security testing suite. It runs a static analysis on the current code and checks for potential security issues, for example buffer overflow.

A

The review app it actually creates, like I mentioned before a container, deploys the application on eks. This is our integration to two eks and the stakeholders. Actually, at this point can review updates to the app before it gets merged into the main branch, improving the fidelity of the solution that will eventually get deployed to production. We are on our way to doing the same integration with ECS.

A

Let me show you real quick. What we have for that, so this is our the pipeline that deploys to ECS, as you can see so our first iteration that pipeline is, is really simple. It's just a build and then deploy, and we are, as I mentioned in our first iteration of this work and and we will be adding more functionality as we iterate more and more on this implementation.

A

So at this point, I like to show you the way to protect variables, sensitive information, so, for example, in this case, you can see some variables in this project related to you know, AWS, access key and, for example, you see a service and test definitions etc, and you can protect them and mask them by checking when, during the creation or I, feel you can edit one. For example, it's let's do this one.

A

This is not protectors or masks, but if you, if you were to change this, you would just click on these checkboxes and obviously masking them. Well, you know is useful so that you don't show the actual value in any of the jobs or output that is generated from running the jobs, so the ability to protect and mask these variables shields organizations from potential security issues.

A

So, let's go back to where we were here and let's continue going over right here to the next stage is DAST and that's were dynamic: application security, testing capabilities, the the Dyess analyzes, the current code and checks for potential security issues. For example, you know cross-site scripting and then the performance is called or it uses the auto browser performance testing that that's the name for it, the complete name and what it does. It measures the performance of a web page.

A

It creates JSON report as well, including the overall performance score for each of the pages and it uploads his report as an artifact.

A

Alright, so let's go back to.

A

The EMR here- and this is the EMR.

A

All right so here one thing I like to bring up to your attention, is that this M R is related to the issue that that created it and and when the M R is completed, it will also close the issue.

A

Okay, so let's say at this point: I mentioned before that all did you know as a jobs run, data is generated and art and reports are generated that are uploaded as artifacts and on this screen you can actually see the results of a lot of a lot of the jobs, the job generated in this case I like to, for example, if you expand here, the security vulnerability security scanning I'm, sorry you'll, see a bunch of you know the results for each of the tests related to security.

A

Here you can click on the full report. If you want to get more in-depth information about that specific report- and in here you can- you know in this specific case is showing you the different types of security issues that it found and you can actually dismiss the vulnerability run a little bit bility. You can create an issue or you can get more information about that specific vulnerability.

A

All right, so, let's at this point this this specific, mr, has been askew the pipeline Brian already and now we are ready to is waiting to be merged, and the assumption here is the stakeholders have had a chance also to to run the the environment. In fact, if we go here, this is the review environment. So if we go to.

A

To the review environment, we should be able to see the deployment which you have. The changes here see so the review environment is actually a running environment. Has all the updates applied to it, and it's the the point or an environment that the estate the stakeholders can use to check that all the changes that were made are correct. Basically, so, let's see so, let's, let's merge the the BMR now notice that the merge button here is not it's not available. It's grayed out, and the reason is that this, mr, has been marked as working process.

A

Okay, so this is something that is built in within gitlab and, as you know, to prevent someone for you know it's an extra safety mechanism to prevent an accidental merge and to indicate also that this is still being worked on, that don't merge it. Yet it's working progress, so people are working on it are reviewing it, so don't merge it. So, first we need to clear the whip status and we do that by resolving the whip status here and now the whip is gone from the title, and now the merge button is activated.

A

Also, you will see that there is an the elite sword- source branch checkmark here that is checked the. What this will do is it will clean up all the resources used by the review app, including the eks container, that was spun up. This actually helps with container sprawl, so you won't be left with eks. You know, environments running an idle, so this cleans up after after you know, cleans up all the resources, basically after the merge takes place.

A

So, let's see so, let's, let's merge.

A

And this merch is going to kick off another pipeline. That will basically run you know all the checks that you saw earlier and it'll take the the.

A

It'll take the deployment to a staging environment that will also be running on eks.

A

So let's go at this point. This is going to take a few minutes. So let's go back to let's go to an mr that has already completed, and that is over here now.

A

Let's go here, it is this one here.

A

This one here so this is the exact same, mr the is executed on the other project, but it's already come. It's already completed, okay, so, let's let's go to the pipeline, so you can see it it's this one here notice here that you know the changes were merged into master and the source branch has been deleted and the issue has been closed.

A

So, let's go to the pipeline now notice here now that this is showing you the there is showing some jobs here that we're not there before, and this is basically the the CD portion of the pipeline, the continuous delivery portion of the pipeline. You can see here that there is a staging environment.

A

So again, like I mentioned before all the what once the updates have been merged, an environment is spun up on eks and this is called a staging environment, and at this point you don't see it in this case, but these are manual steps, so this will be grayed out this actually we're great-great out, and you have to manually click on these to do an incremental rollout. You can choose 10, 25, 50, 100 percent.

A

An incremental rollout into production, reduces the risk and contributes contributes to a better user experience and reduces the danger of an outage and after the application has been rolled out to production it to the production, environment and other performance tests is run in production.

A

So, let's go back to.

A

The completed merged, mr and I want to click on production. Here.

A

All right, so this is showing the production environment. This is. This list is actually an auditable sequence of changes that have been applied to the production, environment and.

A

This is the last one that was run. The nice thing about this auditable sequence of changes is that at any point in time, if you introduce a change and you notice that this is shown an unexpected behavior in production, you can always roll back the environment by clicking on this on this button right here, the rollback environment, which will take you back to a previous and previous known state,.

A

At this point, you can click on view deployments, and this is actually the application running in production. As you notice, it has a need, a new color, that we apply it orange and it has the AWS prefix so that you change the way introduce introduced into production.

A

The other thing I wanted to show you was the.

A

The monitoring we were here so, let's go back to production. Sorry I shouldn't have closed that window, so I'll show you the deployments so now, let's click on monitoring and we have Prometheus running on the eks cluster collecting collecting metrics, and this is showing you metrics not just for the cluster, but also for the running application here and one thing I like to show you is: let's go back and let's redeploy: let's redeploy this environment.

A

And then let's go back to.

A

The monitoring window.

A

And what you will see here now is this symbol here this little rocket here: let's change it to actually 3 yeah 30 minutes, actually.

A

And the rockets that you see here are actually a change that was just rolled out to production and it's you can actually see the job itself that executed as a result of that that update to production and the nice thing about these indicators is that they can help you troubleshoot production issues because there's a correlation between the behavior of, for example, total memory here and the introduction of the of that change. So let's say you apply a change and you see that memory is going out of whack.

A

Then you can actually from from looking at the graph. You can see. Oh that changed you know was highly. You know likely that it that changed cause that issue in production, that much higher concern consumption of memory. You know again, like I mentioned this, helps you troubleshoot issues, production issues faster, so these demos so far have used eks and ECS as the deployment platforms. However, you can also use a get lab with bare metal and virtualized employment environments, as well as different operating systems.

A

Like Windows I'd like to show you a peek behind the scenes of how easy it is to create projects and work with pipelines, so for that I'd like to take you to my get lab dashboard here, these are all my projects and to create a new project. You can clear a new new project, click on new project here and here at the top you can say create from template and it's a variety of different templates here. That will help you.

A

You know we get you easily jump started on your development effort, so we've been using this spring one here, the what you've been seeing this spring project that you've been seeing came from this template, so you can use that template now. Also, you could import a project from github, so you know this allows you for the easy migration of your projects to get lab, and you could you know, and you could also integrate, get labs, CI CD with get lab. For example, you could do that from here.

A

You get import projects, as you can see from other repositories as well. Alright, so, let's go back to create from template. We're gonna select the spring template and, let's just call it a prac for step, one I think that's what I called it now. This point you can select it to be private or public project. In this case we you know we can leave it as private and once you create project it'll create a brand new project. That will look like this right here right here.

A

This was actually populated, I'm, not gonna press on create because I already did a little it'll reject it because the project so that project name is already taken. But this is basically once you click create this. This is what you will get it populates your project with sample code that we saw before and and at this point, what you can do is you can start creating a new pipeline for this project? Okay, so let's create that.

A

So you would say here plus- and you say new file and in this case let's do get left CI llamo and again you can choose for a variety of templates for pipelines here for different languages and different frameworks.

A

But in this case, what I like to I like to pick the sample pipeline or template pipeline for auto dev ops, which is what you've been seen all those but the pipeline's that you've been seeing so far are, are based or were the the auto devil's pipeline, which is this one here, for example, the stages you were seeing graphically earlier are listed right here, built ezdeploy, review, etcetera, included, the incremental production rollout to production, etc. They are all here, so you can start with this pipeline.

A

If you want you can easily, you can modify it if you want, but even if a template doesn't fit your needs, you can create your own to modify this one. For example, there is some instructions here, but if let's say you don't want tests, you want to disable the tests in your pipeline so that they can run faster.

A

You can just copy that and paste it here in the variable section here of the of the CI mo file, and they would just say you set that to true and if you set that to true, then it'll disable all the test related jobs and you in this pipeline.

A

Okay, so just like itll app helps organizations, speed up the development and delivery of their applications. You can also do the same for their infrastructure.

A

So let's discuss here, get-ups so infrastructure as code is a discipline of using gits or get-ups is a discipline of using kids as the main source of truth for managing your infrastructure of code where, for example, the Yambol files defining your infrastructure are like the source files in an application. In this group, you can see two sub groups, one for the applications which is apps and one for the infrastructures code, there's a nice separation of roles and concerns between development and operations, which optimizes the work and collaboration among teams.

A

In addition, gitlab granular role based permissions helps to accomplish this in a safe and controlled manner. So, let's, let's go to the infra subgroup here. There is a project called AWS, and this project right here contains files that are describe your infrastructure s code, so we're using in this case terraform and, for example, we can review some of these. The VP CTC describes your network using a terraform file here, and this specifies in the number of subnets you're gonna have and what they're gonna be.

A

And is using the module VPC here.

A

Let's see, for example, the eks terraform file, it actually is the Amazon eks configuration for the cluster here, for example, we'll use justifying that the that's gonna be the instance type CPU type. You can select from a variety of of eight sense types from Amazon here and also it's gonna, be a starting a number of nodes of five for that specific cluster and we're using the module dks there in here then other files, for example, the back end TF, is the state file location configuration the gate, lab admin terraform it.

A

It adds the kubernetes service account and the group cluster TF. It registers kubernetes cluster, to get lap group, for example, so to show you the running application, we can go to the apps here and again. If we go to this project, this is again the same project that we've been covering through this demo. It's the Java spring project using the template. If we go to operations and environments, you can click on production, and here you can see the application is running with all the changes that we applied through the EMR.

A

Now this is this project is a separate project, but it's supplying the get-ups concepts to the way the the groups and circles have been organized.

A

All right so to wrap up, we have gone over how gitlab can help organizations speed up their software development and delivery. We discussed Auto DevOps and how it can get. You started quickly with pipelines that are germane to your project, language and deployment environment, whether it is c KS e, CS, bare-metal, virtualized environments or even windows. We went over how easy it is to create a project and create and modify a pipeline, and, lastly, we cover how gala can help you in managing your infrastructure s code.

A

Lastly, gitlab is a complete devops platform, delivered as a single application, fundamentally changing the way: development, security and ops teams, collaborate, gate lobby helps teams, accelerate software delivery from weeks to minutes, reduce development costs and reduced the risk of application. Vulnerabilities will increase in developer productivity. Thank you so much.