GitLab ⛅🌱 Cloud Seed, 18 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Production-grade Infra and DevSecOps in under Five Minutes

Description

A technical overview of GitLab's "Five Minute Production". Achieve Production-grade Infrastructure and DevSecOps in under five minutes with GitLab and AWS.

https://gitlab.com/gitlab-org/5-minute-production-app/deploy-template/-/blob/master/README.md

#fiveminuteproduction #gitlab #aws

A

Hello, my name is shri, I'm an enterprise solutions, architect at gitlab and in the next 20 minutes. I would like to explain everything I can about 5-minute production.

A

If there is just one slide, I could show you it would be this one. It would be this image that explains everything that's possible with five minute production, but let's start with a simple definition: what is five-minute production?

A

It's a way for you to achieve production-grade infrastructure and production, great devsecops processes for your project in under five minutes.

A

It brings together the features and the offerings of gitlab terraform and aws, and tries to deliver everything in a simple, easy to consume package.

A

It all starts with a gitlab project, as you can see on the top left of your screen. The gitlab project is where you have a git repository with source code for your web application.

A

It could be a ruby on rails, application, a python or a django application. Maybe you use node.js or java. It really does not matter which language or which framework you use. This will work for everything, but the assumption here is that you have a web application inside a git repository now. What we will do is we will try to achieve perfect, get ops, we'll start with the notion that every branch on your git repo should translate to a web app deployment environment.

A

So, for example, if you have a production branch that should translate to a production environment, if you have a staging branch that should translate to a staging environment, if you have temporary feature branches, they should end up creating temporary feature environments.

A

You decide you decide if you want a long-lasting branch and long-lasting environment, or if you want short-lived branches and short-lived environments, you can achieve that by deleting your branches.

A

The five-minute production pipeline itself has got three steps: build provision, deploy, build stages where we create a build and a package for your application.

A

This application package is usually in the form of a container image which gets stored inside the container registry of your gitlab project, then, is the provision step where terraform skip script actually gets executed and it provisions all sorts of hardware or not hardware. I uh forgive me. It creates all sort of infrastructure resources that you need on aws and finally, is the deploy stage which basically takes your packaged or the release version of your application, and then it takes the infrastructure that has been provisioned and actually merges to do.

A

It takes your application and deploys it on the infrastructure.

A

As you can see what infrastructure gets created we'll go into details a bit later, but this is the rough idea, a pipeline which will build provision and provision and deploy your application, regardless of which technology you use, which programming language you use, which framework you use. It should really not matter, and we want all of this available for you in under five minutes.

A

Let's see how this is done first, who is this for?

A

Well it's for somebody who has a web application, it's for a small startup or a team that has got a web application and they want production-grade infrastructure and production-grade developer operations as soon as possible. That's the first target user.

A

The second target user is somebody who just wants to understand how to automate infra provisioning, how to automate software delivery and wants to deliver something similar for their teams. So this could be something you use or it can be something you learn from.

A

Where does it all begin? It all begins with a commit. You should have a web application. So just imagine that you are now a development team with a web application and your web application, of course is hosted and pushed into a git repository.

A

And, let's assume that your git repo is actually hosted on gitlab, it does not have to be hosted on gitlab per se. It can be hosted on any git provider that integrates with gitlab pipelines.

A

Things are usually pre-integrated with gitlab pipelines. If you you, if you are using gitlab itself, so for the sake of this talk, I'm going to assume that you have a project that is running on gitlab.

A

If you have a project that runs elsewhere, look up the gitlab documentation on how to integrate gitlab, ci, cd and gitlab pipelines for your alternate git provider.

A

Now the web application part, it can be any language, it can be any framework. We really don't care, we don't care as long as it can be containerized or as long as it can be packaged with heroku built backs or as long as it can be packaged with cloud native buildbacks you as the development team for the web application.

A

You decide how you want to build and package your application. We really don't want to make this decision for you, but as long as your build process gives us a package we can use, we will deploy it.

A

I will be showing you some examples and, in the examples we'll go with the containerized application, which uses a simple docker file, and I use this as my example, because it's the broadest possible broadest, possible source or means of creating packages. I think most people will understand docker files, so it would be very well or easily consumed by the audience.

A

Let's talk about the automation part you've heard me mention gitlab, ci, cd or gitlab pipelines in the stock. So far. What is that? Every time you create a git event, pipelines are triggered, which means that your git event could be a commit. It could be a push, it could be a merge, it could be a rebase, it could be creating a tag whatever. Whatever is your git event, it triggers a series of automations known as a pipeline.

A

A pipeline simply is a set of jobs and it's executed in a particular sequence, and these are basically the building blocks for automation.

A

Let's put these two: together we get the five-minute production pipeline, which is a pipeline that lets you build your application provision, infrastructure for your application and three deploys your application optionally. You can also run step four, which is to destroy that is undeploy your application and destroy all provisioned infrastructure resources, but that's an optional step. We don't turn it on by default. The documentation tells you how you can turn it on. If you need to so I'll switch to my first demo, let's have a look at the gitlab ci aml.

A

So this is my gitlab project. It's a rails, application of basic bare bones, rails application and I go to the gitlab ciaml file. You will see that I'm, including a template, called the five minute production app template.

A

This is all you need to do by including this template in your gitlab ciaml file. You have already achieved in complete integration with five minute production. All the variables you are seeing defined here are optional.

A

Let's just quickly have a look in this case. It is a rails application, so I've got a db migrate variable where I execute my bundle command to actually perform migrations on a database.

A

This is a post deploy script that gets executed after every deployment. Then you have some variables to configure various properties of the infra resources that are created.

A

uh We have a variable to basically enable destroy job so that I can actually show that to you in this demonstration, and we have some variables that I want to pass to my end application.

A

That's that the second part I'm going to demo to you is the branches. So you see that I have a get repository which has got right now: five branches, five active branches, they're, probably more branches which were created in the past or which were which are no longer used. But this is just a quick example of the branching flow.

A

Five minute production does not require you to use any particular branching strategy, use what you like. In this case. We have a master branch, we have a stable branch and we have a couple of feature branches.

A

The third thing I want to show you is the pipeline itself, so I switch to the tab here and you're, seeing an example of the five-minute production pipeline. As I said, it's got a few steps. It's got a build step. It's got an optional test step. In this case. The test step is not being provided by five minute production. It is being provided by something configured particular for this example project. So just ignore the test step.

A

If you will you've got the build step, you've got the provision step, you've got the deploy step and you have the optional destroy step, which is a manual step that you have to press a button to execute.

A

Let's come back to the slides.

A

Let's talk about the build step, as briefly mentioned in the past. It builds and packages your web application source code. The generated package in this case is a container image. It is stored in the container registry.

A

Every gitlab project has built-in registries for packages and containers. This could be, for example, container registry npm for your javascript modules, maven for java, pi pi for python, etc. All of this is there inside your gitlab project, so we just make use of it.

A

One package is created for every pipeline execution, that is, every code. Change and distinct packages are created for every git branch and git tag.

A

Packaged container images is later accessed in the deploy stage. We will see how it gets used later. Let's just have a quick look at the container registry. I go to the container registry over here. It's inside my gitlab project packages and registry section and then the container registry link.

A

You will see a list of image repositories here and you can already. If you look at the name, you know that it's got a container folder or a repository for every branch. If I look at the container images built for my master branch, you will see it's going to load a list of container images that it's been built. So here you have a list of container images built for the master branch.

A

Let's move on. Let's talk about the second step provision.

A

What happens during the provision stage? It reads: pre-configured aws keys, so the access key, the secret key and the default region. These are keys that you have defined for your project. So it just reads them note that if you are gdpr sensitive, you can define the default region, so you can say that please store all my data, all my applications. Only in eu west 2, which is the london location for aws, you have that sort of control.

A

Anyway, it reads these variables and it passes it on to the terraform script that actually gets executed.

A

This is the script that provisions your infrastructure. It treats every branch as a unique environment and it's stateful, so the infrastructure is not deleted and recreated from scratch each time. No, it's stateful and stateful infrastructure is maintained for every environment.

A

So when I create a new environment, it creates new infrastructure, but when I push changes to the same branch or the same environment, it reuses the infrastructure that got created, the terraform state itself is managed by gitlab, which means you have a unique state per environment, managed in the same git repo as the rest of your source code.

A

And, of course, uh you can connect to this environment via gitlab pipelines or, if you wish, you can also connect to this terraform state using your local cli.

A

The second part is the aws resources. What is it that the terraform script actually provisions for you? The first thing it does. It creates a secure private network.

A

That means it creates a vpc, a subnet route table, etc, etc. It creates them for every environment within that environment. Within that secure network, it creates a relational database. It uses aws rds to create a postgres sql uh database. For you, this database has got daily backups enabled and each backup is preserved for seven days. So at any given point of time, you have daily backups for up to seven days and, of course, if you perform the terraform destroy snapshots of this database get created.

A

The next thing it creates, for you is a caching layer. In this case it is elasticash with the redis cluster. The cache, of course, is not persisted, it's a cache. So if you destroy your environment, the cache goes forever. Then, of course you have the s3 bucket or the object storage.

A

Your application might need an object. Storage. We also configure an email service, which is the aws simple email service, so your application is enabled and empowered to send out emails to your users.

A

And finally, we create an ec2 instance, which is an ubuntu 2004 lts instance, which has got port 80 and port 443 exposed to the world wide web.

A

Here's a diagram for you, you have the internet and the internet's edge service. So to speak. Is this ec2 instance which can be accessed by two parts?

A

This also lives inside a vpc, a secure network which contains various other resources. All of these resources, so the postgres or the elastic cache or the s3 bucket, or the simple email service live within the secure network and the ec2 instance is able to connect to them, which means when your application is deployed on this ec2 instance. Your application can also access these resources, while the rest of the internet cannot access them and just want to highlight daily backups being enabled for your postgresql.

A

This is possibly the heart of it. So please treat postgres sql as data, that is to be persisted and by persistent it really gets persisted and you have backups and snapshots available for it.

A

Now, now that we have the infrastructure, let's talk about deploying deployment. What are the preparation steps for it? The first is: we have to fetch the deployment secrets from the terraform state. How can we connect to the ec2 instance? How can we actually deploy our applications there? Those secrets, those secret credentials- are available in the terraform state.

A

Second, fetching of a deployment environment from the container registry. So now that we can access the deployment environment, what is it that we deploy that artifact? That container image that gets deployed can be accessed in the container registry.

A

Step two is to actually deploy itself, so we have a simple deployment: script that deploys the application to the ec2 instance while deploying the application.

A

It provides your web application with credentials for the postgres database, the readers cluster credentials required to send out emails, credentials required to access the s3 storage and any other environment variables that you would like to provide, as you saw in my brief example earlier, we also have the ability to create post deploy triggers and in my example, I was creating a trigger to actually run the rails, migration I'll just refresh your memory. Here's an example of a trigger being used to run this command after the deployment process.

A

In this case, it's just rake migrate.

A

Finally, what we do for you is we set up an uh ssl certificate with certbot, so let's encrypt provides you ssl certificates. You can connect your domain, so you can just define a domain name variable which connects the domain and ssl certificates get created for your domain automatically.

A

I mentioned terraform state and I mentioned environments. So let's have a look again what that means. Let me first refresh your memory with the branches we have these branches over here.

A

So now, if I go to my gitlab project, I select operations and I like, and I select terraform, you will see that I've got terraform state maintained for each of my branches.

A

Every time a pipeline gets executed for these branches. It looks up these terraform state and downloads it within the pipeline and actually works with it. So we completely enable stateful application deployments, of course, through your terraform cli, you can connect to the state as well or you can simply download the json and do whatever you want to the second thing I'd like to show you is the environment, so these are environments.

A

In this case I've got one available environment and six stopped environments, and these are environments that get created when a pipeline gets executed and it completes the execution. So your application gets deployed onto each environment.

A

As you can see for each of these branches, you have a live environment with a url that you can click and you can access.

A

Let's move on I'll talk a little bit about autodevops, auto devops is a series of additional reusable jobs, part of a larger pipeline, but they are reusable jobs that you have available, which means, if you take auto devops and you put it in five minute production, you can get a complete workflow for your application.

A

So these could be running tests for your application. It could be running source code, quality checks on your project source code. It could be running security scans, for example, static analysis, first testing, dependency, scans and so on, or you want to perform things like performance tests or license compliance.

A

So you have all of these uh reusable pipeline jobs within gitlab part of the auto devops bundle, and you can pick and choose the ones you like.

A

Of course, once you pick all of these, your pipeline gets a bit complex, so it started off with three jobs, and now you can see, a complete pipeline will have a lot of different jobs and these pipelines, of course, depending on the number of scanners you're running they could take a while.

A

But once your project reaches a critical mass, you probably want to enable all of these.

A

What is the takeaway like? What can you take away from this stock? The first thing you can do is use 5-minute production. If you have a web application and you want to deploy to aws without any headaches use it as it is, it works, it works great.

A

The second is, this can be a learning example for you get inspired, create your own infrared code, setup use, terraform, use the wonderful products. Hashicorp has made available to you and try to achieve git ops in the true sense, which means your source code and your infrastructure live in the same repo, and I would like to leave you with this thought.

A

Have you have we really achieved the complete convergence of git ups with the five minute production? Let's see what we've achieved, we have the source code in the same git lab repo. We have the infrared code, the terraform scripts. In the same git lab repo.

A

We have our release packages. The container images that we built in the same git lab repo.

A

We have the infrastructure state. That is the terraform state inside the gitlab repo, and we actually have the running environments with the with all the configs in the in the get lab repo as well. So every aspect of your application is in one place, so in theory, or I would go a step further, at least in my experience in practice, this is possibly one of the most complete examples of git ops, I've seen so far with that I'd like to wish you to a great day. I hope you are a little inspired.

A

I hope you have the necessary inspiration to at least test it out one time and once you test it out, of course, we're happy to listen to feedback like most or like almost every piece of gitlab project or gitlab, product or gitlab solution. This, too is open source. So please access it. Please use it and please share your feedback.

A

Thank you. So much.