GitLab ⛅🌱 Cloud Seed, 12 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-11-12 5 minute production app sync

Description

Direction: https://about.gitlab.com/direction/5-min-production/

A

So this is the first meeting for the five minute production app and our goal is to make it easier to do uh development with, uh for anybody, and one big inspiration for us is uh heroku how easy they made it to get started with an app.

A

However, there's a few problems you faced. If you want to get started today, I still think heroku is a great place to start an app it's the best place. We want gitlab to be a better place, and that means that not only is it great for the development app, but it's trivial to do to have a production app.

A

You shouldn't be limited to heroku infrastructure, which is much more expensive than the cloud providers and in the end you want to host your app on the hypercloud if it gets popular anyway.

A

You want to have flexibility that if you want to change the ci or cd thing, it should already be structured in the right way. The model way to develop apps is on a hyper clouds, with gitlab um by using a technology called terraform and using managed services of those hyper clouds.

A

Those managed services are especially important if you run a production app, you don't want to be managing your own database or your own redis or your own object storage.

A

So we want to make it so that someone who's already signed up for aws or gcp can go to gitlab.com, create an account start an app and have that production ready and up in five minutes. That's our project, all right. It's called the five minute production app or our theme song is little mo.

A

So we want to do a little more for uh developers and uh dimitri is now gonna give a short demo of what he's done so far, and sri is in here to help us further full time during the quarter and michael's going to help us reach our audience and have some people rooting from the sidelines dimitri. Take it away. Thank you.

B

That's it uh yeah. I must know that most of the work was done by three to my illness, uh kudos to him, but yeah. Let's get started so, first of all, it's still day one. So there are quite some manual steps involved, but uh I still think it's worth showing so what we usually have uh when we start with web app.

B

Usually what you need is a database and the application itself running. So if we speak in the infrastructure of uh amazon of aws, it's let's say ec2 instance and rds instance, both of those instances have three tiers, so that should be interesting for developers, since it uh allows them to run their first app at at minimal cost.

B

So we started with uh this project five minute, docker the name in yeah. It's uh just random, but idea is uh here. We define a oh wait. Wait wait, wait sharing the screen yep now much better, so we started with a simple project that has few key elements: it has a ci yaml file which constrains stages of deployment, infrastructure, building your application deploying it and it has infrastructure file, which is a terraform file which creates corresponding ec2 and rds instances.

B

And if we just go to the to the me here, yeah there's like two two requirements that we should start with, that you have dockerized web applications so having a docker file pretty common these days and your web application should be expected to run port 5000. However, this can be customized.

B

So, let's start with uh yeah, I was creating a new project uh as a rails developer. I like it to be rails, so.

B

Let me just use gitlab templates and make a rails application just to make it faster and let it call.

B

Dc rails test five minute like I have quite a few test examples already, so I need to be creative with naming for now, we'll use a public project. So, let's, let's start with.

B

B

As you see now, gitlab created for us a new ruby on rails project it all here, so we can start with that. Let's see if we can deploy it with our five minute production, app initiative. So first thing: first, let's check the requirements they have.

B

It says that you should have a docker file, so let's create a docker file. Then.

B

I'll just use as much gitlab help as possible, so I use a docker file. Ruby template.

B

Now it asks us to be on port 5000, so I'll just change it here right away to save us. Some.

B

B

uh Okay, now we have a docker file, so that should make the application runnable. um What else we need? We have it on port 5000. Let's move to the next step, so yeah. This readme describes it by default. The infrastructure contains of ec2 instance and db instance, both of them t2 micro, which are in amazon free tier. So it won't be expensive now, uh because it's still early in in development, we we need to have some credentials set up.

B

So we ask user of three three variables: access key secret key and the region, those variables you get on aws uh console websites.

B

I prepared them in the visual studio for me right from the start, so let's go and set them.

B

We set those variables in project settings ci cd section here we can add variables. Let's go. We need aws key, a double secret, key and region.

B

Let me start with region. I will use eu s2 since I'm in eu location, so I'm adding aws region now. I need secret.

B

Key I'll mark the variable, since it uh consider it as a secret and.

B

We need the last one.

A

We are uh live streaming this, so you might want to reset everything after you're uh done, dimitri.

B

Yeah, of course, otherwise I'll have a build tomorrow. uh Okay, so that's pretty common procedure. So every time you work with terraform you work with whatever the code that builds infrastructure. You usually need access, key secret and region, so that should be pretty familiar to user. So we have those variables set. We have a docker file.

B

Now we need gitlab, see a yaml file.

B

In future, we hope it to be part of uh existing gitlab template, maybe within after devote maybe a variation of this, but for now we just.

A

Yeah, I think it it should be part of autodevops that hey, if we find you have those aws things set, we'll just use this. Instead.

B

Yeah yeah uh for now for for a speed of development we just use in uh makes.

A

Total sense we have an entire quarter to get there.

B

Yeah, so it's as easy as including this two lines which basically says that hey take the ci yaml template from this project that we are using right now and.

B

We commit the changes and, after that we should see a pipeline running.

B

And it has three stages: yeah, don't don't put much attention of the naming or anything, but.

B

Aws provision job, what it does, it will run terraform and deploy your infrastructure. Your ec2 instance. Your rds instance all is going in this job.

B

At the same time, another job happening docker build which is exactly the same job as build in autodevops. It takes the application and just con, creates a container out of it and starts it in gitlab container registry.

B

So before we do a deploy, two things need to be done. We need an our application container and we need infrastructure in place.

B

Terraform run can take 2-3 minutes, so we would need to wait for.

A

That and we appreciate our viewers not using these credentials to start the bitcoin miners just yet um so can we talk since we have to kill some time? I think so far. I've made some notes.

A

I think there's four things that, like first of all, I'm super impressed with the progress so far, so this is super good things that we still have to do and we have a whole quarter to do this, but integrate this with other devops, get the aws credentials from aws and put them in ci variables like automatic automatically or something get the urls and search from the terraform run and put them in ci variables and allow people to sign into gitlab with aws.

A

Those are some things that come to mind. What do you think dimitri.

B

Yeah, the reason why I'm not focusing on those synths is that it will require some gitlab work, which uh yeah I'm uh using three knowledges for for terraform for aws, for all that expertise, and that's why we are doing much of the progress in this direction.

B

I think for, of course, creating those cicd variables uh is is annoying part, but it's a known concept yet for developers. So, even if we release or the first iteration is like that, it's still useful for developers because they can yeah, they can get those variables pretty easily from aws and from there on, like the most complex part of actually deploying we do. That.

B

uh Second is logging with aws is one thing is getting all this token and linking account? Second, one is that for the project we would probably want it. uh We want to get those key and uh also swelling specific for a project or for a group.

B

So you can, you know, have multiple projects using five minute production up, not reusing.

A

The same okay: you need to create an iem user and the whole.

B

Yeah, yes, yes! So for now we just like relying on our users to like you know to manage that. But it's uh you know it's a lot of. uh At least I think it's a lot of work and it won't give us such immediate benefit, like you know, uh like those terraform scenario and gitlab yaml, that you can include.

A

Let's focus on that first, I totally agree with you with the order in which you're doing it.

B

So aws provision succeeded docker, build succeeding so now it's trying to deploy our application into the ec2 instance that it created.

B

How application would know uh how to connect to the database so we're using a usual uh generic concept of database url environmental variable and what happens here in this deploy job? We have easy tool instance running. We install docker there, and then we run your container image inside docker on this ec2 instance, and we pass a database url to it, which we know from the terraform script. That was happening in the previous build when we permissioned rds so well.

A

You're already getting data from terraform back into gitlab ci variables. Yes, three, that is uh pretty sweet, well done.

B

Since three for doing that, it was uh yeah pretty fast on solving this issue.

B

B

The job seems to succeed, and that means that we should be able to check our application now I'll, just go to operation environments, and I would see our environment here from branch master. And if I press the button open live environment, it will says that you're on rails, that's.

A

B

And that means that our application is running, and that means that it also connected to the database. Otherwise there would be a database timeout exception from rails.

B

So on this note, I'm going to pause the demo and give it back to whoever is next on the gender.

A

Well, I just want to first congratulate uh street uh like that is really cool. I think that getting the terraform data back and then using that for the subsequent runs is one of the most pivotal parts and I'm really impressed that's already done tree. I want to give the floor to you anything you want to add or celebrate.

C

No just thank you. I must tell you that I cheated so I did not do everything in one week I had a working example in the past, so I just used most of that dc also had a lot of examples here and there, so we just put them together, and we have this.

A

Awesome what um I think we'll do this call on a weekly basis. What what can we expect next week? What's the ambition.

B

uh So there are a few things. One of them is. uh We want to further improve db migration, support and dbin utilization. So that's.

A

Great, what is the second thing, the utilization.

B

Not utilization initialize, so the binary.

A

The setup run and the migration room yeah great.

B

So we want to have this we'll, like we already have some on progress on it, but I'm not uh yeah. It's it's not completely finished so db. Initialized. We make great, that's that's what we're uh looking at for next scene, and uh we also want to give our users ability to change their instance types through the variables in ci yaml file. So let's say they over group t2 micro and they want something bigger they need.

B

They need a way to just change from t2 micro to t2 large, commit yeah, made a comment, push it and have terraform rebuild the infrastructure with larger instance.

B

So we want to look at that and there's like some polishing still need to be done. There was a few issues that I noticed that yeah can occur. So that's the plan.

A

That's really awesome.

B

And there's, of course, further plans like we want. We want domains, we want ssl, we want a lot of stuff. So that's like that's further further ahead.

A

Great yeah and maybe other services like redis and s.

B

A

That's great other frameworks, maybe make it work with a node thing. It works with a knot. Okay works.

B

In the closure.

A

Maybe if we maybe, if we do the db migrations, that's going to be real.

B

A

But good point that today it works with any framework.

B

I tested it with closure and rails and three tested it to be spyton and node. So we get four projects running with, with the same amount of efforts.

A

That is very promising.

A

Anybody else who wants to add.

C

Yeah, no, I just dropped a link to the readme, which contains examples across different languages and frameworks.

C

The readme also contains kind of like a step-by-step guide on how you can actually work with it. Let me actually share my screen and see if I can work you through, that.

C

All right, so uh the idea should be that you can just read this readme and that should be all you need, so you don't need to speak to dc or myself to get this working. We looked at. We looked at the steps that dc performed, but it's basically create a create a project set up the aws credentials and then include a pipeline.

C

Then we looked at what each stage and each job actually does that's been explained here as well. um Just a few things that weren't mentioned, um we already support multiple environments per project. So right now, if you spin up multiple branches within your project, so, for example, master for the main branch, but also you can have a staging patch or a review app branch. All of that will work. It is going to create new infrastructure and deploy a fresh instance of the app in a new environment per branch.

C

That's that a database url is being passed to the application. In each case, you can also define the two environment variables db, initialize and db migrate. It doesn't work perfect right now, so we trust you to not try this or try it but expect issues. But it's something we're working on uh what else uh yeah you can customize the port so, for example, if you, for whatever reason, don't want to run on port 5000, you can change that here.

C

You'll find a few examples: uh ruby on rails, closure, node and python with flask, uh just a quick document on what we want to do long term. Like I guess the goal is parity bit auto devops. In some sense, uh the biggest of that would be having domain names and ssl uh working really smooth. um Then probably you want to roll back deployments you that needs a lot of thinking, but it's it's supported in order to devop.

C

So the expectation would be that we support it here as well um link the logging and monitoring into the operations tabs that we have already. So I don't know if that is possible, but then to set a really ambitious goal: it'd be nice to try and do that in terms of integration with existing gitlab stages and features we integrate with the environments we integrate with container registry.

C

We integrate with the managed form state what we can quickly do, which will be like an easy win, is to in include all the code, quality scans and security scanners and the testing, and all of that. So these are jobs that are created for auto devops, which we can just include right away.

C

uh We kept it for the end for a reason, um and finally, there are like a bunch of aws free to your services, uh which we can add support for so uh a lot of things to work with, but then, like you said, the first goal is to get the db, initialize and migrate working and then build on that step by step.

D

C

D

Like to actually collaborate on the uh adding the security scans and items to this five-minute app, so I actually created a flask application for this very reason for testing out all our all of our security scans. So I've linked that there uh with you and if you get some time later on uh next week, we can talk a little bit further about that. Once the db migration is complete, making that easier, yeah.

B

Thank you perfect. Can we just delay all the security scans and everything, because this is like a bit out of scope, and this uh is something that we can do as a last stage when we introduced gitlab.

A

I think, instead of getting the security scans into this, let's get this into auto devops. At the end,.

B

Yeah yeah, that's that's.

A

That should be the ambition, not like another template we have to maintain. We already have problem between our default templates and other devops. If we create another set of templates for five minute app, we just made it worse.

A

Cool this is very impressive. Our mission is, everyone can contribute. I'd, encourage three, some some proposals instead of having the subprojects three stuff, maybe just move all your stuff to the root of the five minute production app, have an issue board, link it from the readme and then consider sitting setting up a slack channel or discord channel that you link from the readme as well, so that people can can engage so that we grow community around this.

C

Yeah sounds good, uh yeah I'll dizzy I'll. Let you move the project to the right place because.

B

Yeah, I will organize things yeah. I will do.

C

Hey but uh so we can use discord officially for gitlab business. That's nice to me.

A

I I think, uh I think, as long as there's head there's no customer data here involved or anything else. This is just. uh I think I I think uh discord isn't seems like an up-and-coming way to create a community around something. So I'm supportive if you want to use that.

C

No there's actually already a really. This is completely off topic, but there is a gitlab discord, channel or group and it's full of gitlab gamers. So that's that's pretty active already, but anyway I'll stop hijacking. This meeting.

A

Cool wall street awesome work thanks, dimitri for the presentation. This is way above where I expected it to be at this point, so really well done looking forward to next week. Thank you.

B

Have a good one bye thanks! Everyone.