GitLab ⛅🌱 Cloud Seed, 19 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-11-19 5 minute production app sync

Description

Direction: https://about.gitlab.com/direction/5-min-production/

A

You uh want to start with a demo.

B

Yes, hi everyone, my name is dmitry, and this is five minute production app uh weekly sync. So last week we showed that with a few lines in gitlab, say yaml you can get your application deployed to aws using ec2 instances and using rds for database.

B

So let me share my screen first yep now, for today demo, I created a new project which is pretty much the same stage that the previous week project had so the rails app generated from template.

B

It has gitlab, say a yaml file with just two lines of code, including uh five minutes production up, deploy, template, and it also has uci cd variables, which are for aws.

B

Now I want to start with the first essential thing when you do rails development, like once you have your application running, you probably want to to have something more than static yeah, and for that I'll just generate some some code that will require database migration and.

B

I'm going to push it and I'm going to make a five minute production app work with migrations.

B

So I pushed a new code that contains a migration in it and it triggers the pipeline right. But right now, five minute production app knows nothing about migrations about how to initialize a db schema. So we need to to make according changes in cimo file.

B

If we go to the deploy template the readme that I used uh during the previous demo uh yeah, we have quite a lot of information here, but uh in the section of database it says that environmental variables to be initialized and db migrate if sat, are executed right after stock run. So what that means is that we can set up two variables for scripting one to prepare the database one time and another one to run on each pipeline to update it. So, basically, migrations, dbini shall lies and dbmigrate variables.

B

You might recognize them because they are also used in autodevops. In the same way, we rely on users to find and debit initialize and dbimigrate for themselves, and we use the same approach the same name in here. So we, our first goal, is to make five minute production app uh compatible with auto develop. So we can integrate it back easily, so uh yeah, let's, let's add those variables and in the readme we have an example for rails and since I'm using rails, we just use those.

B

If you use whatever framework or other language, of course, those scripts will be different for.

C

B

um Time, let me let me check how the pipeline is.

B

B

Okay, now, uh when I made appropriate changes uh yeah, I just wait for the pipeline and uh while wait for the pipeline uh yeah, we can do two things. uh You can ask me questions right now. If you have any- or I just uh show you uh other functionalities- that we have.

A

Other functionality.

B

Yay good good choice, so uh yeah I'm using master right now in this example but yeah. We know that we work in future branches and let's say I push some feature in the branch and yeah. If you use this five minute production app, you can just go to the merch request and your pipeline will run here too. So if you open the app from here, you'll see actually your review app running and you can notice the difference in environments.

B

Here's our master running here is our review app running now, when you, when you play with it, and when you accept the merch request, it will automatically run terraform, destroy destroying the review app infrastructure created for it.

B

So this way you can just use the general workflows and merch requests and it will create review apps and destroy review apps, as uh as you create and merge, and accept merch.

B

B

So right now I yeah I just accept merch requests and triggers uh destroy job which will uh be equivalent of terraform, destroy and remove all the feature branch infrastructure created while still keeping master alive.

A

That's super cool the michi, so you run kind of one ec2 instance for the master branch and then the review branches spawn additional ec2 micro instances uh yeah exactly that. That's so simple! That's so cool.

B

B

About the behavior, so those variables aws success, key region secret key they can be protected, they can be unprotected, depends on the availability of the branches. So uh let's say people don't want to run this on the feature branches, so they just make those variables protected and if those variables are not available in regular branches, there will be no deploy happening, no, uh no provisioning, nothing. So it depends on your needs. You can tweak this behavior, but simply telling uh through cicd variables, at what branches you want uh visibility of those variables.

B

So if you want to run it on stage in production and only protected branches, you make them protected. If you want to run this on every branch you make them unprotected.

B

Now, let's go back to our first project and see if the pipeline is good, so the first pipeline was without changes to the kit labs here. Yeah also, there is nothing happened here it just. It deployed the code, but uh yeah it's not enough. Now we updated, say yeah movies, db comments. Let's see what happening here.

B

And yeah, it's still.

D

B

B

Good now now it should try good good okay. So this is a failure and the reason why it fails is not because we did something wrong, but because rails uh has credentials and uh to make deployment safe, rails, encrypts those credentials and keeps them in config, slash, credential and encrypted file and in order to decrypt them during any rake task rails need a secret key passed to it.

B

So in this particular case we want our application running, so we want to give secret key base uh variable to our application within the docker container, and we're going to do that. uh I have the variable locally. So let me just copy paste it, but now, if we just add the variable it won't be available to within the instance of the docker within the ec2, so like uh terraform, we have a different concept specifically to pass variables to the application to the internal site.

B

Internal part of uh docuron, you just add gitlab wire as a prefix and do like rails master key, and you put the value here.

B

And now we just retry the job and see if it will work- and this is a another feature that we introduced is like. uh We, you want the ability to pass environmental variables to your application running, otherwise, you can't pass certain credentials. You can't.

B

Can do much of flexibility.

A

Sorry, why why is there a difference like a database, uh aws region, I can just define and glvar. I have to put glvar before it. Why is that? What is the translation step? I'm missing.

B

Yeah so aws credentials, you define it for terraform right. So it's available right inside the ci pipeline, but this variable is needed for rails, which is run inside docker container inside.

B

The environment is not passing through layers down, so uh what we do here in gitlab, we just uh reuse a terraform approach, so terraform has this tf bar prefixes and what it does it's like. Okay, well, we'll just take those prefix variables uh for you know for configuring stuff for terraform, and we just use similar approach here and says that everything that has jlvar prefix will pass down through to your web app within the docker.

B

And with the change it's uh now rails is working, it actually can decrypt the credentials and it runs a db initialize and it successfully creates the database. So if we go to the environments, if we open.

B

The live: if we go to posts, if we write something down, it will actually create it. That means that database table is created and yeah that actually enables us to, like you know, to proceed with further development, makes features, and so on.

A

Really awesome: what's the progress.

B

Thank you, I think I'll just stop sharing at this point.

A

That's your agenda, dimitri.

B

uh Yes, right number two uh organization wise: uh I moved the project to the subgroups and now we have everything under one name. uh I link to the project readme, so everyone who wants to try can go straight to readme right now and.

B

Try it out uh issuer board. I put a link to the issue board. So if you want to create an issue, if something is not working, if you have recommendation, if you want to contribute, you can find everything there and we also have a slack channel with the same name: five minute production app where you can reach out to me to three and yeah just.

B

Ask any any questions real time.

B

Okay, now, next to the next point, is a bit about uh our plans of the next week, so we want to introduce s3 and it's it's somewhat there, but we need to try it out and get it uh ready to be presentable. We also need to do some bug, fixings and more complex. uh The project coins and more uh side problems appears there need to be some fixes done there and yeah. I think that's.

A

Cool thanks for everything, uh you're you're able to work full-time on this there's enough to do yep. That's good awesome, darby at 0.4.

E

um Yeah I just wanted to throw this in here. I created a mr to add some of these links to the direction page, so other people can follow along there um and then I, uh as part of this, I created a a youtube playlist. So we can throw these meetings in there and if someone wants to catch up, they can do it that way. So if anyone has any feedback or other things to add in there go ahead and add that add a comment in there and I'll get that taken care of.

A

Awesome thanks darby 5.5 um gl var, it's like a docker fire, but we call it gl var, but because why not? But maybe maybe someone can add to the to the readme, which is excellent by the way? It's absolutely me that this this variable goes into the docker container, so that wasn't super intuitive to me.

A

Can you take that on.

D

D

Yeah, so let me just, uh let me just think about it a little bit, because uh the readme is titled variables exposed to the web application and uh and jailware eventually.

A

Gets exposed, it's a lie because some of the variables in there are not exposed by publishing no.

D

It is it is, uh it is absolutely 100 true, so all the all the variables here are actually provided to the docker container. That runs the web application.

A

Aws region, we supply that to the web application.

D

Yes- and you will uh you will know by next week: okay, yeah, cool, okay, think about it, but yes, so we can. We can definitely include that it goes to the docker container because that's also 100 true, but I just want to think about whether it's a detail we want to expose to the end user or not.

C

C

A

um Promotion wise: what are the plans you have? First blog post or what other people thinking.

A

Thinking, I michael, what do you think.

C

um I think it would be a good idea. um I just want to try it out myself. First, I didn't do that. This week I was waiting for the project being moved.

C

I will draft something up next week. I think yeah. I should have time.

A

Yeah and and the first blog post doesn't necessarily have to be about what is being what we're making here. It can also be a rent. Why does it take 20 minutes to make a production, app, 2020 speed, run between the different options heroku and the other ones kubernetes. The cost is too high. It's not in any free tier three tiers of the hyper clouds benchmarks.

A

uh Stuff like that. So I.

E

E

Hey michael I'm happy to help with any blog post stuff, too. uh I've got a couple ideas that we could roll in there. So um if you want we can we can sing up offline.

C

Yeah that would be super nice. um I have some ideas myself, but probably it makes sense to just say: hey. I have this problem in my experience and- and you have that and we are trying to solve it together.

E

D

Do we have any volunteers to actually test this out? Because I want to test out what we've built so far and also if the readme is useful or not.

E

Yeah, I I was gonna, hopefully uh take a crack at it tomorrow.

D

Perfect perfect, so if you have a look at uh the group five-minute production app, you will see that uh we already have two subgroups in there for examples and sandbox. So examples are so sandbox is a place where you can just dump your work. It doesn't really matter if it is in a running state or not, but the example. Subgroup is actually linked through the readme and it's probably a place for curated well-maintained examples. So if you can just do all your work, there that'll be great or else you can do it wherever you like.

D

It should not be a problem, but if you have an example to contribute to the examples, subgroup that'll be great. Okay,.

E

F

Hey darby, we can also uh advertise on the weekend review. There might be other engineers who want to try it out.

E

Oh yeah, it's a good idea.

F

The big thing is is uh credentials uh to basically run it on aws, but um you never.

D

Know: hey uh since I have a senior engineering manager and vp of development on the call. Let me uh just raise a question or a concern, or if I don't know what it, what to label it, but just a thought.

D

This pipeline uses the gitlab terraform wrapper right and uh that is of course mapped or that wraps over a specific version of of the of a specific version of terraform against which the state gets created. Right. So if, let's say the release next week in 13.6, the rapper decides to use terraform version plus one what happens to the existing state, because the way this works, uh the state is bound to the specific version of the binary you're using so something to think about.

D

I don't know if my concern was clear, but uh yeah.

E

I think that makes sense. um I I don't have enough knowledge of terraform to know like what would happen, but I can. I can track that down um I'll sync with the team that knows the most there yeah.

D

It'll be just you know like in the perfect world. What's going to happen is the minute we bump the gitlab terraform version uh maybe run some automated tests that check out the five-minute route as well. So just link that up, because state is key right. If we yeah, if we mess up the state, then uh it just stops working one fine day, yeah.

E

F

Good call: trey, uh don't worry about uh titles, uh we're here to help.

D

Please yeah no, but I'm going to hold you to your title. So hopefully I raised it with you and we can get something good out of this.

A

Fair enough he's not intimidated, he sees the possibilities that.

F

People have I understand my role in this organization. These days.

A

More title, more responsibility, great well, three really really well done. um This really amazing.