GitLab Enablement, 17 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Manage / Reduce CI/CD Minutes Usage on Shared GitLab.com Runners

Description

Greg Myers from GitLab supports deep dives into how you can manage or reduce your CI/CD Minutes Usage on shared GitLab.com Runners.

A

This is greg from gitlab support in this video. I will go over how to manage and reduce ci minute usage on gitlab.com.

A

I will start out by showing the user interface and user experience for visualizing ci minute usage. Ci minutes are used per project and the usage quota is at the name space level, so that would be either at the user name level or the group level.

A

Next I'll summarize, several options available for users looking to streamline or optimize get lab, ci cd jobs and pipelines to help improve efficiency and reduce their ci minute usage. Last I will show how to bring your own get lab runner and connect it to gitlab.com as an alternative to using shared runners.

A

Only the getlab.com provided shared runners count towards ci minute limits, so when using your own runner or runners on gitlab.com for gitlab cicd ci minutes are virtually unlimited. The page I have pulled up in my browser here. This is the namespace view for my group. My group is called just greg and includes just me here I can see these are the different subgroups and projects I have in this namespace to see my usage of ci minutes. I go to settings and usage quotas.

A

This pulls up a per project breakdown of ci cd minutes.

A

I can see here that, overall, since august 1st, I have used 208 minutes in this namespace and looking at the per project minute breakdown, it looks like this distance runner is taking up most of the ci minute consumption at this point.

A

If I click on one of these projects, then it pulls up the project page from here I can go to cicd. This is the pipeline view.

A

This is a valuable source of information here I can see the time taken per job, so it looks like I'm sorry per pipeline. It looks like this pipeline took 39 minutes. This one took 31 and this one took 31 as well.

A

I can see I have a limit here of 2000, so I can also click to buy additional minutes. I'm not going to go over the purchase flow in this video. Just wanting to point out that that's where to go to get some extra minutes now that I know which projects are using my ci minutes and how many ci minutes each project is using thus far, I can think about how I could go about either optimizing or streamlining my git lab cicd jobs and pipelines to improve efficiency and reduce my ci minute usage.

A

One way to restrict your ci pipelines and jobs from executing only when you need them and if you need them, is to use rules in your ci job and somewhat intuitively.

A

We use the syntax for workflow rules of if and when. Additionally, there are options for when manual manual is a good option. If you only need to execute this job by manual intervention, this is a good way to avoid jobs that are costly in terms of ci minutes from executing on every push.

A

If you use when manual, then you can define when you would like it to occur and specify that you would like it to occur after manual intervention.

A

Otherwise, there are the rules which also have, if and when in the syntax, combining the power of rules workflow rules when we can pinpoint specifically at what time we would like to use ci minutes and ensure that it's only if we need to and when it's helpful in the same vein, there is the risk of scheduling pipelines too frequently. This is not for scheduled pipelines.

A

They are a bit different than a regular pipeline in that these pipelines are configured generally through the user interface and they use cron syntax, and we can see here that this is something separate from the if, when rules and syntax for gitlab ci, this is in the user interface itself.

A

For an example. Here I can show you this runner here aptly named runner on a schedule.

A

This is set to run every five minutes. I have a pipeline that is scheduled. We can see okay, yes, it is running every five minutes. Now it's pretty rare that someone would need to run a pipeline every five minutes, so seeing no uh project activity.

A

uh Last update two hours ago, I can deduce that this is a scheduled pipeline, and here it is. We have the per minute weather report pipeline and this is chrome syntax. But this says every five minutes execute this ci job. I'm gonna change it from every five minutes to every day.

A

Now I will keep this tab open and we'll come back to it later, but I would expect that this no longer executes every five minutes that it changed it to once a day, so in doing that, I've effectively reduced the number of jobs and therefore number of minutes that I'm using ci minutes on a shared runner.

A

There is also a pipeline schedules. Api endpoint. This is good for automating, either getting or editing deleting modifying pipeline schedules in a programmatic way.

A

Another somewhat common way of eating up ci minutes is to have a job that has no time out, or it has a long time out and that job is freezing or failing silently and for whatever reason the job does not terminate.

A

It always hits that timeout on gitlab.com our default timeout is one hour, but timeouts can be defined. A few different ways: timeouts can be configured per job. This per job level timeout can exceed the project level timeout.

A

If, instead of extending the timeout, I wanted to shorten the timeout, I could make it the timeout on a job, 30 minutes or 10 minutes, and this will also ex overwrite that one hour default and this can be defined at the job level in the ci template it can also be configured on the level of per project.

A

Now per job will override the per project timeout setting, but the per project timeout setting can be configured right in the user interface go to your project, settings ci cd, general pipeline settings.

A

So let's find one that I think might be taking too long to time out. Think marathon running that sounds like it would take longer than a distance runner. Let's see. Oh there's a job running right now,.

A

Test job wow, so this is a good example. This job has been running for and 110 minutes just under two hours, and it has a three hour. Timeout.

A

I can fix this by adding a timeout uh for the sake of demonstration. One minute to this.

A

Now I've not actually modified the job. That's running I've just added a one minute timeout now we see there's timeout of one minute coming from the job and I would not expect this job to take longer than one minute, whereas the previous job was taking just under two hours and would have gone right up to that three hour.

A

A

Okay, job failed execution took longer than one minute so for jobs where you want to have a safe check, make sure they're, not just freezing and using all of your ci minutes. That is adding a timeout can be very valuable. Go back to the beginning here.

A

The last tip that I'll go over in this video for reducing ci minute usage on shares. Runner is to take advantage of get lab. Runner caching features get lab. Runner has a caching mechanism that can be used to save time and ci minutes when running your job.

A

Caching will speed up the time a job takes to execute by reusing content from a previous job. This can be particularly helpful for caching dependencies.

A

We have this caching documentation here, which is comprehensive with good caching practices, common use cases, and we even have examples of ci templates demonstrating how you can cache your dependencies to reduce your build time.

A

These are useful examples on how to cash dependencies for reuse in jobs, so you're not constantly downloading another method to improve the efficiency of your get lab, ci, cd jobs and streamline the minutes you use is to use a docker image that comes bundled with all that. You need right off the bat instead of starting with a base image and building up every time.

A

To give an example of this, I have this ci demo job and I have two different ex jobs within this job. Template first is called from scratch, and what this does is it starts with this image.

A

It updates and upgrades the software. That's currently installed it downloads, the software that I want to use, and then it executes the software to give me the results that I would like out of my ci job.

A

This second example. This uses an image here that is essentially a snapshot taken of. If you start out with this and go up to here and then pause, that's where this job starts and therefore the only thing that is required instead of these four steps is just one step.

A

So, let's see what that looks like in reality here going to run a pipeline.

A

And I see here I have two jobs. One is all in one and one is from scratch: they're both running.

A

And the timer has begun I'll pause this and come back as soon as they have completed.

A

Okay, so the jobs have completed and we can see the all-in-one to one minute. 39 and from scratch took three minutes 37 seconds.

A

Now these jobs are relatively simple, but this could definitely be the same case with a much larger amount of time where just installing the dependencies and software to get up and going in a job can take the majority of the ci minute time. Let's look at the difference in execution here. This is the job log, that's for the all-in-one, and this is for from scratch. Now from scratch. We see it's a much longer job look. We are installing.

A

It just keeps on going and going 2040 lines versus 248 lines, 1 minute 39 seconds versus 3 minutes 37 seconds, and that is done just by taking advantage of the container registry we have built into gitlab.

A

I have this image here that contains all of those goodies and here's the docker file extremely simple. This is just what I used to create that image, and that is what is in my gitlab container registry. Right here.

A

Now I mentioned that there is the option to bring your own gitlab runner, which is essentially get lab runner software operating on a server or infrastructure that you control hook, that up to your gitlab.com group or project and use that for virtually unlimited ci minutes.

A

So I will demonstrate how to do that in the documentation for this we so first to get things started. We need to install get lab runner now. Installing gitlab runner means you have a server a virtual machine, a something running, docker kubernetes, open shift. You can even run it on mac, os and windows uh freebsd, some infrastructure on which to run the gitlab runner software, and then we start from here I personally like using docker, because I think it has some advantages over other types of runner.

A

I know kubernetes is going to be the best choice for auto devops and uh continuous deployment. Kubernetes runners are very powerful and then there's also some situations where get lab shell or these just the package here. That's plenty sufficient or is preferable, I'm going to demonstrate how to do it using git lab in a docker get lab runner in a docker container so to install get lab runner. This is pretty easy. I just I can copy and paste this command and here's a terminal I'm going to ssh into the server the server greg runner here.

A

This is a server I'm using for this demo. It's a droplet and digitalocean and it has docker already installed. That's the only thing I did so far, so I just copy and paste this command into the terminal.

A

I see this big number that is actually a good sign.

A

Because I can see here that get lab runner container is created 12 seconds ago up for 11.

A

Seconds- and we have some logs coming in from that now next step is to hook this up to my group or my project, so that I can use this runner on gitlab.com.

A

To do that, I register the runner.

A

And requirements first, you install the get lab runner which we already have, and then we have the choice between project, specific or group runner, I'm going to do. Group runner and I select docker from this menu here to access the group runner settings. It's settings for my group runners and there's a drop down, see this heading set up a group runner manually, so install get lab runner, that's the documentation. I followed previously specify the following: url: okay, that's easy enough!.

A

Here run the register command. I can just copy and paste that.

A

Now it's asking me for the.

A

A

Asking me for a registration.

A

A

A description kit lab ci tags. These are very helpful. Most of the time for demonstration purposes, I'm going to leave them out I'll choose my doctor, executor and.

A

Default doctor image now it says runner registered successfully.

A

And here we see, I have a group runner, it's currently locked, that's my ip address. That's the description.

A

A

Click that save.

A

Now we see, I have a dedicated runner for my group. Now I'm going to demonstrate running a job.

A

My group, using just that runner.

A

Let's choose gitlab at the ci.

A

Project to specify I don't want to use shared runners, I go in, and I click disable shared runners for this project. Now that I have my shared runner or dedicated runner, byo bring your own runner registered. Let's see how I can actually use this in a project so see I have shared runners disabled. I could click enable if I want to but nope. I just want to use my group runners, so it looks like this one is available.

A

Let's try and see if we can run this job using my shared runner and then in this case it would not count toward any usage of shared runner consumption ci minute.

A

Consumption, I can see right here. This is my version of gitlab, and this is the description for this runner, so we can see right here. This is executing and this will not count towards the ci usage as calculated in my usage quota.

A

Here to summarize, we started out by having a look at ci minute usage in the usage quota tab. We also took a look at some ways to reduce ci minutes and consumption of ci minute for shared runner, including rules in the job template to define. If and when a pipeline should be triggered.

A

We went over scheduled pipelines and making sure they're not scheduled to run too frequently we touched on timeouts and how to prevent jobs from running longer than expected.

A

We also touched on rules and caching and using caching to reload assets instead of download them fresh every time, and we, I demonstrated how having a doctor container with your dependencies in terms of software and executables built right in, can reduce the execution time for your ci pipelines, and last I demonstrated how to install your own get lab runner using the docker executor register that runner on gitlab.com for a group and then showed how to run a pipeline using that runner and no shared runners.