GitLab Solutions Architecture, 28 Mar 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: GitLab CI for Monorepos

Description

Learn how to leverage template jobs, rules:changes, includes, and other CI features to build applications in mono-repos.

A

Foreign everybody Welcome to gitlab CI in a mono repo context. My name is Sam Morris I'm, a senior customer success, manager and I want to show you how you can leverage our verify as well as secure features when you're using a monorepo.

A

So essentially, what we're considering to be a mono repo is a gitlab project that has multiple different applications inside of it. So in this case each application is housed in a separate folder here, but they could have been in separate gitlab projects. Some corporations decide to use a monorepo structure and put multiple different Deployable applications into a single project location.

A

So, let's kind of go through how this works essentially I have two different applications here: one it's a Java application, one, it's a python application, that's named Java and python uh here is my gitlab CI yaml. This is uh quite simple, as you can see.

A

um So essentially, it has only three different stages and a top level job. So this is indicating that there's a job running at this top level, dot gitlab ciml file- uh it's only you know putting printing to the screen hello world. It's not doing anything too fancy.

A

However, you'll notice that I also have an include and local keyword for two other files. One is a gitlab ciml file in the Java directory and one is a py.getlab ciml file in the python directory. These could be named anything all this line is doing is including the files in these two locations, so I named them, gitlab ciml as well, and just prefixed, with like a j or a py, to indicate the language of the application that I'm, including, unfortunately, we do not have a conditional, includes for rules based on changes.

A

Ideally, we would only include these separate files in some situations, such as, if there was a change to a file in the Java directory or a change to a file in the python directory, then we would choose to include the file in that directory However. Unfortunately, our rules include, as of March 2023, does not have does not support that specific keyword that changes keyword in the rules if clause local also means that this is local to this project. It's not pulling from an external project, because otherwise we wouldn't have a monorepo all right.

A

So, let's back up and see what the contents of those files are that we're, including so in this case I I, don't actually have an application in this folder for this, for all intents and purposes, I just have like a dummy file right here, a txt file uh that we can change and essentially, if I change the contents of this file, it should trigger the Java gitlab ciml, specific Jobs versus if I change the python one, it would trigger those jobs.

A

Let's see the logic for this, so This has the same stages, because it should map that top level stages and not have additional ones.

A

I also define a class here at a class in gitlab, is denoted by this dot keyword, so the dot Java common, it could be named anything but I, just called it Java common, because these are rules that are going to be common to all of the jobs specific to the Java application and in this class I Define. The rules changes here so essentially again, you cannot do a conditional includes so I couldn't conditionally include the contents of this file.

A

In the case of an update to a file in the Java directory, however, I can create these common rules in which, if there is a change in the Java directory to any file indicated by this wildcard asterisk right here, I include these common rules in the jobs right, so I have Java build job. This has to extend that class Java Dash common. So it's essentially going to inherit these rules and I'm doing that, because I want to be able to kind of group together any rules. You could also do variables or whatnot in this class.

A

I wanted to group it together, just to promote code reuse, so I only have to write, extends colon and then the class name instead of rewriting these same rule changes for every job because that's going to get a bit tedious, especially because these gitlab ciml files can get quite complex quickly, especially given a mono repo context. When you know you might have even more complicated well far more complicated, build tests and deploy processes.

A

So it's best to just kind of keep this clean and neat. So I use my class and then I say that it extends to have a common, so it will inherit this rules. Changes I do that for all of the Java specific jobs, Java build job, Java, test job Etc, all right, as you can imagine it's the same deal for the python.

A

As you can see python common. This is just watching for any changes to any files in the python directory again extending that common rule set so you'll see that pattern there of you know you want to improve this code reuse. However, um perhaps you create jobs and you want them to extend some other class.

A

This is just one way to implement the same logic, but my top priority here was I want to run certain jobs um if I change a file in the directory right, so that if I'm making a change to my python, app and I couldn't care less about my Java app, then I will only you know, build and test the the python application right.

A

So it's to save some time and your pipeline runs because otherwise the jobs would be triggered in parallel and you would have to wait even longer for your entire pipeline to complete, which is not ideal right.

A

So now that we've seen this, we can kind of see it in action here so I'm going to make a quick change, I'll go ahead and do it to my Java file. So this is Java Dash example. There's no contents in it! So, let's open, um let's open in the web IDE here, just make a quick change to the file.

A

Right making a change.

A

So now I'm indicated I can commit so I'm going to do. Java change commit and push and I'm going to commit to a new Branch, so I'm making a merge request as best practice.

A

Let's call it Java change right now: click on create merge, request.

A

My pipeline's already been kicked off. I'm going to click, create merge, request.

A

So, as you can see that the Java build job is indeed running, and then the Java test job as well will be running next. So essentially, what happened here was you know during my merge request. I I only have this job, the Java job and I. Always I only have the Java test job running as well, so I'm not running my python. One I will say, though, that given you are using the rules, changes and you're watching for changes.

A

There are some caveats to keep in mind here, so essentially, if I just trigger my CI CD pipeline, so we're going to shift gears if I simply clicked run pipeline and I'm running on the main branch, you will notice that all the jobs are running right.

A

So this is because of that rules, changes, keyword and the logic is that um essentially, this isn't seen as like um a push or no push indicator. So the rules changes is only working on merger, Quest pipelines, otherwise, rules changes will always be identified as true.

A

So, if you're just directly running your pipeline, which can happen if you're doing a scheduled pipeline, run, for example, a manual pipeline run where I just go in and click it, it's going to run all of the jobs. Okay and that's not the worst thing in the world. It just means that you know the pipeline might take longer and that that logic of the rules changes doesn't take place. That only occurs in Branch pipelines and in merge request Pipelines.

A

So just something to keep in mind um that you won't see the same logic if you just go and click run pipeline from your CI CD tab. However, in the merge request, you will see that this ran just the Java specific jobs right here and to secure this. You can leverage all kinds of features you can leverage your scan result, policies which you know run due to vulnerabilities being found in your Verge requests and requiring approvals from your security teams.

A

You can also run scan execution policies to run security scans agnostic to the gitlab cieml file, which might be great in a monorepo context when this can get a little bit confusing and you can also apply a compliance pipeline as well.

A

So if I go ahead and click into my compliance framework and I just select one for example, example compliance framework I, believe this has secret detection, container scanning and one other scanner.

A

I click, save changes and I go ahead and Trigger. My pipeline.

A

Again, this is going to run all the jobs, because that rules, if changes is going to evaluate to true but you'll, see it automatically adds the scans to my test stage here.

A

And that's that and you don't have to manage this within the gitlab ciml. As you remember, the gitlab ciml file from before doesn't have any test. Jobs spelled out at all and it doesn't include any scanner templates. You certainly can include you know if you want to include SAS or secret detection or anything in this file, you certainly can, but you can still also leverage that control of the compliance pipeline as well. Another thing to note: is you don't if you don't want to set up your own gitlab, CI yaml files?

A

Here you can always set up your Python and Java or any other language, specific jobs and then set the rules to um only changes Etc in this top level. Gitlab ciml file, but this is a best practice if you want to keep things kind of modular and clean, and that's that hope. This was helpful in understanding how to use gitlab, CI and secure in a monorepo context.