GitLab Solutions Architecture, 29 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Required Pipeline Configuration Alternative

Description

This video was made in response to conversations with Jason Yavorska & Matt Gonzales

https://gitlab.com/groups/gitlab-org/-/epics/3156#note_387526890

A

Hi, my name is tim poffenberger and I'm a solutions architect at gitlab, and I wanted to talk through a an alternate alternative solution that I had for the required pipelines configuration to make it a little bit more friendly for developers.

A

I'm going to be flipping back between two different projects. In this example, this separation poc developer project, as well as this compliance yaml project yaml.

A

So keep an eye on the top left of the screen and I'll try to do my best to remind you as well. So oftentimes developers want to have autonomy over their gitlab yaml file, as well as their source code, so they can um in increase the the level of testing or change.

A

How they're deploying out to maybe development level environments, uh which is great uh with and get lab, is very empowering from that standpoint, where get lab might in in some people's opinion, fall short as the ability for a um an external team to really dictate a few uh key things within a gitlab yaml definition to ensure that certain things are always happening for certain types of projects say. Let's say that you have a some sort of compliance or hipaa compliance or pci compliance. It would be great to be able to say anything that requires pci.

A

Compliance always has to have these particular ci jobs defined.

A

So this project here this is the developer project and oftentimes a developer is going to have you know a build and unit testing and deploying out to a particular environment and their yaml file's, going to look like that, and that's going to culminate in this, you know compile and unit and deploy the dev type pipeline, but there's nothing preventing the the developer again from you know, adding or removing you know other jobs, and that includes all the jobs that a particular compliance team care about.

A

So get lab's original solution, for this was this instance wide, meaning that every every applicator or every project on a particular installation would be required. A certain pipeline configuration and again this is an instance wide setting, and it allows a instance administrator to define a pipeline or pipeline jobs that always need to be able to run.

A

Unfortunately, what ends up happening is a developer brings in a new use case that isn't like any other project, and now they are required to run jobs that aren't necessarily relevant to them or the project.

A

This solution is relying on dynamic child pipelines and it gives the compliance team, so this compliance project yaml uh the ability to set uh specific jobs before and after so they have this before stage and after stage, and then these the project stage, so they can do their static analysis before and their dynamic analysis. After and in the middle, let the development teams utilizing dynamic. Child pipelines run their uh their own gitlab ammo file, that's totally dependent on on their workflow and really what you have to do.

A

Is you just set this custom ci configuration path within the developer project, and when that is set to this external location, the developer doesn't have the ability to modify this and they get their jobs included in their their in their workflow. So we're back on their separation, poc, developer project and we're seeing sas happen and dashed happen, and then, within this child pipeline, you see this compile unit and dev, um so the the jobs that the developer cares about.

A

One of the gaps here is that um parent or the these dynamic child pipelines, the artifacts, are not shared between the two, so the sas artifacts are not propagated down and subsequently, like the test results, deploy results aren't propagated back up, meaning the merge request page becomes way less valuable for the the developer.

A

A uh another option for this is- uh and one of the the other caveats here is that the auditing team doesn't really have insight into who's. Who has these enabled?

A

So um this is an additional modification of the the last proposal that I just presented and it adds an additional stage called aldi audit, which is going to kick off a downstream pipeline. Using the job trigger syntax and it's going to record a all of the the projects that are calling this and what so, when you set the requiring record for the developer project.

A

Now, when this pipeline kicks off, we, we actually see a um and now we're in our compliance project, and you can see that um it's recording and we uh we can drill back into this, um the developer project and see the sas run and the dash run and this record job run. So it gives the the compliance team some level of insight into who's who's, utilizing this. This capability.

A

Ideally, though, um we wouldn't have to rely on these dynamic child pipelines for this and there's a couple different options that I think we could uh pursue. One is the an additional key within the includes syntax, which would be relative, so relative would mean that it's local to the project running the gitlab yaml file, not the local, to the the in this instance. The compliance project, so relative, whatever developer project, is utilizing this the gitlab yaml would be picked up there.

A

Otherwise, we could also use this remote if ci variables were included. This project url this commit sha, and you can see that those are automatically defined here.

A

In addition to this, the compliance framework seems to be the best fit for for pairing with this, so we have within a the developer project they can specify which, which compliance framework they adhere to, and if we had the ability to tie a gitlab yaml file. Here, we and pair that, with this include statement, either relative or remote, utilizing the ci project variables, the the compliance team can um globally. You know, manage and monitor those capabilities and um not disrupt everyone in a particular particular gitlab instance.

A

The other nice thing about this is this: has the potential to ensure that this is accommodating both gitlab.com and our self-managed users.

A

One other thing to know is the ci config path which I was referencing before is also available via our get lab projects, api, which gives the which gives the compliance team some way to consistently monitor and ensure that the ci configuration path is what it's supposed to be thanks for your time. Hopefully, this was helpful.