GitLab Secure Stage, 24 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Jenkins CI triggering GitLab Security Scans on Success - Demo

Description

This video Demos how GitLab pipelines can integrate with Jenkins, and poll the status of a Jenkins job in order to trigger further stages in the pipeline.

A

It's easier to find, if you do it on your computer, I.

B

Find that doing a computer and uploaded to YouTube unfiltered is easiest. I find okay.

C

Yeah, that's what I just did perfect all right, thanks shells yeah, so yeah. Let me get started so so pretty much the the use case for this that we wanted to get going. Was some customers wanted to create a job? They had a build job already running in Jenkins and what ended up happening is they didn't want to migrate all that get laughing so a galette CI file, so they wanted the job to build in Jenkins, for example, and then, if that build were to be successful, then we would continue on in the pipeline.

C

But if that Jenkins build would fail, then we would. We would show a failure on get lot and not move forward in the pipeline, and that way we don't have to run all these excess test. If Jenkins job failed, so we wouldn't wouldn't continue down. So is that correct something right.

A

The idea is, there's customers that want to use our security capabilities, but they want to continue using Jenkins, ci, ok,.

C

Yeah, so so so exactly so, if Jenkins CI were to fail, then security, you wouldn't we wouldn't run the security scans. But if Jenkins CI were to pass, then we'd continue on and run the security scans and then even then we can even do CD via Jenkins or we can do CD vehicle app. So you can integrate Jenkins at any point in the pipeline. I'm.

A

Curious fern so you're saying: if, if Jenkins pipeline fails, then we don't even run the scan, but what if we run the scan and we find critical vulnerabilities, do we send something back to cause the Jenkins pipeline to fail so.

C

No, not necessarily so so the Jenkins pipeline, so so the way that it works right now is: if, if a build were to fail right, then you wouldn't move on to catch the security scans, but the security scans, the security scans can run then, and then they can cause the bill to fail, and then you wouldn't continue on to the next step. So you can you can do it. However, you can configure it in a order. So so.

A

If so, if you found a critical vulnerability, you could tell Jenkins not to correctly.

C

Take thinkin's not to run any further, so ads all depends just on configuration scripts that the customer can write and I'll show you that now I'll show you it in the build stage. So let me share my screen.

C

Okay, so can you see him the Jenkins interface mm-hmm.

D

C

So I created this Jenkins pipeline, her Jenkins job to be more accurate and the way that I configured it is so it does say, github project, but that's just because of Jenkins wording, but I was able to connect my get lab project which I have hosted in this URL and I said the good luck connections gilad for us so for source code management.

C

This is just basic set up that I'm just going through the how you how a customer would set it up, they would set the repository URL and the credentials and I'll try to get something up. The demo systems has a good write-up on this, but I can probably make it more. You know nice and customer centric, so that would be set up.

C

We wouldn't set up any build triggers because there's nothing that's going to trick we're gonna trigger this job from running manually from the from the CI from Gil at CI, so this wouldn't be configured in my build environment. I just have secrets, like my CI registry password, where customer would create a service user which would have all these things on there. So this is just for testing purposes, and so here's my actual job that I have in Jenkins. So in this job, what I'm gonna do is on any error.

C

I'm gonna exit out with error exit code and all I'm doing here is just on building my container and then pushing it to my doctor, posit or e, and then I'm exiting with the zeros to show success and then in post build actions using the get lab, plugin I'm telling it to publish the build status to Jenkins. So that's where you see the actual external job come and run right after the job is, is successful or failed and it will link to the Jenkins job.

C

So now, I will show you that in action, the job burning in action, then I'll go I'll, go on the Gil up, see I fell so let's run a pipeline so see SUV pipelines and I'm gonna go ahead and run a pipeline on master. So remember this pipeline is gonna, run this raker twit, so I'll add exit one later, so you can see it actually failed. So this is a success case. So you see that the build job kicks off and what the build job is doing.

C

Is it's running this custom Python script, that I wrote that just triggers the Jenkins API? So what it's doing is it's running this Python script that is calling Jenkins is starting the job and then it's waiting as polling for the job status. So it'll keep checking the job status to see if the job you know that's failed or if it's thora, if it's still running it'll, keep checking it and I added a timeout to it and if it's successful, then it'll show the result as successful.

C

So now that it succeeded, we'll go back and we're refresh, and we can see that now the security scans are running and if you look at the external job now popped up because they've published the build status and if we do click on the external we'll get to the Jenkins job, where we can see the actual job that ran and the output that came out from that job.

C

So now, just to show the error case if we exit one at the start of the job, so we're just going to throw a failure to start with if I go ahead and run the pipeline again,.

C

So it's going to trigger the job and then the job will automatically fail, and then you will see that none of these security scans will run, and this can be done in the same order we can. We can also have like that. Let's say the next job would be deploying a deploy job. You can do this. You can have this. If any of the scans fail, then we won't go to the deploy job so and that's what you're, what you're mainly worried? The customers are mainly looking for its end.

A

Right but the deploy would be back in Jenkins, so.

B

I'm just gonna ask so is that is that do the same thing as you did with build, where it kicks off a Jenkins job for correct good boy, okay, correct.

C

Yeah it'll be exactly the same. The ordering is up to the up to the user, but it'll be exactly the same, so you can and I'll show you that I'll show you that in depth in the CI, a no flaw can.

B

You go back to your first bite by that debt that passed it I, so that the security scans weren't done running yet when it output that a job that was after in the second of the stage after that yeah this one. So those external completed before the tests completed. Yes,.

C

So external, the only the only thing external is is, is just because of this right here because of this post build action, but there can be different post, build actions, means differently and and they'll just pop an external job will always pop out at the end. It doesn't actually run a job or do anything it's just posting status, so it's not really relevant to it's just so you can go back and check the status of the Jenkins job easily, but it doesn't do anything else. I see okay, so you can have multiple.

C

You can have like 10 different ones with different names, so see how this one called. The build name is Jenkins I can name a Jenkins build and then I can name another job. Jenkins deploy and I can I can name a whole bunch of different. You know post build actions and then do all those external jobs will show up at the end. It.

A

Might be good to clarify this with Jenkins status and then it's clear what that is yeah.

C

Like it could be, Jenkins build status right and then I can save that and whenever I run, so you can so whenever I run it again, the next time I run it it all. Those Jenkins build status and then there'll be one for deploy status. Well,.

B

Would it make sense to have that in the.

C

Stage prior to the tests, so the thing with external jobs is you're not able to do that yet so, there's it'll always show at the end, no matter what I see. Okay, that was always confusing me, but that makes sense, Thanks mm-hmm.

C

So this you can see that it failed and then in the latest pipeline and now none of the scans were run.

C

So none of the security scans were kicked off because the build that actually failed so and then you can. What.

A

Triggered the build to fail, it was because it failed in Jenkins, so it showed up here right, correct.

C

Yeah so since I put since this job since I just put it to exit one a failure status, then right, then it just failed. So if any of these things were to actually throw a failure, like I put the run password or something it'll, throw a 1 and then and then the job will fail, got.

A

C

And so now, to show a little bit more of the of the technical side hmm and the yeah and see I yeah mo. You can do these things where you allow failure, which can be a boolean value. So if you say allow failure equals false. For example, in the security scans, I can say a la failure equals false. If any of those tests were to actually fail, then we wouldn't move on to the next stage in the pipeline.

C

So that's the way that you can configure it so so for the case of you want the scans to run first and then do the deploy job. What you would do is you would you would have your deploy job, for example, and you'd run your script but and deploy job would be in the stage after the test job, for example, and what you would do is in the different security scans. You would say a la failure equals false that way. If any security scan fails, you wouldn't move on to the deploy stage.

C

But then, if you did get to deploy stage, then you've run the script that runs the Jenkins calls and.

B

It's a failure in the context of a security scan. Are you just talking about like the disk fail to actually run the scan like you didn't didn't properly, execute or I mean, because the other side of the security scan is that you know you may have vulnerabilities that you don't want to pass but comes back and you you, you might want to fail against. Like hey I found the crit vulnerability, that's considered a failure that this is just just literally talking about the job not operating correctly, though right uh yeah.

C

That's a good point: I I thought that it would show red what I thought it would show red if there was.

B

No, it will show it'll show the past, as it was correct if it were, if it, even if it finds any type of vulnerabilities, it's always going to say the job itself ran correctly. It just gave owner abilities that maybe not acceptable to the clients criteria right. Okay,.

C

I see actually what you mean yeah it, so it would be if the job itself fails, it doesn't throw a failure if it actually finds vulnerabilities. Can you.

A

Make it do, can you make it do it because I mean we're we're capturing that somehow, because if we have a critical or high vulnerability, we can require a an approval. So now we're capturing that information. Somehow can. Is it a matter of just changing the variable that you're interrogating I'll.

C

Post on secure and I ask, there's probably there has to be some way through the API. They can probably be like another script through the API that just checks exactly what was processed.

A

Yeah cuz, that's what that's really. The requirement is based on the the vulnerability findings, but.

D

That's a that's a setting on the pipeline itself. Isn't it I mean that's, that's a setting that you would set up a security group for approval right if there's Creole.

B

D

B

Process things, so it doesn't really. It doesn't really correlate very tightly to the pipeline job statuses. That's I think that that's one of the kind of confusing parts for customers right now, I think all right.

A

Yeah I think we need to figure that out and walk through that yeah.

B

It feels like we need to have some sort of threshold that you know, checks the severity levels of findings of all the jobs and, if it, if it you know, is above whatever their threshold that has been defined as part of the job, then maybe then it fails that based off of that, even if the job past, because they operated correctly right. Yes,.

C

That's what I would imagine fern.

A

You might want to get with Nicole I think she's done some work on that with the white source integration, mm-hmm.

C

A

She might have a quick answer for you: okay,.

C

Yeah I'll give him the call chat after this yeah I.

B

Guess the last thing I'll say about that specifically: is that I think this is because the way we do it inside get lab it's okay, because it still goes through a merge request, and you see those vulnerabilities listed there now. If we're talking about this in context of Jenkins, now all the sudden, you start to make that a little bit confusing, because you're not seeing that, as necessarily as part of a merge request anymore, because you're doing that yeah.

A

That's true we're so if I see that you're running the scans, where do the scan results show up so.

C

Let's go to another.

C

So, let's just show of murder cause actually, but.

A

Do you get a Marg request? You cannot this way, yeah.

C

You can it'll kick off, even if you, if you just make up merge request, so I was just showing a view because I've committed it myself and just ran it on master, but you can do it the same way of a merge request and you can see it. You can see it like this. Okay.

B

Yeah would be if someone was going with this model where they needed to use Jenkins. Would they be using the flow through the merge request, or would they I mean.

C

Not that's right so that that's just depending on the on the use case from most likely they would be using if they're, using on get lot for source code management and they would should be using a merger cross flow and that's where the approvals can be set up. But then, but it'll continue on in the pipeline, because the that's just an approval right for the end result. It's not an approval for blocking at a certain point in the pipeline, yeah.

C

So that'll be one thing: I have to figure out, I mean I'm for sure it can be done by pulling something or accessing the API through like a separate job. The thing is, you know, is that one thing that we want, you know is a little bit too hacky having another job that accesses that get lot of API to look at the vulnerability status or.

C

Or is this something we should push to the dev teams yeah.

B

I feel like the ideal state here with that would be kind of the minimal viable change here would be that we incorporate another security job that runs after these security jobs in a subsequent stage. That then goes and checks the the artifact report of the scans and based off of the variable, that's defining that job of what what is acceptable level of criteria.

B

They could fail that job so that no more of the no more of the process continues on in the pipeline and you know still will show the results from the merge request, but you just don't have to run those extra build steps. So you know maybe a review app or whatever else may inside that merge request pipeline now. It should now be something for the development team that we would.

A

Yeah yeah absolutely yeah.

B

I started kind of sidetrack this a little bit, but that was kind of top of mind. There.

C

All right, you know that makes perfect sense and then one last thing I wanted to show was kind of how you can achieve calling Jenkins. So this is like a little script that I wrote that pretty much what it does is.

C

It compares Jenkins last build number with the newest build number and I added a two-minute timeout, and what this ends up doing is it keeps checking for what the last completed build is and it checks, if it's equal to the build that it's supposed to be on so the newest, build, and then, after that, it'll keep pulling the result and mm-hmm and.

B

C

Result is not equal to success, then we exit with a failure, if it, if it is, we just exit, and the customer would write something like this in order to determine the status of the Jenkins shop to see if they should move forward or kick it or kick it off or exactly what they need. So this was just further start. This is just like if it's running those security scans is dependent on the Jenkins job passing or failing, but if they just want to kick off the job, they can just use the Jenkins API.

C

So just run this command where they will just build the job and start.

C

And Jenkins has api's in all different languages. It's just a simple script that you need to write to trigger it, yeah sure that makes sense mm-hmm. So.

A

Let me ask a question in terms of how how a user would apply this so would I have to set it up for every pipeline or what I set it up like if I wanted to tweak the rules in the gamal that you showed would I have to do that for every pipeline or for every project or could I do it for a group as a template and because what I'm thinking is it's it's easy to do this, but what's the scope upon which its applied, because it might be very difficult if I have to redo it for every every project.

C

Yeah, so if every project has a different job right, then you would have to redo it. You know for every project, because it's its job dependent. So you know you might be kicking off a different job in each pipeline, and in that case then, then you will have to you know you can you can definitely do a template and probably write like a helmet chart or something for for our something in ansible or something to automate and just change the names.

C

But since the names are unique, for example, for the jobs, that would probably be the only change that you need to make so well.

A

The reason I ask is it's great that we can do this at the same time. I worry about. Do we make it so simple that we then people don't have an incentive to leave Jenkins, but I. Think the incentive to leave Jenkins is twofold. One is we can automate some of that if they'll use our CI and the other is in terms of audit ability, you know if you don't really want a developer going in and being able to change the UML file to say pass everything the security people would have a fit about that.

A

So if they set it up in get lab, then it's all auditable and trackable and and so forth. So I'm thinking, that's the carrot to get them to move that.

C

And that you don't have to manage a whole Jenkins deployment, because that they're in to manage you know two different sets of infrastructure and yep.

A

Okay, well, this has been great thank you for doing it, I'm wondering if we should stop recording for a minute, so we can talk about what how we might use this for that RFP that we.

C

Got yeah, let me stop recording real, quick one. Second I, don't even know how to use this.