GitLab Protect Stage, 12 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Discussion about our approach for MVC for Security Orchestration Policies

Description

MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54067

A

Or okay, all right, hello, everyone we hit record and we're gonna, take a look at things and there will be the whole discussion about the epic for security policy, uh security, registration policy uh and yeah. We are trying to investigate what will be the best approach for us, because we see that we have approaches from compliance team. We have approaches from our team and we've discovered a few things so yeah. Let me let me go quickly through what we have right now.

A

uh So in those previous demos, emails demos, I was recording uh I've prepared something like that. um Like the storing security policies and repositories, I won't go through the code, but it seems like what uh compliance team is proposing right now with their work. It's pretty similar. If you take a look here, uh you can see like it started in the same place and so on, and so they they also have an idea to extend the current uh pipeline service and and to be able to inject something into into that.

A

The difference between us and compliance team is that they have a yaml file with configuration that is stored on some under some project like that and we have to dynamically generate it so that that's the whole difference. But apart from that, we could actually reuse them uh the services they they're using right now.

A

So, for example, here they have like the content, but here they are directly reading those uh those configurations in one service like they're, doing everything in one service that makes a little bit harder to for us to abstract it and have something in common. So I'm not sure still if we're gonna reduce something or not, we have to decide at some point. For now we can.

B

Have some have.

A

Separated some yeah some notes on that.

B

I talked with the compliance team member yesterday and uh so now it's one to one. Okay, so one project has one compliance and it. It will be many uh many too many in the future. So I guess you yes, so it like. So in the future, like you know what they are. Thinking is like. So, for instance, you will have you have different compliance tasks right gdpr. So whatever gbr2 you know so you you will be able to say that, like you know, they call it label. So I my project has those labels.

B

You know if those labels could come from group or they could be assigned to project now it's just assigned to project. You know that it doesn't come from group, so those like labels basically are just yaml files that you you can define whatever you want inside yeah, okay and this this service. They extend this just make sure.

B

Whenever you run pipeline, you run what you define in those yaml files as well just for the clarification, yeah things, yeah yeah, so the same yeah, so it I find very similar because you can put anything in those xiaomi files. You can put branching, you know uh you can even schedule normal pipeline and your compliance with job will run. There's a scheduled job, and you know a lot of similarity.

A

Okay, yeah- that is true. Actually this is this actually great uh that you found found out this one, because, uh exactly like the the main difference between us and the compliance team is that they are preparing the yaml file with uh like the whole conflict by by their own and we'll have to generate it dynamically. But, apart from that, it's very similar.

A

Can you clarify when you say.

B

Like dynamically generally.

A

B

You mean by that so.

A

We have we have our own structure for storing policies like like that right. So yes specify rules and actions and then, in the middle, when we are uh like assigning the project to repository with with policies, then we have to add the logic that will read those policies and then generate the yaml file with ci configuration for it all right, because we need to either like generate the file or extend the current sure.

B

A

So so they're doing is like okay, I have a separate project. I keep the like the compliance, gitlab ci yaml file, and it will be automatically included in every single project. That's assigned to that compliance right. What we are doing is it's a little bit different, but the logic that is underneath is the same because we at the end, we want to extend the current pipeline current ci configuration with something that is defined in different place.

A

B

What what I'm struggling here is, like you know, like what you describe you know. uh I can do this thing right now with their compliance thing, yeah with like right now with the compliance thing by just writing. Yaml, you know. So I can say you know, run uh run dust.

B

You know on those branches with those profiles with just using which just using compliance framework, not even not even writing a single line of code. It's just writing yeah for those branches. So I don't. I don't understand the difference.

A

You know I I believe for now, if we're talking about mbc like, of course, you can.

B

A

What what we want uh by using their component, like the compliance uh feature yeah uh at some point, we have to build the ui on it. You have to simplify those.

B

A

Believe we're doing like at the beginnings are very similar, like, of course you can do that, and the other thing that you could do uh like you can go to. You know instant settings in your admin area and then you can specify required pipeline configuration and then you can specify. Oh, I want to do like fast test on demand, scan and that's it and then you can save it, and then it will run for every single project that you have on your instance of gitlab like this is the other way you can have it like.

A

Gitlab is unique in the way that you're actually able to achieve similar things by multiple, uh like features, and it's good. You know sometimes it's confusing, but I believe here the difference between compliance and the policies uh for us is like compliance is talking like it's a broader perspective. In my opinion,.

B

A

Are focused mainly on security stuff, compliance is more like. Oh, what kind of features would would I want to have enabled in gitlab uh or enforced in gitlab? That allows me to achieve certain like certification or or compliance right? So, yes, this is what it is for us like for now we can. We can go to uh to the nbc like for now. What we need to have is being able to associate the project.

A

Okay, maybe uh do we have a goal here um here: uh enforce require dash scans to run whenever a project pipeline is run according to the configuration and specify that I can be confident that I'm regular, detecting new vulnerability. Yes,.

B

A

Actually, this problem we can solve for compliance easily. Yes,.

B

A

You described yes, the thing is we want to improve the user experience on that, so that that's the whole difference between us sure and compliance team, um so, okay users will be able to associate security policy project uh with similar, how we have cluster management project so being able to assign the repository with policies with given project. That's one thing: the other thing being able to create policy in that project: okay, so that that's what we have right now, like you, specify the policy as yaml file.

A

You specify rules that okay for whenever pipeline is running on these branches. Do dust with given profiles that that's it that's what we we have in our proposal, uh then users will be able to reference an existing scan profile site profile, okay. So this is what we have here.

A

I'm not sure if you want to keep the names or we want to somehow think about uh ids, because I'm not sure if I'll go to uh security on demand scans, if we can get the ids, probably not probably ids are hidden from the user. uh We can think about. Oh, I don't have anything here. uh I think at some point. We can decide if you want to keep the names.

A

I've used the names because we already have it, but maybe we should come up with some idea of keeping the ids or some unique identifier, or something like that. So so, you're not relying on on names that can be changed, but on something that uh yeah.

B

This is a yeah yeah, you know, but.

A

B

Yeah, I I still have struggle for our mvc, like the the difference we are creating, I'm I'm struggling for real, like I cannot see the difference that we trying to achieve in context of this small iteration, oh okay, why what we were gonna do different like what what we gonna do different so that you know I I don't even have the the implementation comes to my mind, because I want to do this different, so I can think of ah okay. Then I can achieve like this, but I I kind of I cannot think.

B

Okay, so because my mind goes to the compliance framework, then yeah, you know it's like you know it's like you know. I tell you the solution of an algorithm and you cannot think anything else, because that's the one you know.

A

B

Like you, you use hashmap, you say, okay and then you can't solve this problem with array, because hashmap is just fits. You know, is what I am having at the moment. There.

A

B

A

Know, that's good that I I I wasn't aware of that, because maybe I was not biased towards any solution. I was thinking how we can solve that without using anything that we currently have in gitlab. So yeah.

B

A

What we could have using this compliance, like compliance, uh git lab ci, ammo file, right.

B

A

B

A

It I'm not going to use it because, I'm pretty sure it's not.

B

A

Not able to use it yet but, and I can specify okay yeah include.

A

B

A

And so okay, maybe let's use it as dust because if you want.

B

A

On on demand, that's what we want to achieve best on yeah.

B

A

And then what else do we want to achieve?

A

Then it becomes tricky because you need to specify variables so.

B

Actually, those.

A

Variables by default, um but.

B

You have to make them.

A

Like sites yeah site, url like this is.

B

A

On demand, site, url and so on, so you have to specify those yes.

B

A

Storing them somewhere on the yes yeah.

B

A

Of giving them like from the database where they can be stored more securely, you know, but.

B

You know you: do you see the potential we can also say like. We can also say if you open that, mr that I send you that the compliance works, uh the mr that wait a minute.

A

Is this no, it's this one right, yeah.

B

So, for instance, let's go let's go like, can you go down a little bit? The the var they inject include. If you go down.

A

B

Go down if you go.

A

Down the description.

B

Yeah, it is a bit more so here right for the pipeline full configuration path, so you can also say if it, if it has security type compliance, do you do your magic? Do your logic in there yeah. You know like uh if you, if you have security compliance, that you want to fetch the dust from database. Do it like you know you want to inject something you want to change. Do it. You know.

A

You know this this solution with compliance would work like the main thing is that the ux uh user experience for that would be bad uh at this level, because you need to configure it on your own, like you cannot click it. I believe the whole goal for it is to be able at some point to be able to drag and drop things or select. Like yes, rules like uh like we have for other policies. I want to do this drop downs and so on.

A

For now I agree that maybe, instead of having like separate structure-free, ammo files and so on, and then uh converting them from from one format to another is tricky and it would be easier to actually specify the compliance with proper desk configuration and so on. I believe, like those first steps, are the same at some point. We're gonna move absolutely in different ways, because we want to have a simple user experience and the rich user experience. At the same time you just you would like to click it and click through it. Yeah.

B

A

The whole thing and they will have like the compliance team, created something that's so powerful, like you can do whatever yes, but at the same time, it.

B

A

You to have some knowledge about the ci thing and the configuration and so on, uh of course, for us you also need to know the structure of our yaml files, but it's a little bit easier and at some point this will be converted to um to ui.

A

So I believe yeah, so I agree we could. We could leave with what they are proposing, but we want to build something that we will reuse in future. That's the goal sure I.

B

Mean, like you know, just to clarify, I don't mind doing this. You know as long as I know. What's the difference uh you know just the problem I have is just. I cannot see now the difference I know like it's like we did like. We did in alerts feature right. We used alerts backhand completely, but it's in different ui. You know yeah, it could be the same. It could live in different ui, different ui experience different view, but at the end it would be the same.

B

You know, because what they have is, from a point of view, super powerful. You know got it and here here right, you don't like so because they use ci directly. They don't need to do this, yaml, conversions and stuff. That is true like so you you will you will like you know, you will pass the branches, you will try to inject those branches in the code. They don't need that because they just use valid ci ammo configuration for branches. So it's like 100 percent, like you know, if there's something.

A

B

That that is going with the whole gitlab. You know so yeah yeah, you know yeah that.

A

Is the risk that we have to take yeah like this is, I believe the whole thing is more. Those are questions, probably for sam. You know, because this is like a product vision that he has. um I yeah.

B

A

You ask me, you know we are both engineers. We are thinking about simple solutions with what we already have. It's like no need to reinvent the wheel to develop that if we can reuse something that's already in gitlab, but I believe for for end user, it would be beneficial for them to to have simplified process of configuring. It now sure.

B

A

Because, like this is also a simplification for the ui, you know it's with iteration.

B

A

About oh, we take the small step and try to see if that works. If if customers will say, oh, I don't like it, it's it's pretty bad user experience. I would have something different, then I'm pretty sure we're not going to to do that. You know, but it's all about being able to quickly get the feedback. That's that's my vision.

B

A

Okay, um all right, so let me go to uh to this mvc once again. uh Okay, I'm just going through things that that are.

B

A

In the proposal, and then we'll we'll see how it works and I'll go.

B

Through the code.

A

And try to elaborate a little bit on that users will be able to reference an existing scan profiles. Okay, that's what I was talking so this is this. Is this one uh being able to select those we're thinking if we should have id or names for now we're going to stay with names? Users will not be able to edit or delete any scan profiles or cyber hosts that are referenced by an active policy, and that's something that I was not.

A

I was not yet thinking about that. Yes, um I wasn't thinking about this because you have those names here and then you need to create some kind of hook and uh whenever you try to delete something like delete, site or scan profile to see. Oh, is there any uh configuration uh like this police security or position policy configuration assigns to given the project where I have those profiles?

A

If yes, then go through policies and see if there's any of those profiles uh mentioned and then prohibit from deleting, like that's my that's the simplest solution, I can think of right now. Obviously I.

B

Think I I don't know we should really like focus on this now. You know. That's probably.

A

B

Probably not this.

A

Is yeah? We should all omit.

B

That uh proposal from there you know proposed, because this is something that we.

A

Have we have a video.

B

A

B

A weak link now it's just a name right.

A

B

A

That is true and actually um an mvc and not nbc, but then the source code. I I've proposed whenever you write something wrong here. uh It will not include the the dust configuration, but it will fail like the job for the dust on demand will fail and say: oh, there is something wrong with the configuration. Take a look that that was my vision just to be able to to give the feedback to the user okay. um So I have a like simple project that is defined here.

A

um I'm gonna just hit run pipeline. So I'll just show you what what we have in the policy this this okay, this project is assigned to to disk uh configuration project so with policy. So I have two policies here: one two to have a test and the second one, second, one for sas um yeah sas for every branch. So that's.

B

A

And I will just go quickly to to the ci gitlab ci project, because it's very simple: it has only one job that is doing only echo as a script and that's it. No nothing nothing's more and then I hit run pipeline um problem run, will start okay, so we have it and we have. We should have uh this job that was defined in gitlab ciaml file. Then we have sas and then we have uh dust, uh that's what we have defined and then it will run.

A

I can go to to the previous uh like how it was run before this is the one.

A

So we have all drops passed and when I look into dashed it was it uploaded the artifacts. Everything is here, but I don't know why this is something I need to investigate or we can investigate together.

A

uh When I go to security, I do not see any vulnerabilities found, even though they are some found for for this job. So probably it's it's about like naming or or some other stuff, and it was not treated as as something less vulnerability. A report I don't know like this is simple things to. We need to take a look, but this is one thing that is missing and I definitely need to look into it.

A

um Okay, and I was telling you that whenever I modify so yes, this is what you demoed before right. I I demoed it, but I uh if I look into the code a little bit too different because I was looking at- let me let me just grab this one.

A

um Okay and the compliance theme some time ago. I don't know when uh I was called like a required rp.

A

uh Let me where do I have the pro okay required? Processor? Okay, so I was telling you previously about the feature that is um that is here: the required pipeline configuration that can be specified and and so on. So actually, yes, uh it's all being like the magic is happening here and and that's in that class. So it takes a look if it is configured for the for.

B

That functionality.

A

For that functionality for that functionality, yeah- and I was.

B

I was looking into that.

A

Code- and I was thinking- oh maybe we could do something similar for security policies and that's what I did with with the processor for um for security policies, but like I, I prepared other, mr, like dmr, that you already have seen, and I I was demoing and the approach that I took. There is similar to the approach that the compliance team has, and this is different approach.

A

So I have two approaches and I wanted to discuss with you or maybe you can do it offline, which would be the better uh for us, because, obviously we need to choose uh if we want to go in similar way uh to this feature or similar to compliance feature both will solve the problem. uh We simply don't know yet how we can solve it.

B

A

The thing is with.

B

Compliance team that we need to be aware of as well, so they the way they do things their over writing mechanism takes presence of everything. Okay, uh so you know they. Actually they do like. So when you say, like you know, if you open a compliance. Yaml example in that, mr, if you open that mr like, if you go like the description of that image,.

B

They have examples: if you go down, maybe they just put okay, they didn't it's in the video uh you.

A

B

It's in the video sorry so so you know you described compliance uh gitlab.

B

You are actually running that ci, you are you are. We are running that yaml. That includes the project. Xiaomi, you don't run projects yaml anymore. They they override all that.

A

Yeah yeah, that is the difference. We will. We that's, certainly something that we'll we'll have to have this. We're gonna extend.

B

A

We're gonna extend it: we're not running going to run like a different yaml file. uh We're gonna run the project yaml file with some extensions to it, so that that's the difference that yeah that's something we need to decide how to do it properly. I I prepared two versions and I'm to be honest, like both are: okay overworking, uh obviously, we'll have to get some mrs checked by ci team, because we don't want to break anything here uh and we are doing.

B

Like so when I point out this, the way that we want to do ci team, that's how the conversation start. The ci team cannot see the difference. How the compliance team is doing. Our hours is doing. Okay, so see you know, so this is how all the conversation started. So I said we want to do this and the ci team says: what's the difference with that, you know and I yeah like you know, I will come back to the same thing though.

B

Like you know like okay, we do we achieve the same functionality with the different algorithms, but like like user-wise. What will be the difference like? What is the difference for the first.

A

Mr, like first iteration only the experience like so yeah, I agree that's something we'll have to tackle. I hope we'll link this video for them and they'll be able to understand like the whole, like like the bigger picture, I know because that's that's the product questions I believe like.

A

What's the difference in terms of what user wants, I don't know to be honest like we could, we could use uh something as you described, but at the same time this is a little bit different in terms of how you use it, because you have to specify the project you have to uh like the separate project and that separate project you have to store yaml files with policies, and then you have the logic that converts that to ci uh configuration a cml file and then we're running it. So that's the difference. Yes,.

B

Like for the for the no not for the recording, like I'm like concerned as as well just to take off the diverging, if you just diverge backhands, you.

A

B

Duplicate a lot of work, because what compliance things teams, long vision is really aligns what we want. They have two-step approvals. They want to improve a ui, they want, like uh you know they, don't they don't want also just yaml. They want compliance jobs to be configured by ui as well. You know. So, if we, you know, the first step is kind of important.

B

If we diverge there too much, we will have to implement what they are doing like you know, somehow, all the time yeah like they have two-step approval, for instance, they have like, because you have groups you have like you know, attend all this that you can configure.

B

uh You know you can give these permissions to people which can configure what so they.

A

B

Cover those inheritance of groups they already covered those. You know it's just I'm just telling for the sake of this recording, whoever.

A

Will watch that dude? I believe matt from compliance team is coming back from paternity leave today and yes, he'll talk with sam.

B

A

That because that's definitely like they have to agree on the product level, uh what we are going to reuse and and how the process is because if it's the same then obviously we have to. We have to go and and like join forces and build it together. uh For.

B

A

You know it's always the code that we can remove at some point like we, uh we don't.

B

A

For them uh to develop something, they don't want to wait for us until we develop something. I believe, especially, that our plan is at some points to replace the yaml repository stuff with ui and I believe, for now we are trying to deliver something that uh we will be able to ship and someone will someone will be able to run it and see if they like it or not, because that's the product that yes.

B

A

That sam has to take anyway, if it's worth investing our time and investing like his time and talking with customers and so on so yeah. This is the valid questions you know we engineers are. uh We have a tendency to simplify things and do uh or do things with with the tools that we already have, but I believe for the.

B

A

People it's a little bit different, so yeah I'll go quickly to the code here uh for further recording, so maybe you'll be able to to better understand what's going on here and how I achieved that I will go maybe file by file so.

B

Another just another note excuse me: I want to interrupt you just the sake of recording right uh like if we, if we do like you, know mvc again, and uh you need to make sure that you're backwards compatible as well. Oh.

A

Yeah, you know yeah, that's true.

B

That that's also a cost yeah. That's always a yeah yeah. Just to mention that right, sorry.

A

B

A

No worries I'll collapse, things that are not really important for the conversation like, for example, you're sending the finder with being able to find by names like this is just to be able to find scanners uh like profiles and scanners by name like this is the same. uh We have okay, I've added the license for it, because you need to be on ultimate to have it. I believe I believe, you've asked this question to samuel.

B

Yesterday, yes,.

A

uh And uh another quick note: I think you need that the feature flag as well. I cannot, I believe, I've added it here uh in pokemon I'll, just that's awesome, I'll I'll leave it as it is right now I'll go to it in a second another service and another service. Oh okay, uh I looked into I looked into that services and, for example, you have run dust scan service or what this service was doing previously. It was building the template and then it was like running it.

A

I I just extracted the logic, that's responsible for preparing the uh the configuration for running on demands test scans because I needed I need to reuse it, and I I wanted to to just extract that. So I extracted that to that scan. Ci configuration service yeah, so so that that's one thing, uh and here oh yeah, and that's the logic that that was extracted from this run.scan service, I'm going to collapse it as well not needed for that conversation. This is needed uh to create okay test on demand scans, create service.

A

That's another thing that I uh extracted from from this class and moved the whole logic to params, create service. So this is the service that actually it's taking the uh the test scanner profile and that's a pro like test site profile and they are they're converting that to uh two environment variables that are used in the scan, so that that's also something that I had to extract, because that was all happening in one service, and I wanted that to to.

A

I wanted to reuse that code, not only copy paste it, so I'm I'm closing it as well, and this is the service I created to extract that logic, not important for our conversation. This is the service I've developed uh to prepare the configuration that that's, okay, okay, I have a feature flag here. Of course we don't have like the url on the of the mr, because we don't have it yet reload issue. I believe that's the one that uh you already prepared uh yesterday.

B

A

Unfortunately, the test fails, if you just put the future flag, anyways yeah side.

B

A

It's true yeah, okay, and then we have this file and the processor and the config.

A

Okay, oh here uh I had to modify a little bit the dust on demand scan template. I'm not sure why we have build test and deploy stages when we're running on demand scan because on demand scan, do not have those those stages. Yeah.

B

A

Only that probably it's a mistake, because that was copy paste made uh like from that kit laptop file. I removed it because um yeah, I'm I'm using only one stage actually uh from for it, but anyway like this is this is not really important for this compressor. Maybe we should talk with last uh team to understand why they did that. uh Actually, I don't believe this change is needed anymore, because I changed a little bit the way we could.

A

We uh generate the yaml file with the desk configuration so I'll get to it later on to see if it's needed. Okay, so.

B

Here, let's just let me just summarize the overall uh achievement that you are trying to make to un confirm my understanding yeah. So you you are just getting. uh You are just putting your job at the end of the pipeline by just coding it yes right, yes, by extending so.

A

You don't know.

B

You don't know whether you have like a running dust job. Do. How do you check that? I I'm not checking.

A

That uh I don't care okay, it already has because we are enforcing it and the difference between having dust configured as a step in the pipeline and having on demand scan is that you can have different uh like profiles, you can you want to check different things and so on. So that's the difference.

B

A

You have to deploy it to review app and then dust will run on reviewed app uh here with on-demand scan. You have to have the url for the app you would like to test okay. So we don't. We don't.

B

Check dust jobs: no, how do we check the branches? That's given.

A

Our dash, we do we check it, we do I'll show you in a minute. uh I will just collapse last things that are not important for the for the whole conversation and we'll go to it uh just quickly. I had to extend the gitlab ci config uh with ref ref, a simple I'm specifying it from here. I need to have an information on what branch are we currently running the pipeline and it was not passed to the uh to the class that has generated the configuration, so I had to add it.

A

So I added ref and rev was added here.

B

A

B

Okay, this is used general class. We are extending right.

A

Yes, yeah yeah um we're not removing anything, we're extending it, so hopefully it will not destroy anything okay and then we have, and then I can start talking about changes I made in into the code to to to make the magic happen right. So we have this config ee file and previously oh shoot, will it?

A

Okay all collapsed. Yeah gitlab is working fine! Okay. Here, if I go to config.

A

I don't I wait a minute config.

A

Okay, let me just go back to uh to master, so uh I, as I told you previously, I've reused the the logic they had for requiring, on instance, level requiring the the templates to be included into into ci drop.

A

So I'm I'm extending the config e file with two with one additional class, so here how it was before we had build config that is actually like the build required includes and add soup like and we're using what we have defined in config, um and here I'm I'm just doing the same, I'm just adding additional step because we're still having this and we need to have additional step to to extend the config with things from security, orchestration policies, that's it so I have the processor and what it takes. It takes the config.

A

The config is hash. With the configuration of the of the ci, uh then the project we need to have a project to properly find the uh desks profiles and ref to know. What's on what branch are we currently running the pipeline? So we know what jobs to include or not and not? Okay. So that's for the config and then you have for the processor, and this is the the place where the magic happens. So, as I said, I have a conflict project and ref and a feature is not available.

A

uh It's on for given project we're not going to extend it and if it's not enabled we're not going to extend it enabled uh it's defined here if the feature is enabled so the feature flag, so we're actually checking if the future flag is enabled or not for a given project. That's it nothing.

B

Else, no okay. I mean uh okay right, okay,.

A

B

Talk about this details in long time now.

A

B

uh Yeah, so what I want to ask like, why don't we just inject our code to what compliance has.

A

Okay, let me let's take a look after, let's.

B

Discuss that, instead of like you know, going through.

A

B

You know so why don't we like you know like uh that's, so one methodology is yours and for me like, why don't we say like forces here there right another line? If you have security compliance uh configuration, you know security compliance configuration model like it gets the content of that yaml. You know at the content of and convert it into a prop, like you know, valid sci job. You can convert into a proper ci job with variables and everything you know, even though user experience is different. There, like you, said profile, convert everything.

B

Is the valid yaml file and add as a job.

A

So, uh okay, let me let me take a look here.

A

Let's see how it could look like, because what they currently have it's a little bit. Okay,.

A

So, okay, so they have right now to include something from the compliance yaml file. They have this path and yes project and they, yes, they simply are um replacing the configuration of the yaml file to have to have only this include project and file right. So for us, what would be a little bit different is okay, if, if it's compliance I'll just uh if com compliance, yaml file, then do this right and the other solution would be to else or.

A

Else, if compliant uh security, orchestration.

A

Policy exam like that right um and then you have and and then we need to do something here and you're, saying that what we could do- and that is that is a valid point which we could do is like generate ci uh yam files.

B

Yeah for policies.

A

Yes, yes, and some somehow included here or maybe uh okay, let's like the config duration, look like that and then configuration.

A

um Include and then we need to include what what they have so like project and like I will do the same path. Project and file will be the same okay. Actually it will be. It will be similar to this guy, but the only difference would be to uh probably.

A

B

Yeah merge because we want, we don't want to break their stuff right. So.

A

B

No, we we need to like in align with yeah.

A

We need to take a look, maybe that's something that you can do, because I will be biased towards my solution already and you already are biased towards compliance. So what you could do is to investigate that. Take their code take the code that we we have for for the policies and see what how we can do that. I believe, if you go to the code here, what will be interesting for you to do that job?

A

Actually, let me go to this, mr um yeah, actually here, if you will use that code like gitlab ci, config, secret specification pulses processor and instead of providing the like proper configuration of the like current configuration of the ci uh ciaml file for for a given project, you specify empty hash and, if you'll specify empty hash here, what this class is going to generate for you is the yaml file with all jobs configured according to this security acquisition policy configuration and then you can, you can take that yaml file and inject it into into this place and see how it goes.

A

That's the other solution that we can take like combine those two together, because we still need to be able to generate the the yaml file, but the only difference is where to put the the yaml file itself and how to do it. So that's the other solution.

B

Okay, I need to get myself.

A

Familiar with this yeah.

B

A

Definitely like because I'll just go quickly and tell you what what certain things are are doing here, so this processor is actually taking the like by default, it's taking the configuration current configuration for the project and then extending it with uh with templates and that's it, and those templates are here like it's defined like that.

A

Merge, security, orchestration policy, template and actually it's it's pretty simple- uh we're just reading those templates uh by using uh template, fender and reading the content of it, and then loading gamble files from it now, one one thing to mention, and and that's not needed for uh for embassy, but to be able to to think about the whole solution as it is. I I was not thinking only about the dust, because uh dust is very specific to gitlab like sas.

A

We need to think about like other uh scans as well, so in this mvc, I'm supporting all of them all security jobs, because that's simple, uh those jobs are specified here. That's what you can have like. Fuzzings container scanning coverage license test and so on, and this is the template that's associated with with given uh with given type, and these types uh are actually those types from from that one.

A

So I have action scan sas and then we take that sas and and convert it to a template name and then, in this model, we're reading that template name, template vendor will help us here and then we're using yaml file to load it and that's it and then uh here we are we're waiting for every scan that is defined. So, okay, I'm using like scan templates method here, if you'll take a look at the scan templates, is go to active policy, select those that are applicable for the branch you're asking about that.

A

So here's the logic for applique applicable for a branch. It will just take a look if the branch is matching the wildcard.

A

That's it take the actions uh and from actions get scans and plaque is a method on hash that is taking the value of of hash from array of hashes like I know, that's tricky, it's actually working the same as black on active record stuff, but it's doing it on hash and then going and taking a list of templates that we should use and that's it um and then we have it then we're reading uh the hash we're generating the uh we're parsing the yaml file.

A

uh Then we are deep merging all of them, so we're reducing them. So, instead of like having like multiple templates, we're combining those templates together um and we're using config extendable here uh to properly um how to call it like okay I'll go to sas quickly- uh and I will I'll show you that, maybe that's so every single sas job has this extends, and what it does is something that it's it's addition to.

A

What yaml is offering, uh so it will just take that go to the zest analyzer, which is defined above and we'll include the the things from that into into the the job, and then it has another like extent which will take this one and so on so on, so that that's the the way it inherits the configuration by jobs right um right and and this extendable will simplify the whole yaml. It will remove all extents and it will include the things it.

A

It will actually extend the yaml file right so that that's how it is- and this is this- is it so as a result of that method, you will have a yaml file. You'll have a hash with the configuration for for running certain selected scans, okay and for on-demand scan template. It's a little bit different because we have a project like a class that is doing that for us. So I'll go quickly here. I'll just tell you.

B

Okay, I think we should just do a specific solution album. I think it's too general for the first iteration.

A

Probably no, I I always try to think big that that's the difference.

B

And we discover every day and you think and making this general solution with a lot of moving parts. uh You know, like you know we're just talking about. We should think at international and we're talking about, like you know, making a general design like if you're gonna think about future. We shouldn't be talking about this at all. We should be talking about compliance teams, think you know yeah yeah,.

A

B

A

That's why that's.

B

A

Discussing it just to be.

B

A

To have something in the middle, you know, because you know I I'm thinking about that you're thinking about the compliance. We need to combine those ideas to be able to develop something in next. Yes,.

B

A

So yes up to us how we're gonna do that like okay, I was talking about things that are not important for now, uh but here is the class that it's important for now on demand, scan pipeline consideration service, and this is the service that will take the project, uh take the list of actions defined in policies and it will prepare the configuration.

A

So it's it's very simple: it will load the yaml file with the configuration and it will extend it with all variables uh from uh from from profiles and and that's it and if there's any error instead of having the proper configuration of desktop, it will create something like that. It will create a script, a job that will say. Oh there was an error during the execution, this error message and it will exit with with false.

A

For now, I'm allowing failure to be true so so, like the error could happen, but it will not block the the pipeline, but that's something that that can be changed at some point. Yeah I'll I'll, create a mr for it and you'll take a look. If you like it or not, I will not create any tests for it, because that makes no sense to invest.

B

Yeah what you can do is like also, I would suggest, like not thinking to support other security scans, because we don't.

A

B

Direction we're gonna take like this is my opinion. You know just do it the dust and create an mr, you know and just shoot it to ci team, because uh you know we are changing like it's a you know, a general uh interface as well. That's used, you know and, uh like you know, versus compliance and didn't have to do any of this stuff. You know that's true. They.

A

B

A

B

Let's create a simple simple with one, because if you put other security scans and stuff with people who confuse, I was confused there. You know like.

A

B

Don't we don't try to communicate overall idea that we are like? Oh, you have our overall solution. Security scans are doing great. We we want to communicate the overall strategy that we want to take so with other teams to make sure we don't uh mess the most important part of gitlab. So you know.

A

B

Minimum bear dust solution. You know I'll look into that, mr I'll try to combine with others like once. You have this base. Mr, what do you think I'll try to combine with the other, but you can shoot this, mr to ci team, say guys this is you know? How is this thing looking like to you like? What watch out.

A

That's a great idea.

B

A

You know that is a great thing that you mentioned, that definitely uh and.

B

I can I can look into both like mrs, I can look. You know if I find something uh you know I just that you know. If I find some pitfalls or something yeah, I am personally I'm supportive of any idea that will move us. You know forward yeah. That is great forward. You.

A

Know but I thought.

B

Yeah, you know, that's it like you know, I don't I'm not like. Oh it has to be this or you know it has to be that you know whatever the quickest and you know the most. You know make our team move.

A

Forward is the best that is great. That is great to mention that, because you know I yeah, I have a tendency to think about the whole solution like the final solution, but thinking how to build it in steps. So I I was, I was not needed actually to think about other scanners, but yeah. I proved to myself that we've given solution, we were able to like extend it in future.

A

Maybe maybe it's also happening. Yes, it happened because I truly believe that supporting many scanners at the very early stage uh will help us get better feedbacks, because not every project needs to have tests. You have projects that are simple like libraries that they they are not really deployable to any website that you can run dust on. That's why?

A

Oh, maybe, if we allow users to select sas instead or like other types of scanners, then it will help us get better uh feedback or have feedback at all, but that's that's definitely something we need to discuss with with sam as a step to our towards our iteration great. Thank you so.

B

Like you know,.

A

And, like all in that.

B

Like you know, that's thinking right. So, if you make one mistake, you will make five mistakes with all analyzers. You know, that's true uh you if you like, you know, go into that step and you will have to support five different analyzers in backwards compatible way. You know. So you know we are doing like a happy path. Kind of testing we are doing like you know. Looking like okay, it's working. It's it's working, our gdk. We did like you know. We don't know extensivity like how extensive edge cases we have.

B

I mean you know that has a pros and cons. That's.

A

B

A

That's my: I appreciate your feedback. This is this is great comment, but.

B

Yeah, that's why yeah? What do you think is it like? You know it's actually great.

A

I'm easy it will simplify the workflow. It will simplify the review process uh yeah. My belief. What we've proved here is that we're able to actually achieve what we want with other scanners. With this approach, which is good, you know we we have invested some time into it.

A

However, I'm gonna just create a separate branch and I'll just keep the copy of it and we're gonna reuse it at some point, so like nothing is lost here, I'll I'll, just jump on, create the the project uh create, mr I'll, send it to you for the initial review and then we'll decide what to do next, because and uh what do you think about writing tests for it?

A

I don't believe they'll be needed right now, uh because so we want to just show the general idea, but maybe tests will help us and others understand it better. So what I'll? Okay, what I'm thinking.

B

Is so, let's imagine you made a minimum, mr now I don't mind writing tests to it. Obviously, and uh we we show it to c uh ci team. Okay, you show it to ci team and uh I can look into writing tests to that. Also, I can look into how we can put that thing in you know we're working with the compliance framework as well yeah both that.

A

Is great we we do like an options: evaluation at that yeah yeah- that is great, so maybe maybe.

B

We can do it like.

A

That I I'll prepare them.

B

A

You'll take care of looking into how to combine it with compliance framework uh and then we'll see which option would be better and we can I'll. I can write some tests for the for the things that will be generic anyway. That will be working for both solutions. For example like generating the dust configuration will be uh working in both either the compliance framework or for us, because that that's the same thing. What do you think about that?.

B

Yeah yeah, an important thing is like to get the ci teams uh feedback as quick as possible. Definitely yeah.

A

B

That was, there were the ones that pointed out that thing to us instantaneously. You know that's great, so they know they know. They know their.

A

Quote: cool okay.

B

A

Thank you for that feedback and thank you.

B

A

For making sure we we're not redoing ourselves and we're doing work for others, so this is great great discussion. um Okay, all right.

B

Thank you for your writing. Nbc's man. Thank you, for uh you know, um making the first step you know. First step is the most difficult one right.

A

B

Yeah, you know we can always build uh top of on something, but you you are the.

A

One who made this something so thank you. Thank you. Thank you again for all your feedback and your help. That is tremendous work. Thank you. Okay. I have a nice one. Okay, why? You close? Okay,.