GitLab Protect Stage, 27 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Security Orchestration Policies as Repository

Description

In this video we are explaining the idea of having Security Orchestration Policies as Repository with YAML files instead of other idea to store them in database.

You can read more about that idea here: https://gitlab.com/groups/gitlab-org/-/epics/4598 and the code that was presented during this video is available here: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52661

A

Hello: everyone, my name is helen percivsky, I'm the senior backend engineer and trade management team, and I would like to go quickly through the idea that we have for test project level, scan execution policies and vc. uh This idea is written down in the whole requirements or written down here in the epic, but you could take a look there. The whole discussion.

A

It was really interesting how we are trying to to ensure that we are we're building a great solution uh so here and I'll talk specifically about the implementation plan of that so more, it will be more back-end discussion uh back and talk about things that we can do and how we can achieve that, um and I will specifically talk about alternative solutions. So we were talking uh about to give you some context: we're talking about two possible solutions.

A

First, to store the whole policies in database, uh so create new models, extend the current database model with those new fields and so on or store them as yaml files in the repository, which gives us a way to quickly deliver things. At the same time, few things that git repositories are giving us like being able to audit being able to see when given change was made or to being able to fork or publicly expose a repository with with those policies and share them across the open source community.

A

You have enormous possibilities with using yaml files, the same as you have with using m-files for kubernetes and any other solution that is currently using yaml files to configure things. So we wanted to to make it the same way in the declarative way. You use yaml files and then you can reuse them for many projects, not only that, so that was the solution that we're thinking of right. Now I prepared a small demo how it could work and what we could achieve with that and so on.

A

I'll show you that in a minute, um if you want to take a look at the code that I'm showing you today, I would like to test it on your own. There is an mvc uh like work in progress, mr with with that code, I'll just go quickly through the code at the end of that call, so you'll you'll see what's going on.

A

um Okay, I believe we can start. uh So, let's start with uh with showing you the general idea right. So we have security, orchestration policies as repository. That's the general idea and idea is simple: you have a repository with yaml files and then you can assign that repository to multiple projects and use the yaml files from that repository to apply policies for given projects. It's a very simple idea- and here I have an example of that repository that we could use. So I have a repository.

A

I have gitlab security policies directory here and I have multiple. I could have multiple files with policies, but I have only one: uh we are focusing on scheduling desk scan right. So here we are. We have a type scan execution policy, name description, every metadata that you could think of if it's enabled or not, because you can store things that you're not yet enabled and will be in the future, the user.

A

uh When you're scheduling a pipeline, you need to know which user is actually doing that it's it you need, you can either create a bot user. That will do that for you or you can use your own name.

A

It is connected to to permissions because you need to have a user that will start the pipeline and you need to make sure that that user is able to do that. You, then you specify the rules, so we have one rule here right now. That is a schedule so schedule something on given time, we're storing it as cron to simplify things and then action what to do so.

A

We would like to schedule a scan desk scan with with those profiles on a given time, so that that very simple policy uh that we want to have: okay, um let's go quickly to death scans and to show you what we have uh here. We have defined uh for the project that we would like to use this policy, for uh we have defined the site profile and scanner profile here with those names that are explicitly mentioned here.

A

Okay, so we have it and it's very simple uh side and very simple scanner profile, and I won't go into details here. um Okay, so, let's take a look at the demo part. uh Let's take a look first, how we could uh save the information uh about security, christian policy project for your project, where you would like to use given policies. So now we were thinking that in the project settings you could go and you could have a drop down and then select.

A

Oh, I would like to select this project as my security description policy project. They need to ensure that you have this uh directory structure, as was mentioned uh in this repository right, so this is it. This is similar to what you could do with kubernetes uh cluster configuration gitlab, where you can specify github managed, apps project and so on. So you can find more examples of of using that now for for many things in gitlab, okay, so and now we can go to the code itself all right here we are in the code.

A

uh So, first of all the configuration I was telling about the ui, but obviously on the on the back end. It works a little bit different, so let's go and grab the pulse configuration as you can see. I already have uh have it it's very simple: it stores like project id for which project you'd like to specify the uh the repository with policies and then the id of the of that project. Where we store all those policies, it's very simple, then we need to somehow have a way.

A

Maybe I'll go quickly here. When you have that configured, the project is assigned to your security policy repository. We need to have a way to create a schedule in database, so we need to ensure that we have entity in database that we'll use to actually do something to actually run a scan or do any other action that's assigned to given schedule uh in our schedule. Our action is to start the scan, so we would like to do that. So whenever we have, we have that connection.

A

So we have a connection between project and the repository with policies.

A

Then we need to create something in database that will help us schedule some tests, and that will tell us, oh: when is the next scan going and then we can have a worker that would do that for us right, okay, so I'm gonna, I'm gonna just create those policies for for ourselves. So we have create orchestration policy worker, and this is exactly what we'll do. It will create those schedules and database, and you can see I can go to rule schedule.

A

And what it was created, it's that so I I have created that for uh for this policy in this project, and we have a crown and then based on that cron we calculated when is the next run so because we have this crown is very specific that it will be on monday on the 1st of february and so on. We would like to update that just to make sure that our other worker that will take a look at those schedules and database will be able to actually schedule the test so I'll.

A

Just update that I'll update the column and now we have next run at uh is set on 10 minutes ago. So just 10 minutes ago, then we have another worker that will actually take it. um Take that amount take the time and see. Oh, it's already past uh current time. That means I need to use it to to schedule a test right. So now I can go and actually use another worker that we have here. Orchestration policy rules, casual worker that will iterate on those schedules and it will actually initiate a scan.

A

I will show you more in details um in a minute. In the code, but you can see it created a pipeline, it created a build. This is the whole configuration of that pipeline and so on, and now we can go and actually see if, in this project we have a new pipeline that is actually doing what we really want to have.

A

So, let's take a look, we see a pipeline that is running, let's see if it was just scheduled, yeah yeah, it was just scheduled, and we see that that's, oh, it was even finished because that was very simple and he's easy test wait a minute uh I believe I haven't refreshed the page with with the pipeline, would like to actually see.

A

So, let's do that quickly.

A

Yeah this is the new pipeline that was started and was already finished, so you can see it was. uh It was created a few seconds ago like 55 seconds ago, and then you have a job desk job that was actually run and you have some results and so on, but we only wanted to focus on things that are actually running right, so we have a job and everything's working as expected here. Okay, this is this is great, uh then I'll go quickly to code and I'll show you what's going on uh in the codes.

A

So if you're not interested in that part, you can you can just turn off uh that video okay. uh So, let's see, if there's anything found like this very simple, uh very simple site profile and scanner profile, but you can see that csp was not set and that was a passive scan.

A

So we were expecting to have something like that: okay, let's quickly go to the code and see here what's going on uh so we have settings so we specify the crown job so every 10 minutes we run this create organization policy worker which will create those schedules and database and then, as like every hour on 19 minutes, we are starting like orchestration, postural schedule worker. So, let's first start with this create post worker. It's very simple: it takes a a look at the purchase model stores per pairs uh so paired when one is project.

A

Id of the project would like to have your policies enabled and the config project configuration id is the id where we store. Now. What is the id of the project with with those policies? Well, so we are iterating those pairs. We take a look at the policies that are uh defined there, uh so you can see we have active policies here. uh Will I be able to see that the security works yeah? I have active policies, it will.

A

This is going through repository and take a look at the files, and then it reads those files uh and and for every single uh active policy it will create a new. uh It will start the process rule service and this processor resellers was actually will create those scheduled rules database. So it will take a look.

A

Oh, it will create a new, a rule uh schedule so with this configuration with this policy path on this time, for this user create for me the schedule and then we can go to to another worker that will actually iterate on those schedules, take a look uh which is currently runnable.

A

So I'm not sure if we'll be able to see that here, runnable um rundle schedules where next run at is less than time now so in the past. So all schedules that will should already run okay. um It will find them in batches and then for each project that is assigned uh to given schedule. It will initiate uh a test. It will initiate uh some kind of job that is defined in the policy. We have role schedule service.

A

So, let's go quickly here and you can see we are just reading actions for a given schedule and for each action we're doing something so for now we have only one action type so that action is scan and we have only dashed uh for now. So we can schedule dust on demand scan and this is what we actually are doing.

A

So we are looking for uh profiles, uh site preference and scanner profiles, we've given names that are defined in the repository and then we are initiating uh a scan using desktop demand, scan, create service and and and that's it and that's, actually it so based on those policies, we were able to actually do some codes in the background and initiate the test scan so yeah. Thank you. If you have more questions, please let me know see ya.

A