GitLab Release Group, 17 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Solution Validation: Prevent Maintainers from adding/editing environments to projects

Description

Release group had a sync meeting for the solution validation on the issue https://gitlab.com/gitlab-org/gitlab/-/issues/262011.

This video contains:
- Problem statement
- 7 LEVELS of securing CD pipelines presentation
- PoC for the potential solution

A

A

Thank you for everyone uh joining this call. This call is for uh talking about this uh issue. uh Products uh provide a configuration option to prevent maintenance from adding editing, invertebrancy projects, and we already talked uh discussed about the solution and somewhere in this thread. Yeah in this thread um that.

A

A

Yeah basically, this I propose to dissolution and then um the victor confirmed it with customer and then it probably works for them. So uh uh I already created a poc, so I'm gonna demonstrate it later, but just you know making making sure that if the solution actually works for them, um if it's truly, you know correctly implemented for uh for them. So I just set up this code.

A

uh Yes, now um already, do you wanna quickly wrap up the purpose of this issue.

B

Yeah sure so um the problem we're trying to solve is that for this specific customer. But this is just an example because a lot of customers work like this. They have a segregation of duties and the people that are developing are not the people that are touching production, and so we we already support protected environments which don't allow people to deploy to production unless they're specifically stated in under the protected environment rule.

B

But this didn't solve their entire problem because they also don't want developers maintainers, creating environments. Okay and when we're saying created environments, we're talking specifically on protected environments and not review apps. For example, uh any developer can create a development environment, but they don't want customers to access this newly created environment uh by maintainers.

B

So what they asked us to do was kind of find a solution to block maintainers from the ability of creating these protected environments and there's two different locations inside our product. Where you can create environments, one is through the ui and the environments page where you just create in a new environment, and then you can go ahead and set it as protected or through the ciaml file through a job you can create that environment.

B

So we kind of need to find a solution to block both of those scenarios for people that are not specifically stated as permitted to create environments.

A

Okay, cool uh so.

A

um Now, uh actually, I was thinking that um the solution could be different per um customer, see the cd workflow, the current workflow, but, uh um like uh you know, there are many uh features, are involved with the um the securing city pipelines, uh not only protected environments but also uh protected branches, protected variables, etc. So, actually I was wondering um if the locking down the protected environment rules is, is sufficient for them, um like a.

A

The basically protected environment rules can be created after environment is protected, so, for example, if uh let me let me just demonstrate.

A

Let's say: here's a test projects and then.

A

Like uh what do you already just referred? Is this button right new environment button should shouldn't be this button shouldn't be accessed by maintenance by default.

A

Right and then like, uh for example,.

A

Like they create a staging environment, and then uh you know it's really uh possible, but protected environment rules can be defined after this yeah.

B

You are right, uh shenya. um Actually I discussed this yesterday. um So first of all, I.

A

B

I tried to write on the google doc kind of some of you what you said, and I can't edit. So if you could just get permission uh on the google doc.

C

If I could get that permission as well, that'd be great.

A

uh Okay, where can I I forgot a way? Can I change it?.

B

um On the bottom, it says a change and then just change that all right um now going back to your question: junior um you're, right and ian- and I discussed this on our sink yesterday- and we have some assumptions that we want to validate with the customer, but we assume that protected environments are going to have some kind of prod in the name. So like we were thinking of adding, maybe a regex rule here saying: if anything has prod inside it then block the creation of it. Something like that.

B

Maybe someone has a different convention, so they can change the regex.

B

But that was an assumption that we have- and um I asked yesterday uh from victor to validate what the customary fire assumption is correct, because you're absolutely right, a developer can create an environment and then later on, particularly um so, we had two options that we thought about yesterday and it's in one of those notes. Let me look for it in the issue is one based on regex and the other one was um that creation of environments.

B

Would you would get a notification um when someone creates a uh an environment and then one of two things either? You just get a notification and then you can decide to protect it later on, if you're, the sre or the production responsible person, or we would insert this new workflow, which I highly not recommend. But maybe that's what the customer needs where you need approval in order to create so a developer, creates an environment and then it needs to go through an approval cycle for it to actually be created.

B

But I think that's a really bad practice, but we're.

A

B

For the customer to uh to confirm.

A

So the as a complexity comes from that everyone can change kilo of cmo. um Like uh you know, you love cmo. They say that, um like this type of environment, name is specified in, in this case production this production, this deployment job, can be changed by anyone.

A

The problem uh that for this solution we provide, we provide a bunch of uh security. Related features like, for example, victor pointed out that how about changing into product production or prod, whatever things that can be avoid, you know, can even skip the validation of regex like something yeah. So this cannot be prevented if they are maintenance, but uh the only solution for this case is that setting variables.

A

So here, environment, scope, setting environment is cool for variables, so in this case only production secret is injected into production environment so that, um even if they change environment name, this deployment job will differently fail because they can access it. uh The deep this diplomacy of cannot access to. You know that the deployment target like aws or gcs, whatever things uh they cannot access it. So that's kind of you know um assumption that we have.

A

uh If there have typical deployment job the kind of this secret workflow, then we can, you know, secure their cd workflow.

A

B

I think that's fine and I think that they this as well um on the issue.

B

And then we talked about having a follow-up issue that will kind of explain to the user why their pipeline failed, which would say: hey you don't have permissions, um go to ask someone that does in order to use production right.

A

uh It depends on the case yeah, but but it basically makes sense.

B

So can you um walk me through again the secret, so I am on my ammo and I created a something environment.

A

So sorry yeah um like uh actually I was thinking that, maybe you guys don't know really about the cd related future. It's really protected. uh It's called protected features for the protected branches, protected tags, protected environments, so actually prepared um click presentation for the features um like it. This is there really um I'm gonna, just I'm gonna, try to finish within 10 minutes um uh yeah.

A

So uh by just seeing this presentation, you have an idea what features are related to cd workflow okay. So uh this is a seven levels of securing cd pipelines, continuous deployment pipelines.

A

You might already know that say pipeline how ci pipelines works, continuous integration, so like testing jobs um like our space jobs, would work um but city pipelines, city deployment jobs are a bit more complicated than these testing jobs. Because it's um you know it has secrets, uh it must be secured by developer developers or some. You know the lower level uh like uh to sell other people. So uh there are uh here's the civil levels of tips now level. One set the proper role to project member uh now in glove.

A

There are a few roles: guest reporter developer, maintenance and owner and then the permission. The giving permission increase from left to right so guest has a list permission and the owner has a full control on the project. Now, what's important here is that maintainer role has permission to access to project settings all project settings, including production secrets. So um you know in gilaf you can access this settings page maintainer. All maintainers have access to here.

A

So um what's important here is that if the person should not have access to production, then the person should be developer role of below to develop guest reporter or developer, and if the person should have have access to production, for example, operators, sre release managers and the person should be maintainer or so. This is a level one at a level.

A

2 protect the deployment branch uh so, for example, in a typical c continuous deployment, workflow matchbreaker is merging merged to master branch, and the master branch creates a new deployment pipeline and deployed to aws or whatever place and then predict the deployment branch means only allowed. People can merge or push to the branch. So, in this case, only maintainer role, people can merge or push, but more importantly, actually, this is a kind of secret feature of this thing is that only a lot of people can run a pipeline on the branch.

A

So by setting this protected role, developer cannot run pipeline on master branch, okay, so differently. You have to protect the diplomat branch at the next level. Three is protect the environment uh production secret.

A

Now, uh like we discussed um all deployment, jobs have a secret um to you, for example, secret to aws security, ccs secret to other whatever place, and then um okay.

B

If I'm doing an on-prem deployment will I still have a secret.

A

uh On-Premise gitlab.

B

No on-premise, wherever I'm deploying to to a server to.

A

uh So it really depends on the external service how external service authorize authenticates a request: the by default all xr deployments, the actual deployment resource servers have um some kind of horse mechanism, so there is definitely like secrets has to be. uh You know the secrets has to be prepared in the deployment job, and then that means that a gitlab users have to define the secrets, so they use environment variables right so user. Here's, the environment variables uh the ui, though um so the point of the laboratory is that protect.

A

This variable here is a production secret and then mark protect it. This protected means that this variable is injected only on protected branch, which just we just covered at level. Two. Okay, this variable is only available in master branch so that, for example, developer trying to steal the secret. They cannot do that because it's protected, they don't have access to uh master branch. They don't have. They can run pipeline master branch so that um this is a level three now level. Four protect the environment.

A

The this is our main point in this issue, though only a lot of people can execute the diplomat job. So in the screenshot production environment is protected that can be deployed by john only okay, um so this is a one step further from the this level. One set proper role project uh you know like maintainer is maintainer can deploy, developer cannot deploy, but this is one step further maintenance can deploy, and only the john maybe join.

A

Is there also one of the maintainers, and probably there are a bunch of maintenance, the other maintenance, but only john can deploy so they can set it. But now, please consider that this is also project level, setting okay, that we just covered that main general has a permission to access to project setting. That means that, even though the the production environment can be deployed by john, the other maintainers can change this setting.

A

Okay, so this rule can work on the trust that all the maintainers are reliable or trustworthy. Okay, now level, five said environment scope to the secrets. Now um we just have either protected um the protected variables, but also on top of that there's an environment scope. The secret will be injected into developmental jobs to the target environment.

A

Only so in this case, this secret is injected into production environment only so we just like made their uh assumption that what if they change production uh change, the you know, environment, name, to production whatever or production one production tree, or something like that, then they're gonna the deployment job going to rules this production secret. So that's uh you know the uh how we, how users secure their deployment, jobs, okay and the level six is protects. The runner- and this is a bit different aspect um from level 1 to level 5.

A

We covered guillaume core, the the server side logic, and now we are covering the executor side. uh Now, for example, um deployment of pipelines created a deployment job is the status transitions to pending? Now then, gitlab runners coming the job agency day is coming to gitlab and then hey? Is there any pending jobs? And if there is they fetch it and then actually start it executing a job? But the question is that: can you actually trust the runner? The which is technically external problem could be?

A

You know the martial's program that developed by marcia's user, um so that in this case, you are make sure that only trusted runners only trusted the job executors can run the deployment jobs and then, of course, this runner should be protected, that you know you can see that in the description of that uh protected checkbox is checked. This means that the runner can execute the job on protected branch only which we covered level two or something yeah, okay, so protected runner at the level.

A

Seven like um this is completely new level that you know like uh we, uh for example, there are a bunch of maintenance in the project. The one of them is john, who has a his release? Manager who has access to production and the dslr maintenance, don't have a shouldn't, have access to production, and then, in the case that you cannot trust the other maintenance, um you know, might change the protected environment rule setting.

A

You should create a completely new deployment project somewhere else and then assign only john to the project so that project settings sorry.

B

That's similar to what we're doing with chat ups today.

A

It's kind of similar yeah uh gitlab infrastructure also does this uh do this practice and also, I recently created a new issue for a kind of similar, proper cell for this.

A

Yes, so um I actually explicitly mentioned in this presentation, because uh you know in the issue the in this issue. The big story mentioned that um technically, guillermo yama should be locked to two right by this practice uh like completely separate by separating deployment project. Actually they can secure uh gilab si ayamo as well, okay, uh because only in this case, for example, john- can change the c yeah yama, so he's at level 7 and then that's all like maintain.

A

Our law means that the person has access to production, the protects the deployment branch, protect, the production secret, protect the environment and they said environment, scope to the production secrets and the protect runner and a separate deployment project if necessary um thanks. This is a presentation I prepared.

A

B

You have any questions yeah. I hope this.

A

Helps you this helps you to understand that you know completely.

B

It helps, I think, it's very expensive, and I think this is actually what they're looking for, but I do want to go back to the original question of creation. Okay,.

A

B

um So we talked here about secrets and production and how to protect branches, and that's great- and I understand that if someone changes the name of the production environment since the secret variable is linked to that specific name, the job will fail, since it can't run, which is great.

B

But what happens if I am a maintainer and I'm not john, but I go ahead and I create a production environment either through the environment dashboard or through the yaml.

B

Do I have access to the environment, to new environment variables that I can set and a new secret token for the new created environment.

A

uh So, like a, I maintain, I created a new environment and then set up a new secret variable yeah.

A

If it's completely new, then it's possible.

A

Yeah, so basically the maintenance has all access to all of the you know, project settings so like the point, is that what we should lock down um so like in this issue? We are talking about what, if we lock the uh we allow owner to lock protected environment rules, so that um uh the even if the arsenal maintainer created a new environment, then the environment cannot have a new secret.

A

I don't know it can't have a secret, but.

A

They don't have access to their only important projects. Only they don't have access to important.

B

B

Okay, ian you're quiet. Do you have any questions.

C

I am taking it all in. um I wanted to call out. Thank you so much. That was really. There was a lot there in the presentation. I appreciate the back story. um Do you think, is there a way that we could do something along the lines of an owner level? Admin has the ability to.

C

I'm almost feeling like enable a mode like a secure environments mode where some of these things are locked down. So it's only owners and then owners can add additional people.

A

uh Yeah, so um that's uh uh the first purple style uh to this issue that uh we introduced this log. um Here's a poc part that I created for the issue that owner role can see this setting the protected environment environment locks and if they, the owner, can check their log product data environment rules then save changes.

A

Then this setting is completely locked. Even maintainers can visit to the setting page and then see their current settings, but they can unprotected were adding new protected environments.

A

A

This is a the solution to secure, for example, uh the production environment, uh which users don't want, uh the other maintenance to change the secure the production environment uh from the other maintenance.

B

C

Ask just a real quick question: um are protected environments always production environments? Are they synonymous or are they two different sets.

A

You can, you can add any environments and to be protected uh like we just created a staging environment right and then, um for example, if we allow only tom to deploy to staging environment, then uh you can add it to there.

A

Actually, it's locked. So I cannot.

A

So staging only tom can deploy to it and then the new protected environment is added here.

C

A

Then, to change.

C

That protected environment list you would have to be an owner or assigned. Is that what we're implementing sorry I'm trying to work through it all.

A

So sorry come again, please.

C

um So, with the proof of concept that you've implemented in order to unprotect an environment and be able to do anything with it, you have to be an owner or one of the people who can deploy to it. Is that the that's? How we kind of lock it down is when something becomes protected?

C

um Is there anything stopping the maintainers from creating an unprotected environment and then moving it to production, or does it have to be protected to go through that.

A

um So in the issue we're talking about that, um unprotected environments should still be able to be credited created by the developers and maintainers. For example, this allows users to use review apps like seeing the environment in the margin request, so the customers they still want to use that so uh yeah.

C

Perfect that answer my question. Thank you.

B

The um john that you showed before with the checkbox to protect the to lock the protected environments was that on the chat ups project, like the deployment project.

A

B

The previous page that you showed before, where john could uh the poc that you he could uh check.

B

Who has access to this checkbox.

A

B

Owner so so that means that if I'm a maintainer, that's not john- I won't see this or is this the chat, ops project, the deployment project.

A

So sorry, I don't understand, I don't follow the white chat. Ops project is coming here.

B

Sorry, let me just uh go through the flow again okay, so I am john I'm the owner of a project and I want to lock the protected environment rules, and so I go to the settings, and I see this lock protected environment rules and I check that all off. If I am a maintainer that is not john.

B

Do I have access to this check box.

A

No, you are maintainer right, then.

A

So like, if you are a maintenance and then um like you are a new, for example, you're, a new release manager and you need access to production environment because you're taking responsibility of the previous release manager. Then you ask the current owner project owner that hey, please unlock, protect the environment rule at first and then add me to the me here right and then order uh like unprotect it and then chancer, adding the new release manager to here and then lock. It again save changes and then okay, that's a that's a yeah workflow. We expect.

C

I really like that, but I do have one comment if I am the owner of the project and I have clicked uh protected environment locks, so that is now enabled, would it be possible for me the owner to then also just edit the protected environments? That's like a right as an owner, even if they're locked that way, there's not unlocking and locking that usability pattern tends to cause problems.

A

uh Yeah, so that's a that's a good point. uh We should discuss it further. um Like uh you know, owner can edit protected environments, even if it's locked, that's uh yeah, totally possible yeah, but uh I mean like at this poc level. uh If it's locked, even owner cannot change it. uh It is.

C

That's totally fine for the poc, I'm sorry, I'm getting a little ahead of myself! I'm excited because I think that this solves a lot of yeah.

A

I mean before that um we should make sure that if this is really what customers need, I mean.

B

Yeah, so in the issue you'll see that we asked for a sync meeting with a customer um we'll see if it'll fit your timezone I'll I'll ask for you to join as well. uh If not, I think we're good. If what, once you post this video, I think we'll also send that to them asynchronously and see that they approve.

A

Yeah yeah, that makes sense um so uh yeah. That's really helpful.

A

Cool, so uh I think uh I presented everything I wanted to tell um I mean, given we only have huge information about the customers. I really not sure if this the right solution, though it's gonna variate it so yeah, that's it. If you don't have a question, then I'm gonna end. This call, okay, all right cool thanks. Everyone thank.

B

You for watching, I can see you put in a lot of work, so I really appreciate it.

A

Yeah you're welcome, so yeah see you bye.