GitLab Compliance Group - Meetings, 13 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Discussion on alternatives to "required pipelines"

Description

We discussed the problem our customers have to ensure certain jobs or tasks are run with each change they make to production or other environments and how GitLab can solve that in a way that's not disruptive for developers or organizations.

A

Okay, cool so we're talking about the whirlwind of a feature, request or need from customers centered around what they're affectionately, dubbing forced includes or required pipelines. My opinion on that has not quite changed. I think the discussion I had with CID was largely a semantics disconnect on my part, but like I, don't think I presented the case very well.

A

So what I took away from my conversation was Sid was multifold and I want to just share with you all a quick diagram I put together, which I think, at least for me helps visualize what the requirements are so for these regulated customers. They they basically want to ensure that before it change gets to deploy that they've met XY and Z requirements that manifests as jobs within their pipelines. There's an opportunity for at least some or most of it, to pull that out of the pipeline entirely.

A

In this particular case, I spoke with our largest financial customer and we defined you know they were very good about articulating very clearly what the process looked like and so I trying to get a follow up, call with them to dig into it. In more detail, but basically they have they treat gitlab projects as applications, and so within their external systems. They referred to Gilad projects as an application.

A

Those applications map to something in a catalog which is a list of identifiers, as well as an app directory, which my understanding is separate from that catalog of identifiers and then based on these identifiers x'. They call out to n number of governance systems, and so those systems check for various things, all kind of like internal proprietary checks for their compliance and regulatory needs, but, for example, if they're, if these IDs are lacking, then they want the build to fail because it doesn't meet the requirements to then go through the it's checks.

A

There is a separate check called their passport system, which is led by their tech risk team in their IT department, and that is a separate form of requirement or a separate set of requirements for each build. They also have the what I would consider standard traceability requirement if here is the business case or the jura ticket, with the change described for what we're trying to change a push to production. So these are what they call the pre build checks and in theory they should all pass or have their requirements met before a build is completed.

A

These are the things that they have indicated could be pulled out of the pipeline. So I have an issue that exists to allow them to provide an API endpoint, and so in theory we could send them some sort of payload data payload. It says here's all this data that you need to process internally.

A

You send us some response back that says yep all of these passed and then, if that's the case, the process continues at the end of I guess at the end of the CI pipeline, when they have their deploy artifact, they wanted them, perform multiple scans, they've described them as a dependency, scan a first-party code scan and then I.

A

Think there's a third one on me to clarify from my notes, but these are kind of like the hard absolute must requirement before something is deployed to production, and so this is the one that I suspect, there's probably a way to pull this out of the pipeline as well.

A

Maybe, but the general intent here is that a first iteration should allow customers to say these jobs should be run for all things are, these are for, like all workflows and these regulated projects that does not mean that we or they should or need to take programmatic action to block a deploy or block merge request or anything like that, I think. For now we can focus on just reporting and say you told us that a B and C requirements had to happen. They did happen.

A

One of them failed, but we'll still allow the work flow to continue and I think. Maybe the second iteration is then okay. Now how do we solve for that enforcement piece? We also have that override or escape hatch. That allows them to say, even though this process failed, I need to get this to production and so I'm going to bypass it. So hopefully that's a helpful summary, but I thought that might be a good place to kind of kick off, because that's my current understanding and thinking on this. What's.

B

A

B

That this touches, you said so this is for a specific customer I'm interested in understanding like how much of our use cases are. This.

A

How much of our you mean like how many customers share that use case? Yeah I have spoken to I, want to say seven or eight customers that are. We would consider that we identify his enterprise that have shared a similar requirement of I need to make sure that a B or C requirements are met within my CI CD workflow.

C

Can I you use the word pre and post build, but nothing that you described in in that language had much to do with actually building and like building a code into an application. Is that true? Is that just how they'd mentally framed it as pre and post build or how are they connected to the actual build process? So.

A

That's how they framed it and all I can go look for the link here in a second when I have a little bit of downtime so that you all have that context as well, but that's how they framed it to say we have these pre build checks and we have these post build checks, which I think maybe highlights that you probably don't have to exists in the pipeline right. Like we mean we can pull those out in some way and it's trying to figure out how that manifests. Yeah.

C

And I think that that is the using the word pre build, makes it sound like it should be in the pipeline process and should be like to find in your CI mo, but that I think is partially what it was reacting to you that all those things about pre build checks like I would describe them as not pre build checks as checks that don't require a build. So you could do these checks at any time and they don't necessarily they just you.

C

Don't have to have built the image yet the post build checks all require you to scan something, but those other ones.

B

C

And so that I think is part of the reaction to you. Those types of things are better than left that are left to, like our merge request approval workflow, rather than a define this in your pipeline and whatever just because you would be kind of running a similar job to a pipeline. They might translate it into you. I want to define this in CIE mo, but if you could do the same thing as an approval, workflow for a merge request, approval would work.

C

One thing that I'm reminded of has anyone on this call ever had experience with the garret review system. It's a similar alternative that sometimes paired with Jenkins to do code review, and that system has this concept of like in order to merge this I.

C

Don't necessarily wait for a pipeline I wait for these like approval checks and some of those approval checks our programmatic processes, so one of them might be like I'm, gonna, lint this and so I need that lint approval to come through and tell me this thing: is that yeah that that like check has passed, but that allows you to mix like I'm gonna call this external system and I need to make sure that there's a JIRA link to it. That has nothing to do with your CI build process. It's more about the approval.

C

Don't let this thing through! Without that check, it might help to separate those two to not use the build word and instead just say these are checks that need to happen before it can go to production. Yeah.

A

And and I think you framed exactly what they did, which is they share refer to about allowing them to plug in an external API endpoint that they control was what the customer has said is yeah those that would actually be satisfactory for these checks that are required to happen before a merge request, because they described them as kind of like slow and complicated and that they can envision an experience where yes send us the data we need, and then we can crunch that outside of gitlab outside of the pipeline and then as long as it's fed back into gitlab as part of that merge request widget so that they're informed and have that data all in one place for like an artist, they can export at some point and I think that satisfies those those initial checks.

A

Then it's the I, guess, pre-deployment or like right before deployment that we then need to focus on.

C

Yeah, and so just my understanding of that video and I've been through the Jeremy knows this too, like through the forced, includes discussion, is that the concern of using a tool like forced includes to say hey: all projects have to have this. These included pipelines is that you are skating what that is and have the potential to block production way too easily so I'd.

C

Rather, if you take the instances where you're actually not needing to touch the code at all and move that to a merge request approval, then those are the kinds of things that you can say: oh and by I'm, bypassing this required approval, I'm still going to merge it or yeah. Even if I don't understand it, I know that there's a merge request, approval happening and that approval is out-of-band from anything that, like I, can go. Look at the logs at in my pipeline logs yeah.

A

I think that's aligned with my thinking is that we already have this existing paradigm, that oh there's a merge, request approval process, you're already gonna get held up to meet some requirement, and so why not incorporate these external systems, but just in a way that fits the current precedent or paradigm for it and as I thought about my conversation was said, more I think somebody please correct me if I'm wrong, but I think part of the the misunderstanding was that we're not wanting to mess with the CI process right like we want developers, I have a great experience.

A

It's the CD process that customers are ultimately seem to be concerned with which is before this actually makes it to production before it's deployed. I want to make sure that okay, this word request has those like those preliminary checks done, but then also right before deploy once I. Have that build artifact I want to make sure that I have these scans then, and all these other checks that are required right before I deploy is that right.

C

Yeah and I think you could actually the way you just had to there's. Probably one other solution, which is your post merge pipeline. That is doing some sort of deploy. Action might have a manual step there. That just says hey before going further I mean you could have one you could do this today. Probably I'm like you have ones one step in that post, merge pipeline.

C

That says, collect all the important information and runs the script that goes and calls api's, and then the next one that says you know take an action as a result of those api's I think that difficulty, the reaction is less about where that's happening and more about forcing the inclusion of those two where the user cannot experience them or override them. So yeah.

A

I agree with that and I think you.

B

Really agree with that, because what Kenny just said is that a user is empowered to override their pipeline and in these cases, where I've talked to other regulated customers, their desire is for users to not override very animal file and to do a deploy to production. I agree.

A

With it, because, ultimately, the the paradigm exists already within regulated environments, a compliance framework so like yes, we have these processes. We have these things, these requirements that have to be met, but we also have escape hatches. Those escape hatches are intended to be kind of like the emergency. You know oh moment, they're not intended to be used regularly and I.

A

Think that's the nuance we'd have to hone in on is: if we implement this cyclic feature, how do we then empower customers to make sure that this is not happening like with every single deploy, because every single deploy is not an emergency? So so there is a balancing act there, but in terms of solving the original.

A

Misunderstanding or miscommunication internally, between myself and Sid I, do agree with that approach, because one I think the merge request. Widget is sort of like the single source of truth for releases or deploys right like changes, and so the idea of like hey. We approve this to then merge it and then post merge we're going to come back to single source of truth and co. There's also these additional things that had to occur did those in fact occur.

A

I think, creates a productive cycle that the audit and compliance folks would be happy to have, but there is that new on Thursday I would want to make sure we address yeah.

C

And I am I, don't speak the language of compliance, but in previous lives we've called those compensating controls, and so if what we're talking about is enabling enabling the documentation of a compensating control for our customer so that they can say yes developers can override. But we showcase every time that they do or create some report for every time that they do. That might solve the compliance need of this customer from this class of customers. Yeah.

D

That's exactly what I was about to say, like I'm, curious man, if you have received any customer feedback on whether or not that's kind of an acceptable approach, because for me, like the the the revelatory moment, that City conversation was I, think probably was feels like twofold, which was the first was around. You know, allow always allowing an escape catch for human beings, because when we talked about enforcement before we always thought about this as a very strict control.

D

But then SIDS approach is like well productions down like we know, we don't have the opportunity to wait for it. For you know someone else to get off. You know who's out of office, so we need like to do this.

D

Deploy right now, I think is, is is definitely worthwhile and I think that just like you know so that what does that mean that neither means that we have the ability to override, like that, required that sort of a required approval with like the help of another human being, or we write some evidence, artifact like when the deploy happens, and we do the override to say, like hey, we're, gonna, you know to write this note or we'll create this issue and like to you know, have evidence.

D

This thing happened for someone else after the fact to go and didn't check because they like whether or not this is acceptable. So that's kind of one thing and the other thing was: maybe you know it is.

D

This is the the aha moment for me was I agree with something you said earlier, which is this is fear a little bit dependent on the type of environment that you're deploying into, whereas all these controls need to be more strictly follow it for a protected environment like reduction, but maybe we don't care as much about these these artifacts in place for something like staging. We just want to go ahead and get something into the environment. We don't really care as much about following these.

D

These exact controls when it comes to actually at deploying to production, then absolutely we want to make sure that these things are checked. So I, guess those are the two things that I took away for the conversation. Have you had a chance to talk to any customers and get their take on whether or not this is kind of like an acceptable direction?

D

A

About that specific concern, I don't have conversations the initial conversations that I've had have touched on it. So to address your first point about, should we generate the evidence? Artifact have to person control any answers both? It follows the same logic of that issue. That's open for two person acts approvals for sensitive project, setting changes, I think it is without applies to this right. You shouldn't have a single person, who's capable of overriding something by themselves at their sole discretion. It should involve that second person. He says, oh yes, this is reasonable.

A

This is needed or necessary. We should make this change and it should maybe Auto create an issue for that project. With some format that says, merge, request, number one was overridden and the description has some data about the merge request and we prompt the user for a description at the time so that they can justify why they're overriding it towards this. For the second question about help me remember, you touched on Oh at the different environments. I think it depends right, like some customers may say, and have communicated that yeah. We only care about production.

A

In my personal experience, and from maybe one or two the conversations there was mention of the requirement to also manage changes to the staging or testing environments, because my opinion on that, because I didn't dig into it with them. Specifically, it was just kind of a casual mention, but my understanding of that would be.

A

Yes, we ultimately care about what change is making it into production, but that starts with changes that are made into the testing or staging environment and like what, if something changes from where it wasn't staging to how it ended up in production. You know those sorts of things, but I don't have specific or explicit feedback about that, which is what I would like to collect after this type of conversation. Where were we have a much better idea, I think of the direction we went ahead for this I? Think.

B

Is just an interesting thought, because I'm undergoing this environments- validation, right now and I'm, learning that customers have different expectations based off of how they've defined an environment, so, for example, staging for one customer may actually be where they compile a quarterly build that they then push to production. So, therefore, staging is a production stage rather than like a dev environment where we're able to test things in a CD world.

B

So if you're, if they're not using CD, this becomes very different than if someone's using CD and they're constantly deploying to a target environment from master. But if master is in gold or if they don't have a branch that they're working from that, they promote that really they use a whole environment. To do that, then this becomes a very different use case. Yeah.

C

It's kind of like depends on if staging is a stop on your way to production or as a where's like a validation cycle itself.

C

Now one thing that I want to make sure- and this is I only am aware of this context, because if I history here Sid mentioned like in that video, the importance of this similar paradigm in how we approached our security tools and like I just want to make sure that stop me if this is already like you're aware of it, but I want to make sure you were aware of what that meant for our approach to our security tools and how that impacted.

C

How we did in there might be some tools that are useful in this same kind of like compliance regime. We had said like most security tools, people install them, but don't use them because they are set up to block production, deploys and people just say. There's too much here, I can't do anything I'm turning the dancing off, and so we intentionally from the start said our security tools will never block deploys.

C

The production will give you the report, you what you choose to do with that, for it is up to you and since then, we've evolved for some of these compliance regimes to have things like dismissing vulnerabilities, where, when you dismissible loner ability it does create an artifact that then gets created, or you can dismiss vulnerability and create an issue as a result or you're required to dismiss vulnerabilities or there's required merger of crust approvals.

C

If there are vulnerabilities, found that if, if we found any of you've got to have some other person come and look at that merge request, those are all compliance standards, we've they're like yeah compliance or enforcement standards that we built that weren't it avoided us having that check block production automatically. Those similar patterns might be useful here to just think it was similar compliance check. That then, creates this list of compliance.

C

Vulnerabilities are like non-compliance report and then the developers can choose to create issues dismiss them. If they dismiss them, it creates an artifact if it requires a merge, request approval as a result that happens so that might be a useful pattern. Yeah.

A

No I am definitely aware of those patterns and precedents. The interesting thing for me is that it is, admittedly, a bit of a struggle coming from a compliance role and that industry and mindset and having an appreciation for the customers who say like no I, need it to be exactly this way and that's at odds with what we would like to do, which is allow developers just very smoothly operate and have no hindrance whatsoever, like those are literally mutually exclusive things of what the compliance world needs and what developers want for. A good experience are.

C

They so there was another point that I would actually would have said was a big takeaway from that video that it's it phrase it. You said something like what you might need to do is not talk to the customers, but talk to their auditors, which would be like put yourself in the shoes of I'm. Looking for my customers best interest, they might be telling me I, want to block all production, but I know that that's not going to lead them to success.

C

So if I talk directly to an auditor and say what, if we allowed this like this more smooth operation for the developers, but we had these compensating controls summer would never come to you and say that, because they haven't thought through that. But well I, don't think what yourself in the position of trying to convince their auditors sure no.

A

I mean it's a valid point and and I don't disagree with it. I think the challenge there is gonna be that it's not so much that the customers haven't thought about it. It's that they're managing risk and liability, which has like the financial impact and consequence right and like it's all about managing that risk or that impact. But like the auditors, we can talk to them and we will. What I suspect will happen. Is that we'll get a wide range of responses, because every auditor is gonna be different?

A

Some auditor is gonna, do it by the book dot every I cross, every T and say yeah you can have compensating controls, but ideally I, don't I want to see is very few of those as possible. Other auditors would be like yeah if you've got compensating controls. I can follow that breadcrumb trail. That's great. So so I agree like it's not necessary.

A

I guess they're not necessarily mutually exclusive, but it's certainly a spectrum we'll be dealing with as far as the tolerances from either the auditors or the customers, and that's why I'm, okay, taking like a passive approach initially to kind of temper my own expectations and try to align better with kind of get labs core values around this particular context and as we learn from what's actually needed beyond that, we can then iterate and work towards that, and we may find that. That's perfectly acceptable.

A

B

Companies are managing right, sorry, Kenny. Whenever we slow down development were telling their developers they're not able to derive like deliver value to their customers, which might have an impact on their revenue and development and ultimately their risk is we're going to slow down development, potentially not deliver features for our customers. So it could have the same negative implications of getting an audit and need to pay a fine, because they're not able to deliver features fast. So something to think about there now.

C

You said something that made me wonder: um do we have a do we dog food this today, like we, we ourselves report to some compliance regimes, I wonder if we could, because that would be in a response to an auditor saying. Oh, this is not enough to be like well, no. We we passed that now. You can't just say: I passed it with this other auditor, so that means it's okay, but you could say, like this passes.

C

For us and I know, we have different we're talking about different compliance regimes, but I wonder if they're yeah.

A

I have a recurring meeting with Jeff Burroughs on the security compliance team internally I he is my internal customer, so we talked about a lot of this he's supportive of basically, all of this that I proposed I haven't yet talked to him about this particular pivot and I'm, not I'm, not entirely sure if it's the same since we're kind of operating within GAAP gitlab using git lab, and so I'd have to do some digging there in this new context. But it's a good point that I can follow up on.

C

Yeah, that's not I was just wondering if we, if we're providing reports to auditors today, based on how we ship software yeah, we might have had some workarounds that our customers would find useful yeah.

A

I I can certainly follow up with them. On that specific, ask for sure and and Jackie a quick response to you. I know we're at time. I agree with you and I think at that point it comes down to do. We manage the short term, the short term risk or the long term risk and I don't know the answer there. I I, don't even know if I assume our customers are managing that and calculating that risk. But you have an absolutely valid point like.

B

The interesting to do like an a/b test on this to create like a scenario and validate with customers on if they feel like they're, managing risk better by blocking production or by providing a really strong audit trail of these events, and it might be better for us to provide the audit trail in the value in ROI that people are able to ship superfast and they compare it to a scenario where people were being blocked by things like compliance and security scans, because this again would be getting at the point of wood.

B

Blocking production really mitigate risk, or are they gonna end up overriding it? Cuz I can't figure out what they're actually violating and pushing it to production. Anyways yeah.

A

And I suspect that'll be a broad spectrum, because this / customer we've referred to has said I would rather block everything going to production, then risk something getting into production that is not compliant and.

B

Are they doing that today? Are they blocking production.

A

Today, I'd have to go reread through it, but I believe so I'm not entirely sure, though,.

C

I know we're over time. The discussion was originally about this. One issue about continued support of forced includes I am I just say personally, I am on the side of us. Not adding new functionality to forced, includes, I, know it's out there and we've said it's deprecated. Is that where we're all at or is that still an open question? No.

A

I agree with that, my original point was like well: if we add these override mechanisms and make it passive does that satisfy the requirements but I but I agree we should not under Kate or continue to evolve it. We should just work on what this new solution looks like. Okay,.

C

Cool so Jackie, will you make that comment on the issue and close it.

B

I actually have one going right now so well.

A

Thank you all. Thank you all for going a little bit over I appreciate your time and your brainpower on this yeah.

C

Man, lettuce, I, don't know if you got what you were looking for from this discussion, so I understand, there's a very tricky situation where you're put in between customers who are essential to the revenue and ongoing livelihood of this business and some product principles and I appreciate your skill and navigating, and so thank you. But if you didn't feel like you got what you needed look we can schedule another conversation. One.

D

Parting question that I, might they may have for anything about, is like I'm, not sure where the in terms of the responsibilities of like developing this functionality, where they lie, whether it's manage compliance versus like verify, and so what I would suggest we do is that maybe we validate kind of the direction that we're going do some solution: validation with some of these customers to make sure that this is the right direction, and then we can kind of collaborate on which groups make sense to do kind of what and in terms of iterating, on that fish towards a vision but I think that might be stolen, open question because I don't know if compliance says all of the you know the expertise to be able to iterate on this solo, yeah.

C

I, can we say that Matt's, that's the drive for it and if it comes down to hey as a global priority, we need other teams to help work on this. We can have it in that in, like that frame of mind, Jeremy I, don't want it to be left on and ambiguous about who's driving it forward because I think yeah totally. Okay, thank you. Yeah.

A

No, but yes, Kenny I did get a lot of valuable information out of this. So thank you.