GitLab Delivery Team, 10 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-10-02 - Delivery: Fire Drill

Description

Delivery: Fire Drill (security version)

A

uh Thank you for joining. The purpose of this meeting is to go over one real scenario regarding security release and release management and to analyze what options do we have when facing such a scenario? And what can we do when dealing with that with that and after the call? Well not during the call, I hope to also get some action items about. How can we improve our process?

A

So, let's get started uh the scenario that goes like this, so um we are in the run-up on the security release in the last days. uh The due date is scheduled to be soon basically the day after tomorrow or in two days. uh Let's assume that this is friday and one of the security fixes that reach gitlab.com.

A

uh It broke one of the workflows of a very strategic customer, and you can read that as the customer cannot do his daily operations because of it.

A

So the customer has raised an incident and the technical account manager, also known as the tam has also has joined the incident and has asked us to revert as quickly as possible.

A

Now this is all you know as release manager. What are your remediation steps and what are your next? What are the steps that are you're taking um if someone can help me take notes that I will also appreciate.

B

um I guess to answer your question for the purposes of remediation.

B

It sounds like we must perform a revert, so I would immediately call upon the appropriate development team to initiate that revert, because I'm not going to do the revert, I'm going to make sure someone else does I'm going to ensure that the appropriate pick into and severity labels are added to that and since we need to expedite this I'll, probably keep track and watch that merge, request and watch you get pushed into place and watch for green pipelines and, if necessary, manually, kick off an auto deploy pipeline after I see that it's been merged in sc green pipelines, that entire process would play take well over an hour.

B

So, whether that's quick or not, is debatable. I guess another option would be if a simple rollback is possible. um Maybe that should have been. The first thing I looked into is see if we could simply roll back, because that would have been the quickest way to revert or excuse me. That would be the quickest way to roll back their problematic merge.

B

And then we can proceed in a non-rushing method. Proceeding with a revert and allowing for the next auto deploy pipeline to kick off a schedule, potentially.

B

Or if it didn't get picked up by the next auto deploy pipeline, we could simply skip that one and wait for the following one. For example,.

B

um That is my answer. Thank you.

A

Yeah thanks uh henry: do you wanna verbalize your question.

C

Yeah, I just wanted to know if, at this point, we already um merged any um backport security on us like. Are they already uh merged, or did we even or detect the security on these? uh No, we haven't.

A

Only the ones targeting master have been merged.

C

Yeah that should make it easier, then yeah, then it's mostly about to remediate the issue as soon as possible for the one customer and then clean up afterwards, and maybe even um considering a rollback. If that's really a big issue, instead of waiting for a fixed amount to make it even faster.

A

B

D

Sorry, I'm good.

A

No no go ahead. Please.

D

I was just going to say so, probably first I would check uh if a rollback is possible just in case the decision is to roll back um then check if a reward can be quickly created so that we know both the options and then it will probably take some discussion to decide whether to roll back or to revert, maybe check with the imoc and then based on the decision and based on what is available either the revert or the rollback. If both are available, then it will probably require more discussion, I suppose but yeah that's.

A

Awesome um ahmad, do you wanna, add something or robert.

D

From my side, I'm good.

E

I'm just focusing on attacking your thumbs.

A

Okay, um how would you deal with this being a security issue, both all of you mentioned, reversing and rollback, but the problem or the subtlety about this? The tricky part is this: as a security fixed and the security release is coming up. How would you deal with that.

B

So because this is recorded, I can't say what I want to say, but what I would attempt to do is try to enable the tam and the current security um um whoever is assigned to the security release, get them together so that we could all understand and set the expectations for we need to do what we need to do for the security release um that's going to entail either not including this patch that has been decided to be reverted or it's going to include the necessary work for us to work with a development team to rush an alternative fix for this specific patch.

B

Being that it was reverted, I would lean towards the former, but um because the tam is sitting here initiating the desire for us to revert. I want them to be responsible for working with the security team to decide how we approach this particular issue. That needs to be resolved in that security fix appropriately.

B

Is that the wrong answer.

A

There are no wrong answers here is carpet.

A

um Anyone else.

C

um I think we yeah cool go ahead. All that.

E

Sorry, uh I think we've seen that, like the security release, dates can be pretty flexible and if delaying it is kind of the best course of action, then I think we should remember and make others aware that that's kind of one of our options is to delay it. And until we have the proper fix in place.

C

Yeah, I want to add that I mean we try to do responsible, try to be responsible with um making security issues public right, and so that basically always means we are vulnerable for something. But it's not exploited right now in most cases, so we just wait until we have the fix in and then we publish this, but um we often stay for months with known security issues um because they are not public and not exploiting not being exploited.

C

So um it's not unusual that we have known issues that are known to us, but we don't fix them immediately right, so we shouldn't get um forced to to do any kind of actions, just because there's an issue that we know but which is not publicly abused.

C

But this is then really on up security to decide, I think- and not by us. It just can be be involved by the trade of functionality and time for secure release. What other releases are security fixes are held up and things like that which needs to be discussed right, because if we delay the release, maybe other important things are also delayed, and I had the second point. ah The second point I wanted to mention is, um I think, especially for this kind of incident.

C

um Apparently, we fixed something without testing how this would affect customers which could have been tested before right. We could have, I think, just looked into logs to figure out um what kind of requests wouldn't work anymore and how many of them we would get. So I think this would be also thing to think about. um How can we improve these kind of situations when we introduce a security fix, which is maybe disabling some functionality?

C

um I generally have some kind of uh I don't know, run book or workflow. That makes sure that we first try to figure out what kind of requests would not work anymore and confirm unlocks how many people would be affected by that.

A

Cool thanks um anyone else.

A

I'm gonna take that silence, as I know um so, just a couple of things here. um All of you mentioned uh robertine and possibly rolling back.

A

So I do have a problem with rollback, because the security issue is an s4. It is not like an s2, it is not an s1, it is an s4 and obviously, if the customer is um presenting problems, it is kind of still in my mind that this is an s4. We normally roll back for only s2s and s1 when everything kind of is broken.

A

But this being this and s4 I mean this is a security vulnerability, perhaps rollback it's an option, but it is not like the end of the world, because the information that we know so far is this is an s4. This is known as one and only one customer is being impacted so far. This is all we know um now um what happened when this actually occurred?

A

um I'm not saying that this was like the best set of actions, but what happened is like the first step is to get as many people with topical knowledge as possible, like you need the team that owns someone from the team that owns this security fix. Someone from the team that participated in that the other person that you need is obviously appsec, whether is the engineering manager on call security manager on call or someone from upsec and then.

A

Once you have that many people involved, you need to analyze the well. That could be step two right analyze. The security fix. Is this actual an actual security that fix a vulnerability? Is this a bar? um Is it is? What is this? What is the impact of this, um and then probably um you probably want to escalate this.

A

This is probably already an incident so mostly like the imoc and the il eoc engineering call are already involved, but also uh involving someone from support, because chances are that if this- or this is already breaking one customer workflow when this is mostly spread, it is going to break more workflows and notifying them ahead. It's always like a good idea um question. So far. This is kind of the first initial steps that should happen almost like in parallel like the first 30 minutes or something of the incident.

A

Okay, so um let's assume that it is not a bach, it is actually like a vulnerability that it's a security fix that actual fixed vulnerability. So um what are our options with that? I think one one interesting thing that uh that happened is that it wasn't aware it wasn't known if this vulnerability was already known to the public, or was it something that it was already disclosed turns out that we have vulnerabilities that have already been disclosed and we are just a bit behind and trying to fix them, but this wasn't one of them.

A

This wasn't enough. This was like a vulnerability that wasn't this close so far, so um I think the next step is to actually revert, uh revert the vulnerability or revert the merge request, because that leaves the strategic customer where they were.

A

And then, if this, if someone from the teams or someone is trying to actually fix the problem because there might be people that want to fix the problem, that is always a good idea. But let's remember that in this situation in this particular situation it is friday and it is 2 pm for you. And if someone wants to fix it, they are going to prepare a merch request and taking merch requests merging it and all that it takes around one hour one hour and a half.

A

So reverting is quicker, because if you wait for the fix, then you will end up deploying production way too late in your day for being a friday, and also that depends if you're available. Of course, it is totally acceptable to say that I'm not going to be here at 9 00 pm deploying to production, because it's friday.

A

Questions comments.

D

Also, you have to consider that the fix itself might have bugs.

E

In this case, where we know we want to revert, even though it's an s4 would a rollback still be an option just because it's faster like it's depending on the diff set like say this is the only thing that was in that diff would. Would it make sense to roll back that gate.

A

Yeah yeah, in that case, I guess it would make sense um if this is just like a single merch request that we can from that then, yes, let's do it. That will be faster, but uh that is like an idea. Never.

E

A

Yeah, that is like an ideal world right. In this case, it was deployed along with a bunch of changes and along with a bunch of features, and also since this is a security vulnerability.

A

It is actually like preventing vulnerabilities like it is protecting gitlab.com, so reverting immediately, something that you need to assess. It's not something like I mean rollback is something that you need to assess, something that it's not like, a decision that you need to take lightly, because if you remove that vulnerability, then gitlab.com is already vulnerable.

B

So realistically, absec is probably the persons that will make that decision. Ultimately, not us and probably not anyone of infrastructure either.

A

I think you need to be: it needs to be a combination between upsec and us from our side. We need to give them options. We can roll back if it is feasible or we can revert.

A

But then, when you mention the word revert, a vulnerability absec gets uncomfortable because well it is. It is a vulnerability that needs to be fixed, so here is kind of our job to explain that that, once we are reversing that doesn't mean that we are removing the fix from the security release.

A

We are just reverting to remediate the situation to make the customer happy, but then we need to reschedule- or we need to reorganize the security release, to to give some room for appsec, for the technical account manager and for the team to organize themselves to see how they are going to reintroduce the vulnerability or the security fix to the security release, because we cannot ship the security release without that one.

D

I'm wondering did: did anyone want to increase the severity of that of the incident, because if it was an s4, but with all the things happening around it, it doesn't sound like an s4. So.

A

Yeah funny thing: it wasn't really an incident right, because the infrastructure was okay, nothing was failing. Actually, the problem is that we were dealing with a particular strategic customer that he can screen because they were using a vulnerability as a feature rather than a vulnerability so yeah after a long discussion after this took like one week and a half, everyone agreed that, oh this wasn't an s4. This was actually like. Probably an s2 ride like the vulnerability should have been categorized like that, and that was an action item for upset to better understand the impact.

A

The security fixes have.

C

I think it's it's really an incident if we break functionality that that customers are using right, so we broke something, and so something was was not working anymore, and this is worth to be an incident I mean, then we need to see how many and which customers are affected to just the severity of that, but an incident. It was for sure.

C

um Another remark I have for for these kind of incidents is really um what I find always very stressful is if you need to deal with um incidents which have to do with um a lease management or security releases and stuff like that, and you have to involve a lot of people to make decisions and work on things, and I think this is very important to then really reach out and get an imoc onboard to to deal with that, instead of us doing most of the work trying to coordinate people here, because I think this is very stressful and we should be there to give enough input for the imoc and the upset and other teams and people involved to understand the options.

C

But the ultimate management and decision making. um We shouldn't be the ones who need to drive that. And so we need to make sure that we have someone as a di for being the manager of this one right. So because it's I make the experience. This is stressing me totally off. If I need to constantly.

C

If I have the feeling I need to go around and try to align people and bring things forward, and so I guess we shouldn't, or we shouldn't get too much involved in driving things and uh doing decisions. We should inform people to make decisions.

E

I think it's also important to keep in mind that the vulnerability was an s4, but then once it became an incident that was probably more like an s2 right. That's those are two different things. We need to think about.

C

Yeah, I think the one thing that happened is that this one um strategic customer, then you probably about the vulnerability right so because we did something- maybe they figured it out, maybe not, but at least we a little bit disclose something with that action.

A

C

A

Yeah, I do agree that uh we are not. We should not be like the dri of the incident, because that is not our our main role. Our main role is to to be there to manage kind of expectations, so they are going to basically the imoc. The time and upset is going to want some different options and what we need to provide. That is okay.

A

We can't revert when we say that we need to be very explicit because they, my upset, might have the idea that okay, we are going to revert, but that might disclose the vulnerability to the public and we need to explain that it's not happening. The reverse is under our control because the reverse happens on security, so we can revert and we can still be saved when we are going to or when there is a chance of us revealing this vulnerability to the public is when we publish, but that is also on our control, so yeah.

A

The plan in broad scale is to revert over back if possible, if it's visible, well revert and then work with upset to move the due date of the security release. Probably move it a couple of days, I don't know four or five days to give enough time to all of them to assess how this security fix needs to be reintroduced.

C

I mean: wouldn't there also be the option to just um take it out of the next security release, do the security release and then just when we revert or fix proper fix for this is ready to do another security release.

A

It really depends on upset and on the impact of the security fixed again, if the, if the security fix is actually a security improvement, and it is already known to the public, it is already disclosed, it is most likely we can just remove it from the security release. But if it is not known and the impact is breaking workflows, then it is unlikely because it can reach a point in which in which it can also involve legal about. No, we cannot disclose this, because you know zoos and things can happen so yeah.

A

It is a very sensitive decision.

B

So I have a question that kind of bends. This scenario slightly. What if this was a security vulnerability that was undisclosed, is probably considered high priority, so it might be say, an s2 style security issue, that's being fixed reverted for similar reasons, but now we've got a situation where the development teams have requested more time. So there's no way we're going to make the next estimated due date for the security release.

B

We've now got a situation where the revert is going to be published, so people who are keen on the gitlab source code they're going to know that there's gonna there is a revert and they're gonna see that revert in our code. So they know that there's a security vulnerability being worked on.

B

A

That is a great question, because that is also what happened. um So we can continue moving the security release due date until, of course, it is too late. um Delaying it for a couple of days, two or three days is fine, delaying it for 10 days or 12 days. Then that should trigger an alarm not because of the work that it involves. Of course it puts pressure on release managers, but it also could impact the major release. 14.8, the omnibus images are disabled. That means that the night list cannot be tested.

A

Italy might also be disabled. So as long as we continue pushing the due date, the release the major release gets in risk. So when it comes to that, it is basically a decision from the director of security and from I don't know, the electoral infrastructure.

A

We need to escalate that we will need to escalate that to marine and to give options and say: okay, we can continue delaying the security release, but that could put in risk the monthly release or we can just grab up the security release and perhaps trigger a critical security release to deal with the disclosure or something right. But that decision is no longer on us. We will need to escalate.

D

I know we normally don't do this and probably for good reason, but would it be possible to rebase the master branch and remove that commit rather than reverting it so that you know when you publish the security releases, it won't be revealed or will that cause other problems.

A

Yeah, that was one of the options we considered one of the last our last last resource sort of options, because well that implies releasing master and it is kind of a sensitive operation um and also will imply for everyone that already branched off from that map from that branch to recreate that merge, request and will also imply replacing death, because death is mirror from security.

A

So we will need to also push force to death and probably it could affect our operation. So it is not a decision we could take like slightly. It is an option, of course, but it comes with its difficulties.

C

Wouldn't um other people who are forking our project then also get into issues later. A theory based like people who are mirroring.

E

C

E

I think at what point you rebase like we would want to do it as close to when the original mr was introduced, so that we're not rewriting the entire history, just the history since that so yeah. This was something we considered on the day. I was trying to think if we could, because the problem is canonical masters, constantly merge the security master through the merge train, so those commit shaws would be rewritten on security. So we would want to avoid that.

E

So I think what we would have to do is take master on security from immediately before we introduce security, picks rebase it against canonical master, to kind of bring everything in from that with its original ids and then reapply security fixes that have been merged since then. So it's really hairy to do without rewriting everything.

A

Awesome so um we have two minutes left. um Is there any action item that you see that we could do to improve this process.

C

I think we have two things here that we need to separate. One is the decision making of um what we want to do like playing the options, and I think we can only inform people to do that, but that needs to be done by upset by product by times I don't know somebody in the company who convey those different interests, but we can't do that, but we can look into technical options right. What are the technical options to deal and what do we have available and maybe to look into?

C

What can we improve in our workflows and um scripts and automation to deal with these kind of leaks like what we just discussed, um trying to rebase, for instance, or other of those things? I think one of the action items then would be to figure out what is possible in the worst case right. Could we really rebase the security master and and then try to get everything in line again if we really need to prevent something leaking, um I think those are the things we can look into.

B

I feel like we need maybe.

A

B

Of like a little run, bug for like any time there's an incident. We just have a small runebook of hey. You know check the severity of this issue. Do we know what the issue or the mr is what's the severity of that?

B

um Is it a security thing because, like having a checklist, helps me not forget all of our options and helps me make sure that I'm at least taking into consideration everything, because I only do this so often, sometimes I'll forget a step, and you know I may not think about it until someone else chimes in about an option. So I want to make sure I'm kind of ahead of the game, and I feel like just having an overall checklist would be beneficial.

D

Yeah and uh also a run book for what to do when the decision is not clear.

D

You know when it's not like a clear case of you revert this and move on with the incident, who do we contact or who do we escalate to stuff like that, or even just you know, just making it clear that if the decision is even a little complicated, it's okay to escalate? I think that also would help.

A

Awesome yeah, thank you for all your ideas, um so we are on time. uh Thank you for joining. I hope this was helpful and well, if you have any other scenario that you want to discuss regardless the security release any other like release manager scenario um we can continue discussing. I do found these sessions very helpful. So, okay,.

A

Well, I guess I see you around. Thank you.

B

B