GitLab Create Code Review Group, 20 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Create:Code Review - FCL Closing Ceremony for Incident 8608

Description

A short and sweet closing ceremony for the Feature Change Lock related to incident 8608

A

All right, this is the closing ceremony for the fcl for the code review team. uh This is related to incidents 8608 um notice. For a little recap, there was a incident at the end of March. We introduced a security fix it and there was an edge case that caused.

B

A

Errors that customers were seeing uh we rolled that back, um but then we went into an fcl to to try to address that. So um this closing ceremony, where the idea is to kind of just do a retrospective and talk about what we've done.

A

So there is a retrospective issue, that's shared in the agenda. If you have I put a couple things in there, if you have any other thoughts, feel free to add them as well, um so I think just in summary, um we had already before the fcl started. We already reprioritized and started addressing our our availability and reliability issues.

A

um It included the fix for this security issue, but then also there were a number of infra Dev issues that um were high priority. So when we started the fcl scope of work, didn't change too much, um but I think that's good that we already recognized the need to address these reliability issues beforehand uh without needing an fcl, so I think that's good.

A

Yeah I'll uh think I think the it was a little confusion about whether or not we actually needed an fcl um kind of decided. We didn't need one um at first, because Well Carrie, you can correct me for one I, don't think this was ever posted on a public status page which is kind of the triggering event for an FC hell. One of them is that correct.

C

I believe that is correct. Yeah, okay,.

B

C

Happened, it happened, uh we, it happened outside of major working hours, um major traffic hours, I should say and the impact we identified what it was or yeah the the people who are addressing it identified it quickly and the fix was pushed out really quickly.

A

Right! Okay, um so because of that, we didn't think we really needed an fcl. um There were a number of other uh unrelated production incidents um in the days after so to be safe. We took all.

B

A

Started fcl's for everyone um just to make sure we were focused on availability and reliability.

A

um I think the other important thing to call out was that um our team was already at fairly low capacity. During this fcl um we had a couple team members focused on the AI initiative, um others on vacation, so we still were able to look into a few reliability issues, but uh our capacity was pretty low.

B

A

Yeah does anyone else have have thoughts or comments on on this fcl.

A

C

um Yeah I have kudos for Gary for stepping up and like shipping getting that change put together um and working on it. I, don't think I, don't know that it's merged yet, but yeah thanks thanks for taking that on I.

A

C

A

Yeah I would say thanks.

C

Gary, hopefully, I don't also trigger one.

A

Hopefully not um good all right well, I mean for the the low impact that these fcl had in our team. I think the the closing ceremony can also be uh low ceremony and low impact, so um I think I'll I'll call it there unless anyone else has anything else, they wanted to add.

B

Thanks Matt and Gary, both of you for uh taking care of all the things that be taken care of as part of this process, the team is team is everywhere, so this is helpful. Yeah.

A

Got it all right? Well, thanks! Everyone.