GitLab Delivery Team: Security Releases, 28 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-07-28 - Security Release as part of auto-deploy.

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

Okay, so welcome to another meeting uh just very quickly: let's go to the progress updates. I am currently working on the implementation details. They are all outlined on the issue and well, I have some up and merge requests that are on review right now, robert.

B

Yes, so continuing the auto deployment security experiment. Last week there was a problem with um deployment tracking that we seem to figure that out now where we, when the flag is enabled we track both in security on the auto deploy branch and then also create another deployment on canonical, but using the master branch so that the merge requests still get notified.

B

There was a problem at the end of my day yesterday that I've just investigated and figured out what cause was um because the previous deploy in security prior to this was a month ago, the latest deploy thought we had just deployed like 8 000 commits, and it timed out trying to notify all of those merge requests so um yeah. We can avoid that in the future by just copying the latest, deploy from canonical to security and continue on with the experiment.

B

I would like to turn it on again, probably after this meeting or after my one-on-one and then kind of just babysit it for the rest of the week. So we can kind of see the full breadth of our auto deploy stuff working with the flag on before we conduct the experiment. For myra's work, which leads us to myra's experiment.

C

How? How are you sorry, robert, how are you confirming that it's working and it's covering the whole of breath like? Is there like.

B

Yeah, mostly just lack of errors.

C

Okay, maybe uh if it fails but.

B

Also, verifying everything works, the way we expected them.

C

Yeah, maybe if it fails this uh experiment now, like the next time, we go through a set of action right, creating attack, creating a branch, cherry picking, blah blah and so long yeah ever driven development.

B

Yeah no, my resident, I think we can um kind of build a plan of what should be working and what we expect to work.

A

Okay, uh so moving on to the discussion, so I have uh broke down depending items that we need before starting the experiment.

A

So uh the first one is going to depend uh on the thing that robert is doing right now. So I think it is all leading down to having the auto deploy branch on security.

B

I agree, I think that's probably assuming nothing else goes wrong catastrophically and there's no blockers like we thought the deploy thing was. I think that kind of gives us the best option.

A

Awesome, so I think we are all in agreement okay. So uh the next point is updating, like the diverge branches between canonical and security, and if we only have master branch between those two uh master receiving regular fixes on canonical and master on security receiving receiving security fixes, I think they can be updated through merge train.

A

What do you think.

D

So the main issue that we identified last time is hot patching right like in the event we have to hot patch production, is that the main risk.

A

um I mean in the event we have to, I hope not, but if, if, if it happens, um production is going to be hot patch, a merch request needs to be created on canonical and merge into master, and then the merge train is going to bring those changes into master security. So I believe we are cover unless I am missing something in the big picture. There.

C

And how does it get from security master to security canonical how to deploy sorry, security, camera security, auto deploy.

C

So current workflow is, we have a hot dash blah blah s1. Someone creates a merge request targeting canonical master label, it speak into auto, deploy label it as one right and we merge it to master, and we cherry pick it into autodeploy.

C

What's changing now is that that merge request is still on canonical, it's still merging to master, it should still be labeled with picking to auto deploy.

C

Will the merge train when it merges it when it merges master, secure canonical into security? Will it will auto-picker have the same shot to pick from security master into security, auto deploy.

C

It means that's basically what you're asking.

D

Yeah, I think so yeah.

A

Yeah, that is a good point, um since we don't have the auto deploy branch on canonical. I think, if that happens, we so far. We are going to need some manual action from the release manager to pick it into auto, deploy on security, unless we can think of something like this 20 minutes to make it more automated.

A

um I know this is something that we have to deal with, but I also think this might be some sort of edge case.

D

Yeah- that's probably fine, oh! So if we enable your um experiment today, robert so we're saying that for the next week any hot patches would have a manual action on them.

B

By half edges you mean like a merch request on canonical.

C

Let's call them, as uh speaking to auto, deploy, merge, requests, something like that. Like.

B

Not even security related just stuff, that's broken an autoplay, no yeah. It will be picked into the auto deploy branch on security.

C

How look did we already confirm this.

B

Yeah, it gets mirrored because it's in master- and it gets mirrored to oh wait, you're saying if the merge strain does it.

C

B

I believe it will have the same shot because it's doing a not fast forward merch, but I'm not sure on that.

C

I think to confirm where I get your point about this being an edge case, but the edge case in a situation where you have an s1 or s2 bug is amplified by the amount of people that are involved in this.

C

That's the main problem uh I mean what we could do also is to see what how big of a problem this will actually be meaning take a look from the moment we created uh daily, auto deploys, take a look, how many actual cherry picks we did in the auto deploy branch compared to.

C

I don't know a month before that when we had two times a week, but that feels like uh just a lot of work together already what we kind of feel in our guts, which is it's gonna, be really annoying and frustrating in a situation where everyone is already almost in the incident-like situation.

A

I understand so yeah yeah, I I understand uh so if we are going to have enabled they have to deploy branch of security after this meeting or after roberts one on one, uh I think we can just confirm how it's going to behave right. We can just cherry pick something and see propagates to security and well analyze that situation.

D

C

Sounds great error-driven development.

D

Hopefully this is the opposite.

D

Is that the worst scenario that could happen.

A

I guess that we are going to discover that in the next two days.

C

What exactly do you mean amy? What if I.

D

Mean there are way more other things that.

C

I can imagine just like for this situation.

D

In the next week, I'm just wondering: are there any other, like scenarios that we maybe haven't considered, that um we could anticipate.

A

Probably, if someone well not someone if a critical security release is triggered, that might complicate things a bit because the fixes will be already on master and then a critical security fix will also be merging to master. And then the critical release will contain all the security fixes merged so far, plus the critical security fix, um which it might not be. What is expected, but.

C

We already have a stable branch, so.

A

Oh yeah yeah, no.

C

Yeah, I guess we are not going to pick anything.

A

Yeah right never mind the packages are not going to contain any other fix, just a critical one. So it might be a bit troublesome to have like to merge all the security fixes into master and to also deal with the critical release, but it shouldn't affect actually the critical release. You're right.

C

It's going to be challenging for you and your to to wrap your head around five different processes at the same time, but you're strong people, so I think you can handle it.

C

You didn't quit yet.

C

C

Okay, so so sorry, just to to to answer amy's question, I can't imagine anything so um we have a situation covered where we need to rush through a fix to deploy we're going to confirm that next there are no other situations that we are handling day to day. It is deploy what is deployed in autodeploy branch during the day right, like security releases from the sides. uh Critical security security release is the topic critical security release.

C

We already figured that the challenge will only be to track the changes and the process, and all of this thing.

C

I guess the worst possible thing that can happen is that I think I have a the worst possible thing that can happen. Merge, train, merges from canonical uh master into security master and because of the way merge train is made it's going to ignore the source conflict. So there is a conflict between the merge and I think we use ours strategy in merge train, which means that the the the target wins, no, the source wins, which means source wins, which would mean that canonical would override some of the security fixes inside of master branch insecurity.

C

Robert correct me, if I'm, if I'm mistaken here, I think it's ours in merge, train the strategy. So what we used to do is we merged.

B

C

Oh, I got lost now.

D

We should confirm that, though right so like that sounds like a great one to. Is it easy to confirm that like to know whether.

B

I was verifying the thing that, where the shot of things stays the same, if it's merchants, I miss the question.

D

So if there's a merge conflict up from the merge train going from canonical master to security, will the target or the source win.

A

Wasn't the target like when we're doing when we did c e to e e? I think it was ee.

A

C

The target we were bringing things from ce into ee and ee was always winning. I think.

B

um The strategy is currently it takes hours, but we can easily just have it fail loudly on a conflict and not do anything.

C

I think that would be great for this specific case. I know we're not doing that for foss, but for us we have a bit more room to react for auto deploys. We are on a super tight schedule and anything that would slip past us and us not having the test coverage that we need to have the integration test coverage. It would be very scary.

A

So if it fails and um how do we notice that it failed.

C

That's the thing we need to have an alert of some sort.

B

We do get uh pipeline failure, notifications in slack for merge strain, but they might not be. I worry, they're not um forceful enough like it. Just it just looks like a pipeline failure. It's not necessarily like hey. This failed because there's a conflict in this critical branch.

B

We've seen before, where there's like a string of them for whatever reason and they're, just literally ignored.

A

Are the failures the notifications you're talking about on the delivery channel or on the broken master channel.

B

I believe the delivery channel gets the merge stream.

C

We haven't had one in a while. That's why you're asking this, um because it usually works fine, but you know that as soon as we enable this next week, it's gonna break right. That's how it works the world in the world's possible moment, so it might be, might be worthwhile verifying that as well, that it failed fails loudly and that we have some notification and then I guess the action item. What is the action item then create a merge request from canonical mastering to security master, resolve the conflict and have someone review right.

C

That's what we did with the e2c emerge long ago. A year ago,.

B

So yeah the merchant pipeline failures go into delivery. They do show the job that failed. So if that says, merge security that might be actionable enough.

D

Cool and that it I mean it's something to resolve, but it wouldn't block the auto deploy right, because it would just mean that something committed to master. Oh, I guess something later might commit, though right or would it stop the whole thing.

B

It would it would block it in the sense that we're not getting updates to auto deploy. So there's like nothing new to deploy, it does lock it in that way.

C

As long as it fails loudly and stops like if the merge train stops- and we don't have anything new- that's an action item for us, the worst part would be for it to succeed and continue and still override things. Then it means we have a security hole.

A

Okay, so if we know that we have a notification on delivery, is there something we need to do besides configuring, the merge train and make it like fail if something like, if there is a conflict.

C

I think that's it.

D

We we might need some again. One of the slight challenges of this, I think, is that yorick is going to end up potentially hitting a lot of these things when you two are offline, so I think it's probably also just making sure he knows what's coming and what to do.

A

uh Yeah, that was a part of my next point, um so there is like another item that we need to do that are like the implementation details that actually like to implement this in our tooling, but uh well, we are working towards it, so nothing that complicated okay. So uh the next point is like actually like running the experiment, so the merch request, targeting master, will be merged when the chat ups command is executed.

A

So now that, given that me and robert are both like in a in america time zone, uh I propose that we execute that command on our time zone, uh whether manually or by a pipeline schedule.

A

I don't want to execute it like on europe and then just have joric or whatever like running around, or perhaps I'm overthinking this. What do you think.

B

I agree we can run it during our morning and babysit.

D

A

Marine, you look worried.

C

um No, I'm not I'm not worried. I think it's a good approach to be a bit more careful.

C

Probably I would I would like a full, like the full, full duration of the experiment to us, so I'm already at point b of maybe worth enabling it on friday. We don't deploy anything over the weekend. We have blocks left and right.

C

So trying it on when you are around both of you and then leaving it through the weekend, should give us enough data by or not enough data, but it would give us any edge case failure that we might not have anticipated and then for your ex monday, you can have a this is what you need to disable this right like if anything is broken, just disable it.

C

This is how you go back and we'll take a look at how to get out of it rather than leaving a decision um in the europe time to try to dismantle this whole uh situation.

C

Am I making any sense? I guess my question.

A

Yes, yes, you are uh so I think it is worth clarifying that there are probably two things here. The first one is the auto deployment security that creates the auto deploy branch only on security, but it does not execute the command the chat ups command. We still don't have an implementation on that, so that one, uh I think we can lift, leave it on on friday. Unless something happens and uh the chat ups command.

A

We don't have like a pipeline schedule to run it so far. We can implement that or we can execute it manually.

A

So for the first part that you were saying that about leaving it leave the feature flag on, I think that is possible, but when it comes to the command to the actual thing that merged the merge requests, I think probably that should be done or on our time zone.

A

Does that clarify things or did I make you more confused.

C

No, no, you didn't make me more confused. um I think I guess the answer is both like. It is worth doing it in your time zone and if time permits it might even be worth doing the merging into master on friday as well, just see all the edge cases that are broken if there are any over the weekend.

B

Myra one thing we talked about that, I don't know if we ever made a decision on is what do we do in the event that we like? We merge something: we'd merge a security, mastermind request and then at some point we have to abandon this experiment. How do we? What do we do with master at that point? Do we just sync it until it's time for the actual release.

A

Yeah, I think we talked about it briefly on one comment and well, my answer was not a very good one. Right like I was saying that, since the security release is about to start, uh perhaps having diverge branches between canonical and security for two days is not going to be the end of the world.

B

Yeah, I think that's better than the alternatives of like force, pushing the actual master and then reopening merge, requests or yeah.

A

So, returning to your point marine, uh I mean if time allows we can try to merge something on friday.

C

What dates do we have again? Can you remind me so monday, tuesday, right.

A

Yes, monday, tuesday for the experiment and then starting the actual security release on wednesday. That is something that I actually need to clarify with jorik.

C

So the thing is the the thing that concerns me there is that will one day be enough for this experiment, which is effectively what you have if you do monday, your time, you have your work day and then the second workday as well is two days enough um to do this. If we start a tiny bit early, no one is going to be affected, um but we have a bit of a longer period that is also safe um when it comes to deployment.

C

That's the only thing, I'm thinking about right now,.

C

What would extra time give us actually is a question. I guess that I'm asking.

A

It is a good point, so perhaps.

C

Because we are covering two different ways: ways of working during this period right like we're covering weekend where we have a lot of gates, left and right.

C

Which we won't hit, if we do monday and tuesday only.

C

If it complicates things, it might be easier to just keep on with your plan. I think your plan is good. I was just thinking like how can we get a bigger bank for a buck um and we want to do these experiments like to wait for another month to do the next one, because we forgot an edge case or something like that. um I would want to get all the data.

C

We need right now, obviously as much as possible and then have the next security release, be not an experiment but a regular process, because the our return cycle right, like feedback cycle, is really long like it's a a one-month um cycle which you know exactly matter. How that affects your development, I don't want you to work on these things for the next six months is basically both you and robert robert is gonna turn a year really soon um on dealing with security staff.

A

Yeah, I understand your point. I guess we can also implement a pipeline to merge to start the command on joric's shift and just make joric aware of the things that can happen and how to deal with this problem like, and that probably can give us some more hours. For this I mean.

C

You started in your shift. Sorry if I misunderstood you, but like you started in your shift on friday right, so you have a full friday. We have the weekend and then, if something does happen, past your schedule over the weekend, we give you some directions on monday morning or in his case afternoon, to um to react right like to do something to unblock the deploys or right until you come online. Basically, both of you.

A

Yeah, I think that sounds visible. I mean if we finish every every implementation detail like we can start on friday. If not, we will start on monday, but yeah, something we can try.

C

Consider it, I guess, is my suggestion, then just consider it.

A

Okay, uh so we have four minutes uh to wrap up the meeting. I guess the next items is that well robert. I think you already confirmed that something.

B

Yes, so the merge train doing the security strategy will bring in the canonical commits exactly as they are with the same show and everything. And then it's just got an extra like merge commit at the top, but the containing commits are the same.

C

So that means that if we.

B

C

This we should be able to cherry pick, something that is labeled in canonical. Let's confirm that um this. This makes me confident. It's gonna be fine. It's just. Let's confirm that path. Actually,.

A

Works. Okay, uh so robert, can you take care of that of confirming that all this auto deploy path uh works? And uh if you can also help me with the merge train like to configure between the canonical master and the security master, because I think last time we configured to have security security master right?

A

Yes, yeah, okay, yeah.

B

We should probably do another call where we're just preparing for a little while.

A

Okay, and um so I can continue working on the implementation details.

A

Does that sound okay.

B

Sounds great, I'm a little unclear on what the additional thing you want to confirm with.

A

Oh yes, uh so I was saying that if you can't take care of uh confirming that, uh when a merch request is merged into canonical, it can also be like included on the auto deploy branch on security, because the branch on canonical won't exist, which I think you already confirmed. Something.

B

Yeah, as long as the commit we're trying to pick is mirrored to security, then it cherry picks the same.

A

Okay, so it is going to work.

C

But that's what I want to confirm right, like label a merge request with that after you've enabled all of this and let's, let's see it, succeed.

A

Yes, so that is one thing uh and another one is to configure the merge train from canonical master to security master, because uh I think the last. The last example of what I saw is what that we were using security, slash master on security, and we now need to use security master.

A

Does that clarify things a bit robert cool and then I will take care of the implementation details that are missing, and I also need to update the agenda without the things that discussed awesome. So I think we are on time uh well, thank you, everyone for joining, and I will see you around.