GitLab Delivery Team, 11 Jul 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-07-11 Delivery:Orchestration demo - EMEA/AMER

Description

Featuring a demo of a possible solution to use an MR to sync repos following security releases

A

Okay welcome, so this is the 11th of July. This is our email, India America's timed orchestration demo and we've got Alessia article not showing another live stream, but only because it's got video well internet troubles, so videos are off um also I want to just mention as well that we only have one more demo uh of this time start in this quarter. So let's keep that in mind as well as we sort of plan actions and think about next steps.

A

um But Alessio. You have got a demo item.

B

Yeah I do so. Let me start by sharing my screen and hoping the thing breaks, but yeah. So here we go.

B

This is more of our walkthrough than a real demo, because in this condition is not really easy to actually I will show what happened so uh yeah, that's the thing right. So uh uh how fast can we do uh merge requests a real gitlab, merge request that does Master security to master canonical sync.

B

So yesterday, I was testing. This and I actually went down to a seven second pipeline, which I think is a remarkable outcome.

B

um So what are we looking at here? So what we are looking at is um first, there is a merge request, a real getlab merge request, which is this one. This should be in the screen right now, which is quite simple in terms of changes, but it basically what it's doing is um is introducing the ability to change the shape of the gitlab pipeline.

B

If a merge request is coming from the security repo and is targeting the default Branch as a source and as a Target, so this clearly identifies our Master to Master Sync from security into canonical and has the City I would say: Advantage is not the advantage.

B

It has the guarantee that everything in those branches have been reviewed by upsec, maintainers and everything, because we are talking about things that are already in get the club, so we're nothing talking about uh someone contributing something from a fork or from a feature branch and asking to be merged in, but we are really identifying a very specific thing, something that is on security, mustard, that wants to get into canonical master and because both branches have uh cannot be pushed onto. They just go through the merge request, uh code owners, approval and everything we are.

B

We have very high standard on what is on those branches. That's the point Okay. So if these conditions are met, so this is emerging- is coming from Master branch on security that wants to merge into the master branch of gitlab, then we can play this little game down here and say we are including this skip yamo, which is something it's um it's um it's a yamo ci-fi that is already in the Pro that is designed to have a green pipeline. That does nothing.

B

It basically prints out a message and we are into including this only in this case and if it's not a security canonical sync, we are not including the regular pipelines. So this, basically this acts at the very top level of what defines the the the overall merge. Github CI yaml file, we're just changing the inclusion, we're removing everything, and only including a special file that that does this no operational, no operation job. So, as we can see here, this merge request here, which is a regular merge request, has all the regular uh long pipelines.

B

So all the jobs are here, but then we are testing this uh here. So that's the basically uh in order to test uh I, had to change the behavior and removing the CI default branch and putting a feature Branch here so that I'm can I can test this and show the end result without having my change merged. So, basically, uh now that I've shown this. uh If we go into the overview you, what we see here uh is loading. So basically, this is a cro across project merge request.

B

We see that the source here is coming from security is actually the test. Ci sync and the testing CI Branch. It wants to merge into a master, and this is canonical.

B

Merge request on gitlab and the pipeline.

A

We're just losing here, but at the moment it's here.

B

Yeah, when computers want to load- oh, oh yeah, the connection is table.

C

B

C

Okay, so we can see your screen, but we cannot hear you Alessio, but I think we are getting the general idea of this.

C

He dropped oh he's back.

A

I'm gonna, try and.

B

Summarize my back.

A

A

I think we have the gist of it. Let me try and summarize what I think yeah.

B

Yeah I I, just put a link, I've put a link in what I wanted to show at the end, so the thing that the final thing is that the job there uh uh that's the only thing that is in the pipeline and is kind of saying this is um this- is a merge sync for master, and this is a no no operation. uh Ci now.

B

This has some advantages and some disadvantages that are interesting to to discuss because it it scales with a number of changes, because, basically, the merge is being performed by GitHub itself and that's good. It also has another I call it benefit, which is uh even though it may not merge immediately. For other reason, it already exposes the content of the security releases, because anyone that now can go on that merge, request and see the changes that are coming through.

B

Basically, what we are doing is we are leaking the security release with that step, which is actually what we want to do, because we want to close the security release by merging. uh So that's good as well. It's okay! uh There is the problem of the code owners and approving that thing, which is a problem. uh I mean it can be overcome by having uh a Bots that can approve everything, which is something compliance will not like, but at the same time compliance don't like having a body that can push the master so.

A

I think we have to balance right. We have I, think we'll have to find a compliance process for Bots because at least for example, will have to exist. So it's a problem, but it's probably not a bigger problem than the one we already have.

B

Yeah and I wanted to point out uh final benefits that I realized when I was working on this, which is in case of a conflict of something that breaks so it. Let's say it can't merge, because there is a conflict, so this will clearly show up in the UI that there is a conflict. But the nice thing about this is that the regular security merge drain process will fix the problem itself.

B

So what what I want to say here is this so because this is a merge request if the source Branch so security, Master advances, then the merger request refresh itself. So, basically, if there is a conflict, the solution to the conflict is doing the reversal merge. So you want to incorporate canonical into security, fix the merge, and this will also sync at the end.

B

Basically, because basically, the the regular merging process will keep running will identify the the merge conflict will open the merge request, the way we are doing today or if we can improve it, the better and then as soon as that thing get merged into security Master. This merger request will auto update if we were able to have it pre-approved and set as merge one pipeline succeed or something like that. It will just merge itself. It will rerun seven seconds and done it's. It's closed.

A

So delivery valid code owners to approve this Mr.

B

No, we are, there are two problems here. One is that uh there is a blanket uh approver for almost everything and um I think we are not there, but that's not a problem. uh The thing, though, is that there are special files that require special approvals, one being code owners itself. Another one is I, think the the Italy version file, so it is possible and that this is what happened.

B

There are a sub pumps that have there are overriding the the general rules, so basically, for one of those we need to have the bot in there.

A

And you're proposing this as a uh better approach compared to the merge train. Is that correct.

B

Yeah, basically, yes, uh yeah.

B

So it really depends on how much we want to build on top of the emerge train, because the merge train, as is works, and so I say we can keep it as is, but instead of doing improvements there, we can try to build this and have this either long-term solution.

C

Yeah I think uh dissolution scales better. When it comes to the git traffic, it removes the requirement of us trying to find a window in which no one is pushing to master, because it is creating a merge request and it removes the necessity from pushing directly to master so I do think. This is a long-term solution that could be implemented.

A

Do we have any problems unjust, going into the Mr I know on one of the generated Mrs that um we had sort of some months ago? We had too many changes um in the security like uh repo that we needed to sync back and we had those split over different Mrs, but I don't know enough about how that got generated, but is that going to be a possible problem on this.

C

So this one should only include the security changes uh and, depending on the security release size, the changes might be large or maybe considerable, smaller, um but yeah I. Think if we want to implement this I think we should test it, at least at the very least, and also I, like the idea of having just one job for the merge request, because technically it has been approved from security.

C

The pipelines have run Etc, but perhaps we should also ask um I: don't know engineering, productivity or quality if it is a good idea to to confirm it. If it's a good idea to push all of these changes without running a pipeline, but.

B

We are not running a pipeline today.

C

Yeah, it is the same scenario.

B

Yeah I mean because we merge locally and we push right. So basically, the pipeline runs after which changes already on master, and so this would be the same thing actually exactly the same thing. So the the pipeline itself will not have the pipeline and then like Midland lands on master and we'll have its full pipeline.

A

But just to come back on that limits thing like. Are we like, if we, if we did, have a big I, I I? What I can't remember if anyone knows uh about that case that um we had a few months ago, where we had more changes than could go on a single Mr Right? Would there be a limit to how many things could be synced in this one Emma.

B

But do you have a reference to.

A

I I only remember I'll have to find it because.

B

I do remember the um the problem that we had, which is uh sorry give me one. Second,.

C

So are you talking about? We have too many changes to feed in our chocolates yeah I. Don't think there is a problem. I have seen merch requests uh with a lot of changes when I review them, like 40, 55s change, with thousands of lines being introduced and the interface of the merger Quest is a bit slow, because merge requests cannot handle a lot of file changes like when loading the merge requests, the files start to be collapsed and the page takes a bit to load.

C

But the merge request goes through like the ux experience is not the best, but the merge request works.

A

B

But also this is more. This is because this is moving security into canonical, so it's only the concept of the security release, while what we do, which is when we have a conflict, is the opposite which is bigger. We have everything on canonical that goes into security and those mergers just they work they, the the UI, can be slowed. That's true also here we're not planning to interact with the UI, because it this is kind of designed to just go through by himself right.

B

Just it has this simple job that runs and gets approved, and that's it uh I wanted to say that maybe but I don't know if this is what you were thinking.

B

The problem we had was when we were trying to do the the sink in release tools and we had a limit on the number of changes we were fetching and.

A

ah That could be it's.

B

Created the problem so because we were fetching instead of fetching everything just the last uh 50 000 commit 20, commit whatever it was and trying to merge, and if the merge base was not in that range, the the operation failed.

A

Yeah, okay, yeah, it makes sense and I guess then. My follow-up question, perhaps is: how does this Mr that you're proposing get created Alessia.

B

I mean this can be created by uh the the finalized pipeline in release tools, because it's just uh it's a single API call to create, and then it could be. Another couple of API calls to uh approve it and set as merge when pipeline succeed. It's just a matter of the code owner's things. That's that's the real problem.

A

What do you think Myra should we uh like, because, uh basically it is a um does this make sound like it, makes more sense to test and try and go down this path versus doing the um the sort of next iteration that you were proposing.

C

uh The next iteration that I am working on. Do you mean the.

A

One yeah the one about getting the med trained so that it has the um turns on and off and we have all the pieces to adopt it fully.

C

um Well, I think the merch training solution that we are currently using, the one that we just tested it works for the now, um but I I think the solution of the merchok was having a like a single job. It could work for the long term.

C

um Are you think we should figure it out the code owner approvals and all the logic and perhaps tested in some way in a security release, because it sounds like a valid and long-term solution that could be scaled, but it might be worth it to test that.

A

Do Alessia, do you think you could figure out the code owner stuff before the next security release.

B

Which is in two weeks or so yeah you've got a few weeks. Yeah I mean I. Can try I think the hardest challenge will be the having it disapproved, yeah right. So, okay, it's just a matter of having it approved right so that that's the because.

A

Then what we can perhaps do is is try and get that piece um figured out, and then that gives us a choice of for the next security release. We either do the merge train as we did this time or we switch that out and try with this Mr and then perhaps once we've tested both of them, we can actually decide what makes sense for next steps if, if any, are needed,.

B

So uh it makes sense. I want I, also wanted to say that uh probably we can still move forward with my merge request and see what engineering productivity thinks about it, because we we don't do that type of merge request in any case right, so we don't do Pros, so it will just stay there and in case we want to use it.

B

Then there will be the problem of the code owner, which is an it's another problem that and we can try to solve all those things uh on on a side note, uh a couple of minutes ago, Myra have sent your attention the a merge request to fix the the retry logic in in the merge train approach to this problem, because within one line change so already did that and we with that thing fixed we already yeah I mean we're already removing another problem from the current solution.

C

Awesome. Thank you.

A

So if we want to test- and this approach, um does it make sense, then that we pause on the uh any of the outstanding issues on the merge train implementation? I'm just going to have to look at their epic because I also don't know what they are. But.

C

Yeah so I finally started but I talked with Steve and the next step was to actually uh automate the inclusion of the merge train because we have it a bit manual about. Let's have the release manager to trigger a pipeline, and if this one fails, let's have to trigger it again. So we wanted to automate those tiny bits.

C

um I haven't started, but if the plan is to search for a solution and test a solution that could replace this, then it might not make more sense to for me to spend time automating this. So perhaps we should focus on see if this works and test it on an executed release, and if we decide that okay, this is a better solution than a more scalable one than the merge train. uh Let's focus on this, instead of the merge train.

A

Does that make sense to you as well.

B

Yes, it does okay,.

A

Okay, great um so.

A

Do uh let me just say to start to that is that issue you mentioned the last year? Is that already on this epic, since.

B

I think it was removed.

A

Yeah because we said it was long term right so.

B

A

Let's um do you want to maybe just do a little rewrite on that and just make that similar to the one that Myra was working on the other week where we basically say test out.

A

um You know an MR merge and then we can put that back on and we can then have the work, that's related to actually getting to a test point where, whatever we need to do with code odors and whatever we want to do with EP uh and then coming off the back of that test, we can then work out what the next steps will need to be for either solution.

B

Okay, yeah, so is there so when we start talking with uh compliance about this, should we talk with people that are already involved with the the bot ownership problems.

C

B

We're trying to explain what we are doing here and if they think this is better compliance, yeah.

A

Agreed agreed I can I can connect you to um the people that we've been talking to about the other bot stuff. Because for me, this all kind of hinges under we have things that have to be able to be approved and merged by Bots, um which isn't great from a compliance perspective, but I hope we can find a compliance compatible approach, which kind of has this two pieces to it? One is who has access to the bot credentials so it when you set this.

A

If you do set this bot up a SEO make sure only delivery has has the access to this can.

C

We use the release toolspot no.

B

Because it's the one generating the merge request, so you need to which is really what we are doing for the Giselle version bomb. The resources is playing uh two Bots. At the same time, one creates emergency press, the other one approves and set for the for merging.

C

Oh, the bot that creates the merch request to solve the conflicts when syncing canonical to securities also the release tools. But yes,.

B

It is, it is okay, it should it should yeah I think it is okay or maybe, in that case no thinking in that case could be the GitHub, but I.

C

Don't yeah because I suppose we can use the git love bot, the one that is already embedded on git love pipelines to create a merge request and perhaps half the gitlab release tools. But uh I.

A

Think it needs to approve. Only we have the Keller bar is owned by ep right. Is that right.

C

Yeah, but it is already like implemented on the gitlab pipelines in ways that the token is there and the trigger is there, so we can perhaps we can reuse it I.

A

Don't think we should do that I think from a compliance point of view, we'll have a lot of an easier time if this is a delivery, bot and only delivery have access to it because EP are not allowed to um do this action.

A

C

Mean the merging or the creating the merge request.

A

uh Oh, would this be the creating the merge request box.

C

Yeah yeah my area through the approval.

A

The release part the.

C

Release bot yeah.

A

uh Yeah, okay, yeah, that as long as it's that way around yeah, that sounds good.

C

A

And then yeah and then we can um figure out uh like what we'd need to do with with the compliance. It's probably fine for us to test I'm. Sure they'll, probably be okay with us running a test to see if this works and then, if it does work we'll put a process in um so customer.

B

So I hope uh that I'm coming through now, yeah so looks like you're looking at me at the screen. So I'm talking- and you hear me so the that, from a compliance perspective, if they are willing to implement all the things ability either to API or whatever they want. The thing that we are proposing here, as well as what we are doing with the guitar version bump is easier to identify.

B

So if even if the bot is shared, but that bot is designed to only approve Mercury was coming from security Master into security canonical, they could probably see. Did this bot approve something that was not meant to be approved because I mean there's not there's nothing in the future in the product itself that allow them to limit the approval, the use case, but then can be audited right. So it's still better than nothing. Yeah.

A

A

um Across so let me I'll just write this down as an action, so I'll help you out with this.

B

Actually, I I keep cons on the same uh idea. uh Recently I did a contribution into the triage op reactive uh software, which is basically the thing that when we do something on a merge request, it automatically comment or do stuff, because when I was testing, this I found a bug which basically was called the by the bot because I've merged something without running the pipeline. I've.

A

B

One yeah, but it was but wasn't running on Masters. So that's, basically my thing was doing on official Branch into another official branch, and so I got scolded and I didn't like so I ended, I went there and fixed the the automation, but actually it it is really interesting how it works, because, basically it can react to things like approval, so maybe to make compliance happy, we can interact. We can make something in that in that software. That say, if this bot is approved, moving something and this something is not what what we want.

B

The brought to approve, remove the approval or just uh I, don't know if they both can remove the approval, but do something like just prevent the merge from happening, because, even if it happens, our discussion items that is unresolved. This is already enough to to stop the approval and I think, because that that software runs with gitlab bot, it can even remove the approval, because it's just removing it.

A

Okay sounds good uh yeah worth repairing.

A

Oh okay, um so just on a few, so a few of the actions I've just jotted down, whilst you're talking through there so uh Myra, yes, agree, pause on your work on 19458 um I have also just seen that on the Epic we actually do have another issue about using an MR for this syncing lseo. So I've just dropped you a an action there for B for 4B to just consider. Are they the same? Maybe we can close one of those out.

A

um I know we have the bot compliance work.

C

Unless who's gone.

A

So what I think would also be worth discussing is um with so, if you're poised on that task, Myra um I guess like like with say three weeks to go. What are we committing to completing so.

C

I told the last status from Steve, he wanted to focus on the merge, train uh or well on the solution to figure out this problem, whether it's the merge train or the merge request, and to also complete uh the final steps.

C

The final steps, the steps of the finalized section, uh which we have a few issues already there and these issues are very similar to the prepare pipeline.

C

So these were the two items: okay,.

A

Great so you're gonna get started on the final steps. Yep.

C

um Yeah the final step, and if Alessio is able to break down uh the other issue, I can also take some other issues that he creates for the merge train.

A

Cool okay, great um sorry, let me see if I just take I'll take one nine four five eight off the boards.

A

um Is there a? How are we doing the final steps um work? Is it a separate it's going to be a completely separate pipeline? Yes, yeah, okay,.

C

Yeah, basically, security release pipeline is split in two stages: the start and the finalize, and then everything in between is the target. Merging and Publishing.

A

Awesome, um do you know, which is the first issue which has the pipeline creation I.

C

A

C

Also read it on the.

A

Final steps already exists: yeah, okay,.

C

Totally implemented, I think the next one is either the mirroring that goes after the sink, but it might make not much sense.

C

um I, don't remember, which is the next one. Perhaps the versions were creating the versions.

A

The versions are part of the publish all.

C

Right steps, uh perhaps closing the issues that are associated with us, spiritually extracting, is.

A

That move them over as well.

C

Does it move over.

A

Any uh it just literally closes that one issue.

C

Yeah, it literally closes the security implementations issue associated with the tracking issue. Okay,.

B

Is it already done because I was planning to do it tomorrow.

A

It's next stop. Next up.

B

Okay, oh yeah, because it's under it's the the one in the ready EQ, the only one in ridicule.

A

I think okay I've put that one on the board so that one's definitely good to go. We've got the um we've got lots of little steps that are probably worth thanks. We've got to notify security release is completed.

A

So if these are all quite small, then um do we feel quite confident that we will have a final steps pipeline that we could demo in the next demo so like in two weeks time.

C

I think there's something that we can show definitely well.

A

I guess I if these are all small, what makes you hesitant to say or be done.

C

It's not that they will not be done it's how a we can demo, because we can test it on a security release or we can just create a fake pipeline. That just shows the jobs, but basically does nothing, because we are not in a security release. Data.

B

C

Yeah I mean we can test it, but testing. We need to create certain environment like a fake tracking issue, a fake slack Channel, a fake Omnibus tasks. So we are testing it, but we are not testing the actual style I.

A

See: okay, okay, yeah makes sense. Okay,.

A

Cool okay, so are there any other things? We should talk about.

A

No okay, um Myra for you as well. um There are Leslie I left a comment on the LKR issue, so um I didn't want to bring it up too much on this demo, because I'm, assuming you haven't, had a chance to read through it yet so have let's take that async and if we need to um grab some time to discuss things, then we can just pop in and ad hoc um sync.

A

Awesome: okay, great, in which case thanks for doing the demo um Alessio and thanks for the chance and good luck with the final steps pipeline.

A

All right. Let's go.