GitLab Delivery Team, 11 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-01-11 - Delivery:Orchestration Demo

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

So welcome everyone. This is January the 11th 2023. This is our um emea, America's timed orchestration demo and, let's see so, we upload demo items today. uh Steve you have the first discussion.

B

Thanks um so I worked through some issues about stable Branch configuration, um and this one specifically is regarding managed components and what I found is. There is a bit of inconsistency in the different settings that these uh projects have for merge, request, settings and just a repository settings uh who can merge and um and who can push, and while this doesn't have a huge effect on our ability to extend the maintenance policy, um it does present sort of the opportunity to have differing Behavior.

B

So when we're trying to debug something it's, you know, we have to maybe re-look up some of these configurations or understand why certain things are happening um which causes extra time spent. So so I'm kind of just wondering if this is something that we should look at standardizing more or if there's certain types of things that we might consider foreign.

C

So I think this is a very interesting point.

C

um There are, there are other things like this, like the um all the compliance work right so I know the compliance were checking about configuration across projects, who's allowed to merge where push and all those type of things uh so and all honesty.

C

What I see here is kind of um to me looks more like a lack of a feature in the product, so there's an opportunity here, a huge opportunity to have some sort of an Enterprise policy enforcement type of feature. We're going to say those projects belongs to a certain Enterprise policy and even maintainers are not allowed to change those uh those settings. That being said, so this is kind of a strong, long-term desire.

C

That being said, what we have here is that, uh if you are maintainers or one of those projects, you are allowed to change those things. I, don't I'm, not even sure how many of those are logged through the audits logs and so how, if it's possible at all, to figure out that something changed and who changed it. So I I think compliance is running some sort of automation which I don't know if it was some sort of one-off and generate a report out of this and generate those issues where we are all involved into.

C

This project is not configured properly according to compliance and I, don't know if someone is going to change something if this will be detected by their system or if it was one-off right, so an option could be to figure out what they're doing there and if there are opportunities to leverage something like this another thing that we were doing, but it's a bit different for mirroring.

C

We were toying with the idea of generating mirrors using terraform, and so we have a terraform um repo telephone project on Ops that is taking care of mirroring, and the idea there is that you are. You are not supposed to change the project configuration because the the thing will just every time it runs. It will reset everything according to the the the the state that is supposed to have, but yeah. If we don't run the script, the the thing will simply diverge right.

C

So the it's not designed to keep a system consistent is that one is more design as I have a project and I want to have mirrors all those mirrors must have a given path in a given name given schema and we want to propagate configuration from the canonical to the other one. So we kind of just generate those and create tokens and mirroring, but it's more about applying it's one of our one-off uh configuration more than making sure things stay aligned.

C

C

Long, the short answer is I, don't know the long one already so.

B

Okay, um yeah I mean I think that that long-term solution that future sense definitely really nice um I'll search around and see if I can find. If there's anything proposing anything like that, already um yeah I think the the one area that I found to be the most like sort of like messy it was that uh I think the the mer yeah, the merge request can have or merger class uh settings can have settings uh regarding approvals. But then the repository also can have.

B

The repository settings can also have settings regarding approvals, and um but this shouldn't.

C

One is that should remove the ability to edit uh merge, request approval at merger's level. So if something like.

D

C

In place somewhere means that this is outside our compliance. Rule.

B

Gotcha, okay, um I'll take a closer look, let's see if if there was any of those cases, um yeah, that's some good context. Thank you. Some.

C

More of a intermediate solution could be um exposing those or let's say we I'm not familiar with the issue. Okay, but let's say we we're checking three four things like the has this type of settings: yes, no, we could even consider um doing some scraping this like some sort of a matrix, so when in delivery metrics we already have are scraping something.

C

So we may consider scripting settings of the repo under say: I, don't know what is that manage the versioning or whatever is the thing that we want to check and surface as a zero or one metric and say this. This repo does not uh comply with the rules that we need and from that we can create alerts because it goes on Prometheus and then yeah. That's the idea right. So if something diverts from the expected configuration, then we can uh notify using the alert manager of Prometheus.

A

B

Thank you, you're welcome, um I, don't think I have anything else on that topic, so I'll send it to Myra.

D

Yep, thank you. Steve I just wanted to take advantage of this goal. To give you like a nice status of the maintenance policy work, uh so this work is basically split in I'm gonna call it five phases. uh The first one is regarding building or having enough information for the last for the partialist pressure. This was one cup was completed and we can see the end result on the release.

D

Manager.Com, which reflects uh like the overview of what is the battery is pressure, and then uh the battery is pressure to air version and perseverity as well.

D

um Since we just perform um a patch release and we are doing some pattern releases right now- we don't really have any pressure. So that's good to know the next one is about our tooling adjustments. It is basically making for these tools to be compatible with extending and making patch releases to account for three versions, and that one is almost there. We just need to make some templates that just means that depends on some links, but it is basically minor work.

D

uh The other one is the gitlab configurations, which is what Steve is currently working on. It is making um or more than making it is preparing the git Love Project settings for maintainers and developers to merge changes into stable branches. We have introduced some guidance and we are enabling settings to make them safe and the other one is the quality adjustments which is currently in progress for the quality strategy. We are waiting on quality to get back to us to discuss what will be the best quality strategy to use to validate and release packages.

D

Our proposal is to use the packaging QA pipeline or the patraction keyword test pipeline. I, don't recall the name, but we are waiting on the response. We are also uh collaborating with engineering productivity to set the project to set the process for broken stable failures, and to summarize, we are going to rely in the current process that we have so far, which basically is the mitigation dri or the three hdri to cherry pick failures into stable branches.

D

uh So that is what we are going to rely with some perks like announcing the failure or surfacing the failure in the releases channels and probably being the emerge request outer to notify that a failure was introduced in a stable branch and.

C

I have a point about this.

D

C

Came to my mind uh regard because we were talking about the patch pressure, so we may consider if we don't already have having a label that will not count those cherubick into the patch pressure I'm thinking about. uh We know there are uh only error. Spec fixes things like this: a known flaky tests, and things like that.

C

So if we are asking the quality to proactively uh back back Port those those uh tests, then we we may end up seeing a higher pressure than the real one, and maybe you decide to make a release that only contains a spec.

D

C

D

It does but I think we are covered, because those multiples are Blackboard wouldn't have like a severity labels, so they are not going to appear as origin. They will appear.

C

They will, but they will be in the known right, so yeah.

D

Exactly which I think from the release perspective I suppose we can have a lot of those, but there is no really urgency in trying to do a patch.

D

Well, I think it will be up to the release manager right there, yeah.

C

I just had to see an extra step, I mean it. What I'm afraid of is something like this. Let's say we have um uh broken stable, broken Master. Okay, we have a broken Master broken Master is a priority. One task severity, one priority, One everyone all hands on deck, so that's the way the master Mr uh get fixed and then someone's okay. We have to do the same thing for the stable one, stable Branch, because we know was affected and we kind of do.

C

What is the slash copy metadata thing, so we are just applying blindly the same labels on that we had on the original one right so and then this will show up as some sort of uh high priority thing.

C

I mean it's a nice Improvement. We we can live without it, um I don't think we will ever reach out the point when we are going to make a release. That only consists in aerospec failures and broken Master back posts. Things like this I, don't believe, but yeah.

D

C

D

Think this could be like based on the observation. If we notice this is happening, we will need to do something about it and also I. Think uh patch releases are going to be automated, but still I'm sure as a release manager. We are going to still review what has been Patches at some points and we are going to notice like hey. There is just a bunch of our spec things to be patched. There is no really need and then readjust the labels.

D

um I do agree, it's an improvement, and this could be like a follow-up to try to improve this process.

C

D

Thank you, yep um Steve. Do you want to verbalize your question.

B

Sure so, um with regards to the broken, stable Branch process, um so one thought that I was thinking like with the dri being the merger Quest author um and Then followed up by possibly the release managers. So if we have the case where stable branch is broken, um let's say they've merged all of their back ports and then either that you know that triggers them making a change to what they did or someone reverting it um like. What? What is the expectation for the other matching map back ports or how do we?

B

How do we make sure that that fix either goes across all of them or they all get reverted.

D

Yeah I think that is a good question and it might depends I, don't think this is a black and white um response or process. It might depend on several variables, for example, if this is a S3 or S4 and suddenly fails in just one stable, Branch I think it can be reverted and one enjoys that branch and then we can continue without it on the other branches.

D

um I, don't think it is a big deal to just revert that and continue with the process in just having just the other two branches, but if it is an S1 and we just revert and it is causing a problem uh well, we cannot continue without it and with the broken, stable Branch. So in that case, I think we will meet a fix and we will need to wait, um but yeah I think to answer your question. It might depend on some other variables for the patch release.

B

Okay, so if I'm hearing that right, then then it will be okay, in some cases to release two versions with effects and one version that we took it out of because we reverted it. It.

D

Might be, it has happened for security releases, okay,.

C

Just something more elaborate: on top of that, which is, uh we may also try to aim for a situation where making that patch release is a trivial operation and if something doesn't work on a given stable Branch, it means that it need that specific fix is not back portable and needs more work, which means we can just release what we have and the next one will include it, and we even to the point that we can simply drawn a patch release with only that single fix if it was important enough.

C

So right now we are coming from a situation where there's a still quite some manual work, so we may feel not comfortable in doing something like yeah. Let's decide to do one now and one tomorrow, but if we reach the point where it's just a matter of running chat, ups and waiting for things to happen, we can even say yeah. uh This didn't make today, but there is great benefit to releasing all those fixes to the customers that are running on, because what I'm thinking is that is, it's likely that those will fail.

C

On the older version. I mean you develop on Masters, so likely the the the the closest stable branches will have the same. uh They have. They looks like mustard, so they should behave more or less like Master as some, maybe something back behind that. Maybe there's some missing feature or something in between that affects the result of the merge request.

C

D

Okay uh and then once we have completed these all four phases. Well, the battery is the touring adjustments, the gitlab configurations and the quality adjustments. We are going to be ready to start testing. I am thinking of at least um well I'm gonna put it like three spaces for testing now, the first one will be some sort of try run to test or tooling.

D

That is working correctly, with all the commands that is generating the right, template and etc, etc, and the second one could be some sort of beta test in which we are going to ask some developers I'm thinking of asking some developers to perform a patch release and allowing them to merge into the stable.

A

D

And perform a three patch version with a new process, just to get a sense of how it feels and to see if we can spot any holes and then basically to Roller out to well the rest of the engineering department by now. Of course, we should have all the documentation, ready and everything prepared.

D

So that's it. Does anyone have any questions.

B

So so the beta test would result in a in a in a real release.

D

Yeah I'm thinking we can do like a patch release for three versions.

D

Probably we're going to cherry pick, something that is easy, like a spec change or a documentation change or about fix, depending on what it is on the backlog.

C

um I have a question, so are you thinking of involving some of those Engineers working on those fixes and asking them if they're willing to participate and so creating a say, a smaller uh patch release with say some some feet? Some some bug fix across all the tree or to uh say take a look at whatever our today is labeled. With pick into last three version make a selection and just author, the things by ourselves so kind of removing the label on their merge requests and creating the backboards. So we have interior.

C

We have enough Engineers to create, merge and so have a more I'm thinking more about. We have more control over it.

D

uh I think ideally, I would like to ask Engineers Engineers outside our team, to take a look at this process because to get a fresh perspective of how they feel I think all of us are very familiarized with what to expect and what to do. But I would like to have someone that is not familiar with this process to get a sense of how they feel and what is the process from their perspective.

D

If we can find any person that is available because of reasons I suppose we are going to be the guinea pigs in this case, yeah.

C

What what can a concern is a big word here. What came to my mind uh about this is that uh when we're talking about dry round to validate our toolings, it's uh in in the new world, it's more about validating that things on the stable branches will end up in the release, so it's kind of after the new workflow for Developers.

C

So what I was trying to so my idea was more about before putting the new process in front of Developers. That will may have their own question about. Why we're doing something in a certain way and then also surfacing eventual problems with the tool themselves was more about. Let's test the final part of it, which is if something is on the stable branches.

C

Will we be able to create three patch releases at once, and this is all about orchestration, there's nothing for for development here right and then, if that piece of work is okay, then we can say: okay, now development, the we can do this, and in order to do this, we need you to open the backboard, merge, request and work with this new workflow. Would you mind, participate in this and then doing uh so? We are not mixing the because one test is about, does is the tooling working and another one is?

C

Is the process reasonable for backend Engineers there's something that doesn't work as expected? Does it create more complexity, and so that that's why I was uh thinking if we may want to split the two tests.

D

Okay, so if I got your idea right, the first test would be to actually merge something in the stable branches from our side, then that's not the whole workflow like uh functions and then open it up. The second test will be to open it up to developers right.

C

D

Yeah, that also makes sense yeah, but.

A

I think did I interpret right. Let's have a suggesting almost an additional test. So first one is a dry run. Then a test with our ourselves and then.

D

Well, I think the first one would be like a dry run, not in such a way. That is a dry run, because we are going to end up merging something in the state branches which should be fine. As long as we choose what to merge, it could be a documentation or a spec or something so yeah.

C

Yeah, that's that was not more about the tray run, but more about yeah us cleverly, selecting ourselves, something that makes sense. That is still at a feature, a bug fix or something, because we will end up releasing three things: three versions of gitlab, so people would go through the hassle of installing it and things like that, so something meaningful, but something that we can control.

C

We can check ahead of time if the back Port applies and all this type of things and not involve uh the the developers at all at the stage, because maybe I don't know I'm thinking there will be uh some of those triage or the bot thing that will interact and will complain about stuff right. So, yes,.

D

C

Things that we can Surface ourselves and I will like I mean I, don't want to have to give a bad impression on whoever is testing this on the first run, and so we can just try, try to catch some of those easy win by ourselves.

D

Yeah yeah for sure that is a good column.

A

B

A

Other other discussion items that we should go through today.

A

No okay I jumped to uh final bit, then our collaboration check um as our action from retro so quick show of um thumbs. Are you personally getting enough collaboration to complete your current tasks.

A

A

D

A

Yeah yeah for all the folks. This is this is an orchestration level one. So not just this okr. So yes, personally, do you have do you have what you need from other people to complete your tasks.

D

um I'm just gonna kind of refresh my question: is this limited to collaboration with my team with the people on my team or with the people outside.

A

The library this is anyone, so this is, do you as a person in orchestration, have what you need.

D

um I'm gonna answer with my hand like this okay.

A

Well, I wanna just give us some some some words for that as well.

D

Yeah for sure I'm just kind of luck. In the quality side, uh we have been waiting for the response uh to uh have a quality strategy for this backboard uh first world for the yeah for the patch releases and I know that you talked to wincy and I know that she's she's very busy and her team is very busy. But uh she.

A

Said we should come back to us by tomorrow.

D

So that is why I'm going to answer like this. Instead of this.

A

And actually that's something else to think about, as in the testing thing, because um I uh I think Vince is Keen, that a set will take uh sort of a part in this testing with us um to be able to confirm that yeah. The the test that they want to see running are running so um I'm hoping we'll get a fairly quick response which is sort of a yes. We can go this way and then it will be more of a. How do we plan a test when, uh when everybody's available.

A

Oh, and um what else can we do to improve on collaboration.

A

Nothing, it's all all good I would say probably on this one. Actually, this one was a little unfortunate on the quality because of the holidays, um but I think uh for your original issue. Myra Vinci said that you know she had seen your ping um and it it. It was just one of the many many things that's very hard to get time to go and read an issue so I think probably what we should have done. If it, the holidays, we definitely would have done, is spot.

A

That was going to be an issue for us sooner and get the sync course set up, because actually, once we talked through what we were asking for and what this looked like, then it was all quite straightforward, um so I say that's probably just the the takeaway from this is that we should probably recognize these things like, potentially are going to become blockers sooner. If we can.

A

Awesome: okay, great! Is there any other stuff anybody wants to go through in this demo call.

A

Nope; okay, in which case, thank you so much everyone great to see you and thanks for the discussion items and uh have a great rest of your day, take care bye,.