GitLab Delivery Team, 26 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-04-26 Delivery:Orchestration demo - APAC/EMEA

Description

Demoing the current status of release environments

A

Awesome welcome uh so this is our APAC email, timed orchestration demo on the 26th of April and welcome our first item. uh Graham, your demo.

B

Sure so I'm just going to share my screen. Everything will. Let me is this going to work perfect cool, so what I wanted to demo is basically where we're at with the lease environments. So what can I'm looking at here is the release environments pipeline page I'm, just looking at all pipelines that have run on master and we can actually see we are having mostly green, not quite sure what that, but that one failed, but we are basically having green pipelines.

B

uh Oh, that was canceled, that's right, um which means these are actually all coming from two stable branches. So I've done all the work, all the fixes and backboards and everything now into 1511 and 1510, so some of these pipelines are actually coming from them. So it doesn't surprise me that the release tools, bot, is doing some work, although that's just the renovate warming once a day, I should say: source I was hoping, I can say it wasn't scheduled but anyway.

B

um So if you look at some of these, for example, like you can see here this like if I look at this pipeline or say this one from an actual gitlab developer- and we follow this back up. So this is the deployment you can see here. It's actually deployed to 1511 stable and then it commits back to change. But if we actually look, this came from and we'll look at this one as well, where these came from Dave came from this one was me merging or Steve I should say, merging effects.

B

I did um oh did that work, click the wrong button yeah. Here we go. Here's the stable range pipeline, so the job on the the actual, stable Branch itself and the change was merged. uh We can see there there's the actual start release environments pipeline, which is what we were just looking at, uh so that was 1511., and this one is a cherry pick, for example, into 1510. You can see here by these two Downstream This Is Us, building the container images- and this was us actually deploying it.

B

So you know we can click this and we can.

B

We can see here that it's gone, triggered the deploy and because the deploy was successful, it committed back to change to git so that the get State matches the deployed state, which we can also see here. If we look in the commits of the Repository um you'll, see these update environments update environment skip CI. This is basically that commit back working.

B

You can see here it tells us which pipeline it came from, and um this was the first one that was deployed to 1511, so it used so I originally deployed the 1511 environment with the last version of 1510, because I got myself an interesting situation where I wanted to get the 1511 environment ready before we made the branch, but I had nothing to deploy to it because we hadn't branched yet so I just put the last version of 10 on there and that's it getting upgraded to the first I guess the first commit on on the 11 branch uh I mean we can follow that and once again see this and.

B

Yeah there's the 1511 step employee commit deploy um yeah. So that's all working the.

B

15.11 techn9, so 15.9 has an MR open um to also get it up and running and fully into this process. However,.

B

um So this is the Mr. It's been looked at and approved.

B

um By Myron Steve, however, we're having problems with the packaging QA job on this pipeline.

B

um It's failing quite badly and I'm not entirely sure why? Because none of my changes touch the application itself, so I've tried to pee out to QA of peach quality and we're trying to figure out if we can, what we can do to kind of move this forward, because it should pretty safe to merge. But I do also feel a bit nervous about merging this when um packaging test is is uh in such a broken state. But anyway, this.

A

This also may not matter as much right now Graham, because um this version in a month's time will drop out of security support. But actually we are um since we're not going to go ahead and extend the maintenance policy in the next month.

B

um Yeah, it's actually actually.

A

This version will never, even if we extended it immediately, like you know in a month after this version won't come into that, so it.

C

May not be doing this now.

A

C

In a week, right, yeah.

A

Sure, but isn't the problem that.

C

We and that's the problem with the stable Branch, so the security release will not work. Oh.

A

C

A

Have other problems like yeah, exactly I meant more for this specific yeah.

B

A

Absolutely we should definitely get the um well.

B

Yeah, it's unsure why this is failing, because this is very curious, because all the tests are failing with an internal server error and this is against the um just the little Docker image, install I believe or it's an Omnibus or something I'm, not I, can't remember what it is, but it's something that runs in CI so anyway,.

C

That that is actually kind.

B

Of I guess in a bit of an aside once I um once I get that Mr merged. We could in theory, then have the last three stable branches, um basically automatically deploying to a release. Environment, um I'm also happy, obviously, with the it's potential focus on when we do extend the maintenance policy. I could just drop 15.9 and just say: we'll only do release environments. The last two releases just for Simplicity, like.

A

B

Picked three because the original intention was extending to three, but if that's, if that's not happening, we can just do release environments for two. We can do school for one. You know it's. The choice. Is ours, um I? Think it's nice doing it for anything. We kind of do think we will have um that's what's going to just because it does it's. The point of it is to slowly build confidence in the the pipeline that you bring I guess exactly.

A

And what we will probably uh we need to figure this out, but what I, what we may end up doing for MacBook requests is even if they get approved saying to the uh author. Yes, it's approved, you go ahead and do the merge and using this same process um would probably not be a terrible idea. You may have to think about that.

A

Do you know Graham so I know you've got a stack of other stuff coming on uh on your plate already, but do you know in terms of that stable branch uh failure? Is there an issue or is there something that.

B

A

Can make sure that, like there's an actual dri that maybe a quality dri getting that up and working.

B

So all I did was follow the standard process, which is a paint quality and said. Look I've got a backboard here, that's failing thing and Dave. Sorry excuse me, they're um they're kind of I think is as.

A

Much of a loss.

B

A

My Mr is the only one: that's failing. Okay, it's just that one! Okay, let.

B

A

Up with Steve and just get because I'm not super worried about your Mr, just because, of course, maintenance policy stuff. But I am more worried about the point that Alessia just raised, which is, we do need the stable Branch to be working with tests. But let me let me get Steve to um worry about that. I mean all.

B

He should need to do right is just literally go into their pipeline, create screen and run a pipeline on the stable Branch now and see if it comes Grand.

A

Because either either.

B

It's green and then, in which case there's something funny with my Mr who knows, or it also goes red and in which case again, let's get ahead exactly because, um but.

C

It will not generally the same result because the same so we do, we need green pipelines on the backboards for the security release and those are merger's targeting stable branches. Yes, and if we run a pipeline on the stable branch- and it is.

B

A

But let me uh I do think yeah. We probably want to look into it, but let's get uh I'll chat with Steve and get him to to take a run on that one. uh As he's the release manager also sounds like today, with the staging upgrade. He hopefully has a really quiet day coming up because he probably hasn't got a staging environment to deploy through. So actually it might be a good day for him to look into this.

C

I have a question: yes, so you were mentioning the generation of the 1511, a reasonable environment. If I remember so that that's the question that I have is uh how much admini work do we have in this release environment to actually spin up new things, because uh I remember on the day we did RC 42 that I was kind of looking for the the new release environment to be there right, some kind of regenerate the separate Branch so wow. This is the first time we have real RC without having overseas right and it wasn't there.

C

So I wanted to ask how that I. Suppose there is a manual uh thing to do. Maybe creating the yaml file or something like this, but is it documented anywhere.

B

Yeah, it's no I've got it, it's I've got it to do and that I I was hoping to have it done well. I wasn't sure if I was going to get deploys working by the time that it basically.

A

B

It didn't fall out as I expected, but yeah by then by 16.0 it'll, be at the moment. It'll be a manual task, I'm, happy to automate it because we're using Jason and schemer and like we've, got very strong schema support. We should be able to automate it very quickly if, if there's a place so I don't know, is there like a piece of automation when we create like I'm.

A

Trying to figure out is there a.

B

Place in release tools, I could automate, like I, could put put this into like even just open an MR that there's just a release. Manager clicks, approve kind of thing, I'm guessing it's like in the tag. It's.

C

The public release classes, so there is a okay set of classes in release tools which goes under the name of public release and are all managed to do. They do the monthly release overseas, everything that goes on-premes and not on.com, so there you can hook into the yeah. That's into that thing, uh I would say if you want to automate, don't do an MR, but just go with the the generation of the I. Don't know what what you need. Maybe it's just a yaml file because.

A

That thing is automated right.

C

So there's no there's no click or approved things in that process. The release manager say create rc42 and it will do everything.

B

Yeah yeah sure, okay, all right cool, so okay.

A

B

Enough to kind of feel out something obviously I'll have to look into it and put up a proper RMR for review.

A

But I feel like.

B

For the moment, I'll make sure it's a at least a manual or something. So if, if I get distracted and don't get to it, that um you know it's done for the it was made sure people understand to do it for the next time. But.

A

Yeah, ultimately, I think we.

B

Should it should be relatively easy to just automate you.

A

Know it's you know just to get that API call and.

B

Passive some stuff and then okay yeah. Once again, it will be the question of you probably want to do it before the stable Branch itself is created, so maybe there's a bit of trickiness of what version We deploy, but even just deploying the latest stable release so like if it's 16 we deploy 15.11.4 or whatever it is, and then you know- and that gives you that means your first pipeline is also like a Mini upgrade test like going from the latest one to which also it's probably not a bad idea to slip in.

A

Yeah, okay, uh sorry alessia's! Behind on my typing, uh what did you say that thing was called uh we're going for.

C

Public environment, public release all.

A

Right yeah sure awesome cool thanks.

A

Just as an overview game, because I um for for this recording, like so roughly on the released environments, um did you just give like a very, very high level overview of like what what happens? So someone commits um someone's trying to commit to this to the stable Branch, like all of the pieces.

B

Yeah, so this work will at the moment it only happens when someone actually successfully merges to a stable Branch, so any Mrs to a stable Branch doesn't matter. We could expand to that use case later, but I think there's a there's, a different level of problems with that. So someone has got a review they've merged to the stable Branch. We've got our own child pipeline in the gitlab model, mono mono repo, which is where these stable branches live right.

B

So now in that repo there, for example, there's a child pipeline that will um set up a review environment if it's on an MR and now we're saying, we've got our own childhood pipeline. That says when this is a stable branch that matches the stable Branch pattern. Add this other pipeline, which will do the following things: it will do a trigger into the CNG repo or a mirror of the CNG repo.

B

That's used exclusively for these uh non-official builds and currently there's a mirror, because um you know, distribution, we're just unsure of generating new builds in the official repo I think they just didn't. Have enough background information to feel confident with us just directly, diving in and building into the official one, which is fine and changing that later is easy enough, so our pipeline does the goal is to have three big things: build the images deploy them test them. So we've got the first two don't know.

B

So we have a child a trigger pipeline off into CNG, it's kind of like a mini, stripped down version of Auto deploy in a way or that's. Why I'd like to see it end up looking like, um but yeah it just goes build.

B

The images comes back triggers release environments passing in the data it needs to deploy, because now the images are built and can be deployed, and then at the minute at the moment it just stops, but eventually there will be one more step which is running QA because QA setup they do the whole repo per environment, they're not really Dynamic, and things like that I won't be doing a trigger to a quality. Job I'll actually be running the QA Gem and code mice in an actual job in the pipeline.

B

um You know in the future of quality, do offer what I guess what I would call QA as a service where they they could make it a bit more dynamic. Then we would just trigger into whatever whatever submission they had there.

B

C

Does this mean the the running, the QA as a from the code that we no longer have the problem of the?

C

um What was the if you remember, I'm, trying to remember as I speak, so we mentioned that in this setup, if we ever add to backboard a QA fix, we will not be able to have a green QA pipeline, because the QA select the the the the docker image they're using or whatever is downloading the The Code by himself by the last stacked release on that environment.

C

So we say if we backboard something and we don't until we tag, we will not be able to run, and so is this thing that you mentioned running from code, uh overcoming that limitation. It should so.

B

I think I figured out a way where, because we already built the QI image earlier in that stagger Branch pipeline, so I will wire that up. So it's actually that code, that's getting run so yeah. If it's like, oh wow, here's a flaky test. You just do another Mr to the stable Branch. You merge it and therefore the Q, a the QA, is one to one. So it's not like. Oh yeah, yeah,.

A

B

That problem, hopefully I'll, put a big disclaimer, hopefully until I see it working, I won't say it's fixed, but from what it looks like that. That should be it and that's what we that's what we want. We.

A

Want to be able.

B

To say whatever the QA state is on stable branches. um That's the test. We want to run, which does mean that when we do this and start to see QA failures, we should be able to immediately go, and we because I mean at the moment the packaging QA I think what happens a lot I could be wrong is when things go wrong or flaky people sign off, but they don't fix. If that makes sense. They're like.

A

B

Know, that's like you, ignore it and just continue on, whereas what we can do is we can say well because you can, you could actually fix this and we would deploy it again and make sure it's green this time and you.

A

Know actually get you know what I mean like it's not just saying you.

B

Ignore it, it's actually getting the back points into fixed QA, the fixed QA.

A

Nice, that would be a huge I'm keeping my fingers crossed for that, because even just that piece alone will be a huge Improvement yeah.

B

A

B

Great, like I, understand, I understand what the current setup of how it is, but yeah it isn't great where we're like. Oh yeah, we just know this test was bad and there's nothing. We can do.

A

So exactly okay,.

B

And it's for poor quality. It's a lot of rubber, stamping like even when you ping them. You have to Ping them to get them to rub a stamp. It.

A

But you both know.

B

They you and them both know that you, you know you.

A

Both know it's a known issue: okay, yeah, exactly cool okay, great stuff, um and we are still sort of ironing out all the kind of next steps, but um on the maintenance policy extension we are going to um hold off on making the actual extension. A little part of the reason is.

A

We would rather have more release environments um up and running before we do that to reduce the workload for um Quality, but it's not only that um so from uh the sort of the things I was thinking of it's really about the unknown workload that it will generate once we start having three patch releases for um for release, managers like yeah we're already overworked. So we have to deal with that first, uh but actually, Sam has also got some questions really about.

A

um How much would this extension affect upgrade Behavior, so he's going to do some more investigative work on actually the the business sort of side use case and chat to some users and actually go and do a bit more investigation on uh on. What's going on over there so or you'll go ahead, we've got the um patch release. Process pilot is running through this milestone and before we get to the 22nd, um we should adopt this process um and actually just get this like locked in and have an easier.

A

The stats show that it is quite a lot easier for us to do with the without having to do the merging, which is amazing.

A

um So we just need to wrap that up basically get that documented. There's a few tools tidy up, there's some tech deck clean up tasks that need to happen um so I'll catch up with Myra. Later we'll get the Epic organized to show that stuff.

B

So does that mean like that's.

A

B

Sounds good and let's say that happens, does that mean we would still draw the line and say 15, um 15, 10 or 15 11? What was the pilot for 15 10 or was it for.

A

15. we did a pilot on 1510 we're now doing it for 15 11.

B

But so will we say 15, 10 and 11 is developers merging, but 15, nine and earlier should any back. Parts need to be done. That's still on us in this the old way, or do we just open it up for all of them to be developer, learn I.

A

Think, that's probably to be figured out um yeah fair enough, but you'll probably do it on a bit of a case-by-case. I, it probably makes sense to open it up, um but it may depend on on that Mr uh from your siteground I. Think at this stage, probably whatever is easiest um is probably the way to go. I think all of these things are about workload driven so yeah.

B

A

um We could just do it on a case-by-case basis to be honest and actually see if we get something that's older, um we could maybe just decide at that point like. What's going to be the quickest way to to pick this up actually accepted.

B

But it definitely sounds like 15 10 onwards, yeah. Well, just that's the process now yeah. That's.

A

Right yeah, um so yeah feedback from um engineering like across engineering has been largely no no specific complaints, I think everybody's been reasonably happy with the changes and I think from a release management side there's been like it's much easier for us to just not have that big delay. So that's great.

C

Yeah things from engineering is more predictable. They know when it's done. It's done it's just a matter of when we release it, but it's definitely done. There's.

A

C

Gonna get back to them and say: oh if.

A

Yeah or things like that, yeah yeah, exactly that's it um so yeah, we'll uh we'll, try and get that stuff tidied up. But hopefully we can just you know, lock in what we've got and and move on to some other uh other projects.

A

Is there anything else that would be good to discuss today.

A

Yep: okay: awesome have some time back thanks, very much uh go to church base and thanks for the demogram and.