GitLab Delivery Team, 15 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2023-03-15 - Delivery:Orchestration demo - APAC/EMEA

Description

Demo and discussion about the new release environments - https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/837

A

Okay, welcome around this is 15th of March. This is our APAC Emir timed orchestration demo and we are live streaming privately for now. um So are there any? Is there anything useful to demo or do you want to jump straight to the discussion.

B

um So actually I might have something to demo a little bit to um demo. If you just give.

C

Me one second.

C

C

It should actually thought about this, a little okay. Let me show my screen.

B

Cool, so uh everyone can see basically a copy of the release environment's Repository, um as you can see here, we'll try and make it a little bit bigger. This is just the private Fork I have of it. uh So it's not the actual real release environments. uh So this you can see here it's under my name space. It's the fork of the project.

B

What I'm going to quickly talk about is what I've been looking at over the past two weeks and just a slight changing direction in how we're actually going to get deployments with release environments working for the first iteration of the kind of release environments work.

B

So originally what I wanted to happen is I, wanted the stable Branch pipeline, so the pipeline and the gitlab or get lab repository to essentially do a full I guess, get Ops kind of workflow where it might push a commit or open a merge request into release environments set that merge request to merge when it passed, wait for it to merge and then track the pipeline. That was, you know, running on Main, deploying the actual new deployment.

B

Doing all that automated is it's not impossible, but it is tricky and what I discovered when I was doing. It is the experience with the developer or the person is not great, um and this just comes back into some very big issues we have with gitlab and how we kind of can glue, workflows together or how we can't glue workflows together.

B

If we were to do that kind of workflow, essentially what we'd have to do is um you know you would have a job that would do all this, and so the developers would see some deploy job that would be spinning away and um they wouldn't get the downstream pipeline at all, because you can only do that through triggers. You can't do that from like you can't just attach to unrelated pipelines together Downstream, even though I actually think you should be able to somehow join them together and likewise, gitlab CI has no way of saying this.

B

This job is waiting on a merge request over here to be merged, or something like that. Like we're very limited into the workflow options we have with gitlab CI, it's not a very good general purpose. Workflow tool unfortunate um so yeah, so in the interest of trying to move things along quickly and provide Pro, possibly a little bit of an easier experience, um especially for people understanding. What's going on, I've gone back to the traditional trigger model, as in you trigger a release, you know trigger a pipeline on release environments.

B

You pass in two variables uh the environment you wish to deploy to in the version you wish to deploy to, and it will basically deploy that for you. So that leaves the problem of well. How do you still kind of reconcile that with what's in git and make sure you don't have this kind of drift? And so what I've kind of done is this process of essentially developing the pipeline for release environments in two separate ways?

B

It detects that if there is a an environment and a version set in the pipeline, so they can pass those environment variables. It decides that this pipeline process to deploy an environment is uh basically just deploying one environment. It's going to deploy the version you've passed in and then, if it's successful it will commit that back to the git Repository.

B

So if it fails, the git state is of the old version, so another pipeline coming through won't kind of like try and um because what the other problem we have is right. So if something goes wrong, we don't want all the other pipelines coming through and continually to try and like do the wrong thing. Essentially, um if those environment variables are not set, we treat this like a normal pipeline, just like any other merge to master or merge process.

B

So if I was upgrading updating like something in global values, I do that in an MR I merge it. The pipeline detects that it's just there's no variables being passed. It's just the standard, merge request pipeline, that's being deployed, and it just does the normal thing. If I run a pipeline uh like I can run a pipeline here, but you know it will be from a trigger in the gitlab org repo.

B

If, if I um pass those variables in it goes okay, there's someone's trying to trigger me to do an actual deploy of a new version. um So you can see here. Scarbeck was playing around with it as well, um so I'm, sorry I'm, trying to actually so here you can and I've actually labeled the two types of pipelines. So if you see a pipeline like this, ignore the deploy failures, um that means that this is the late. This is a standard pipeline.

B

That's going off the latest commit to Main, and then you might see a pipeline like this, where I've passed in two environment variables, um you see that deploy tree pipeline trigger deployed version, 15.6.6 to environment, 156, stable, and if we look over here, we'll see that it does the deploy and that it does the commit deploy. So it commits basically the change back. So a little bit of you know, scripting and stuff there to do a very simplistic job of saying.

B

Okay, we've deployed successfully I want to make that change back into git State and we just put skips the eye. So we don't get in a CI Loop where we're kind of deploying multiple times it's rudimentary. It works not too bad. The biggest problem is the security model, because essentially, if you allow anyone to trigger a pipelines onto Main, essentially you're, basically saying you have to give them permissions to more or less be able to work anything in the repository, they can do anything with the repository deploy anything uh change.

B

Anything I believe, like the permissions model. Gitlab has around I, want you to the the ability to trigger a pipeline, but that's it is really not there.

B

All our permissions model is Branch based, so you have like a developer Branch or you could do protected branches, but branching model does not work with infrastructure code because you, if you try and have it a branch per environment if I want to change, say the global values which version do I change the one on Main, the one on all of the branches, and then it starts to spiral out of control. You can't have two Dimensions, unfortunately, anyway, getting a little bit distracted. So that's what I wanted to demonstrate? I mean I.

B

Can I can do it here again? I can run a pipeline and go.

B

Environment is 15, 6, stable and version. We can just make this up because it's not actually doing employment. 15.6.20.

B

You can see here, deploy pipeline should trigger deploy version 15620 to environment, 156 stable this. You can ignore the renovate job. This will take a little bit to run. So that's basically it so I think this is good enough for the first implementation. It gives us a simplistic model that we all know and trust, which is trigger pipelines.

B

It gives us the downstream pipeline. So when a developer is merging to a stable Branch, they will see the downstream Pipeline and we'll be able to follow it. The pipeline into the release, environments, um I, don't think it's the best for security and I do think the model of switching back to a kind of an automated merge request, workflow or something like that. I do think it's something we should look at doing, probably as part of the second second iteration epic for uh release, environments.

D

Thank you dreams, so this is something we have been trying in the past uh with um Helm charts, so the when we developed the helm, charts integration for releasing how to deploy um charts that we've never ended up. Never using that, but we had this.

C

D

We basically implemented the same thing, because the level of complexity of address, of upgrading the the reference inside the charts were so high and was already implemented in the chart itself. When we say we're gonna trigger a job, charts will uh I'm talking about the helm, repo, not the CNG right. This will run an internal.

B

D

The thing was, uh this will run its own internal script, that is capable of figuring out the proper versions and everything upgrading every file and self-commit yeah right. So we were doing exactly the same thing. uh What happened there is that we were not owning both side of the thing we never ended up using the outer deploy the charts, because the change in the chart may break the deployment. So this kinds of things break and we are no longer using we never used.

D

An action is not even working, but here with us in being in full control of both projects. I see uh opportunities to make sure that these things continue working right. So that's not a problem. What I wanted to suggest is trying to figure out if we can work around this, not the right term, if we can strengthen the security model by using um something like an example. So, as you said, uh you need the maintainers of gitlab to be maintainers of the release environment as well.

D

That's the first thing right, so they both need to be maintenance because they have to trigger Downstream.

D

So that's the first thing um and that's okay, but then uh maybe what we want to do is using um uh kind of a bot token in the I. Don't know how you generate the credentials for self-committing, let's say if you're, using a project scope token. Basically what we can try to do is say only this project scope token is allowed to to directly push on Master maintainers. Cannot they.

C

D

So you can work on that. You can work on uh I, think there's also the get hook, so I I never done this with gitlab, but there should be a way for preventing things to get into a branch. So something like if this is a direct commit to master. It is changing things that I should not change in terms of file. So then uh denies.

A

D

If it's emerged, okay, okay, so I think there are options. Maybe not all of them are feasible, but there should be as well as if possible. This is not sure, I'm not sure about this. There should be this thing where only owners can change I think so we don't have this one. Only owners can change the protected branches and rules around branches, because otherwise maintainers can circumvent all the restrictions.

A

D

As long as it's in the audit I mean I'm fine in keeping this open right, so it's the product, so it's a product doesn't support. This doesn't support this. So probably there are options to.

D

uh Because of this, uh we may consider having a fork of release environment I mean that doesn't have to be a fork. What I mean is that, as long as release environment is only addressing the release environments, then I have no problems with gitlab maintainers being able to affect release environments, because that's the thing we want to give them power to to own things, uh hand to end fine.

D

If we end up exploring this uh concept with other type of environment that have stricter uh permission, then we can afford and gonna have another and so working on the kubernetes level right. So the credential inside the project will be restricted only to the namespaces that can work or on those environments. So I think we can try to work uh on this level of um defining the the permission model.

B

I think yeah. No, that makes sense to me. I didn't think about that. But you're right I could at least protect the commit part a bit better, probably with um rules of like yeah. Maybe the bot token is the only one could commit that that means they could still deploy, but it wouldn't commit it back to the repo yeah um I think the big key is is that gitlab has and possibly not just gitlab. Honestly, it's really good security around branches right like so you know.

B

If you've got this Branch, you can protect it, if not worth that, have you, but what we need is security around files in a repo, and so what I started looking at was what code owners can do for us, because really what I want to be able to say is hypothetically.

B

um You need to do a merge request if you're going to change like the global values or you're going to change like the memory settings or something like that. But here's a file where you specify the versions for this environment right so each and by each environment might have a versions file I'm happy for you know. Xyz people to change that, but but that doesn't work with the pipeline model.

C

B

So that's why I'm thinking the second stage is we just have to kind of sit down and get a good uh understanding of how to automate merge requests that like work in a way where it's like I trigger a job. It does create a merge request and like merges that using the security that do you, like, whatever the job is running as it's talking to the API of that security can set things to Auto, merge and therefore I can go okay, gitlab or can can talk to these.

B

You know environment or they can change these files. They can't change these files and what have you and that might give us? The question will really come down to um talking with Amy about this earlier, as well is like what, as these release environments start to become more important to the process. So at the moment we're doing the first iteration is really just getting things up technically making sure we can kind of see this happening that works. It makes sense.

B

Then we need to concentrate on like what do we have to do from a security from an auditing perspective to make this?

B

You know to the level where we can say: yes, this can be a trusted piece of the the process. I mean technically, you know it's all fine but like. If there's controls we need to put in place. If we need harsher rules, it might change the implementation in the future, depending on what people come back with, um but.

C

We'll just see.

D

I would say this is in any case, this is a test thing in quality environment. So, as.

E

D

As there's no way to escape or say, the important thing is that you can't uh escape the namespace in kubernetes and escalate into the production or things like that right. Otherwise, uh the the gate is at the beginning, so maintain only those allowed to merge on stable branches will be allowed to propagate battery yeah, and so we also need to make sure that then um we we have the right permission in place.

D

Right so I mean maintainers are so we have groups for maintainers and things like that on on githra project, and this should be the same rule. The same maintainers.

B

D

B

Yeah see yeah, so I'll just quickly wrap up the demo here you can see here. This is the one commit that was just created, update, environment, 1560 version 15.6.20. We can have a description says where the pipeline, where the commit back came from. If we actually look at the commit it's uh yeah, you can see there, it's updating it um all the different pieces.

B

I may expand this as well. Sorry.

D

Sorry I have unrelated and I was related to that one before I forget about this. If we are going to put metadata in commit message as that one, we should use git trailers, because then we can query by the API and we get them parsed. So we don't. We don't need them.

E

D

We may need them in the future, so if you put them only all at the end and with the columns, so there are trailers. When you ask gitlab to give you the list of commits, you can ask to parse trailers, and so you can do interesting. Logic on on API.

B

Okay, cool, okay, that's also good to know yeah briefly mention um if the deployment model becomes more complicated, I will actually replace the variables being passed in at the moment. It's just environment version, it's very simple, but if it becomes more complicated, I'll actually replace it with a variable. That's like deployment Json or something which is the Upstream password pipeline- must actually pass in some kind of well-known Json object of, like all the deployment targets and stuff that it wishes to hit, and then I'll pass that Downstream and do the appropriate pipeline.

D

There is an option in our pipeline system for trigger to download artifacts from the Upstream.

B

Yeah I saw that as well, which is another similar thing like if we generate an artifact that says this is a deployment Target like we can define a schema, that's more complicated than just environment version, I! Guess that's what I'm trying to say we can if, if the need arises, which might we'll just expand upon that.

A

C

A

As the environment comes up great, um do you already have some anything around kind of um checking the health of the environment like how do we determine like it up and healthy.

B

So beyond just how I'm doing its basic checks at the moment, that's it uh I, like I, agree. That's the thing, I've kind of pushed that into a second phase kind of idea, because really so let me kind of uh give an overview of what I'm really thinking is like. What's phase one and as I said, the focus is just trying to get some pieces working technically, so there's three really main pillars to this: getting images built so the interface with CNG and really getting the images built and making sure.

B

Yes, those images contain the code. We expect there's doing the which I've already got merged now to gitlab or gitlab. There's the deployment of that which now I've got that trigger interface. So I've got a method of doing that. It's not the best method, but it is a method that should work and then the final piece is running. Qa I have the pieces together for running QA. So from what they've, given me, I think I can piece together something and and not piece together.

B

That's probably it I'm confident I've got enough now to build a job that will run QA against that environment, um and so that's kind of the like the next pieces. I'll actually build together that job so where we are at so the way this kind of all works from a developer perspective and in the stable Branch perspective is on on the pipelines that run on stable, Branch right. So whenever you merge an MR to a stable branch, we we have a few different. You know. You've got your aspect tests.

B

We've got some child pipelines doing bits and pieces. There is now a new child pipeline, specifically called release environments and it's a child pipeline that lives. It's just a um Dynamic. Oh it's not even Dynamic. It's just a child pipeline that lives in the git lab or gitlab repo.

B

But what that means is you look at that little Pipeline and that should be this three-phase end-to-end kind of process build images deploy QA. Now we can ex now, once we have that there that pipeline we can expand. Do we want to check Health metrics? Do we want to do this? Do we want to do that?

B

We've got the basic building blocks there to expand that out and because it's a child Pipeline and it's an own files in the gitlab org repo I, feel like it'll, be easier for us to iterate, because we can kind of own that pipe that child Pipeline and own that pipeline, if that makes sense like it's hard like once putting it in the right spot in the big pipeline, obviously affects a larger group of people.

B

But now that we are like we've got this on our own pipeline I feel like you know, we can hopefully get more reviews and change that a lot quicker which, because it has been a bit tricky to get that change, getting reviewed and and uh moved in So yeah, so that's kind of where we're at now.

B

What this means is it's been merged to master, but it hasn't really been 100 tested yet because the pipe the child pipeline only fires on stable branches and because that changes merged to master, you have an and not a stable Branch. Therefore we haven't really got it happening. Yet I did an aramara up that I know um with that. I think is sitting with Alessio at the moment to backport that change into the 15-9 stable branch um for testing, but the other thing that will happen is when we do the ass.

B

No someone could correct me here when we do. The RC for 1510 is when we create the 1510 stable branch, and when that branch is created, it will pick up the copy of gitlab CI from Main, as all branches do when they're created, and then that will have this change and that will also you know, start basically building out the you know. It'll start running this release environments pipeline this.

D

Is all allowed to fail? Isn't it yeah.

B

100, all of it, if it's not it I, should go fix that I, I'm I'm, pretty sure I set everything allowed to fail, specifically because I do not want this to have any actual real impact on.

D

1510, we will have the first for the first time the.

C

D

The pre-environment and the release environment, both installing the RC they.

B

Will well, they will be at the moment. They'll just be building images and.

C

B

Hoping is and then I'll actually because the tricky part is, is a kind of want. I I do need to do the other parts, but I kind of was hoping to make sure I could validate the child pipeline worked and the image there's no point me putting the trigger deploy in until I know. The images are coming out kind of like so so.

B

If we, if we merge the the back Port, then we could test it today on 15.9 um or we can just wait until 15, 10, I guess in what five days or less four days, three days um when the when the 15 10 RC is created.

D

So for the 20.

C

D

B

So either way, we'll um I'll actually get some real feedback about uh the images I will go in I'll, validate them myself that you know they're, giving us the code that we're right and then I'll probably do an MR to start adding the trigger like to the deploy trigger and that will probably then need to to actually get it working.

B

We'll have to need to back all these CI changes we'll have to get back ported onto a stable branch of it, obviously, to actually make them useful because being merged onto Mouse, CI changes merge to master, won't really affect things um until they it's kind of like I have to develop them on master and then I kind of get them all back. Ported onto stable branches, I guess.

A

What would this look like on the actual MetroCast, so the on 1510 is also the first stable Branch, where we'll be having developers merging stuff in um like what does this look like if it fails sure.

B

So what they will see is they'll see um just like any of the other child pipelines we have now. Let me see if I can Show an example.

B

Let's have a look: it's go to gitlab or gitlab.

B

Yeah, so the big thing is: is this job will be allowed to fail, so it nothing we do here. I do want want to actually cause anything to break for people to actually hold up work or anything. This is just a purely um so this might be a good example.

B

So what it's going to look like uh and I really wish I could show the real one um actually I wonder if I can use the no no I can't. We need to wait for 15 uh 10 stable, maybe um you'll have another Downstream over here and it will say, release environments.

C

B

There'll be a box to the side and it'll have uh one job which is uh trigger CNG I. Think something like that. I can't sorry I can't remember it's build images if it breaks it's allowed to fail. So it'll do the whole uh exclamation mark and you know I. We definitely will will you know. Obviously we need to start developing documentation of like Okay, so this pipeline breaks. What do we do kind of thing?

B

But at the moment it's just trying to get confidence that the pipeline's there and it works okay, but we can tell people to ignore it for them. I'm happy for people just just ignore the failures and.

B

But then, eventually, obviously as we get confidence, we will you know that will not allow failures, we'll actually want the QA results to pass and you'll have to demonstrate that QA is passing.

A

Yeah, that makes sense Graham. Are you also going to be affected? Are you also um working towards running the package and test test? Suite? Is that the one that would be run on these release environments.

B

That is a good question and I, don't believe so so packaging QA test six correct. Oh okay,.

A

B

Build days, uh all-in-one Docker image and runs it in in the gitlab runner, it's great. It's actually crazy that it works I'm very impressed, but it runs just as Docker container with everything inside it inside the runner. Vm, that's like running over in in gitlab, CI um and I believe the test Suite is much smaller. I could be wrong. I.

C

B

The suite is smaller because of the limitations of that setup. Okay, so, for example, right um with release environments because they are full release, environments, we get cars, we get Pages all those things that live off subdomains, so we get like uh Pages, Dot, 15.6, 15-6., stable or whatever, like we get the full. You know we can. My goal is to try and build these environments, which is with as many of the components enabled as possible so that we can run the largest test Suite possible.

A

Because what I was I just wanted to see if we can, maybe if we, if we have a thing we need to get ahead of, is whether we have a stable enough test. Suite so I'll take a I'll. Take a.

C

A

Because if we one thing on the work Myra's been doing is, but we know that the packaging test um test Suite isn't going to be set as blocking because of the number of uh the percentage of flaky tests in there possibly becomes too noisy. So so you may want to sort of uh review that as well.

B

uh So QA tests, uh so this is the issue I'm just trying to figure out where to put this in the.

C

B

Qa tests issue, so we can have a look at that issue. So what I believe um what I believe we've landed on is.

A

ah Okay, gitlab QA as well.

B

Yeah the kit gitlab QA Jam, but so that gives me and I've actually like been running that myself trying to like recreate. How is this gonna look and there's some discussion there about Flags I need to do and settings I need to do, which is all fine.

B

That's that's just configuration, but the tricky one is um or what I need to do is: I still need the stable Branch pipeline to build me a QA image right, so that I think we do that as part of this packaging QA test, I I will validate that, but it whatever whatever it means, is that we're already building some image with QA tests and I will just be already. So we already do that and I will be reused.

D

It's not published, that was the problem that they were addressing so as far as I understood it gets published where it is supposed to be only on tag. So this is what they were mentioning when they said uh you can kick start this process, because the stable branches for us begins with rc42.

D

So there is already a QA image for that and that's okay, but at every single things you built on top of that, will still run on the previously tagged image until you tag again, which means that if you have uh an issue in the QA image itself, so not in the code but in the QA, you cannot uh uh get the green uh QA until you you. But this is the technical details and it's it's already in the pipeline, so it just it's just designed to only publish those images on tags on stable branches.

D

So it's just a matter of um I'll.

B

Change that yeah change that and make.

D

Those image available to us.

B

Yeah yeah I think that that that sounds about right, yeah I think they weren't being you're right. They weren't being published and we'll just we'll turn that on for the use case, we now need and then we'll just get use that image now the gitlab QA gem.

B

um This is the thing I'm still not 100 sure is like it says: oh yeah, you can run QA tests, I I I, assume that's the full tests. I I looked through the settings. There was no. Besides, there was some smoke you could set smoke only for a subset, but I believe just by running that gem I'm supposed to be getting a full test. Suite.

D

So, if I probably would want to run the smoke test because they're, the only one that are supposed to be reliable.

B

Oh so you're saying we should cut that just to smoke. Yeah.

D

So I mean if my understanding is correct, there are several toggles on how on what you can run. One is smoke which is give me the curated reliable set of tests and then even in the smoke test. There is another distinction about uh knowing, if you ever not admin access to that instance,.

A

You can turn off because yeah apparently on release, we don't run those so probably the same for the release environments here.

D

Yeah, maybe we want to run them because we.

A

D

Think we run them on staging on staging we're on the admin, but we don't run them in production.

D

That sounds right. So.

C

D

Most coverage and most and more stability, let's try that balances.

E

As much as possible, with the most predictable stability, if possible,.

B

And definitely I think you know we'll we'll make this decision, obviously to not have things be painful for us while hitting the max coverage, but I do think and I think we're all in agreement here that if people want to expand that later like that is a us working with quote, like you know, quality.

A

That's totally fine, yeah I think the main uh thing that we should check in on and we might have that information already I'll do a little bit of a cross read on a on a definition just see if we already have. The information is what percentage of known flakiness is on this test, Suite good. That was something so I'd admire.

A

I had a good chat about with package and test, and that's what's determined that on the kind of interim solution, the one that will go live next week with 1510 for developers merging in is the packaging test won't be blocking because.

C

A

Is known to have a high percentage of flakiness and what will be more more damaging is to immediately build in the behaviors where every developer is like. Oh, it failed.

C

A

Just keep me trying forever, it's actually less work to allow it to fail and have quality. Do some manual work if needed. Definitely not a good long-term thing, though, because it creates more manual work, so getting um what I think either we find a test, Suite, that's stable enough or we, this would be I, think a good one for us to be able to give quality a heads up that like hey. This is the direction we're going we're going to need a stable test, Suite that has decent coverage that we can set to be blocking.

B

Sure I think then for the interest of just kind of moving like you're right. We should do this. um We should have. This discussion sounds like what I would do just to move forward. Quick is just do smoke only to start. We.

A

Can always change whatever I mean honestly at the stage game like it doesn't need to be blocking, so you can pretty much use anything and see what it starts to look like.

B

Yeah sure yeah I guess that's true as well. We can just see what what it looks like yeah, yeah.

D

As long as it starts, then it's just a matter of making sure that it actually also pass.

A

Exactly well, what would be interesting is I'm guessing that for for probably for like 15 10 at least, we will have almost all three processes running right. So we'll have the sort of interim one that Mayra and Sonata figured out, which is uh for allowing developers to do anything we'll have the existing one, which is right before we. You know when we tag, we will run things.

B

A

Will have this coming in at some point, so we will actually be able to compare the different test results. We're.

C

A

um And that might also give us some idea of, like you know, did we get two test, Suites passing and one failed did it matter and we can start to make some um judgment calls there.

B

I also suspect, there's going to be a level of flakiness at the start for release environments, because I might need to tune them like at the moment. I've just left everything default and might like. Oh look we're getting timeout errors because I haven't made the pods big enough or I. Haven't I forgot to configure this. You know so like getting. You know just working through the like, oh yeah, The Kinks, to make them run fine in a test environment, um yeah.

A

B

Sure I suspect there'll be a bit of.

C

A

What did you end up doing for security release this great, like do these get triggered when.

B

No I said this is all canonical, so this is all canonical. So when we do the push back and go public, then they will deploy to security to release environments. This was another kind of conscious decision, because when we start talking about security releases in terms of deploying them to real environments as well things, the security question becomes higher. The questions I was saying earlier. The the implications of security just start raising up so I thought, let's short circuit, that if you know in the future, if we actually, we.

D

Can Fork them right so, as we were discussing before right so having our security Fork of release environment that is only triggered on on the security uh uh commits yeah.

B

D

To make sure that they don't overlap, because obviously the.

C

D

Will the push mirroring? But it's it's something doable, because we can distinguish between a merge event, I merged, something on stable branch and the mirroring.

B

D

Things right so what.

B

I'm thinking is, if we get buy-in from, you, know, security and everyone and they're like yep. No, these environments are secure and the information you have in there in that repo is not public. It's not bad like then. What I think we do. Is we just change the pipeline, like literally, that whole child pipeline instead of running on canonical it just runs on security like we just flip it over, and then.

B

D

I would like to end up. Having is getting rid of the automation for merging things and having, let's say, absec or someone that has the right to make the Judgment call watch.

B

D

Working on all the backboards and is is legitimate, is actually a bug fix, and so they can must merge the things right so and and so only those allowed to merge on stable branches on security. Yeah sorry you'll be able to trigger a deployment on the security release, environment.

B

um Yeah, you're, right and I think we can kind of figure that out when we get down the road I.

B

Think too, with the release environment stuff, we can look at um reuse a bit better, like I can take all of the the stuff out of it into a common repo, and then you just like oh include the like: you can create release environments, you can create secure release environments, you can create gitlab.com deployments or whatever, and they all just inherit off the one repo like not even full Forks but just empty repos that just have the environment files, but all of the like Ci logic and all that stuff is just included via like includes or whatever, and that way you can just you know, you're not having to actually like if I want to make a tweet to any of this workflow I'm, not you know having to worry about Forks, it's just all in one place and then all of our repos that kind of work in this model.

B

Just work like you know, just get the updated, workflow.

B

um Well, we'll see I mean you know this is kind of in the future. All right, let's we'll see how that works,.

A

Something else this is just a a a a a small drop in like and definitely not ex.

A

This is not a fully formed, thought and I'm not expecting it to become a thing that we know much about for some time, but just as an interesting one, Graham and I chatted this morning about the fact that we don't really have like an entry point like a land, a single landing page but I wonder as well as we sort of think about things on stable branches and in different places and things whether there's you know something that sits in front of a lot of this, where it's kind of almost uh you know, I can see this Mr I can see which environment it's on right now or I.

A

You know like I, wonder if the the, whether we're going to end up meeting from an orchestration team point of view like just a an orchestrator, essentially to to keep track of all these things.

A

Which has like all the possible environments- and it shows you know, like ticks of where your stuff is, or you know just it feels like pulling all of this together- might be useful at some point.

D

Yeah I was trying to take a look at Backstage, but I've never had time to actually play with that. One.

A

D

Feeling that it might solve some of those problems, maybe not all of them, because it's it can do a lot of stuff. So.

A

I used to work in a place, it was a different problem, but it's something that worked really nicely was we had a lot of difficulties, keeping track of uh dependencies? It was like a microservice environment. We had a lot of trouble, keeping track of, like which version of which microservice was installed, on which environment and somebody in the end just created like a service with a landing page, and it just listed them all, and you basically just got the version numbers for each environment, and it was so so useful.

A

um I wonder if we I I do think at some point. We're going to need to have something.

D

Maybe backstage.

A

Or something to coordinate.

D

We were having this conversation last week or the week before, with skarbick and.

C

D

When I shared them, the my the scripts for The X bar the you want the eugrams recommending the day only run they don't run on Linux, and so what I told them is? Yes, uh we need a page. A panel we'll say just uh on autoplies is different. It's more about from starting setting points, probably is the packages, so these are the packages and what happened to those packages while on release environment, the starting point are the environments, and but still this is kind of the place to go.

D

I I think that maybe grafana can do this, but we don't know enough about this.

E

Has always been.

D

My thoughts about this.

E

I'm interested.

A

D

You able to do some of this.

A

D

The other point of uh foreign.

D

Is that what the script is doing is just doing, API calls on Ops and so all dot API calls can be also made on JavaScript. So you can just build a single page application that just force you to log in on Ops and then, if you can see things on Ops, it can rebuild that same things. If only we knew how to do a single page application in JavaScript.

D

A

We get someone out of a university course I feel like they actually probably have done a lot more of this than we yeah guys.

B

Actually, I was thinking about this as well, and it's like to me like a page for some of this information is good, but what I was actually even thinking beyond that is like an actual workflow engine like we need a proper workflow engine and what I was looking at actually um talking about backstage and everything, I was looking at the two.

B

The two workflow engines, I know of for this kind of stuff is octopus, deploy and Spinnaker, and this is the kind of thing this is what I was saying earlier about, like the limitations of gitlab CI like if we try and do everything via gitlab CI pipelines, but we have the UI feedback, Is Not Great like and if I want to say, go to Ops open an MR go to Dev check.

B

If a package is built, it's so hard, whereas with these kind of tools you can build plugins and actually give the user like an actual. What's going on, oh by the way I'm waiting for packages on Ops I'm doing this I'm waiting for this Mr to be merged. You know these kind of, like actual workflows, that just get the FCI is a fantastic tool for CI, but we're trying to build actual business workflows um the self-managed release process.

B

You know how much have we got sitting in issue descriptions and it's very complicated and it'd be great if it could update dynamically. If the steps would like real time change as things are happening, so you can really get a grip on that, and so it's interesting because I think there there is a whole market for those tools and it's not a big Market, because a lot of times people's workflows are simple.

B

I just commit a thing and it runs a Helm install and Away you go, but when we start hitting the level we're at where it's like, we have complex, workflows, I think gitlab CI is just unfortunately it's just not. It doesn't give us the user experience for complicated things and it shouldn't just be gitlab CI, like we build things like environments deployments like protected environments, uh deployment um like valid. What is it where you can like um control? Someone has to click a button deployment approvals.

B

We build all this stuff, but it's just not good enough for our workflow and the UI is really hard for us to kind of to to actually consume so, in other words, I'm completely.

A

I just want to counter that by. This is exactly where, as we build this stuff- and we see this where we.

C

Are absolutely right? Absolutely it's not unsolvable. You can capture.

A

These, like here, is the use case we have here's here is the problem we're trying to solve, and here you know, here's what the information people need to see and then maybe like we do lean on something like backstage as a kind of like POC type of yeah.

B

A

Something that jumps us that way, because I think this is absolutely the stuff which I've been you know talking to other stage groups is this.

A

End-To-End flow is so difficult for people who are not release managers to really drop their head around because so complicated um that I think we can feed that in one thing that led me to say, my kind of original thought is one challenge: I can see coming up around the uh opening up the stable branches and adding in these extra environments is the the sort of the not a huge shift, but just the slight difference between when we're in a security release and when we're not in a security release.

A

At the moment, that's already really hard for people to see, um but they don't necessarily need to see too much of it. So we kind of get away with it. I think that's going to become more painful I, just wonder if there needs to probably be more ways that people can actually see the status of of their changes.

B

Absolutely yeah trying to give that that overview as I said, like the uh a business like a business overview, not just at a like some CI jobs are running like this actual. This is the phase of like you know that a security fixes in well accepting their Mars we're not accepting Mrs yeah um yeah. It's like how do you capture that and like demonstrate that and I, don't think um I think that's such a custom process to gitlab. How do we write something? That's custom to that?

B

We can encapsulate all of our processes that are kind of custom and tricky in a way where we can put it in front of people and they can see what's going on yeah.

A

Exactly I wonder if we're going to have cases where teams like who have set up like Italy are going to kind of accidentally, merge stuff in and end up blocking themselves, without realizing they're going to block themselves because of like merging onto a stable branch and then we're in different statuses like it feels like it's probably some edge cases like that that we haven't yet um come across thought about we'll find out. Yeah.

B

A

We'll find out.

C

B

Yeah, mirroring is a whole other problem, like that's a whole other problem right, which is just like so much of our products. Functionality, just like the concept of mirroring not only mirroring Downstream to repos, but mirroring to a completely different git lab instance. You know that we we just it's so hard to track and do because.

C

We're not yeah.

B

Because we're the only people who realistically have a process, that's doing that, because why? Who deploys multiple, git Labs right and like throws all their code around and inside them all like.

D

I mean private Forks are probably common, also elsewhere,.

B

E

Everything is instant.

C

D

Yeah also, everything is private by default. Yeah.

A

Mm-Hmm yeah for sure, okay, awesome, so one thing going from the back of this demo is: it would be good to make sure that in Myra's uh documentation and announcements and things we just have the specific release, environments name, so we can say definitely ignore this stuff.

B

That makes sense sounds good.

A

Just so that we can make sure it's super clear, there's one unfortunate thing for the first iteration is people have to kind of do a manual step on that package and test Suite. So if it fails, they have to take an actions they have to watch it themselves.

A

Because it's allowed to fail so just to make sure they know this bit. You can just ignore.

C

It's completely coming later, work in progress.

A

Awesome um great, is there any other stuff we should chat about?

A

Do we need to like? uh Do we make a decision? Are you gonna, wait for the 1510 Branch gram or we're going to backport the.

D

I'm I'm running the pipeline on the backboard.

E

D

Are okay, I would merge.

E

Attack I wasn't.

C

My understanding.

D

Was It was a brand new thing, but it's it's a it's a backboard, so it was already deeply.

B

Yeah, it's already merged to me. Yeah yeah.

D

Exactly so, it was just checking the what.

B

Is it yeah awesome.

D

B

Breaks obviously just revert it. It should like if it breaks it should just be allowed to fail, but if it does cause a big issue, obviously just revert it and then I'll go and have to figure out what I've missed.

A

Sure awesome, okay sounds good um great thanks much for the demo Graham and thanks to discussions both of you um hope you have good.