GitLab Feature Walkthroughs, 20 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-03-20 Background jobs improvements demo

Description

Part of https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/96

A

Cool I think we have Sean as a first person yeah.

B

A very simple one: I'm just gonna, share this real quick, but you can see it in the doc, but for the benefit of the video, so had an issue where we updated some key metrics of sidekick and there wasn't a combined function at the time for these types of metrics so didn't use it, but that caused a dip in our essays.

B

So now there's a danger bot that will warn you when you generate no, when you change one of these auto-generated files, it's only a warning because often when you do that, it's actually intentional, but I also added a bit to read me here about like what you would do concretely, which is use combined, make the change once the change is like been in production for long enough, that you don't need combined anymore, remove, combined and then you're done so that also decouples the it means that we don't need to bother the SRA on call with the come with that process, which is some nice because they do you a lot to deal with so yeah.

B

That's that cool.

A

Awesome thanks, Sean I know that this feels like a small thing, but actually it is not because we do get direct directly affected with these changes. So thanks for doing that,.

C

Cool Bob, you wanna talk about the next one yep that show my screen so I allowed for configuring.

C

I can experimental default kind of for running sidekick cluster as the people's way. The only way to run sidekick using omnibus so I've got an instance here here. That's currently running I think once I click process increase this a bit.

C

Yeah here it is, and I've got the the Copic here and then, if we.

C

Something's wrong with Mikey winning.

C

So if we set cluster to.

A

You like a different editor yeah.

C

Obviously so, if we enable sidekick cluster like this, this allows us to use some of the configuration options that, like plaster, provides, for example, the two groups configurational. We are the q group option using this setup. This should start to psychic processes running all the cues. So let's try that.

D

About something Bob, so imagine that someone has sidekick running on the server like just exactly and then we change the config and the person will like ran the reconfigure thing, and would these sidekick process get killed? So then new sidekick cluster process is started with the child's processes. Yeah nice I was thinking.

C

That what I just did now like the psychic, was running. That's the single process that I showed like just a default thing that runs out use and now, as you can see here, we've got sidekick running with a concurrency of 50 and all of the queues. This is the thing generated by sidekick cluster, and here we've got psychic cluster running.

D

Yeah yeah, basically just just killing the old process and making sure that we use only psyche.

C

This is just part of the reconfigure since then one thing just overrides the other and disables itself. That means that's. The sidekick service itself is disabled, which means that yeah, that's just part of omnibus already. There's one thing there that I don't know. I haven't actually tried this before.

C

But if we enable.

C

That kick cluster as well. This very.

D

C

D

C

D

To just say something.

C

They should now rate anime error during reconfigure because we don't want to run psychic after twice or anything like that. So.

B

You have to opt in to psychic cluster, like psychic square brackets cluster, and if you do that and configure psychic underscore cluster, then you'll get this. Then you get the narrow yeah. If you, if you don't opt-in, you won't get that error. So if you just have regular, psychic and regular psychic cluster for now we're supporting that, even though it's a weird configuration right, yeah.

C

C

D

Was thinking about how how much time should we be running this like as often but for Christmas? Let's say we're studying 12.9 today's from out and then I'm wondering if it would be interesting to like leave this open for Christmas for a month to make sure that people can use them. It's not raising errors, and then, after that, we make this a default like in 13.0. Maybe yes or that would be too much yeah.

A

So what what we want to make sure is that we have at least some you know feedback over a month, if possible, for the people who are upgrading and giving the thirteen is around the corner, and we don't want to carry over this configuration.

A

The important part is, let's just make sure that if all of this works, we can actually make the call to make this a default, but then also have an option in 13 that if something does go wrong, we can instruct people hey. This is the fallback. This is. The old behavior will fix the problem right, but this is what you can do in the meantime. So.

C

That would mean just like right now. This cyclic cluster variable thing is set to false by default, but in 13 we would flip that the true yep, but then we would still have the old sidekick cluster and sidekick kind of starting everything, and then, in a later release we get rid of the old sidekick cluster configuration and the sidekick configuration is the only way to start sidekick cluster which will like it yeah.

A

Making sure that we have a step back in case. Something does happen right, like you, don't know what kind of configurations people use, especially there are like a lot of them, so we just want to have that. But one thing that that you showed there Bob is: there is psychic, cues and then star star. Whatever is that necessary, or you just wanted to use this to show no.

C

This is not necessary, but equal people use star, but I tried this just to see like what starting to processes look like, but thankful for this are yeah just a single star. Okay,.

B

Very one very minor thing: I wanted to check when you start currently, when you start psychic cluster, it shows a psychic cluster in your lab CTL status, I'm guessing when you start psychic cluster as psychics square brackets cluster. It doesn't. It just shows a psychic, no.

C

That's easy, I'll start. It starts as a service sidekick cluster, and this is because we still have both of them. So, okay, miron you're, probably going to have to help me here.

C

B

It's not super important I was just curious about how it showed up in that, let's actually.

D

Do a point that we're discussing right about like changing the service name, because imagine that the default would be psychic cluster. That would be strange for people. That's using that's.

C

I think we wouldn't support anymore, because right now we allow running sidekick and sidekick cluster next to each other. That's the setup that I just course.

A

And well, we allow it, but we did not recommend this.

B

To any, we didn't intend to allow I.

A

C

Kind of kind of suggest that we do yeah like.

A

C

A

Is adventurous is that I.

C

Like it, but if you read the documentation on psychic cluster and enable it in and enabling it in omnibus, it is documented as a way I like the default. Psychic is running. If you want to add one, you can start sidekick cluster like this.

D

C

That's what it sounds like and like, as you can see, shown anywhere.

D

C

D

C

Running both next to each other, so yeah, both services still exist so with the the first setup I had with cluster enabled that would still be the service sidekick cluster that is running, but I discussed it with a swallow. When we remove the option to start side kick without sidekick cluster, that makes sense, then we should name that service sidekick, sidekick cluster yeah.

D

That would make sense, because it's sidekick so sidekick closer or not the naming should be at.

C

Least bad: either you don't care anymore. You shouldn't care anymore than.

B

Cool okay that make sense yeah. It was mainly just thinking about that. From the perspective of like troubleshooting a people report issues, we can tell what they're doing based on that as well. Thanks for talking.

C

About there's a discussion open on the Omnibus, mr just so the reviewer sees it and based on feedback there I'm going to create an issue in our scalability issue tracker. That's then, going to be targeted that 13.1 or something like that yeah remove that as long as we allow running both next to each other, we will need the two services I'm afraid thanks.

D

For the welcome that look, nice great.

A

This could you also humor me and tell me when you run status, so get all control status? What is currently shown.

C

Both like because of the configuration we have right now, it will show both okay.

A

All right cool thanks.

C

D

Screen quickly, this is a follow-up work from Bridget this week, so just like boring compared to box demo I'm running GDK here and as we can see, there's there's running running background jobs and there's the Eid here. If I do a ps3, we can see as I click closer running all the cues here and all the commands workers before GDK restart everything. So basically, under the hood, we are using your using cyclic placer and we pass this cyclic workers to the radios background jobs.

D

Specification on profile would be interesting. I started the discussion about bumping these two Chu by default, two processes by default, because I think that would probably raise a few errors if, like concurrency errors, sometimes that would be interesting to see young in development. So maybe we can start with one running just single process. Then we bow to two, maybe next week making sure that everything worse works well. So.

A

Let's just make sure that we are not making too many changes at once. All over the place. I would be happy if we we even follow the same cadence we do for omnibus. So let's say we are shipping.

A

These things in parallel in omnibus, but we can do the same thing in GDK and say when we start switching to default in 13, we can say we're just gonna bump in GDK as well, so that you can see the impact better, because I'm afraid, if we take on too many things at once, we're gonna be spread to 10. Imagine dealing with judy k, & omnibus n charts. At the same time, it would be a bit too much yeah.

D

Makes sense- maybe maybe on 13.0 we bump this, and I mean.

A

We can always we can always inform people who are interested hey. This is what you need to what you want to do. If you wanna test this earlier, we do.

D

A

D

Just just shared this in back in engine development, so people can already bump this if they weren't yeah.

A

And were there are any problems since we introduced this I mean you only announced it yesterday, but yeah.

D

At least for now no comments, so if nope there, there is no comments. I hope that there all right cool the next one is for very small one and just I'll share my screen again just for show neighbor Japan, because why not?

D

This is just a small one. The problem that we are solving this here is that if we go to the profiler page with data that we had with grow services, we don't show the version of the deployments. That's the version of the actual service. Let's say work hearts. We don't show the version of work horse. If you what we bump the version of work person making you deploy, we won't see this version, so we can compare profiles. So here we are making a very simple change.

D

Since we already passed suit like through lab kit, we passed the version 2 Prometheus the version of this, the actual builds. We can reuse that to pass it over to dprofiler to the go profiler, so the changes is very simple here. So if you interesting I, think I look on the merge request feel free to take a look, boring demo. Sorry.

D

The next one is like just update on behalf of Craig that he is working on rolling out TD profiler, deco profiler, so I'm just sharing my screen again so.

D

We started running this in can canary, so we started this started, seeing a few bumps on. Let's see here on web CPU usage, we saw a bump here and before that as well. So we decided that we should be running a dprofiler, at least for like more two days on, can read to make sure that it's not really related, and we can make sure that anything is alright to to move to production, but.

E

We we know that it's gonna use more CPU right, yeah.

D

Yeah, actually, we know that this bump is like higher than expected.

A

We are expecting 5% increase right, approximately yeah.

D

Exactly we saw these bumps or, but it seems a little bit higher, but you'll keep monitoring that.

E

Yeah I mean I might was to be worth just putting process monitor on to like the workers processes. I don't have go, has anything in its in the Prometheus client. That tells you because, like how much of that is his workhorse, you know that would be at the moment. It could be anything on that machine, so you know that it's work or that would definitely help it might be worth just just wiring process monitor enjoyed and Wow yeah.

D

No comment on that concrete on the Detroit here: yeah.

E

D

I think that's it from me.

D

So next one is this discussion with pop.

C

Yeah there was just a lot of back-and-forth yesterday after rolling out, so the problem is the lettuce seep. You get. Yeah lettuce uses way too much CPU. If we watch too much too many cues at the same time like have a one-sided process. Watch too many cues, because I know you can probably explain this better than I can can you could you take this from me? Okay,.

E

So like what it seems to be, is that, like the because if you read the the blocking l pop-locking our documentation, a one-handed says this is a in a one operation. I so it's constant it'll never take longer or less. But then, if you read the rules, they're actually quite complicated right.

E

So they're like well like, depending on on the left to right and which client got there first and then you know how many things is asking for and if you just read the rules like there's quite a complicated set of rules that are obviously going to have to be applied, and then, if you have a thousand wells, what we're going with at the moment like over 200 cues and then 2,000 clients, but because each process, sorry each thread has got a blocking client right.

E

So that's you know you can work it out by the thread concurrency in the sidekick thing, but also, if you look in read a sidekick, we we record the number of blocking clients and that's mostly side. Well, that's many! All of it is psychic stuff.

E

We can. We can sort of multiply the two hundred two times, the two thousand and that's a hundred thousand whatever's it's a lot, and so as a little test on my machine. I I just pasted into slack a minute ago.

E

It's a really basic test, because it's kind of difficult to run two thousand clients at the same time on my machine, I actually tried to to do it with a hundred while I was talking to Baron in a zoom call and my video kind of started dying, but so, instead of doing I did a hundred clients and two thousand items and the and Ritter's basically dyes it.

E

So even though they say it's an o1 operation, it's very clearly not and it seems to go up the longer the list and how and and the more the clients that are contending for those to use. And so this kind of leads us to a point where we have two hundred and something cues at the moments. And we have the strategy that very clearly goes against the sidekick documentation that we have a cue per worker class and the the psychic documentation says.

E

Don't do that and and we've kind of got brighter now, but I don't know like how much longevity there is in that strategy like in a year's time. We might have four hundred cues and we might have even more clients listening to those cues and then, in that case, like is this gonna, be more of a problem.

F

E

The opening thing yeah so.

D

Let's say we use sidekick closer today and in a separate note, we are actually using a when.

E

You say sidekick cluster, you mean it's right, exactly closer, yes, I was thinking yeah. We.

D

Are seeing this problem in a note that we handle way too many cues right.

D

E

It's it's basically the catch-all note, so the simplest solution, I think this is where you're leading to you is split. The catch-all up into like multiple like I, don't ya, know not well not as well, but it was so like multiple, like maybe split by feature category or something like that. We can do something like that, but I think it's all just kind of beating around you know not fixing the problem here.

E

So that's my take on it. Yeah.

B

There's a couple of other things here so, first of all, the main reason that sidekick recommends against that as far as I can tell is because of the psychic Pro feature for our level fetching. We can't use psyche Pro, but we do have our own gem that Valerie wrote to emulate it like, because you know we can't. We can't distribute psychic for it's not open source and that actually has two options and we don't use the reliable, reliable fetch. We use the semi reliable fetch.

B

So the the other option is our pop el push the reddit command, which also says it has over 1, but the problem with that is that you can only do it on one key at once. So that's where we would, if we had many fewer cues, that wouldn't be a problem, because we don't have as many keys to iterate over.

B

But if we have like a couple of hundred cues like we do now, we can't use that because yeah we're just gonna be hammering Redis with our Bell push where this needs more memorable, more pronounceable command names.

B

So on the queues thing, I think I said this in the issue, but, like my feeling is that we really don't want to migrate queue names if possible, because it's a pain and because there are so many things that can trip you up and I can go into those later when we start discussing solutions, but if we are gonna do it, we should probably do it now like so the way I put. It is the best time to do.

B

It is never and the second best time to do it is now because we were only gonna, add more queues, so I think what would be interesting to know is so. The main issue here was that we added the casual nodes which basically do the same as the best effort, which is listen to a bunch of queues, most of which have very little activity, well, I, say most of which, some of which have zero activity like including all the giro ones. I'll get laughs, calm because we don't have to on Caleb comm.

E

Obviously, they're not running they're, not running right now, yeah.

B

Obviously we intend to but I don't.

E

Know they're literally not running right like as we speak right now. Oh yeah.

B

But I mean it's like this was when we, when we rolled out.

F

We were I know these.

B

And the problem was the specific nodes so, like you said, we could split those up, but if we split those up in a way that we have the same amount of listeners for each queue yeah so.

E

F

B

Have the same problem I tested that out because, as Igor said, it's the number of clients times the number of keys listened to times the number of clients waiting on those keys. So if we split the catch-all up, we would reduce the number of Q's say: we'd have the number of Q's, but we double the number of clients and the number of clients listening to the keys would stay constant, so you've got the same because it's just the product of those three you've got the same answer.

B

So I would really like not to do this and I'm curious about how how it would look if we had catch-all but not best effort and.

B

Beef, your CPU for already psyching.

E

B

E

I think that, like one of it, so one of the things we could potentially do Shawn is wait for the palmera change to go through and then scale that fleet back scale the the pages feedback, which does nothing and there's various other fleets that basically sit there as like heat generation, machines for the global economy alone,.

E

Exactly yeah and- and we could scale it back and we could kind of do this shuffle, like you know, one of those little puzzles. Would you like to move the little tiles around right and we could get it all in and it'll be fine but like in that database?

E

Sharding working group Eric had this like really good points and way he said like look, we shouldn't be optimizing for, like you know, ten percent more, we should be optimizing for like like three hundred three hundred percent more right like like ten times more traffic, like that's, where that's when kid leprechauns going to become profitable and you know like. Firstly, the number of queues is going to go up like that's not going to change. Unless we address this at some stage and then the number of listeners is just going to go up like that.

E

As that goes up, and obviously there's other things we can do like shard, the registers and stuff, but it feels like we've kind of like what I don't want to do is double down on a bad decision, and you know it feels to me like this. In hindsight, I'm not saying at the time it was a bit, but now it feels like this is not the way to go.

E

You know, Redis is not designed for its sidekicks, not designed for it. You know if we want to use the reliable fetch, things become more difficult with the with the swapping queue thing, whatever its name is and but but there was one other point. I think like this. You know when you said the renaming of queues is difficult.

E

F

E

We know whether, if you get a job and like, in fact, I'm pretty sure another answer, but maybe I should just ask it, because I might be wrong if you get a job, the the name that the queue that you get that job from does not determine like whether or not it will run right like if I have a queue called a and a queue called B and I'm listening on a and B, but in the application it's basically designed to send on a but on the sidekick machine, its listen, it's designed to listen to be, but the sidekick queue is listening to a and B it'll still pick them up right.

E

B

Understanding is that it instantiates the worker class in the payload, and the queue is just for what the listener is doing. Yeah.

E

So my men from that I imagine that renaming the classes would be relatively.

E

We could do it because we could just keep the old class the old queue names and the new queue gnomes right so, like catch-all, would listen to everything it listens to for the moment, plus the queue called catch all right and right.

B

That's yeah! That's that's um so that the part about renaming psychic queues is that we have to do it for everybody and yeah.

E

B

Means, like you know, making it work everywhere. That's that's my issue. So let's talk through I, don't know so I feel like this discussion could go on for a while and it might not be concluded in this call. But we try to have a sync yesterday and I think having a call is useful as well yeah. Do you want to try talking through the option and then I'll talk through my concerns and then we'll see where we get with that one first and then so.

E

So so very quickly like what it is at the moment is that we've got like these 250 workers in the application, and each of them has got a queue so, like imagine like from left to right elections. Okay, that way we kind of like each one of them has got like a pipeline, that's going to sidekick and then psychic cluster. Does this very clever thing where it takes a bunch of them according to the queue selectors and it groups them up?

E

And then it starts a bunch of psychic listeners that are listening to to those and effectively on those queues. It will pick them all off. You know from from the whole bunch and the problem that we're having at the moment is that some of those lists are very long like a hundred turn in queues and so like that's where the BI pop come problems coming from. So what this solution is.

E

Is we take that clever logic that takes the many-to-one and remove it from sidekick cluster into the application, so that, when a sidekick, you instantiate when a sidekick worker instantiates at startup time of the application, instead of being hard-coded with the queue name like post receive?

E

It basically says: I am a job with these attributes and my key selector syntax, you know run through it's right, I'm gonna, be I'm, gonna, send to the urgency here and queue, and so, instead of there being two hundred queues that are getting sent to sidekick from the application, there are, however, many you want to configure.

E

So so you know in the simplest case, you know in the GDK case, there could be one or two and in the case of gitlab comm, this 10, for example, you know one for each of the not ten but one for each of the priorities that we got at the moment and so we're just taking the the maxing capacity and we're moving it from sidekick into the application. And then we sending to a few a number of Q's and then, when we pick those up, we don't have to pick them up from 200 different queues.

E

We just picked them out one at a time from one queue and that's basically what it is. So you take the sidekick, selector, syntax and moving into the application. Yeah.

B

So the main issue I have with this is because it's determined by user configuration in order to migrate the queues like if you change your configuration we either need to. You know basically rely on people reading the documentation or we need to know what the old configuration was as well as the new down figuration, because we need to know what something used to map to.

B

But even the documentation side is an ideal because we're listening to views and we're wearing them for them to to go to zeroes that you also have things like scheduled jobs which kind of handled differently and that might not go through zero. But.

E

So that's a real little.

C

E

How many people that are are setting up like different queues for different listeners? I mean we were not doing a very good job over at lab comm. How many customers are actually sitting there.

A

Is one right I.

C

B

Think I don't think this can be get lab. Comm, plus everything else, I think the the other problem I have there is that everything that sends a job to a sidekick. You needs to know the entire sidekick cluster configuration, so you might have a sidekick node that says: okay, I'm, processing, high urgency CPU bound jobs, but it also needs to know that low urgency CPU bound jobs, go to this queue and.

E

No more there's lots and lots of stuff that we keep and get lebar be that you know if one machine is different from the rest of the fleet's, everything will break. So that's no different from from you know.

C

Gets up and get up RB and therefore also get about llamo. That means that people can change it, which means that the whole renaming thing like, even though we only plan to do it once more and then we'll never ever, do it again, pinkie swear everybody. Everybody else that uses that can change, however much they want, and if they have queues. That means that the whole renaming thing that Shawn is worried about. We need to be able to do it reliably like all the time. No.

E

I think you can put in the documentation. This is a like an expert feature. Don't do this unless you know what you're doing and if you do anything you get it wrong. You'll lose whatever you had in your queues in the time between you Reese on it. Well, yeah that way, it'll.

A

Just say that, but we were recommending to some customers to use this different configuration, whether they use it or not. That's a different thing, but when they ask what do you do and get a blood come to process this many jobs? We point them to this, so there are chances that in the wild there is someone using.

B

That it's also a pain for when we want to do it. You know, like it's not ideal for us to do it. This way, right, like it's still, it's still painful so.

B

E

Like any any number of things where you know, if we rename a database column, we go through very similar process and that's just how you keep a big site up.

B

That's transparent for all users, but also users can't.

E

Do it you do it through a series of steps.

B

Doing that like if we're saying we're renaming cues based, so we could say, for instance, that we are going to rename iqs to what we think the current get lab. Comm configuration could be, and then we will provide a couple of cue selectors. That will do that like sensibly for other environments. So, like you know, they might not want to quite as granular they just might want, like. You know that you said two notes and we can just provide a cue selectors that do that.

B

So that's fine, but if we want to change that, I think we're actually better off taking the the hits and migrating that's in the application, even though it's painful because I think that's more robust in the face of like what our customers are actually going to do on their self managed instances than having to rely on them. Reading the documentation following it exactly and understanding how exactly this one fits together. So.

A

But before continuing like discussing this one option, like do, we know if we have other options that we could discuss as well. So.

B

One option could be to do the renames based on what we want to see. I'll get lab calm now so like, for instance, we put all higher gianccm, you bound jobs into a higher gianccm and queue, and then we do that going forward.

B

But the problem with that is.

F

Every time you.

B

Want to change it lab comm configuration, we need to change the application, but the advantage is it's always the same for everybody, and we know that we're.

E

Not losing, but we can ever. We can never in an emergency spin something up. We don't have the flexibility that we've got at the moment. So, like oh, this new job is, you know, really kind of like we need to isolate that. We have no control over over that without changing the application. Where this is a you know, this is a this is something.

B

But we would say we wouldn't if we use it, we wouldn't have that if we you psychic with just a couple of cues anyway,.

E

We would because because say say: for example, we knew we had this new feature called, which.

B

E

Know I'm not and.

B

There was a and we needed a.

E

New priority I'm.

B

Not trying to say, if we're doing your proposal, I'm saying if we had used psychic from the start using just a couple of cues, we wouldn't have that possibility anyway. No.

E

No no, but we know we wouldn't have that, but we have that at the moment and I think it's an important thing to to have to be able to. You know, stop listening to a cue or you know, roots our cue to another priority.

E

You know without making a merge request, change.

B

C

Could another on top like that, using middleware in the application, perhaps maybe and.

B

Another option.

E

Is really much with the site? That's what the queue selector is right. It's pretty much a middleware that gives you that.

B

Another option would be to because the problem here isn't really with the queues that are problem queues. The problem here is with all the boring queues that we just have like. If you look at how many queues we have that are lower agency, have an unknown resource boundary and have no external dependencies.

B

We have 187 yeah so like those are all boring queues, so we could put those in just a couple of cues like we could just manually say like all of those that go to default or something and I think I, don't know, I think there are definitely issues with that like.

B

If we've missed attributes to misattributed, it's not tributed a worker, then we've definitely got an issue, but that seems to me more of a boring solution, because I think there are definitely headaches with having to have every node always know exactly what queues are available and also having to do this dance. Every time you want to change that and figuration.

E

You know I disagree on the on the on all call nodes having to know because, like you know, that's a shift configuration change that we roll out everywhere and there's you know the same as all nodes have to have the same. Ridaz configuration the same Postgres configuration if it's not the same everywhere the application breaks- and this would be exactly the same as that so I.

E

Don't think that that's a problem myself and you know, there's lots of things that have to be coordinated on configuration changes and that's that's just the nature of a distributed application with the with the second, like I I hear you there are.

E

There are issues with that, and but you know, if we're going to go to that route, I mean that the other thing to consider is like do we need to like, if we're just gonna kind of take all those kids and put them in there, then do we actually even need the cue selector syntax? Can we not just so well, you know channel everything to default. The problem that I have is that we lose control over being able to like pick out certain jobs, run them somewhere else and not even run them at all.

E

Just leave them even not running and we've kind of taken away a bit from the from the platform right, I think.

B

If we've got jobs that are never running anywhere and also jobs as we go on it.

E

Yeah, it's kind of it's, it's a combination right, so it's not jobs that are only just running know where it's being able to like reroute jobs and saying, like you know, hey, there's a lot of cool Murray jobs like what we're going to do is without making application changes. We're gonna just send pull mirror jobs to its own. Its own, you know, say, say we decided it was really bad idea or you know, taking the pipeline jobs and running them on there on the same infrastructures. Everything else was a really bad idea.

E

We can make that change in like a few minutes or not a few minutes, because because chef and everything takes a long time, but you can make that relatively quickly and route that without having to make application, changes that that kind of worry you know like.

E

Obviously, if we have a set number of queues- and now we want to add another queue, then that coordination becomes something that we've got to do with self self managing stalls and all those people where this is now like a gitlab comm only configuration we don't have to kind of coordinate with with self managed, etc. I think.

B

The most boring solution to get us to a position where we can handle three hundred percent is probably to drastically reduce the number of queues for things that are boring or things that are related. So not every cron job probably needs to. Thank you.

B

You can already disable a cron job in other ways like they could all just go in a cron jobs queue there's, certainly all the ones that have the same attributes to cap new queues so like make it so that, like it's for a bunch of queues, we have namespaces, and then we have individual queues. We could just bundle those together a lot of the time, and then you know if we can also increase the CPU on our Redis sidekick instance.

B

It seems like we have a little of headroom there anyway, because we're already kind of fine. The problem is that we're trying to run to catch all and best effort at the same time, which are the problem ones, but I think we can. We can do a lot with that. I I think we.

E

Go ahead, Bob because.

C

Well now we added the catcher and best effort next to it, but at some point we're going to add more cache holes like that. We're.

B

Gonna spin down best efforts right, yeah.

C

But once we've done that and we've become, we become more popular and we have more jobs to process we're going to spin up more catch hold.

A

On is saying here is that we are going to buy ourselves some Headroom to actually find a solution right well,.

B

I'm also saying that we can make the capsules listen to like 50 queues, instead of like over 100 quite easily, with very little.

F

B

Know I think the risk of that would be quite low, and you know just by like grouping together jobs into the same queue. Now we could do that and that migration is also easier if those queues are not heavily populated, because we don't have to spend as much time in Redis like moving renaming everything. So.

E

So if we do that, we lose the ability to do this on an admin side right and like every change becomes a merge request into the application and like if we want to get something out of there, or we discover that you know like if we want to change something too, to reach it somewhere else. It just kind of feels, like we've, built all this stuff already right with the you selector and it seems to work pretty well like I've, been pretty happy with it so far like. Why not stick with that. I think.

B

Probably the biggest disconnect here is the estimate of the amount of edge cases and work involved in having this routing the ingot lab derby. I. Don't really know how we can reconcile that.

C

I'm very worried for having that be configurable for everyone all the time, because it involves much more than just writing a new string there, because because the queues are then dynamic, I.

E

Mean you, you could have like a catch-all and anything that goes. But yeah I mean you, you know there are solutions to that, but I just kind of feel like we have this lever at the moment and I really like the fact that I can take a runaway cue and I can divert it and I'm losing that lever and I and I. You know and felt like it's been working really well for us for gitlab calm and we can reconfigure things. You know without lots of changes in the application.

E

You know we're doing all the stuff at the moment and it's you know worked really well today. We just reconfigured things so that we don't have a bunch of keys and it just you know, I want to keep that level of configurability. Forget love, calm and it feels like we're going to lose that and that's feel sad to me. I think my.

F

Phone is my vote is for that you know I. Think like we. Definitely you know where the quarantine cues that like in we could have external dependency problems, that we don't anticipate, and this could cause problems for specific use and we don't have those believers to move things around. I think it could be problematic. I mean.

B

I did I feel like I feel, like the discussions not really getting us anywhere but like I did say we have cron jobs, you can already configure cron jobs and get that Darby or cron jobs could be a queue that would save some space and I. Don't really see how like.

E

Iii definitely agree with that short the con jobs, like that's a easy one right like I, don't think anyone's disagreeing on that. Well, at least I'm, not I, think you know, cron jobs is easy win and it's like a bunch of cues and I don't create a budget. Is.

C

That a good place to start figuring out one like some of these super boring things like the cron jobs. I, don't know how many cron jobs 55.

B

C

Queues, we don't need to watch yeah.

B

64 because we'd end up with one.

C

That's like one thing that we could already do and there's probably more if we just look for a few of these boring things and we've moved out, yeah.

B

I think so the other reason I would like to start with.

B

That is because it gets us like migrating psychic uses a headache, though not the contrib ones, but also, if we do it more, we will probably get better at it, and if we do it more, we might get better at it enough that the other option becomes more feasible, so I think, even if we were doing the other option with Congress style would be easy bit, which is, which is, you know, rename queues that don't really have anything in them or have like a couple of things in them and go from there. So.

E

Shawn, if we had like a if that proposal had a thing that basically looked for, say orphan queues and just these pulled them or something along those lines. So there's a bunch of jobs that have been sent to queues that are no longer actively configured whatever then it'll just make sure that those run on the best efforts and be done with that. With that fix, your concerns, they'd.

B

Need to think about it a bit more, but don't forget. We still need to migrate all the existing queues to that like we need to have some kind of like overlap period or something so that we don't accidentally run like a bunch of project export jobs on the best effort, nodes.

E

Like the easiest way to do, that is to start off with the figuration being whatever the default queue name is for this worker is that's its name right or something along those lines. It's basically one to one and you'll, get you 200 cues.

E

That's actually.

B

We need to talk to her a bit more so like. If I have a worker, that's processing, orphaned, cues, then how do I do the switch over to do the switch over I have to add another step because I have to add all my current queue configurations, all my previous cue configurations and then remove the previous ones after a time because otherwise I will send they export jobs to that best-effort catch-all, node I.

E

Mean I I, don't think like I would imagine it would be something in Redis rather than like a configuration. You know the the old queues I would imagine that that could just be something in Redis like this is a list of queues that have been sent to in the last 24 hours, rather than a configuration change, which is. These are all the cues that we configured in the last 24, because that's not going to work right. No one's gonna do that.

E

But if you you know, if you, when you listen to a queue or when you create a queue, you set up a you know some sort of greatest thing with the TTL on it. Just say this is a. This is a job.

E

Sorry, this is a qu and it was there was something sent to it in the last 24 hours. So it's something that needs to be caught. Yeah.

B

I think I think it's possible to make this work. I just think it's you know, maybe maybe maybe Bob and I are like under overestimating how much work it is, but I think it's a lot of work and I think we could get there.

B

We could get a lot of headroom from other options which would possibly help us.

E

If we did want to do this,.

B

Work in future with.

E

It with the cron I agree like we can do that, but like I, just don't want to give up the ability to to rethink like we've, we've kind of built and and the reason why we had this one-to-one one one worker two 1q originally was so they. You know it was a crude way of doing this like if her.

E

If a worker was doing something bad, we could just take it out of sidekick or we could move move it somewhere else, but the problem was that just ended up with too many cues, and so you know we had as an infrastructure team. We had that ability to control that and if we start putting everything in the same queue without configuration for that, that becomes like much more difficult. You know it makes a whole bunch of stuff with infrastructure people more difficult, I.

B

B

276 queues I think it's fair to say that probably 200 of those have never been a problem. We can probably find some subset of that 200 that we can assume will never be a problem. I don't want to like over over optimize for flexible.

E

Education, but the application changes on a daily basis right, like so so, and every release there could be something in a particular queue.

E

C

Example, LDAP worker things to keep groups and think and stuff we're never going to use those of get Lancome yeah.

C

Okay, but when we do start using them, we know that they will have been empty before so. Okay, we don't really need to migrate queues for us, so.

A

I, just I just want to interrupt this discussion because we exhausted I think what 45 minutes discussing and I'm not sure we are making any progress at the moment.

A

What I'll ask everyone to do is first of all consider Shawn's suggestion of starting to reduce the number of queues we have to deal with so putting all cron into cron right like a single one, and then we are not talking about 270. We are talking about 220 right, and maybe there are a couple of those that will get us below a certain line and give us a different perspective. Maybe.

F

A

Maybe it will and it will unblock us also- we have a weekend, so we can sleep on this and then come Monday or Tuesday or whatever we can revisit this this problem. In the meantime, there are a couple of other things that are being done right, Craig, Lee,.

C

I think I think we'll end up doing both things like right now, something that's like the chrome job. Things is something easy. Adding more CPU is something easy and then yeah we've bought ourselves some time, but at some point, I.

A

Just think that we're not going to resolve this in the next four minutes, nor that we have resulted in this coal, but we have raised a lot of concerns, so I'll upload this as soon as the zoom finishes, processing and I'll invite all of you to rewatch it with a different perspective as a you are, rather than a participant.

A

Ok agreeable to everyone, even if it's not sorry, 4 more minutes, all right Bob we want meat, we won't make it to the know.

C

Your item I, will move it to the either the next demo like to be the one.

A

C

D

One in three minutes minute.

A

C

A

What is best, let's move to the next demo, if we don't make it there, Monday. Okay, all right thanks! Everyone for a great discussion, see ya.