GitLab Memory Team, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Memory Team Issue stack ranking - 2GB edition

Description

The Memory team is stack ranking issues using RICE to understand priorities to achieve the GitLab on 2GB RAM goal.

A

B

A

Yeah, it's great. So this is the memory meeting we created to record our meeting and we forgot to so you're missing 10 minutes of content for those that are coming in. We essentially are discussing some sort of organizational questions of where we're tracking issues and how to ensure that we're not missing any any items going forward.

C

So fabian, like my proposal, will be that, like we work these topics from this to be quick, I completely agree with matthias on creating, like the follow-up issues in this area, that you have right now and as a conclusion of these two gig focus epic, I would simply propose that we close it once to kind of indicate that this is like done.

A

Yeah, I think the if I understand correctly, the intention here is just to go through this now and say: have we captured it appropriately or have we forgotten about it, and if we have forgotten about it, we can create a follow-up issue in here.

C

Or- or we don't want to capture that because it doesn't make sense correct, but also.

B

Cool, should we just do it swiftly.

A

Maybe camille you can you can go through this? Oh thank you. I'm getting a coffee delivery so.

C

um Can you just read the topic and probably like just read the topic like the one that is, like the top bullet point? Okay, this is like this is what we are using copy and write flagging in mri mathias, like you work on that, is it something that like we should follow up, is.

D

It no I I had. I have not worked on this. This is like something that came up out of a thread on twitter. I thought it sounded like a super interesting idea, and I tried to poke that person for more information and they never followed up with anything actionable. He was an airbnb engineer and apparently they're working on a custom, ruby dm fork uh that supports this, um and I thought I'd just like bring it up prior to the week uh in case anything actionable falls out of it, but it never did so.

D

um I have no idea how how it works, how it would work uh and I have no like lead, so I don't think it's something.

C

Because, like we did a lot of other preloading, is it still like worth investigating in the future, maybe with the high uncertainty.

D

Yeah, it's a good question. It's it's a really experimental thing right because I mean it would.

E

D

Really intimate knowledge of the ruby vm like this would be something we'd have to write in c in the ruby vm.

C

Okay, so you so you rather know on this one: okay,.

D

I mean I'd like to keep the idea around and you know let's also keep, because maybe here's the thing I mean they are working on this. You know like. Maybe we don't need to replicate this work, so let's just keep it on the radar. Maybe we don't have to work on it. Maybe airbnb will say hey we send a patch back upstream. Maybe it will be in ruby core for ruby3 who knows.

A

Okay, but that feels to me um like something: that's exciting, but in the future, but not actionable right now we can yeah.

C

Sounds really good? Okay! Yes, I I realize, like stupid,.

A

Copy online flagging in mri.

C

Yes, let's leave this one out: okay, investigate use gems, so we actually have find, I guess, finding mechanism to dynamically load terms, which is probably like a very good reflection of this topic. Cool right.

F

C

F

We have an issue where we will try dynamically to load, not just gems, but the code itself. We will see how it goes, but I think it's covered.

A

A

So this is covered. We can move on yeah yeah, investigate memory, content.

C

This is tricky because, like we had these strings as perfect, so this is reflected mathematics. I also saw that you mentioned these empty hashes. Is it something that like we, we aim to pursue, or is it like that end.

D

Yes, so the empty hashes thing: it's not that much! You know it's like a couple megabyte um might not be worth doing this. There was like another suggestion that peter I think mentioned. I.

C

Mean couple megabytes seems like a lot really.

D

Yeah, maybe you are yeah, maybe you know maybe you're right, you know, and maybe in yeah, in some total, like with other things, um yeah. Okay, it's a fair point. We could do it. I think the question is it's: it's not clear uh like where these allocations come from. So we don't even know so in terms of the rice stuff. uh I I don't have. I certainly don't have 100 confidence that we'll be able to fix them all right, um but.

A

Is this reflected in an issue that we should stack rank or should we add it to our list.

D

I don't think we have an issue for this. Maybe I underestimated the impact, so maybe we can create an issue for this.

E

I mean like, like.

C

To give you the perspective, like the stream that application- or maybe we give like around five megabytes in total. So now, if we have the houses that give us three or five megabytes.

D

C

It kind of makes uh let's say noticeable impact across the fleet.

D

Yeah- and there were actually two aspects that came out of this- that because the hashes there was only one thing- there were also just a bunch of other just really um sorry. The empty hashes were only like one thing: there were a bunch of other hashes that were just quite large um where uh we would have to still follow up on like what that is, and if that is something we can shrink or get get rid of, uh there was one, and I think for that other one. We do have an issue somewhere.

D

Let me check, which was, um I think I think craig might have created it actually.

F

D

The um yeah, the review unicode hashes that end up in in memory. That's like this big json file that we load into memory, which is like also like two to three megabyte. I think it was, uh or one and a half something like that: uh yeah that contains unicode mapping, uh which I think we use in the front end somewhere, something we could look into as well. It's just not clear whether this is something we can just get rid of. I mean there's probably a reason why we need to make this available, um but yeah.

D

That's another story that came out of this. So this parent topic here investigate memory. Content was really just us. Looking at you know, uh taking a heap down and looking at what what is in memory at any given point in time, and then there.

A

D

Things that looked like, um maybe we can shrink that okay.

A

So we have these concrete steps covered somewhere, as in.

D

Yeah, so the last one I just mentioned, we have an issue for this. Let me double check whether that is in the uh list of uh issues in the rice.

C

F

D

Yeah- it's not it's not in that list. So so I'm yeah, I'm like uh adding a comment here around things that are not in that list and then maybe we can create a new one for the empty hashes separately, because I think those need to be tackled separately.

F

I'm also wondering, like we covered this with help of eager like visual visualization of gemalog thing and like making it readable, at least, but it's still a lot of data, and we didn't conclude anything. But maybe if we look in the more details and into the smaller scale, we can maybe.

D

Give me the j emalox.

F

Yes, yes, so I'm not I'm not sure like it's. If it's worth like pursuing further or not.

D

I mean I, I looked at it for a while, and my conclusion was um what we're seeing here is the ruby vm allocating or most of the time was spent or most.

F

D

Analogs were spent on allocating ruby heap pages well, yeah, which we need to do if we consume a lot of memories. So I I didn't find it super uh enlightening.

E

D

Look at these yeah, uh but maybe someone else can have another step. Take another step at this. Okay.

C

So this situation yeah I'm going to create an issue like for these empty houses, with some uh like a few words so like. If you could matthias the next standard.

A

Can we move on to the next epic I'm just trying to move it along, because we have a very long list of topics: yeah try to hack spring, to run production, omnibus and, let's say just for your context. We're trying to ensure here that everything that is covered in those topics is reflected in the other issue for stack, ranking, just making sure we're not forgetting anything.

G

C

um So I'm not sure if it's even like feasible that we ever gonna do it like it brings like the memorial benefits of like running from the common process, but there is like still a lot of aspects that I don't understand why a lot of copy and write happening when you puma process, basically so I'm kind of like yes, like it's beneficial on one hand.

C

On the second hand, like it's with the height uncertainty, if we can, if it can be done well because, like it's, like the signal handling and like alaska, scorpion right being not the best as well, so it needs further investigation, but also like the impact. Is it's tricky because we're not going to run that way for the big applicant for the for the big installations. We may only decide to run that way for very small githubs installation, because it's not really like very desired way to run github from like the single parent in this.

C

In the big instance like in the cloud native, we speed the processes even further. So I kind of got to that point. Thinking that like it doesn't make sense to pursue that aspect at all because, like it's usability, is very limited to only very small installations, because in all other cases, we're not gonna run that way, it just doesn't make sense.

C

C

I'm kind of thinking that, like yes, it gives benefits. It requires a lot of more investigation why these benefits are so small right now, because they should be significantly larger, but, on the other hand, like.

C

It's pretty complex and I'm not sure if you are, if you would ever want to support that way. That's my issue with that.

A

To me that sounds like too large and too undefined with too much uncertainty to you know continue. I mean like like it's.

C

Maybe I mean this would be: maybe not a problem if you would see like a very big benefit. That would have the very high reach, but I kind of considered which to be very minimal in this case, because you would use that as a last resort only in a very specific circumstances. We have not used that in general. That's that's my issue with that.

C

So I'm kind of thinking that, like since the rich, is very small and all of these other aspects make it kind of, like I'm sure, I'm not sure if it even makes sense to invest more time into sweden.

A

Do people agree with that.

A

Okay, um can I move on cool check the impact of swap.

C

So I was playing with different swap settings, but it's not really about like anything that would improve github performance, it's more like maybe ensuring, like the same defaults for this for this constraint installations. What amounts of the swap you should be using and in what circumstances?

C

So it's more like the documentation output that we may I mean fabian. We may even like consider this to be like the bullet point in, like recall, of the running guitar from the constrained environment, something that we discussed on the monday because, like this documentation today mentions raspberry pi. But maybe we want to record this documentation to discuss how to configure github for like for the two gig installations, and this could be like one of the bullet points, how how the swap should be used as well.

C

So um maybe maybe our issue here, autofocus of that we should be like re-whole uh the memory running github on the constrained environments, documentation and this being like one of the aspects that we may that we should maybe cover.

C

But I I'm not I'm not thinking that this is much bigger than that.

A

Okay, so it's more of a documentation issue at this point.

A

um Do we need to capture that somewhere else.

C

um I'm gonna create this documentation issue because, maybe like as part of this price framework, documentation is also important.

A

Yeah, no, the documentation.

C

Is absolutely safe? It could be actually like the very first step before everything really to ensure that we have documentation reflecting the current state.

A

Yeah, no, please, um I think, then we can add it to the to the to the list.

A

Next, one puma in single mode.

D

Yeah alexi worked on this. I already took note that um I I'm not sure if we have an issue for this already, but it's definitely something we should look at. We we might just have to create a new issue.

G

Okay, yeah, I think boss me and camille. We tried like to run boom in single mod and it was like very effective, like I found it most effective for constraint, instances even more effective than nakayasha for okay. So.

A

Is this reflected in the in the um the rice issue list? No, I guess no.

A

um Yeah great, then, we should make a note of that.

A

Try to share aiesec.

C

I I I started to look at that, but I didn't do anything meaningful, so I'm kind of considering this for this moment that it's pretty complex to do.

D

Okay, I actually had a quite. We don't have to get too deep, but isn't this completely covered by copy and bright anyway, because icesack is never gonna they're, never gonna change right. So this should just be shared memory for anything that preforms.

C

Yes, like the preferred, will work, but the tricky part about that aspect that I I was thinking about is like. If you can share isaac as part of let's say, memorial mapped file, this would be as versatile between different distinct processes.

C

So like, if, if, if you're on like in the kubernetes board, if you would run from the same container image from the same file that is in the container, it would be served across different processes and not use too much memory as well, so uh it it was like my another way of thinking and ensuring that these 250 megabytes of the let's say ruby code that we load can be effectively searched if the system buffers basically across different processes that also works in the kubernetes world.

C

I just didn't get to actually doing that, but it still feels visible. Just it's pretty complex to do, and it requires a lot of changing in the ruby, vm and I'm not sure exactly yet what would have to change? I'm just sure that it's possible it's complex.

A

So, do we need to capture this, or is it one of those things that is maybe for now not completely relevant.

D

It sounds a little bit like the copy on right flagging. It's maybe a bit of a moonshot to do it short term, but it sounds like a really cool idea.

C

It sounds like a really cool idea, but I think it's a lot of uncertainty, but potentially it has very high reach. So I'm not sure exactly how to tackle that. But.

D

E

Those things topic.

D

That might be better to bring up not inside gitlab, but more rather inside the ruby core community.

C

um I mean like yes like in general, it would have to be upstreamed, but first like it would have to actually investigate if it's actually effective, so github code waste could be a good way to to prototype if it's actually effective, try to load parts of the application.

C

I think it's reflected. One is matias issue which is about gams and the second one. It's like that, one that I created yesterday about loading application, using contacts in which is running.

A

Okay, great try to run gitlab on j.

G

Ruby, I think we didn't even try that and that's why we didn't create an issue, because there was an argument that we're losing too much expertise like with moving to based ruby, which I agree personally.

D

Yeah, that's a bunch of issues with doing this. There's also problems with native extensions and all that.

A

Okay, so I'm going to let that one go.

A

G

Yes, we have an issue, but uh it's kinda included in nakayasha fork. I didn't notice any in gc compact issue, which will be negative work because I didn't notice any specific, with puma update itself when I I played with it. So currently we updating puma it's our priority, but it will result in gc, compacting.

C

There is one aspect of the five point x new, which is like uh worker.

G

Refork, it's kind of broken camille. I tried it, but I've seen a lot of issues even non-fixed ones. I could. I could send a link, it's it's very broken, so it's it's not something that we can really use. No, no. No I've seen like there are still open issues here.

A

G

Like I didn't quite get that I tried working in fork, I I could like discuss it later.

A

So I didn't quite get it. Do we need to open an issue or is discovered.

G

No, I think it's like kind of it. It will be done by by closing the issue. Gc compact, okay,.

B

So yeah super ruby, 2.7.

D

Yeah, that was just the parent theme, uh because we we're already on ruby 2.7, but we weren't necessarily taking full advantage of it. Maybe so.

B

There's a bunch of things.

D

um Like what I looked at was gc compaction, uh there was an.

F

Issue there wasn't even.

D

An mr, but we decided to close it because uh puma, uh why is it 5.1 or 5.0? I got confused at some point with the versioning but like whatever puma version, we're not on yet.

B

The one that we're looking.

D

To get onto actually has a config switch you can just uh tackle, which actually does this for us um compacting the heat before we fork into workers. um So we decided to just um not do it manually, but um just do it by upgrading to puma with the new puma version, which has other benefits.

A

Okay, but we we do have that covered in issues.

D

Yeah, uh yes, that is um an ongoing alexis you're working on this right, the five point: what is the 5.1 bump or.

G

Yeah yeah, I I sent to the chat like there is an issue for for puma 5 5.1. I created it like a separate issue, but it's like acts as a prerequisite for this 2 7 compaction, stuff.

A

Okay, so investigate a big difference between idle and runtime memory.

C

um I mean like uh I was able to partially investigate it, but there was like nothing really interesting that I did found on it, probably at least from my perspective. It would require more time to like maybe look at these mri flags that matthias mentioned, maybe like try to attribute more other locations that we talked somewhere else.

C

So it has potential high impact, but like I'm, not sure exactly why this memory is still like increasing so so significantly.

A

Okay, uh so should we let that one go.

C

I'm not sure like nicola mathias. Do you think that this is something that we should continue investigating.

D

uh Idle and runtime memory.

C

Yes, because, like there's like quite a difference between like idol and the runtime, that also impacts copy and write.

D

It sounds to me, like that's just the result of yeah the application consuming more memory over time or like mutating areas in memory. It's I don't know like I don't know. I can't say I didn't look into this at all. It sounds like this is probably due to a billion reasons. It sounds. I don't think there will be a single thing that falls out of this.

E

C

E

Of the aspects but.

C

I didn't really test that in the end, if this is also affects because, like tuning gc parameters being pretty aggressive, I actually reach to the having pretty constant memory usage over time instead of the growing memory usage. So maybe, like these dc settings, also offers this idle and runtime memory. I just didn't investigate that.

D

Well, I I looked at it from the uh yeah. I looked at it from the copy on right angle and how much we can what the compact, when I worked on the compaction stuff. um So what I saw was uh the the amount of pages we share as like um when the system is idle, it's quite good, but as soon as you put it under load that deteriorates completely like it will almost that effect will um yeah like deteriorate, you're right completely um so yeah.

D

I don't know why that is but yeah it's clearly pretty visible. It's probably just the same thing you you saw there. I don't know what you looked at um exactly but yeah it's just because the application mutates a lot of memory as it does its thing whatever. That is, it's probably.

C

I mean like like at least from my perspective. It could be the outcome of like application. Basically initializing, and I was I mean, aiming to figure out exactly which code paths are still executed like after starting server but before like in the time after starting server and processing. The first request so actually to see like what are the cpu cycles.

C

We are using that for, uh but also you mentioned about the memorial application like I found that, like making dc pretty aggressive and jmalok, pretty aggressive makes it to actually like allocate less amount of the memoriam, because, like in the default setting, we would allocate a ton of memory that we would never free and he would only keep allocating more and more and more over time.

D

Yeah, I also wonder if it could be because of meta programming stuff, where you go back and define methods after the fact, if you hit a certain endpoint and stuff like that, so that um whole class hierarchies that could be shared previously are not shareable.

A

Anymore, my suggestion here would be. It sounds like there's still a little bit to talk about right, but it it is. It sounds like there is investigation potential right, but it's not something that we may need to do right now. So maybe it's worthwhile opening an issue to at least not forget about this right, um but I'm not sure what it needs to happen. Sort of next.

C

uh Fabi, maybe my proposal would be because, like I'm pretty certain that dc settings we are at this reflected and the other angle that we might want to investigate is like um is there aspects that we may optimize between forking and processing? First requests, because this is something that may be affecting that. So maybe I'm just gonna create an issue to measure this particular aspect, which is like, I guess, like.

C

We have done like two aspects that don't affect this topic, which is one aspect: it's like gc settings and jmalok settings that impact memory, research, but the second is exactly what is happening uh between forking and processing, a request.

C

A

Is that something that we should capture in the stack ranking for rice as well.

C

um I think so it's still like it's just a store. It's still like a ranking, so yep exactly cool it might. It might end up like like at the end place of the lease basically yeah.

A

But it's still in the list, then right, um cool, okay measure, the size of features that uses language. I can appreciate a little bit more.

D

uh So this ended up making into the list uh through a slightly different issue, but we have it covered. We only have ideas currently for how to did you end up looking to this nicola.

F

No, I I didn't like, if you think about like marking and coloring thing, I didn't yeah.

C

D

More about later.

F

I was talking about how to dynamically load the files and the gems.

D

So anyway, this was part of the rice um thing. We just don't have it it's tricky. We have a couple ideas for how to go about it, but we haven't done anything.

A

Okay, then I'll move on disable unneeded services in omnibus.

D

Yeah um that that is uh covered uh because we have oh actually, let me check um okay.

D

So in the rice list of issues we had the gitlab exporter thing, which was only one of those components, and I don't know if camille had any other ideas and then I expanded that specific one into all of monitoring, because there's a bunch of other systems that came up as part of the default monitoring settings that we enable, um but that's as far as I went in terms of that, so I don't know because I think camille you looked at this as well right, I don't know if there was anything else.

D

I missed that we need issues for.

C

I'm I'm not sure if there is like anything else that we can disable to make actually still continue making traffic still to continue to work and from this our services is basically all the monitoring ones, because, like at least for the postgres, we might optimize memory buffers for the redis as well and like for the other services as well. But this, like this optimization, is reflected like optimized, gc and memory settings of the uh all other services.

C

So I think, like, like your issue, is probably very well reflects what exactly we might do, but, like you you're talking about dropping, but do we need to finish? Oh.

D

Gitlab next I'm not talking about dropping monitoring, uh just the gitlab exporter, because it's deprecated anyway, but uh in terms of monitoring I was suggesting- and that's really a product decision in the end is to not enable it by default on every for every customer in every environment, because I I I I'm not sure I don't have any data to base this on. But if I was running a very small single node uh gitlab deployment, for you know like a few dozen users, do I need grafana to monitor that system.

D

It sounds a little bit excessive. I mean the request. Volume will be so low like there's, basically nothing to look at right. I would.

G

Still say that we should enable it by default because, like small instances is extreme, but I I believe camille mentions that we could dynamically detect such small constraint instances and disable it for them by default.

D

Well, that's what I mean with disabling by default, like.

G

F

D

And then you they, they can be it's based on templates, so they can contain logic. I mean that's what we do for.

F

D

Defaults we size them based on how many cores you have stuff so.

A

Do we have an issue that reflects that discussion because I think that's a worthy discussion to have? Yes.

D

Let me check if it was in the rice list, because that's the stuff I'm working on this week, so it's quite new. uh No, it's not in the rice list, so I broke so I generalized the one up about dropping gitlab exporter, which was on the rice list to uh revisit and honestly. This is not something like the engineers can really work on, because it's the feature decision. I think it's something product managers need to look at, but um there is an issue I can mention it in the comment that I broke out, um which.

C

I'm kind of thinking, maybe one aspect that we may when we we may work on, is like add a function to omnibus the detects that you are running in the constraint environment, which may be defined by the two factors: amount of the cpu and amount of the ram and automatically disable these functions and print like a big warning message like that, this is disabled.

C

But you can like override this disable and still continue running if you want to so kind of have a way that, like we run same by default and make it like part of the omnibus that, like with that, like this constrained condition,.

A

So my my very high level thought on like how we can probably go about it and that's just on my limited information is, I think, first, you know doing um or making changes that um are essentially efficiency, improvements for everybody right so where you know dropping gitlab exporter, because it's deprecated and uses 300 megabytes of ram overall is great, especially if no functionality changes right, it's just being more efficient and I think there are quite a few things that can be done in that class when that is more or less exhausted.

A

I like the idea of considering sort of saying: oh you are running in a memory constraint, environment.

A

We have a way of detecting it, and then we change some of the behavior, because we assume that certain things are not needed or may save some memory because we know about it right and then, if that is considered, not enough sort of the most extreme step following that would be to say, let's disable certain things in the application that have a user impact right, because we, we know they're, very memory, intensive and they're not available to you right, and but this is sort of uh there's a product decision gradient right.

A

The taking features away from people because they're running on a memory concern environment is quite drastic. I think we, I think we would like to have people use everything if possible instead of my two cents on that.

C

um So like like, we discovered that at least like running in the puma single and like disabling all these monitoring services. As matthias mentioned, it actually frees a lot of memory that we could use for guitar, so it still doesn't degradate github, I mean the product, yes, but they graduate some some aspects of the monitoring functions of the gitlab. So maybe it's still like a good trade-off in this case and like the lowest picture.

A

But we can discuss that in in more detail, I'm just being mindful of time, um so maybe we can. We can make our way forward.

C

I I'm actually gonna, I'm actually creating an issue to like to discuss this omnibus. Thank you.

A

Disable just in time, compilers.

D

I think there was, um I might have created this, but I might have been confused around the direction of this. I think it is disabled by default and because it's very strange, like uh apparently rails is slower with jit enabled for some reason, because the overhead of jit just apparently not um a long story short unless someone disagrees, it doesn't sound worth looking into because I think ruby, core people or rails core people have already investigated that and said. That is not something you should be doing.

C

um I mean, like I read the mention that will be 3.0 max if, when users did like it's three times faster, so I I'm I'm, I'm not sure exactly yeah.

D

But ruby3 is available right so probably.

C

Probably we need to validate it ourselves if it is faster or slower with the ruby 3.0.

A

Happening, I'm not sure what the conclusion is from from that statement.

F

I mean we'll wait for robert.

D

Yeah, I think I think we can put it on hold cool. You can skip this one as well. That was just uh camille wanted to create a a benchmark test results or suite that we can compare against. So there's nothing to do. Do.

G

We have that, actually it would be amazing. I would think that we could maybe catch this sort of in an issue, because when we will be going forward with our improvements, it would be great to compare against some stand or.

D

But why do we need an issue for this already done right? We can just refer to these results.

C

There is, there is one aspect like, but this is more like the quality um thing like. Maybe uh the quality testing should have like one of these more constrained environment, with the smaller amount of the rps that we are constantly testing.

A

I think that's a good suggestion. My my question is similar to I think alex's says: do we actually have a way to quickly evaluate what our baseline is and are we able to then assess how our improvements that we're going to make are going to compare against that baseline.

C

uh Like we have all these, like our receipts, for this to be quick, but it's still like.

A

Semi-Manual process, okay, so, but we can do it, we may want to make it more automated and.

C

But it's not very streamlined. If you ask about it, it's not that like we should change and we can, let's say like in another one or two hours from some kind of automated testing suit and like automated environment creation. That would give us like comprehensive information. That is not be asked by any specific quick of our configuration that we made applied.

D

I mean is this something we could work with the qa.

E

C

Together, I'm kind of thinking that this is exactly what we should do like like introduce like these very small constrained reference architecture that we may not announce, but we may be testing to to see exactly how it changes over time.

A

I think that should at least be discussed, because I think that would allow us to really sort of track ourselves against something and if that's hard to do from my experience that you know will sometimes not happen, because it is just time intensive and other things pop up.

A

Okay, um is this something that I guess this is more of a collaborative effort at this point, so um we should capture that we want to do it, but I'm not sure we need to stack rank it for immediate activities.

C

Alexi, would you mind creating issue, maybe for that I'm just creating a ton of other ones. Okay,.

G

A

Okay figure out what holds memory.

D

That sounds like we already covered that.

F

Sounds like a duplicate.

A

Gitlab matrix instrumentation.

D

Oh wait was that in rice did we, I think we do have an issue, but I'm not sure we actually voted on that. Can we.

F

D

A

It's okay. I mean it was a good suggestion to go through this, so that we don't forget.

D

A

D

Can have an issue, though I'll add it to the list.

A

Remove prometheus metrics.

C

Yes, I I'm kind of like noticed myself that, like our primitive integration, can consume a lot of memory because of a lot of cardinality mapping and things like that, and actually I was able to free a few or maybe like 10 or 50 megabytes of the memory just removing prometheus.

C

So it may be, I think, more for us to take a look exactly how it works and whether we can optimize that basically, this permit is integration to have smaller memory footprint.

C

It's gonna be probably required anyway, because of the puma single mode, because our prometheus metrics consume use, ruby gbl, I mean block on the ruby gbl and like it makes it very bad for the performance aspects. So maybe it's not remove promises matrix, but figure out a way to make it more efficient and maybe figure out to move away from our fork of the prometus client as well. So.

A

Do we have an issue for that.

C

I think that we had some issue, I'm just not.

C

Sure I I'm likely gonna create another issue like with this specific aspect, and then we can maybe link to the the issue from the monitor team. But it kind of gives me a perspective that it actually can. We can free, maybe 20, 30 40 megabytes on like on big github installation, for a single worker. Basically that would otherwise be consumed.

C

So I'm kind of thinking that it's gonna give us a lot of room more.

A

Okay, so you're going to action that one.

A

Tune gc, params of all other ruby processes.

D

I think we have an issue for this right. I think.

A

It's correct discover unused.

C

Code, yes, I I don't have issue for that. I was thinking that maybe we could add a very simple addition to the ruby vm that would pin each metal that was executed ever as part of the stack and then kind of gather match rigs to understand which methods were not ever executed as part of the execution of the application block and maybe see if there are parts of the gitlab or parts of the germ that we steal out, but they are not used. I just like them.

G

Is there a gm which doesn't like coverage stuff? I believe they don't.

D

Recognize it in.

G

Production, but they have this feature once they show coverage of which.

D

But, like ruby is a dynamic language right. How do you even discover that, because things might be load, loaded lazily, um if you have like a test suite that does not instrument a certain path.

C

Yeah I I was, I was thinking that in the context of getting this breakdown from that production, so what coverage works, but it's super slow actually to run that on the production and that I could simply add a single byte for each uh I seek, and basically this is the way how I could very cheaply track.

C

If something was ever executed or not, and then, like you would iterate just an object and each method has like a place where it's allocated it doesn't mean it doesn't mean that we would catch all methods because, as you are saying like, we are lazy loading, some aspects but even like from the existing code that we loaded, we could see exactly if there are methods that we are never executing in some contexts, and this.

D

Is like my idea.

C

D

Yeah, could we make this part of because we already have a story for tracking memory used based on features uh which would require us to hook into, like crate, trace points or something to find out? What's actually going on in terms of code that we load? Maybe we can kill two birds with one stone there and um track also yeah other symbols that are never being being accessed.

D

It sounds like a very similar task. Somehow.

C

um It's similar, but different, but probably from the same category. So the question is like: do we need a category of saying, attribute memory usage to the groups because then, like you, have the attribution of the runtime allocation of the memorial and you have attribution of the code being to the group, so I'm not sure like mathias. Can you handle that and like figure out how to like, make it nice with the issues yeah sure.

A

Okay last thing last thing um extend ruby vm to provide the detailed gc compacting details. I think we have that. Don't we.

C

I I have some parts it works, I'm I'm sure if it actually makes sense to create an issue for that. Okay to give us more visibility, but it doesn't really make us do anything more meaningful in terms of the memory reduction.

C

A

Cool okay, so deep breath.

C

I need to mean also finish: creating these issues. Sorry.

D

No worries, I think I think we should also go through the next epic, uh because there were also a couple stories that were in there and there's actually a whole epic. A sub epic in that which uh I hadn't looked at at all, which we also didn't talk about in the rice, which.

A

One is that in here.

D

Yeah it's um yes, it is uh the one about giddy, ruby.

D

um There's a sub epic: yes, remove the gilly, ruby, sidecar, there's a ton of issues in there uh coming.

A

From a different team, I think yeah I would personally, this is clearly something that's being worked on right now, right yeah! Is this even something that we need to like take into account for us at this moment.

D

I I have no idea like I'm not sure to be honest. The first time I looked at that today.

A

Yeah, so I'm not sure why it is in that epic- and I think um you know that's- maybe another thing- maybe next week, if I have a little bit of time, I'll go through our epics and ask questions about why certain things are where they are um so maybe for for this sake of moving forward. Can we just leave this for now.

G

Yeah sounds reasonable not to like jump into it since there is work is going on, but I think it's in this epic because we expect some improvement in our right. So maybe it could happen silently and we'll see something on matrix if it will be finished cool, but I don't think we should work on it. Yeah I mean.

A

Have I okay, so one second, I think I've lost our epic.

A

And this is the rice, epic and I need to.

A

A

So quick question: matthias, did you add all of the issues that or the the things that we forgot down here.

D

um I added a comment listing those we already.

F

D

But there were at least three that we don't have issues for yet um so I have not created those yet.

A

Okay, that's fine, but we can. We can still. We can still sort of rank them and add them to our.

D

Yeah, so these first five we we can just uh add them to this. I just didn't want to edit the description while other people uh do, but those yeah.

A

D

Think those are valid to be voted on.

A

Okay, so these here, all of those are essentially not yet accounted for in our existing stack ranking. Is that correct.

D

Yes uh and like there's one that we're already working on, so I'm not sure if it makes sense to vote on it. But.

A

I think it does. Let me just do this. Okay, quick question. As this is a relatively long long meeting, do people need a five minute break before we continue.

D

uh I'm okay either way.

C

I I think it was hard for me to finish the issue creation. Okay, then that's.

A

Maybe just a coffee yeah. um I I like that idea as well. Thank you for um going through the issues one by one, though I think that was quite insightful, at least for me, even though you know um you know it makes it a little bit longer so see you in five game.

F

C

Fabian, do you think that, like we should um like score these things like the new ones, uh like alone, or should we like try this?

C

A

Share my screen.

A

So I suggest the following: we just do these scores as a group together, I'm just about to add them on in here and be really swift, and that's probably good enough. You know it's not about scientific accuracy. We can compare them to the average scores from above.

A

That would be my suggestion just to kind of get this get this done now that we're all here, um but I'm open to suggestions from the group as.

C

Well, I'm fine whatever. I think we should also actually release the the puma single.

A

I'm adding all of the remaining things here.

D

Oh actually, that epic, you just posted that that was the one we just looked at with giddy ruby, that's already being worked on. So maybe we can just skip that. Sorry about that.

A

No worries thanks for doing this. I I appreciate it's a bit more more synchronous than often is the case. So thanks for going along.

C

For the swap documentation issue actually created, I I didn't okay, that meaning.

G

Also fabian could represent baseline performance- one it's not yet created, but camille asked me to create sure we can do.

B

C

I'm okay, I created more. I added.

A

Okay, I think let's look at this, it's incomplete, but I think that's fine. This is.

B

B

C

ah Actually, I added we have okay, uh this really hard running heat upon constraint, environments. It's I renamed that we call documentation for running on git clock.

C

Okay, it's fine.

G

I personally found like editing, markdown tables, quite painful, so they're.

B

Really thinking maybe.

G

During the process, we could, like I just spreadsheet, in google docs and then just yeah.

A

Yeah yeah yeah, it's just mentioned it's getting quite unwieldy.

G

Yeah, because all these columns- yes.

C

Like like, like you, can add that in the in the google spreadsheet and then like just copy paste, it.

B

A

That, but I probably can't do that easily now right, it's like well, it should be fine. This is just rice right plus the.

C

The same I I know, but we're gonna have to calculate score by hand.

A

Yeah, that's true! Well, I probably.

D

Should have done this.

A

D

I actually copy and pasted that, and I don't know if you're on a mac, if you use alfred or something like that, I I'm on linux, but I have something similar. You just copy and paste that into like the search bar and it will tell you don't need a spreadsheet but yeah. Okay,.

A

I will yeah we have to. Let me think about what to do to.

G

Have a spreadsheet in in the discussion of rice just comments, I think we can.

C

Figure out how to copy that in the spreadsheet, just keep coming.

C

I'm just just trying to naively copy that.

G

I think marketing table to csv should be like quite common.

D

I would just try to just select the html and copy and paste it into sheets, like it's usually.

F

Smart about this.

D

F

I I was muted, I guess they did it like just copy preview, yeah like that yeah and paste it on the group.

D

I think it doesn't deal well with empty columns, but just give it a go.

F

It's not perfect, but it will work but.

D

I think that you can select.

F

You need to select the whole table.

D

Yeah by the way, fabian here's another life hack, if you type sheet.new in your address line, it will actually take you to that form here.

A

And I'm learning so much today.

D

Okay, this did.

F

D

You didn't select the whole.

F

The whole table.

C

Yes, like, like you, need to select the whole table and then it's gonna select.

F

From the top like select the even the yeah, the text yeah like.

A

F

A

Okay, holy mother of mercy.

A

Okay, one second I'll just do wet wrap.

B

Okay, there we go okay. Can we do this quickly.

A

Impact on performance reach.

C

I think it has like.

D

E

Mean 600 or.

D

Something because, like I think, sorry like, because I think we like just to align on this for me, like all of these things that touch the runtime they affect everyone right. So I think, by definition, that should always be maximum reach because it reaches.

G

D

It literally reaches every user right.

G

um I'm not sure.

G

G

um Let me find an example.

F

Because- and there is an impact, so it can have high reach and low impact on customers.

D

Exactly exactly that, that's what I.

F

Think is because the.

D

Like, let's say, there's a bug in it, then every single get that user would be effective right.

A

Yeah, so so I think if we have something that actually will touch, gitlab.com right and every single self hosted inserts. The the reach is relatively high, but the impact may be negligible right and that I mean like.

C

I I mean like I there is that point because, like the first one, I completely agree with materials that is like 1000.

C

I think his explanation makes a lot of sense, but, like the second one like prometheus matrix integration, it feel like impacts everyone, but it doesn't impact everyone in the same like because not everyone has like the very high instances and like various spread, so I'm kind of saying that maybe in the second one rehard prometheus matrix there, which is lower because even though yeah everyone, but not everyone, equally, that's my thinking.

A

B

Okay, let me just quickly do.

B

This e for divided by.

A

Okay, so reach impact.

C

Either it's gonna be great or it's gonna be terrible, but investigate so it doesn't mean enable it says investigate, so the impact could be awesome on or terrible. So I would say somewhere between one to two from my side. Let's.

A

Be conservative: let's give it a one and the confidence it seems, is really low.

C

Nicola matias, alexi, can you give your opinion on the impact, because no idea.

B

A

Okay, so the confidence here is is pretty pretty low. Is that correct.

C

um I mean investigate confidence in this case it means enable and test. So I guess from that point of view doesn't mean like ship it to the customers. It means evaluate.

C

So I would say that confidence is pretty high, so it's either zero point a or one because yeah it's it's already down there.

A

We don't have to, I think I think the confidence here is like do. Are we confident that this is actually going to do something not that we can enable it or d uh disable it? Like the confidence of you know the you know that this will have the impact that we think it will have.

D

Yeah, I I I'm personally almost convinced it will have an impact. It's it's a pretty massive change in the way that vm operates. I would be very surprised if we wouldn't see any difference, good or bad.

D

We just don't know which one.

A

Exactly okay, that's why we need to investigate so I'll, give it a 0.8, maybe.

A

C

So probably testing when we have like all these qa architecture, it should be pretty straightforward. I guess something between one to two to have like this.

E

D

Yeah, I had a question about this, and how did you go about this when you did your race voting, because because there's effort in terms of it takes a long time, but it's not actually a lot of work because just like monitoring something just letting it sit in production and collect data, it takes longer. You know it adds kind of lead time more than it does um to the.

F

A

D

It does adding work. Do you know what I mean like it's, not really that I need to spend eight. I would say.

A

Looking at it, I wouldn't overthink it. You know it's like this is.

D

This is rough right if you.

A

If this is, you know, going to take two months to actually come to and to a conclusion or three months, then give it that value. It's it's more about t-shirts, no.

D

Sure, I'm more wondering so it's more important about kind of the time to market like in the sense.

G

D

When will we actually have a result right rather than.

G

I would say yes, I think this is.

D

A great example because this is just a switch which it's like a literally a boolean toggle, so it's very low effort in terms of development, but it's very high effort in terms of validating that change.

A

Yeah but then I think that is like, like you say it will take two or three months, maybe two to get there.

D

You know, I would say two sounds right.

A

Cool all right next, one prometheus metrics.

D

Sorry, sorry, what I'm not sure I understand this issue to be honest. What.

A

D

Permit, do you mean re work or overhaul, as in like redoing.

E

C

Like figure out how we can make it more optimal, I I mentioned like a few aspects, exactly why there are some optimal today and, for example, why it's from the memorial perspective.

C

It doesn't scare very well for the big installations running multiple workers, so in this case like right, investigate exactly how we do it and like how we could do it better, because this is this actually has the amplification impact on the growing memory research over time, because we never wipe these metrics when you start counting them until the process restarts- and this is one of those of the reasons why the memory usage grows over time of the application, because we have a few packets of different types of the metrics, and each of these buckets per process requires us to end up at least four megabytes of the file.

C

It doesn't mean that we map order pages of that file into memory, but reading that file actually makes us to. We periodically read all these files every 15 seconds or 60 seconds, depending on the scraping.

C

So it means that you, if you have 20- let's say 10 workers, it's having four buckets of the four megabytes, and sometimes it's even more because you have more metrics that uh over this four megabytes, initial location, you're gonna require that amount of the mmr. So four by four by ten 160 megabytes in this example, at least of the memory to be able to scrape metrics from the third processes. Basically.

C

So what I am saying like that our prometheus matrix integration is kind of reaching its limits of the scalability from the scraping performance from the cpu from the memory usage performance and, like all of these aspects, make it not really the greatest.

D

Okay, thank you.

C

Thanks for the explanation, there is a.

B

C

Standing issue about like reworking this integration, but it's kind of going back and back and like it's. I think it's like it's no, no hanging fruit to do it, but it has the impact on the memorial. It's from our perspective.

G

But by this issue recall you mean: rework it. Do it less memory right or what or investigate what.

C

um That results in reduction of the memory. It could be even like saying that initial allocation is not four megabytes but 550 kilobytes. It could be that we periodically try to wipe this metrics from the memory. It could be try to run the upstream librarian. I don't know what it takes really, okay, that we could do it.

A

Better, I think I think I would suggest we move. We move a little bit quicker because otherwise we're not going to be able to finish this, and I think if, if there is not as much understanding, I would vote for the person who has the best understanding can go ahead and we can always iterate on it later on right and you know get into the details. But I would really like to finish this, so we can have a a result.

A

Does that sound, sound, okay and I'm not saying the the explanation is not valuable or anything. I'm just saying we have 20 more minutes so, okay come here, you have the best understanding go for it.

C

uh 600, it's not it's not um or I don't know so it's either 600 or 1000, because it actually affects everyone, but not the same in part. I would say it will be two okay confidence uh 0.5.

C

I said four okay, um it's like it has a lot of uncertainty. What to take. Maybe there are smaller iterations, but it's not super straightforward to do it.

A

Thank you, and you can see like this- is what I like about it right. You can really see this reflected in the score already: um okay extend omnibus to automatically disable non-essential services, um so this is only going to affect those that actually run in the memory constraint, environment, correct.

C

Yes, so probably like, which is very small: it's like 100, maybe or maybe smaller impact it makes github, maybe usable. So I would say one or maybe two.

A

Let's be generous, give it two confidence.

C

One effort because, like it's, it's reduces memory effort, probably one as well.

A

Okay, empty hashes.

C

But yes, it's more your topic. I guess.

D

Yeah um reach, I would say again, a thousand because it's like literally the whole app and every user, um unless anyone disagrees uh impact, I would say, is smallish. I can't for the life of me remember these weights. um Is it like lowest a half, I would say 0.5, okay,.

G

There is 0.25 also like minimal.

D

That's minimal yeah; um no, let's keep it a little bit more generous because it depends on the type of hash. I think that we're looking at um confidence um give it a point eight, because there.

F

D

Definitely open questions, but I think it sounds like do best doable stuff in there effort uh give it a one.

A

Next one rehaul, so when we say rehaul we mean rework.

C

Yes, am I, using with these three whole world property.

A

I think it is, I think it's uh I think the word rehaul is fine, but I think rework is maybe a little bit more common and more understandable from for poor non-native speakers like myself. So, okay.

C

Okay, like it's still like 100, because there it is like quite limited right. In fact, I think we need to do it so, like I would say it's high, so maybe like even three or like two confidence.

C

Do we need that? We have some discard, like metrics of maybe 0.8.

A

0.5 right, it's like unicode emojis, don't quite know. I don't think we discussed it.

D

This is really hard to rate because I think that's up to another team. Basically it's just. We discovered that I guess for front end.

F

D

So we have this ginormous hash in memory all the time.

F

D

Json mapping of like unicode or like shortcuts to unicode emoji or something uh so I don't know like I mean reach should be pretty high because I think it's always in memory for everyone running gitlab uh yeah, it's making 800. I guess um impact is quite small, because it's also not that massive, give it 0.25 1.5.

E

I think it was a megabyte and a half or two meg or something so how this amount compares to the duplicate, empty, hashes.

D

It was similar, I think it was a bit less. I think it was half of it yeah, so I think that makes sense actually 2.25 here uh confidence. I have no idea. um Actually, I have low confidence because there's probably a good reason. We keep it in memory, give it a low confidence, um yeah sure uh effort, can't imagine it would be like I don't know, we have 0.5 yeah.

A

Yeah use memory, impact of gitlab monitoring.

D

Yeah, so to me, this is kind of the product portion of what camille uh discussed above about extend omnibus to automatically disable non-essential features. I think we need to decide what is non-essential and for who, okay, so.

B

I would say.

F

D

This is more like for product managers. I think to look at this and decide.

A

So I put myself on the spot, but.

D

Not just you, you know, also monitoring at the other teams.

A

Let's say like the the reach is relatively small, though right, because that is for folks that are running in a memory constrained environment. Right, I would say, the the impact is likely relatively high, because you know you're disabling specific functionality right yeah. My confidence in this is probably also relatively high. You know, as in you know, we would that we expect that to happen right the effort. I think this is maybe now like.

A

Let's say, I think the effort here is actually three, not necessarily in engineering time, but more in getting everybody aligned that this is a good idea and that it's important and that we want to do.

E

It in the discussions and.

A

All of that blah blah right. So that's a three.

A

Okay, reuse impact, no drop, instrumentation module.

D

Maybe nicola can take that one.

F

Yeah, why not? But okay, so everybody are affected. The impact, if I remember correctly, was really low. 0.5 yeah, maybe even lower.

F

The confidence I think between 0.8 and 1, because I think it's not used at all, but maybe there are some hidden thing, but I really believe that we are not using it. It is just left over from enoch's db, so I guess it's fine, you guys agree and effort is also low. I would say 0.5 right.

G

I could comment on that, uh so we reach will be low the same as we. If we put 100 and 100 impact is high because like for them it it will give room to breathe confidence. I would say medium.

G

uh Medium is zero point end, because an effort is pretty high. I would even put three or four pessimistic, because we have a lot of things like for clustering.

D

But we we have pretty high confidence that this is pretty impactful right, I mean didn't we look at this during the.

G

Medium, I mean 0.8 yeah yeah. I agree like it's not 0.5 0.8.

D

um And another question: uh maybe that's more for camille, but um we also. It also came up that in production in kubernetes we run oh wait or what's that, for maybe I'm confusing it. Maybe I'm confusing with sidekick no.

F

I think it was puma.

D

uh We we run like a really low number of processes. I don't remember it was one or two, but I because we scale with pods now is read or could reach actually be way higher here in the future.

C

I'm not sure exactly like in the how in the cloud natively run, but if you're on the single process, then I would say that for mine in the single mode, it.

E

Actually makes a lot of sense to run there as well yeah. I think it might be interesting to investigate.

C

Yes, so, but I I'm kind of curious, because this shows like the score is very small, but this is this- was like the single, the biggest impactful thing that we had in the like in running in the 2 gig.

G

That means impact is not that effective in this table, like.

C

Diminished I mean like the reach: it's like it's optimized to the general audience. It is not optimized to the two-week goal of running on the constraint environment. Well, I.

A

Think actually, like I would say that is true, but also it is an interesting question now right if you acknowledge that this actually may reach a very small amount of users right in comparison to many other users.

A

I'm not saying that's necessarily a problem, but it is an accurate reflection of the impact that it's going to have on gitlab as a whole. Right, if you do something for a very small amount of users, you know because you think that that may still be important, but the reach is still low right. There may be other reasons to overwrite it, but I still think that's an interesting perspective here right if there are.

C

A

Hundred you know, but.

C

Like but like really the question about the reach which is like, uh and it's the question to you, fabian like what we I'm kind of like having our optics around, do we because, like we had this 2gig week, it's our optics on actually being able to run guitar from the construct environment, which is like our focus right now, which kind of makes this reach uh here for the constraint environments be for the puma single mode, not 100, because on 1000 it's gonna affect every single.

C

So what you're saying is we may need to rescale to say no, I'm kind of saying that maybe we should consider looking after each like which of like the github as a whole, as us like within account.

F

C

Every every github installation, but second reach to think about from two gig perspective. Only.

A

Okay, that's a good good suggestion. We can. um This is that.

C

Because because, like it kind of gives you a very interesting aspect like the score is 60, but from the 2 gig perspectives, it will be like the first thing, probably that we would be working, because it gives like the highest.

E

Benefit, but also is the effort, maybe a bit too pessimistic.

D

Why does it take four months? That seems like a lot.

G

D

Piece of configuration we need to change right, we already measured the impact of this.

G

D

It will be a lot of stuff.

G

Like well cng only buzz configs are tied up. That's super coupled to like our.

D

Cng was I mean okay, I mean cng. Was it's a bit speculative still right because we don't run kuma single, promote and cng right now.

G

No, I I mean what I'm afraid of like. Maybe we could put three and find this three, but I'm I'm a bit like pessimistic about our callbacks. Many events are coupled towards clustered mode and many.

F

Optimizations are done on clustered mods, so.

G

It will be a bit of a mess and configuration which I find really struggling to to verify.

E

Oh yeah, no good point yeah thanks. I I I agree with father. I feel like.

C

It's like it's, it's gonna take a little time to actually.

D

Yeah, no, maybe.

A

Let's leave it like this and I I like the um concern in the ranking um about local versus global optimization, really like how far is it going to get us to this specific goal right versus? What is it going to affect globally right that we can maybe think about that, but that's also easier to see when you, when you have the ranking for everything and see you know how it actually turned out. So let's do the last two things.

C

I I'm kind of thinking that maybe like when we heard at least maybe we should like then asynchronously to monday. Yes,.

C

Looking at the 2gig versus global, yes, that kind of thing like see how this piece compares.

A

Yeah, I I agree with that, but I still think this is super valuable because we we need to have it first to actually look at it right, so swap documentation issue.

C

I actually added that to the documentation in general, so you may just remove that. It's one of the points in the rework of the documentation. Okay,.

A

So discover unused code.

C

I mean it reaches everyone, because it's gonna reduce from.

E

Our consumption impact, I think it's pretty much the right. The funny thing is we don't really.

D

E

How much unused.

D

It's kind of self uh yeah. We need to play that story to even know how much unused good there is exactly so like we don't.

E

Know really how much we can free.

A

But their confidence in this as in is actually pretty low, because we have no idea what it will do.

E

C

0.5, probably or maybe.

D

Coin: flip yeah, but yeah the.

G

Impact will also be fairly.

D

Low, I can't imagine, there's a ton of unused code in gitlab, I mean we have a lot of churn, but we, I don't know.

C

um Maybe two: I guess it really depends how you approach that, so I I think kind of reflects, like my perspective, compared to other stuff.

D

Yeah, I think camille raised a really good point with this local versus global thing, because now I'm thinking back now because we were very- we were not really focused on dot com right. When we did this two gigabyte, it was really about hey here's, a single note, 2 gigabyte, you know user. How can we run gitlab on on that? And and now we started to reduce reach a lot that kind of down down scoring really like stories.

D

That sounds really impactful, uh but we we down scored them in reach because they do not affect.com that yeah. I wonder like yeah now in these scores, because yeah I don't know I I didn't really think about that when I did my rice scoring.

A

But that's why we like we can do this as like a a global and a local score, for example, right.

D

A

No, I mean this- I really like this because you can have.

A

And I think that may be a balancing in priorities right as well. If we have things that are both good for global and local optimization. Well, then, that's actually really the things we should do first and then there are some things that are maybe much more impactful for the local goal, but that we still need to do, but globally they don't matter, that's interesting to flush out as well. Okay, I think there's one last thing.

C

I mean, like you, may simply have a magic that, like you, pick marks of these two values and you order by them. So you kind of do like the highly impactful things that gonna impact like the global and local in the end right, yes,.

A

I mean that's the optimization. We probably want right so um so the last thing we have to do is this baseline performance bit and then we're going to be done.

G

I would say it's tricky about each of these tasks. We've had similar tasks. Sorry, so it interrupts you, but I I'm.

F

G

To put reach on this tax because we also had a task on making features, but you could say: reach 1000 and you could say: reach 100, because developers only so what's our reach on developer tools, that's my question.

C

But is baseline performance developer tools or like something different.

G

I would say it's a developer tool because it's only used by us, it's a user, won't doesn't care if we have this baseline performance in our two worlds.

C

Because I was thinking that, like the baseline performance, it's like it's more about like having this constrained architectural part of our reference. Architectures.

G

But that's also about us right how how effectively we work, how effectively we test how effectively we could verify our improvements. It's it's not about, like user, effective, it's about our speed of work, so it's very meta kind of task.

C

Actually, like we had each lengthy discussion on monday about like uh gathering as much metrics as possible to see our like the trend of the performance, which may include memory and other aspects.

C

So I I think that I agree with you alex. I. I have also hard time like to understand. I think I think.

A

I think personally, maybe this is just one of those things where in my mind, maybe this is a radical thought. If we can't measure the baseline of what we're trying to do, you know, then I mean what are we like? How are we going to work right?

A

So it's it feels to me like a thing that we just must establish at some point, because otherwise you know it's a little bit pointless because, like we're changing stuff against an unknown entity right, so I don't even think we need to rank it necessarily because it feels like an absolute prerequisite for measuring what we are trying to do. In my mind, sounds good.

A

Yes, that's maybe right the best way of doing it. Okay, so um you know I'm not beholden to like religiously follow these things when they don't make sense. um Okay, so let's recap: I think I can't actually easily get the summary table into this right now.

A

A

E

A

I'll try to I'll try to do this separately for it, but we can already rank.

A

This here right.

A

And we can, we can look at the the top items and we can look at the other top items. So my suggestion is because we're out of time I'm going to make this table in the google docs here and share it with the team with the ranking of our of our items, and then we can. We can make some comments on it.

A

um I think we may need another round for sort of reach within the 2g one, and then we need to look at sort of global versus local, but I think what we already have is we have a essentially we have scores for everything that we tend to do, which is a lot right, there's a lot of stuff going on here. So we should be a little bit careful as to sort of over subscribing right as in you know, there's a long, a long stack, um but I think we can probably come to a I mean.

A

We're not blocked is my understanding and I think, by probably monday, tuesday next week. I think we can derive priorities for the epic from this. I think I'm quite confident in that.

G

Stupid question, but shouldn't certain materials shouldn't be reached, be one thousand on every like two gig stuff, because, like every issue when we created that we had in mind like constraint, environment just do we need to like assess it. Maybe we should.

C

Just put one, I I mean like investigate, gt impact on the performance. I would say that on 2gig is actually bad because it allocates more memory so like it's rather opposite in some cases,.

G

But that's rich, I mean reaches everyone for every issue because, like all issues were created with constraint, environment in mind,.

D

Though right, because not every single node user might be using the same feature set, so maybe.

C

So maybe like uh it's actually interesting point, maybe we need to reach an impact.

E

Do you mean local versus global impact.

G

Yes, I would say, impact will be much more likely than bridge for two gigs might be.

A

G

I would probably put one thousand everywhere: if you ask.

A

Okay, so what you're essentially saying is that, because we created all of these issues with the reach being tailored towards the 2gig initiative, if we there is no difference in ranking here, the difference is really between global impact. You know, as in will it impact all of you know like. Will it impact globally right, or is this only something that goes into um the 2g impact and that changes the rest is stable?.

A

C

I I would like to keep like the rich and impact as a coefficient of the local and global I'm. I need to think about the reach myself. If it's 1000 always or is it not.

A

Let's think about it, we can do all of that asynchronously.

A

So I I would say what I can commit to is I'll get our average scores into um into this table here as well and I'll I'll rank by that, and then I'll do that today, and then everybody can actually look at the stack rank and see if they sort of violently disagree with some of the ranking right and then we can figure out why that is right, because that is also what that teases out right.

A

If, like our perception, is not consistent with this, then we need to maybe think about it right, but I think we'll have a scoring system in place and also we have a place where we can keep track of all of those issues, because I would like to put them into one epic with potentially sub epics, so that we know exactly that. Those are the follow-up steps that we're going to take, because that's important for me just in order to not lose track of. What's going on.

D

Sounds good sounds good. I had one one more thing. If you don't mind, no.

F

D

Thing that bugs me a bit still is that um this one story that got a really low score.

D

I think we're kind of putting the card before the horse here, because it already looks like we'll never get to this, but that doesn't make sense to me because, like the one we have this issue up, there extend omnibus to automatically disable non-essential features or services, which has a much higher score.

D

But we cannot even do that story until we decide what is non-essential, and I don't think it's our place to decide that. So we need to do all this product work to decide what is not essential to actually, because that's just implementation work so to.

A

I think that's also fair. There may be some cross dependencies here that we have to tease.

D

A

If there, if one this, is so just to take a step back right, this is not the solution to all of our problems, right as in figuring out some of the dependencies- and this is a it's a toolkit to take a a long list of stuff and get some relative ranking and also facility facilitate some discussion right, because I think what we've seen here is that maybe on some things, we have no common understanding on.

A

Why we're trying to do that right or who this is for, and I think that's actually also really really important right and that helps do this. So there's more than more benefit than then. The score is because you need to talk about it, but I think that is in its own right, pretty important.

D

Yeah, that's a fair point, I mean all I wanted to say was, I don't think we should just dismiss things outright just because they got a low score. I.

A

A

No! That's that's perfectly true because, but it's it's important to write that down and say this has a low score. These are the reasons for this, but this is the reason why we decide to do it anyways right, because that's something to to document right so that when people come and ask, we have an answer to it right, and I think that is important, and this is why having these things sort of ahead of time is, is really nice anyways.

A

Thank you for doing this. I think this was maybe a little bit new for some folks. um I hope it was useful and we are not done with it, but I think um there was a at least for me. I learned a lot. um It was probably one of the longest synchronous sessions I had in gitlab. So that's exciting.

A

um It felt almost like being in a room with a whiteboard which I actually really like so yeah. Also, if you have any feedback regarding this right, um maybe we can do it I'll actually, post in the in the channel right like for a sort of a mini, live retro. Just to say, you know: how can we improve what should we have done so that we learn from this.

D

Cool yeah thanks fabian, for putting this together and very cool. Thank you.

G

Yeah, yes, thanks again thanks. Everyone all right enjoy your friday.

F

Yeah happy friday, guys yeah.

G

F