GitLab Memory Team, 17 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020 11 17 Memory Team 2GB Sync

Description

0:30 - Gather baseline performance - https://gitlab.com/gitlab-org/memory-team/team-tasks/-/issues/75
- Initial impressions and swap space needed
- Discussed using Spring with Omnibus
- Application loaded only once
- Loading Puma and Sidekiq on top of preforking fast
- Rails console instantaneous with Spring preloader
- Most of the memory is unique to a forked process
- Only 200-250MB shared - this is known
3:33 Are we loading everything?
- Next steps - print out what code is being loaded within what process
- Determine why we have so many copies in memory
- Running GPT against 2G and 2G with split to compare graphs
5:40 Investigate Memory Content - https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/4
- Took 3 different snapshots using rbtrace
- Compared scripts and removed common objects
- Goal - get only the long lasting objects
- Repeated steps multiple times and got different results each time
- Also ran same steps for Sidekiq worker - no discernable pattern
11:25 Discussions about heap and malloc
12:00 Topic - Investigate used gems - https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/5
- List of memory usage per gem
- Maximum is 200MB
- Only includes just Ruby gems
- Re run with cut-off point of zero to see volume of things in memory
- Discussed loading graphql in memory (schema file)
- Discussed not needing to load whole API in sidekiq
- Can we optimize what we load? How much memory does the API package use? We can shave this from the SideKiq process
- Memory profiler - look into retained memory
19:51 Topic - Investigate Ruby 2.7 GC Compaction - https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/1
- Overview of garbage compaction
- Description of fragmentation - only free when entire page is free
- Discuss HeapFrag tool
- Demo heap on our processes
27:00 Experimenting with timing of compacting
- Not sure about memory impact
36:00 Discussing the time taken for garbage compacting
- Discussed different ways to configure gc
39:40 Suggestions on when to call GC
- GC before forking
- GC when application is fully initialized
○ Perhaps within initializer
41:20 Experiment with Puma in Single Mode https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/2
- Initial failures on 4GB machine
- Instructions
- Why we can't use it
- Next steps
○ explain difference in PSS
○ Test different request pressure
○ Describe why we don't use a single mode
52:00 What's next

A

Maybe we can go through topics and then maybe everybody can say, and maybe we can update document later or next time before the.

B

Meeting okay, so like I, I yesterday like spent some time looking on uh this baseline matrix, I'm like it's actually, what is alex also saying like 2g is like very slow because it's like heavy swapping and basically it's unable like to like run majority of the tests that we have in the gpt a piece from what I saw and actually like from what I saw there later looking at the memory usage of the 2g from the baseline matrix is that we are somewhere short about, like 1.5 gigabyte of the memory on like 2g so like it seems that we need to use about 1.5 to 2g of the memory of swap to actually to run anything and it's pretty pretty nasty.

B

Then I kind of spent some time with the spring. I kind of posted the snipper that I use and applied on top of the omnibus, but, like I only got about 250 megabytes more like more space.

B

So it's not really like a lot a lot, uh but from the interesting sites it actually had some very nice benefits because, like application was loaded only once all the like fork in copy and write was shared so like then running puma on top of that or like side key on top of that was pretty super fast, but additionally, an exciting cluster. Like works out of the box like with this preforking on top of spring so like.

B

If you would have to configure psychic cluster using multiple cues, it would be like pretty instantaneous to run because it would use this parent spring process, uh but also like running things like race console. They are pretty much instantaneous with the with this spring pre-loader because of like all the processes loaded. So like you get a spring console in like something I don't know 10 seconds basically, instead of usual like around one minute, so.

C

I mean it's interesting, sorry, just a quick remark. It's interesting that you say it's about 200 megabytes, because that seems to look that seems to be in line with what our pss readings yield right, because we we know, we know already that um most of the memory is not shared. Most of the memory is unique to a four process, a fork worker at least fukuma, uh and and only around 250 or even less than 200 to 250 megabytes are shared. I mean this.

C

Of course, yeah depends on how many workers you're running but like in the default setup with running two workers.

B

Yes, and actually like it's, I actually we have this topic, and I I think it's like it's due to. Are we actually loading everything really? Is there like some code that is then being crazy loaded later, that may impact, like our pages being used. So I think, like really the next step to figure out to do next is like somehow hook into the like the loading, routine and print out exactly what code is being loaded in what process, because, like I mean like I'm, I'm also surprised by this, like in a inefficiency.

B

I was kind of expecting somewhere around, like 500 max instead of 250, but it can be like indicator of that, like uh that, we may be just loading some additional code or like initializing, some objects that are like modifying the class objects. That kind of creates these a lot of copy and write pages.

B

So I think, like it's it's this, like it's part of the story about the idol versus runtime memory uh or like either or like fully forked, maybe like, even even before, to actually figure out why uh we have uh so many copies being done because, like I mean like this, this number is like too low for me, and I don't really. I think I didn't found yet exactly why it is so low. It should be somewhere around like 500 max.

B

This would be like there is an angle number that I think um so I guess this is really like what I spent time on and I'm actually like right now, re-running all my lgpt against, like 2g and 2g, with speak, to have like the the apples to upwards comparison on the prometheus, because I just want to like compare graphs directly like in the exact point.

B

So uh once that finishes, I I just want to like to continue working, probably on that spring um nicola.

D

How about you share my screen for a second.

A

So I was working on investigating memory content. What I wanted to do. I try to get like three different, like memory hips from the virtual machine that I.

D

Was testing and in parallel I was running the actual uh gpt uh performance test on that machine. So.

A

I took three different snapshots using rb trace and then I like this is these are the steps how I did it against puma worker and then I used the script that will like compare those three snapshots and remove the common objects from like first and remove the missing objects from the third one, and then I was hoping to get some clear like the idea was. The goal was to get only the long-lasting objects like that that leave like that will be not cleared like to detect the strings, symbols or different objects.

A

That will like stay forever or stay persisted in in memory, but I'm not sure that I understand these results and I repeated this two or three times just to be sure and every time I got different results, which is maybe okay, but I'm I'm not sure. I don't see any clear like places where these like leaked objects live. I was hoping to get more clear results in the end. I also run this for the sidekick worker as well, but again, I'm not sure about the the results.

A

C

Leak mean exactly.

A

This is just the scripts like that is printing, so the the memory dump looks really unreadable like. There are a lot of small json lines that contains like everything that was in the memory, and I used this script that I found on the internet. That is just comparing the three different like dumps and keeping only the common object that existed like and states like it does. Some combination, some.

E

Smart combination, and then it just leaves.

A

The objects that like lasted.

A

From first to the third one, okay.

C

So that's not a leak right. That just means it's a long-lived object.

A

Yes, yes, that's true! Okay, it's just the wording is like that yeah. I agree. It's not so.

B

Like so the total size, what is this number, like? The total size.

A

uh I think it just summarized the total, the total of those. What.

E

Does it say zero repeated.

A

C

I I'm not really.

A

Sure I still didn't have time to like look at it like that. I will spend a little more time just to understand the results, because the script is not written by me.

A

A

This is the description like what was the actual goal and what this script does, but I'm still not sure like why there are some zeros, maybe maybe like.

A

I will check I'm not sure, and especially that there were some time where I got totally different results with many less objects than this.

A

And I also had trouble to enable at all the arbitration on that machine.

C

A

C

Like you pull, can we what, if we just pull um uh just normal heat dumps, just to see what the actual contents of the heap are? I.

A

Can upload it but like.

C

A

C

A

Be just 600 megabits.

C

Okay, um right right, yeah.

A

C

Mean that makes sense, but.

E

B

E

C

You just like put up a script or whatever you used to pull the heap down. I think that would be useful.

A

Yeah as well to look at I did. uh There is a snippet that I used it's linked. Okay here, it's cool.

C

Yeah, because what I wanted to look at as well is um how large the ruby heap is in comparison to how large the process he is because, like I think, one thing that would be good to understand for us is as well like what is the limit like the kind of the ceiling for any of these things? We look at because there's only so much like the gc compactor can do right because it only optimizes the ruby heap. So I.

A

Can use like some sorry, I can.

C

Put the hip in some.

A

Of those visualization tools- maybe maybe that will help somehow, if, if you guys think it will be beneficial.

B

And I'm kind of thinking, just like probably also look at that um like looking at the heap and like how much memories are located on a heap of that's controlled by ruby versus how much is monologued by the sea. Yes, exactly if this is something that like we can understand and like because like hip, is something that we can control, but I'm just curious like if there is a lot of malloc.

B

Can we like recycle that somehow, like reduce that amount.

C

Yeah, that's what I was hoping compaction would. Do um I mean I can did you have, did you have more nicola or should I talk about compaction.

A

Just one more thing, like I created a different topic like investigate the used games. uh I just created an issue and I just ran quickly this morning the derailed benchmark, because it will give you the like memory usage per gem. So this is just a list, and what I can see here is that, like it seems that the maximum is 200 megabytes and I don't see any huge gems.

A

So I'm not also sure what to do with that, because there are also some suggestions that we should maybe group them and optimize and tweak how those gems are like included. But I was expecting to see bigger numbers here.

B

But does does it number also include like native extensions, or does it only include the ruby.

D

That's a good question. I don't know.

C

I I'm pretty sure it's just uh ruby, so yeah.

A

E

B

E

Taking the space.

C

So, and so it will only ever see 40 by 40 by entities and also keep in mind. There is a um you can tweak this right. So by default there is a threshold below which it doesn't even include uh uh uh file or hm in the total, because it falls it.

C

Has this cutoff point where, if it's too small, it doesn't count it, but if you have a million small things, they will still add up to a lot of memory used and they will never appear in these reports, so I would actually suggest to run it without the cutoff point like just set the.

A

C

To zero or disable it.

A

Good point yeah is that super high right.

C

So then, so that nothing falls through the cracks and then I would be curious to see that might just be interesting to know. Are we dying like a death by a thousand cuts, because we have a lot of like really small things in memory or is it like a few chunky things right? That would be nice because then it's like usually easier to get rid of one and like say, okay. uh This is how much we gain, but um it's usually way harder to fix, like the lots of small stuff going on.

C

Also, I I'm not sure if you saw because I posted it in a different issue. I think a while ago, I think one lead could be here uh all the graphql stuff. When I looked at one of these reports before it was, it was loading a graphql schema into memory, which we definitely do not need in most.

E

C

In fact, we only needed for the api fleet pumas, so we never need it on. As far as I know, we don't need it on sidekick on a cyclical api, graphql stuff. I don't think it does so, but there might be something to look at so this schema file. This is schema valid loads, which was 15 megabytes. When I looked at it.

B

But, but also like it's like in the psyche, we don't need as well like loading the whole uppy package that we have for the grade. We probably don't need to load the uh as well, because they are never gonna be called.

B

So I guess, like this kind of selectiveness of loading parts of the application like we actually, I think, wrote everything right now in each context. So if there is no selectiveness there, so I I think, like graphql is probably something like the biggest offender, but uh I'm just kind of curious as well like hey, like the happy package like in the github, how much more it uses.

B

Is it like? How big is that it's happy plus graphql and things like that? How much more is that using because like if it's using, let's say together, 50 megabytes, this is 15 megabytes that we can save from the sidekick process. Basically,.

A

Any idea how to measure that.

B

I I have no idea. The question is like: how does the right benchmark measure the germs, how it knows how much.

F

B

F

Memory profiler.

C

And I think memory profiler only ever looks at object. Space.

B

So maybe so, maybe like following the same pattern and trying to measure like the happy happy module like with all the objects and everything how much memory it uses, or maybe this derived benchmark has some some way. Maybe it can give you like the memory or such a file, or something like that that you could then just group and like provide like the given path, how much memory it's using.

A

Did it make sense yeah also? I know that memory profile. I I I think actually, that derailed is using the actual memory profiler that we already use as a gem. So maybe I can wrap around the graphql base model or grape api to see because it lists the results, looks really nice, because it's breaking break and breaking it in like number of size with the file with the classes with the objects and everything else. So I think that this can be useful, but.

C

Also another thing: when, when using memory profiler, I would focus on it's not visible in this derailed report, but um maybe just run it manually.

C

I think we should focus on what it calls retained memory, which is the stuff that is long-lived and will actually stick around, because there's also, it will also report things like objects that are only allocated in a single method call and that's probably not useful, to look at.

A

Yeah, that's true, that's true, but the result contains also retained memory. So that's fine.

A

Cool, I will stop sharing.

C

Cool I can, I can go next. um I'm gonna share my screen as well.

A

Sorry, matthias, before you start related to these memory dumps. What should we do? Should we do something more about it, except explaining those numbers.

A

What do you mean.

E

I think like from from what I heard, I think this idea by grouping them by file may be interesting, but I'm not sure that like grouping them by file is easy and possible, because file is weird like unit of unit of information like file could contain many classes right. I think there's other ways to I.

C

E

Hearing it from camille that it would be nice to group with by file, I'm curious, like do you want to group the.

C

Idea is that usually in ruby or rails, you have a one file per class right. So, okay.

F

C

Would give you an indication of um how large is the memory footprint of that class? It's a it's a proxy for it. It's not exact yeah.

B

Yes, so I actually know, like our part, you like we read that, like we have basically most of time single class per file, uh which is also like auto loaded, so it like to like have the the correct file path to be resolved to the given module and the given class.

E

Okay, that's fair because of the loading here file file means a lot.

B

C

uh So I'm going to talk about compaction. Is everyone familiar with heap compaction, or should I talk quickly about what it does and how it works?.

E

Let's talk because I'm not sure I have all the context.

C

Right so uh so I think the main point here.

E

C

um uh Is to recall that the process heap uh in a ruby app actually consists of two parts right. So there is the overall memory that the process claims, which is allocated uh by through a malloc right from the operating system and then within that overall space uh sits the ruby heap, which is managed by the ruby, vienna and managed by the garbage collector. So these are not the same.

C

So one of the subset of another and um the only thing that happens in the ruby heap is keeping track of uh which objects are alive or can be recycled right, and that happens in these um 40 by object slots. So a ruby object is never uh larger than 40 bytes on the ruby heap.

C

uh If it is larger, ruby the gc actually needs to, or the ruby runtime needs to, claim, extra memory on the from the uh up from the operating system heap, uh which will then live outside of that ruby heap space so um yeah and like one of the problems was prior to having compaction.

C

Is that oh- and I should add, like all these slides they they? um So so? If you need extra memory, if the ruby runtime needs extra memory, it doesn't, it doesn't malloc individual slots right. So it goes to the operating system and mallocs these blocks of 16 kilobytes, which is called a page uh and that's how they are like all these slots are grouped into pages. That's also what you see in the gc stats right um when it tells you the heap length, that's actually the number of um page objects.

C

It maintains it's just an array like a list of a linked list of e-pages. So uh so that being said, um yeah so.

B

One of the.

C

Main problems that we have ahead is that um there's like runaway memory problem where, because the heap the ruby heap gets fragmented over time, because you keep like releasing and allocating uh new object slots within these pages at some some point, uh so you will keep like ruby, will keep allocating new pages from the operating system as it runs out of out of space, and it can only release such a page back to the memory.

C

It can only like de-allocate like free that physical page and give it back to the operating system when, uh when it's entirely free so and that's why fragmentation is a problem, because it makes it much less likely for any of these pages to ever be completely free. So there can be no single object be allocated in anymore right. So that's why typically, ruby memory grows only grows over time, right, um yeah, so in fragmentation, sorry, defragmentation or compaction.

C

It's basically defragging your your memory. The idea is that you, you cluster, all the uh you basically move all these object slots around from one heap page to another, so that you have only full or mostly full heap pages and then a bunch of free ones, so that ruby can release that memory and return it back to the operating system. So so that's the main idea of uh compaction and there's limitations to this right.

C

So, first of all, it's because um yeah you need to actually be able to compact it enough, so that enough pages are releasable right, um but also there are certain cases in which you cannot move an object around a good example for this is a native code right.

C

So, if like, if, if, if there's any kind of reference from c code into a ruby object, uh if it will hold an address, a pointer right, so that object on the ruby and if uh so, if yeah, if the native extension doesn't know what the gc does and vice versa right so so, uh then there can be no consensus by by default. So uh you can't always like move these objects around.

C

So so there's some limits to this um and like so, I wanted to get first, a better idea of like how our heap looks like, and I was only partially successful with this, uh because um aaron patterson, he wrote this awesome tool called heap frag, um which did not, which does not work very well. Unfortunately, but it's actually it's really awesome like it. um It will take it. It starts a unix socket uh use the main socket and it will uh so.

C

It runs too um so it runs a process uh on which it receives a uh well heap report. I actually don't know what that means. I didn't look actually what it looks like uh when it's pushing out that data, but it reads from that socket and then it plots it as an opengl program in this like separate window and so basically every line. I think every vertical line here is a page uh and every um uh well. This actually confused me because in the talk that he gave, he uses different colors.

C

So I think, like black means, it is um it's a free object. Red means it's not movable, maybe because it's pinned uh and then I suspect green means it's um it's used. I suspect I might be confusing black and green here, I'm not totally sure it's just not very well documented either um but yeah. So I tried to use that and um it was hard because, like more often than not, it just crashed on me and it took a very long time to um produce this data.

E

But this is how.

C

How it showed uh this is actually a renders in the console. I didn't run this on boom. It didn't even work in in running in puma, but this is from just firing up a raid console. I thought that would be at least it is a kind of a proxy right because it still does it just loads like the application in memory and stuff. So um this is what it looks like and I think red means these are all memory slots that can't be moved which surprised me because it's like mostly red um yeah.

C

I don't know like this. It doesn't look right to me. I don't know why I have to spend a bit more time into understanding like when an object slot cannot be moved like painting is one reason for this, but there might be other reasons, um so I just moved on into looking like how much could we actually save in terms of memory, so my first intuition was, um and now I'm looking at puma enough, like the console or whatever um so my first intuition was just before we fork.

C

It might be useful to run a compaction. So actually I forgot to mention gc compaction is not a thing you get automatically. It's like. I really remember watching this talk. It's really funny as well, so basically uh aaron patterson. He basically said like at first like we, we merged this into the ruby main line and no one even reviewed that code. It's like it was like almost unreviewable, because it was very complicated uh and yeah kind of yeah. It was really hard.

C

It took him three years to finish this because there was so many edge cases to look after. So it's like it's not not documented at all like. If you look at the I mean I found his talks and stuff, but so this is so you would have to go. Read the c code. If you want to really understand what it does, um he explained the algorithm a bit behind it, which was really easy to follow. So that's good, um but um I don't even fully understand if it runs synchronously or asynchronously.

C

I think it runs synchronously and it was quite fast when I ran it as well fast enough to do it like as a one-off thing, probably not on every request right, but um just before you fork. So because if we start with a more compact like a more like cleaned up, ruby object space layout before we fork, then then the expectation would be this is this is good for copy and write behavior right, because then, then you have all the pages together neatly in those pages that will likely not change.

C

That said, I did this and I saw almost no impact so I'm kind of like halfway through this, but there's two ways to look at it. I guess like one is the actual, the the actual total figure in bytes, which I hope would improve, based on the assumption that if we have more compact memory layout and if we actually can free up entire pages, then ruby would be able to release these back to the operating system, and it doesn't look like it does uh so.

C

So these are two I just pulled memory uh before and after we fought with and without compaction,.

E

There's just no difference like.

C

There's no difference in rss used. um uh There is um some difference in the, uh but I'm never sure like how much this is also due to like, um comparing it across runs. I sometimes see fluctuations of like tens of megabytes right so like some of these might just be statistical like fluctuations.

F

So there was a little difference in usf yeah.

C

B

I think like like, if pieces, release to the system, it's actually like configured differently with the end variable. So what you may really want to look at is like how many pages are considered free from the gc start point of view. Yeah.

F

B

What I'm looking at right now.

F

I'm not done with that. Yeah.

B

Because, like uss rss would not give you like very conclusive information unless you tune how aggressive.

F

Ruby should be like with releasing pages, but but.

C

That's like I mean we need to keep the final goal here in mind. Right I mean because if, if the ruby heap has more available slots that that's fine.

B

C

B

I mean but, like you are you're looking at the outcome, but there is like, like this middle step, that, uh like you, are unsure. If, if the page is free or not free and if the page is free, then like you have some chance of like releasing this page, so you may look exactly how much of the difference is like this middle to the outcome. uh Point because, like this can show you like that nothing good is happening, but in fact it's happening just you need to tune some other parameters to actually.

F

That might be true, yeah.

C

um I'm not sure what those parameters would be, but um I I don't think.

F

That's requested to release free.

E

C

Think it does that just by default right I mean that's the coverage collector's job, so I think the final outcome here should be to reduce the memory use of the application right and so far I'm just not seeing that like it's.

B

C

Meaningful way.

B

It depends because, like uh memorial, location is pretty expensive and in most cases it's actually systems are configured to not release memory because it's expensive to allocate new pages. So our configuration can be very conservative on.

F

Like retaining pages, even though they could be free and is that, is that something I need to configure the gc to do or.

B

F

Actually, looking for these parameters, all right- that's interesting, yeah, but.

B

Like but like the releasing pages to, the memory is really like last resort, so.

C

Also, keep in mind like this is just after the workers started, but the application hasn't done anything yet. So I also wanted to see what happens going forward, because one thing I thought I would I was hoping to see- was that maybe the so because the gc keeps this internal.

C

It's not a constant, but this internal parameter where it keeps track of um it, tries to um predict how the memory needs of the application in in the future just based on path behavior, um so it tracks the size and bytes of um how much malloc should uh actually claim at once. I think I said earlier, like it claims individual heat pages. I don't think that's true. It will. I think it actually claims a bunch of them based on this malloc increase byte size, there's like a gc step like this malloc increase biases.

C

So those are the chunks of memory will claim simultaneously and um it will. It will increase this uh as your memory needs uh grow, so this can contribute to uh like runaway memory problems like if you, if you, if you load like a one gigabyte file into memory right, then the gc will know take notice and it will say: oh, like your your memory.

C

uh Consumption needs are very high, so it will increase this value uh so that next time it asks for because what you said come out because it's an expensive operation comparatively uh the next time it will ask for uh like twice that or something like that. Some multiple of this um to not have to go back so often to alloc new pages but yeah. um Oh sorry, you had a question galaxy.

E

No, no, you answered that already, I wanted to ask: did you run some pressure on the memory before.

C

So yeah really different the full yeah just to like give it like, so that has something to do again. This might just be a normal fluctuation but like in the varian, I ran in this experiment that used compaction. It actually is consuming more memory afterwards, at least.

E

C

According to the rss reading, so I don't know, I might have to run this a few more times and see like because this can fluctuate so much like it might not be meaningful, um but 200 megabytes seems to be quite a lot but yeah. um I also yeah, I'm now looking into the gc stats, uh which are definitely interesting and you can actually see uh so I'm not done looking at this yet, but you can actually see the differences here that are kind of obvious, like considering that you did a compaction.

C

The compaction count went up from zero to one. I think this should actually answer my question around. Does this run synchronously? So I suppose it does right, because um I just took the snapshot before and after calling gc compact uh and also you can see how it rearranged. So so this heap sorted length. uh The way I understand it is that's the the length of the page array that the gc keeps it's just a linked list of pages.

C

So I think you need to multiply this by the page size to get the actual memory that it has so far been allocated and the allocated pages. This will always be. This will never be larger than this right, so this will be smaller or equal to this. uh So these are the pages uh that are actually allocated right now. um Yeah, the allocatable pages is the difference in between that. So.

E

So this basically.

C

Means this is your head space or your leg, um your that's! Not the right. Word right. I think, like the leg room right that you have before you have to call malok again, I think, is what this means um yeah, and so after this this looks like it's been aligned, you know, um so uh it actually allocated the same amount of pages it had in the and this actually went down. So it looks like it released a bunch right, so it it released.

C

What is that 1200 pages or so, um which is not that much in terms of raw memory, megabytes right uh but yeah, so so so this kind of like um got um aligned. I don't know what you want to call that uh yeah, I don't know. I haven't even haven't looked into this um further than this, but um yeah.

C

That's what I what I have so far so yeah, I don't know like maybe there's also other like points in time where we might want to run a compaction so, like someone suggested in here, we didn't suggest compact, but I'd like to. Maybe we want to like gc at certain points in time like after running the worker. I.

F

Don't know if that makes sense to be honest, but.

C

He uh mentioned this is something we could look at so.

E

C

Don't suspect this is going to have a big impact, but I I still think it's yeah can make sense to look into it.

B

This is really quite time consuming, so I'm not sure like if executing after that, like each individual worker would be.

C

Well, but not compared to a psychic worker, actually executing that's a fairly small chunk of time right.

C

I mean I mean a major gc cycle is in the order of milliseconds, so I think our average psychic job runs more on the order of seconds. I think, am.

B

I here, I'm not sure, like probably you would have about like if, like the sidekick, if some side jobs take 10 milliseconds and like we take five milliseconds like executive dc, and given that this, like type, you can execute concurrently, make us side effects. Basically.

C

Yeah yeah, I don't know yeah, it's also, I think, there's like way more impactful things we can do with psychic than running that you see triggering that gc, uh because because I think one thing we should definitely do this this week. Maybe I can pick that up after this is to look into. I don't know if that collides with you're doing uh camille, but we looked at the memory sharing problem with psychic right.

B

I'm kind of wondering like because there are like this one of the bunch of the health parents. Maybe there are some better settings to make dc more aggressive to execute, and maybe they would be better uh if you run them in the memory constraint. Environments like to have like to allocate less amount of the slots to locate less amount of the pages in the single in the single round, because right now I guess like the default settings could be that we want to allocate a lot of space.

B

We have a lot of head room at each given moment. So maybe it's like I mean it's like comparting, plus different settings on how much data you, maybe how much slots pages you may be allocating in the single code- and maybe this is like a advance of the n variables- make this uh runtime memorial growth to be smaller.

C

I don't know how much like yeah I need to. uh I didn't.

C

No, the gc in ruby was like configurable. To that extent, I I actually didn't look at this. Maybe I should do that if you have like any links to docs, if you can.

F

Let me know, I I think in the dark like and some.

B

Of them, not all of them to configure dc, there are also, I believe, additional ones to j malloc that you may want to check out. But probably this is like your first try like trying to play with these ratios amount of the slots, and maybe there are some good configuration that would make it less aggressive.

B

I mean that would make it more aggressive and but it would allow pre-allocate less amount of the memory, so it would be more compact by design.

C

Right yeah. I also like notice that if you, if you ask, if you call it explicitly, you can also at least ask it to uh also do a sweep, I think usually it um separates the mark and the sweet phase, um but yeah. I can look into this stuff a bit more.

C

I think I would like to do that separately from compaction, because I don't want to mix too many different things into like because, like I want to make sure that we know at the end what causes you know cause and effect like if you change too many variables at once, it's not really clear like what actually had an impact here or not.

B

I actually have like the suggestion as well, like maybe as part of the application startup. This is like one of the trends, also in the puma with uh nokoyashi fork. I guess, but it like does gc before forking, but maybe like. We should kind of control.

B

This dc to happen like when application is fully initialized like to perform this, like full dc cycle, to like to release them like the pages that maybe not needed, and then I kind of proceed to starting application, maybe like, as as the last step, adding gc uh in the initializer on like on these different lifecycle, events to kind of like perform a full dc sweep and like full compaction that we can maybe do to kind of start. The process like with the there is amount of the memory.

F

C

So my understanding is that the compaction includes actually two full major gc cycles. Right now, compaction triggers a full gc before compacting and after so just by calling compact.

B

uh It sounds like.

C

The this was done for simplicity and they are trying to get rid of that in the future, but currently it should. It should already do that, but I can I can.

E

C

Can throw in an extra gc start to see if that makes any difference.

C

Oh yeah, that's all I had.

E

Okay, let me share my screen, so yeah I was experimenting in boom in single mode. uh To start with, I just found that even running on small benchmark, like five seconds surpass for me, gives some failures on 4dx machine. So that's something I think to keep in mind, at least for my stance because yeah, it's kind of weird that even for gig machine fail. Sometimes I didn't look deep down into that. Maybe this uh intended- or maybe this uh uh in in in denialist by grant, but I will look into it.

E

So I put some instruction how to start boom in a single mod, so you basically put worker purchases to zero and based on the promo code, you it interpret it as not clustered, but we also have this validator, which raises an exception. If running in single mode- and here I have a question for camille camille, do you know? uh Maybe do you remember some of the issues possible issues that could come with that, because I see that benchmarks are passing, but what I currently see that probably memory killer won't work right, because uh it's currently.

B

Three problems that are associated with that, like first one memory killer, will not work because there is no thing to hear yeah prometheus, scraping metrics in native see, so it will create contention running in the single if you scrape matrix- and there is a lot of metrics. Basically, we stop execution of everything that is free and the third problem and the graceful shout down so that you allow to process requests while returning health check. Failure will also not work uh in the single mode, because there is no proper hook for that.

B

Okay, so these are three major reasons why single mode is not functioning. Today properly, um I mean memory killer, probably not being like the biggest issue, matrix being quite significant, because it's actually blocking ruby and graceful shotgun affecting zero downtime deployments. Basically, if you use health checks, these are the reasons why we don't use why we don't follow single mode today,.

E

Okay, but if we still hack it and run it, I found uh these results so quite interesting. That for gigs uh have failed in in a single mod at some tests and compared to for gigs in clustered mode, uh actually single mode failed in more tests than than clustered mod. I didn't look yet deep down into particular differences, but what I looked at is uh memory utilization in clustered mode is close to one, so it tries on for geek instance. This is the red one.

E

It tries to utilize all memory as far as I see, while clustered mod, while single mod sits at 0.8. So so now. For me, the question is: is it like a good memory utilization so it is it utilizes and could do more because of this utilization, or is it utilizes more and just wasting it? Let's say for let's say, supporting the clustering mechanism, so that's I'm not sure.

B

So I'm kind of thinking that, like the in the cluster mode, you're kind of losing some of the memory because of the copy on bright being notes are effective in the single mode. You have only single process, so probably you're saving something like 100 to 300 or 400 max of just running in a single mode that you have just single process.

B

Basically and like you, you have very good memory utilization then. So I think, like your your result is expected. What you are seeing. The single mode is more memory efficient, but also it's slower because some of our uh pursing benchmarks, they are uh cpu bound per heavily on the ruby.

B

So it means that you actually cutting significantly the capacity of running in the single mode, because you can only use the single cpu core because of the ruby gbl.

B

So I guess, like my my thinking right now, it's like it's kind of expected like what you are seeing right. It aligns like with also my understanding that, like single mod is gonna, be more efficient, but it's gonna be sorry. It's gonna be more memorable efficient because you don't really lose your bright pages, but it's gonna be less performant because you are at most using one cpu versus two cpus that you may be using if running in the worker mode.

E

B

E

So I have a couple of questions, so how could I check that this difference is actually like because of copy of right pages? So where should I look at.

B

I I don't know it's like it's about figuring out as part of the project. Okay, like.

C

B

That's why we track.

F

C

Pss metrics right, they should be available. We track it for every gitlab installation.

E

Okay, I will I will. I will look at, and so I mean since results are expected, so here I think we should decide if we're going to proceed with investigating single mod. I I wonder like: are we going to overcome these different difficulties or are they like show stoppers, and we are not going to proceed with it, because it's too much trouble to to fix them for for a possible game.

B

I I'm kind of thinking like that memory killer is probably not a big problem, but the other ones are to overcome if you would have to so so I guess, like my perception, my question is like if you're trying to pull mine the single mode, how much gain it give us.

E

Okay, I will try it.

B

Because it can be like a good trade off like you like, if you run 2g to 2g instance, you already probably run like with two cpus. So maybe this single mode is just good enough to run basically in this 2g. But the question is like how much benefit it gives us in terms of the memory. So like the question of matias, like you can use s-man the tool to check like uss, pss ps has been proportional uh set of the memory.

C

B

As well, you can you can, and there is also like the prometheus metric. So you can like calculate the total memory used by the or puma processes and compare like the total amount of the memory in the psls by all puma processes uh in the single versus cluster mode and like estimate how much savings it brings us.

E

Okay, okay, but but it also comes with worse results, so I wonder how we should proceed with this, so where lies, let's say: borderline and what is tolerable because, as you could see like in single, not some specs are failing, but I mean not much. You see like second more here, second more there, but do do you have any opinion like when to call this an issue and well to call this like something.

B

uh My opinion is like that we are running 20 requests for 1k architecture, so.

A

We are all right.

B

So we are running pretty extensive, like loads uh on this, like presum2g or like the this small instance. So I guess it's it's about like really defining the capacity if you would run in the single mode, how much capacity it offers and like what is designed for. Maybe it's not for 1k architecture, but it's maybe for 100 type architecture or something like that where, like you, don't run 20 requests per second, but you run maybe 10 requests per second and it's fine, it's within our limits.

B

So I guess, like the results, um I think it's worth understanding why these results are worse in some cases why they are maybe similar, but usually it's basically due to ruby, gvr and single threat problem of the running in the ruby. But maybe it's not really the problem. If, if you run on this small instance because like 2g, will never be a big instance, it's not we. It will not be able to handle maybe 1k users.

B

So maybe it's designed for much smaller type of the installation as well.

C

Yeah we actually have a new reference architecture for less less than 500 users. I think they just added that originally maybe have a look.

E

At that, maybe that's right.

E

If I get s, I think it's something like this, uh so yeah, okay, so my next steps would be, I think, to actually like explain the difference in pss trying to like uh describe it in a clean way via metrics and see different, let's say, uh request pressure and see, maybe in 10 requests per second, it would be operating nice in a single mode and then probably I try to look if I will be able to into these issues and try to shortly describe them. But it comes with this issues.

E

B

Yes, probably probably like the biggest issue in the single mode is the prometheus matrix. It would have to be somehow every time to something different, uh but I I guess like it could not be really a super challenging thing to do. I guess it's really about, like the amount of the memorial saving can give us if you would run that in the same mode.

E

B

So actually like, we have pretty interesting investigations from yesterday, so what we are looking at today, I I personally want to continue like with this spring and understanding like loading of the dependencies to see. If I can make, maybe this pre-loading approach be more efficient.

B

I just want to figure out if I can like preload, significantly more to have like much more memory utilization, and I will probably try to use alexi puma single mouse to see if it actually helps with reducing further, because maybe it brings like 200 300 500 more space, so it give you it would already give a lot of more headroom, and maybe I would be able to finish most of the tests on the 2g.

B

So I guess this would be really my focus today. I want to basically continue with the spring figure out like the ordering of the loading. If there is loading happening after the forking, if it's I want to get rid of that uh use, alexi, puma single. If it's possible and see like how far it gives me like the the room.

E

Folder, okay, I set my focus, so I will probably try to wrap up the difference and to like look into single mod uh restrictions so unbox it.

C

Yeah I'll continue to uh look into compaction, wrap that up uh and um yeah. I want to spend some time understanding the sizing of the heap like what I said earlier, like how large there will be versus the whole process. Memory being claimed.

A

Okay, I will try to understand better the results that I got and to understand better the scripts and the numbers that are that is showing and maybe I'll try to analyze a little bit more since I already get, I think, 15 different heaps snapshots.

A

So I will maybe take a look and see any other means to analyze this, this hip snapshot and then I will probably continue looking at those gems and maybe run some memory profiler against graphql or something like that. I'm not sure which topic this is related directly, but I guess it also goes with the gem sizes and everything else.

B

I guess nicola like you, can create a new topic if you want to be as specific as.

D

B

So that's it any other questions, suggestions all good.

E

No, maybe later like somewhere in the middle yeah,.

B

Okay, so thank you thanks.

F

B

Thank you. Thank you. Thank you.

E