GitLab Memory Team, 16 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020 11 16 Memory Team GitLab 2GB Sync

Description

Discuss approach, identify and pick themes
Picked baseline testing configuration
2:04 Garbage collection discussion
- GC tuning more about mapping pages we are using
- How we can configure to be more aggressive?
- Is it happening?
- How many pages are half empty
3:40 Idle vs. Runtime
- Are there parts of the application holding big parts of memory
- Example number of strings/symbols
- Understand why runtime memory is increasing
- Memory fragmentation
- Build a reference tree to understand GC usage
- Can we somehow nullify some usage here
7:00 Discuss examples of how to break down into smaller topics
8:30 Topic selection and goals
- Move topics into dedicated issue
- Thought process, metrics, data
- Next day, read out results of investigation
- How it went
10:45 Discuss baseline topics
- Calibration session Tuesday
- Goals
- Consolidate and join goals on Friday
14:40 Topics
- Dive into specific topics detailed in https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2
- Topic - review Gem usage, remove dependencies
- Topic - COW, compacting GC
- Topic - Puma 5.0
- Topic - review memory content
- Topic - using spring for pre-loading

A

uh Okay, so this is like the first day monday of our like two day, two gig week, um we're gonna discuss like the approach configuration like and uh how we want to measure uh our work and like who is picking what team from the list of the teams that is part of our epic.

A

So, uh like we discussed that um after friday, we discussed that we actually think that using the high cpu machine will give us much faster turnaround time. So we settled on using uh n2d high cpu four, which is like for a vcpu with four gigs of the mmr, and we also additionally configured four gigs of swap. It's gonna, be like our baseline uh testing configuration, but we're gonna be argumenting.

A

This uh vm, with setting of kernel man parameter which actually gonna allow us to limit to whatever memory uh spec we need so we're gonna be like tinkering only with like single parameter, but still having like high cpu count. We kind of like aim to run successfully github on mem two gigs, which gonna, basically we're gonna, add it etc default grab and then update crop and restart vm. To have this new setting.

A

um We should probably discuss like our teams for today uh alexi. I think that you had some questions about our teams.

B

You mean topics.

A

Yes, topic team. However, you name it. Yes,.

B

Yeah, I was curious about garbage collecting garbage collection like tuning, so I remember we mentioned that we probably want to tune like how aggressive garbage collector is. But I wonder if this already goes into this- investigate a big difference between idle and runtime memory, or is it different from garbage collector.

B

Tuning and experimentation so.

A

um I I think, like in my perception they are kind of like the objective is slightly different because, like garbage collection with compacting, this is about like efficient memorial, page use and like being able to like to wipe out memorial pages that are not needed.

A

uh Compiling makes you, like uh I mean any allocation, creates a lot of data, a lot of memory fragmentation so, like you have four gig 4k page that is used, maybe in 10 percent and compacting makes you move these blocks into one page. So you could like recycle that one. So I think, like dc com, this gc tuning is about like figuring out.

A

If we can like map, maybe pages being used is compacting kind of working it can it like move things around to like to recycle these pages? How we can configure like to be more aggressive in recycling these pages, or is it even like happening? How many pages? Maybe we have like these? Let's say half empty or my maybe like even less empty, where, like the second one, um what was like the second one that, like you mentioned the name of that.

B

uh Investigate a big difference between idol and runtime memory.

A

Yes, where I think the second one is about trying to like dump objects in the memory trying to understand. I mean it's also connected like with looking at the content of the memory pages, but I'm kind of thinking more from the perspective.

A

Are there parts of the code base that may be holding like uh became out of the data over time? They are encouraging right, so they kind of be regarded corrected. Can we like maybe imagine, like how many strings we have? Maybe they are like duplicated?

A

Are they like kind of growing over time? Are symbols like growing over time, and things like that so, like? I think it's more about trying to understand why our runtime member is increasing.

A

I mean we know that one aspect is like this memorial fragmentation, but I'm kind of thinking more from the perspective of.

A

Are there like the code in the gitlab that maybe it's holding very big obvious, a lot of references that maybe, if we would like notify a few entries, it would make gc actually be able to to wipe out that so kind of, like I'm thinking more about maybe trying to understand and build like the reference tree, trying to find the wrote references that gc is using to traverse these three of the objects.

A

So it's more about. I guess this aspect where, like one gc tuning is about. Like I mean we know that we hold a ton of memory, but can we make this representation compact so like have much better patch utilization, but then the second one? We know that we are holding a lot of memory, but is this actually needed to be hard? Can we somehow, like nullify and recycle that so make like dc to be able to uh discard these blocks of them are being used?

A

So you would not so it would not even have to like to recycle them or like like it would not have to compact the spiders in the first place. So they are like very see me see like similar, but I think the direction between them is like some somehow different, like you're. Looking at different aspects like one is like uh language uh toning, where the second is more about application tuning. I guess.

B

Yeah, but uh about this gc twink, we currently listed it under topic: ruby 2.7, doing it a separate topic for just outside of ruby 27 to play with garbage collection, uh let's say params, because we don't we. We don't currently have any separate topic for garbage collection like we only.

C

B

Gc tuning under ruby, 2.7, which I don't know like how it it could affect the overall memory usage, I'm I I I'm not sure like how to so. You think like install seven and try compact and try also like different aggressive models and.

A

So I'm kind of like not I'm not sure like how to grab that on the like issues and topics.

A

But if you decide on like on working on that, like you can like rewrite these topics to to make it meaningful and maybe create the issue and keep track of that, because, like I'm kind of like expecting that as as soon as someone starts looking at that you're gonna like discover a ton of other stuff that gonna be another part of the investigation so like I said, I would kind of expect that, like I look at only this some aspect, but another topic is look at this specific aspect that I didn't look at yet.

A

So I'm kind of expecting that like this is like the headline which gonna spawn, maybe two three five sub-topics to this headline- that we may then pursue or not pursue. So I would be not attached to the current topics, because this is okay kind of draft like uh look at the topics as they are try to find like the objective that you are gonna, be focusing on and try to like to create like another topic. So, like another, like discussion, points about this aspect that you look at and like what may be worth investigating further.

B

Okay, should we select on topics now or should we wait for materials until after lunch and then build will say.

A

I'm I think, like we should select like mathias, can can like follow up once he's. Okay pick another topic, any.

B

A

I I was kind of like like about the timeline.

A

I was kind of thinking that, um as the outcome of the day, I mean I was kind of thinking that for each topic that we choose like we move this topic investigation into dedicated issue and, like kind of like start like having our thought process, metrics data parameters that we test this kind of like a log in the separate issue that is attached- and this is like how we migrate this topic into like the like the work item that, like we work- and I was thinking that- maybe because, like we have this meeting at the same time, every day like tomorrow or like this one hour before, like this meeting, each of us would look at each everyone's results and read this log of the investigation and like then, we could, on this daily meeting tomorrow.

A

Discuss like how it went. Was it hard like what else could be interesting to investigate that?

A

So I was kind of thinking that, like our daily thing, would look like that we focused on reviewing what every one of us did like last day by going through this issue, that, like you, populate with the data uh but then like. We pick like another topic or maybe the same topic if it's still worth investigating for another day.

A

So um I'm kind of thinking that like if we choose the topic, we move this topic into like a full-fledged issue as something that we work on, and we keep like a lock of the items that we like investigated as part of this topic, or maybe another items that may be worth investigating later.

A

As part of that issue so nicola, maybe you have some preference for the topic. First,.

A

A

C

Yeah before we start with the topics, I just want to go with a baseline comparison like should we list those metrics and how to use them in the issue where we like set, configure the setup and everything else, so we can always use the same metrics.

B

I think we should, but I'm not sure we figured out like it fully, because.

C

Yeah, I'm not saying that we should do it now, but like.

B

It could be beneficial.

C

To define them so we can like do the compression.

B

Yes, but also everyone could like char, a a bit different like values from primitives during the run, so it may be a bit different like based on issue but yeah. I agree.

A

So I I was thinking that I'm I will spend some time like today on like getting this like baseline metrics from the gpt uh and like we would have like the calibration setting uh calibration session on the metrics like tomorrow, so like, we would like to work on the topic like today, kind of like maybe in our own way. Each of us, but like uh tomorrow, like we would start refining like these metrics. Basically.

B

A

B

Was thinking something about like we will have a bunch of metrics, we need to measure them on idle, let's say like because some changes could make that idle consumption as well like, then we need to run gpt, and then we need to chart how the growth rate makes a minimum problem right.

A

So yes- and it's actually tricky because, like I'm not sure if there is like one, fits all type of the metric like. Definitely if you, if we talk about the gc compact, I guess like you, there is like different set of the metrics or, like uh differences of the data that you are looking at. um I think like from this baseline comparison, I'm more aiming like on this side to be a summary of the week like we have this baseline comparison. We started with that.

A

We integrated all our improvements from various parts and we kind of resulted in this kind of like outcome. This is like how memory we like reduce. Maybe this is how like how many more requests we are able to like to produce.

A

So I would like not be super attached to the gpt details, but I think, like throughout the week, we should figure out a way to make this meaningful comparison after this week, like how much game we achieved since, like now before, like we made any change and once we maybe integrated all our changes because, like I was also thinking that, if we find like some improvements, I would like to join these improvements on friday and see exactly like of our joint effort like how much we can improve gitlab like how much we can gain on the memory usage.

A

So like it's hacking investigation, but like I would kind of like to join all our improvements at the end, so kind of try to like whatever you do like make, try to make it in some form. That is like reproducible from this issue lock to someone else like it goes through these steps and it can to some extent reproduce like this uh investigation that you did so.

A

uh I I think that this is my idea behind that, and I'm not sure like what is yours, but I'm kind of sure that it's gonna be changing throughout the week. The more topics we kind of investigate.

C

Cool make sense.

C

Okay, we can go to topics uh I I look this this this one investigate used gems and I have a question related to okay. We can measure like the biggest one and try to remove them from the code base, but can you maybe explain how we could figure it out that they can be executed externally? What do you mean by that.

A

um I was thinking that maybe in some cases, if maybe it's often or already used- maybe it would be, for example, loaded by the dedicated process. Maybe it would be loaded only by sidekiq or someone else, or maybe it would be executed by dedicated, let's say, bean command.

A

So this was like my idea like if you would like, remove the like the biggest terms, how much gain it would allow us how often they would be like, let's say, maybe used or like in what circumstances, and if you would have to like to move them and still execute them in some different form, maybe rewriting them in something more efficient.

A

How complex it would be and like what methods it would we would have. So I guess like it's about like finding.

A

uh If we can remove some dependencies basically or can we move these dependencies to be executed in some different form, to kind of reduce these idle and runtime memory usage? Basically- and it's also kind of vaguely connected with my uh like suggestion about like uh memory like do these gems like contribute to the runtime memoir over time when they are executed?

A

If yes like how we can recycle the memory of these jumps, and things like that, so uh I guess like this is another like all of that is kind of like interconnected, but I'm kind of more thinking about as part of this topic, we have like the big jump. Can we remove this big jump like what it would take for us to remove that maybe like we, we know that some parts of the github are gonna be defunct right now, but we like we remove like we comment out.

A

We try to run and see like how much it brings us the benefit. Does it bring like, maybe like 100 max in total, which would be like pretty big saving and now, if you would have to make this permanent chance that we removed that how like we still are able to run this somewhere else, maybe or maybe we can replace it with something more efficient.

A

I don't know I'm kind of looking into some. This kind of like investigation like what we can do with that.

C

C

A

Okay, so um what we are focusing on today, alexi, what about you.

B

I was thinking about something small to calibrate my let's say environment, to actually understand how to do this fast, so it would be either like puma, something because it's simple to execute, but I could concentrate on how I measure how how I work with environment, how I produce its results. How I compare so maybe I will take puma if it's okay with you or will be 2.7, so whatever you think would be beneficial for the first day.

A

I mean like pick anything that you want. There is only like, I I'm not sure like which one of them is like okay, so like, but I don't.

B

Have one question.

A

About you think that it's like interesting for you.

B

Yeah I have, I have a question about topic. Ruby 2.7, uh you mentioned: does competitor in gc break copy and write from forking? uh Do you use scorpion right from forking currency? Yes,.

A

B

Okay, yes, so, and how do we check that it will break okay, I mean.

B

How how could we check that it breaks gotten right, I mean we should like monitor. Somehow this is cooking right process itself.

A

So I'm kind of like it's pretty. I know I don't know like how to answer you that question, but I guess it's pretty tricky like to figure out.

A

I would maybe first before starting this one, I figure out how like compacting those work because, like copy and write, we use that in the forking like each each memory page is like uh cloned into the process, but it's been copied on the first right to this page. So if you change any byte, it's gonna be copied into new replica and now compacting, because it reorders blocks so like it moves block from another page to another page.

A

I I would assume it kind of creates more of this copy on wright's um occurrences. I just don't know how to like to uh gather this metric. Maybe there's like some kerner metric on the process like how much copy and write pages you have, and maybe compare that like like, let's say from the outside, like you have the compacting, you know that compacting is working, so you have the compacting gc versus not compacting dc and, like you, compare like how the memory pressure changes, how many unique memory pages is being allocated or like?

A

Actually, you are looking about how many pages are being shared with the parent processes?

A

Is it like how much this value difference between um like having compact ngc and not having contacting gc so something they're, not designs, but now the question is like how you know that compacting dc is working on the ruby 2.7, how you measure that is it like there is a matrix that you could use for that?

A

um Is it effective in our case, if not like, so it's kind of like connected in like understanding how the dc parameters and compacting dc does work as well, probably being for this topic.

B

Okay, I think I think to start today I will take puma uh single thread. Do you know how many threads to run currently? Do you.

A

Remember you you, I think, like you are talking about uh not single trial, because puma always wrong in multiplayer yeah I mean you, you talk more about the single mode, yeah single mod, so like so like the single process instead of multi process.

A

uh I I think the number of the threads is not really that relevant, like you still want to run as many threads as is possible, but because you don't have this copy on the right thinking, you don't have multi-processes.

A

You only have a single processes. It should be like basically more memory efficient.

B

Yes, yes makes sense. Okay, I will try to investigate and it will be possible if it will be possible. I will also try to check the recent fork today.

A

What would be the second one that you would be looking at.

B

uh New fork of puma, since it relies on the same, like uh I mean 5.0, right.

A

B

Yeah, it will be crazy, like connect and easy to check together, and I hope it will be fast.

A

um Nicola, how.

C

About you uh just give me uh like, maybe I can start with investigate the memory content like see what is actually in the memory dump a memory map and like check. What's in there? Maybe we can have a clear picture and maybe it's a good way to start with. All of this.

A

Cool, um so I wanted to look at the use spring for pre-loading, I'm just curious like is it hard? Is it the way it doesn't work?

A

It's not really like production already, but I'm just curious like can it be made quite simple to work with.

A

Omnibus, so I guess we have topics right.

B

B

Anything else we want to discuss before jumping.

A

Game, I think like uh right now, like I'm, not sure what else we may discuss. I think that more questions may pop up throughout the day or like tomorrow. On this thing,.

B

Do you want us to start like forking new issues? No, I mean so until.

A

B

Need to fork like a couple of issues right.

A

Right, I I think like it would be useful if you would like create the issue on the topic that you are working right now and like as soon as you have like an uh details. Investigation interesting, like you, just write a comment in this issue like a lock so like. Okay, like some people, use like the like the notebook to like to note what they did and like what benefited they they give them.

A

So you may use, like the let's say, issue comments like in that form like us as other look of like what you are doing, but I mean like uh alexi like it's it's my suggestion, uh but I'm not sure like if it works for everyone. So if you find like some other way like to let's say to summarize that maybe um like pick whatever it's like.

A

Easy like and like go higher up like with your usual workflow, so um I'm kind of like thinking like I'm kind of looking for something that, like each of us tomorrow, could read, look at and see like. It's actually something interesting for us to maybe to proceed if you would like to proceed with something different right.

A

So I I'm just curious about like having like the summary of like what you what you did like, what every one of us did and like what benefit did it bring.

B

B

So nicola, were you able to run this on mac.

C

B

Okay, good because I'm currently in this stage.

B

C

B

C

With localization, but I posted a comment in the thread with.

B

C

It worked in the end.