Kubernetes SIG Scalability, 29 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021-04-29 Kubernetes SIG Scalability Meeting

Description

Agenda and meeting notes - https://docs.google.com/document/d/1h...

A

All right uh all right welcome everyone to our sixth grade public meeting.

A

uh We don't have a lot of agenda for today, just a few announcements and organizational informations, so I uploaded as promised, like the last four meetings to youtube from the period I was away. So if you want to take a look here is the link to our playlist.

A

uh As I said, I was hoping that apple will be here today. I wanted to discuss the issue we touched two weeks ago about the performance benchmarks he uh he did for priority and fairness. We were hoping- and he was positive about this when we were talking and that he would share them with us and basically open source them, so we can start running them continuously.

A

um So, let's leave it in the next item. Until above is here.

A

Three more things I wanted to share with you. One is that we have this issue with pear dash. We are not able to upgrade it beyond 2.22 version and because the next version consumes more memory than the previous version anywhere out of resources in our a cluster. I open an issue for this. I I mark as hell wanted. I hope someone will uh be able to take a look and help us with that.

A

There is pr actually opened by magic, who is here today to change the aa cluster, to use to use the an app right, no auto provisioning, which would solve the issues we have and actually ship the discussion that we have with the aaa and management team to talk about the actual resources used by the uh by the jobs we run done rather than talking about whether it's okay to add bigger notes or not, uh but yeah anyway.

A

I hope that they will get unblocked like it has to get unlocked, because we we have some updates to partners who would like to deploy, for example, the things that voitek mentioned two weeks ago about our new network tests. uh So we still don't have this data in purpose because we weren't able to upgrade further.

A

um Do you know, what's what.

B

Three good repeat: okay, can you hear me.

A

B

Yeah so the question, so do you know what has changed between like this version and the next version like? Do you expect any like more growth? I don't expect that you have too many changes to the predator, so it should be pretty easy to trace the change.

A

Yeah, like that's another way, to actually fix that uh so to find the change that increase the memory usage. I even I believe it's my change, because I uh I added like so what I did in perth dash is that we had some of these tests that were neither load, neither density test and they uh basically weren't displayed in perth, because paradise has this very complex logic for matching uh for, like looking for the files on gcs, so long story short it they did.

A

The file is looking for uh has to have this prefix or suffix with the load or density in them. So I added like this like third uh group of parsers that are looking for uh like any test, so I believe this might be the culprit because they are not looking for density, although they are just looking for the name of the measurement.

A

So it's likely like that now for each test. We are actually uh like displaying sorry.

B

A

And displaying the same data twice, because one is much for the density or or or load uh parser and the other is parts for this catch all parser. So that's the reason uh uh we can roll this back. That's an option, but I think it's not sustainable long long term uh that we need to. You know, fight for every, like gigabyte or something uh yeah, even.

C

Like that, that's actually.

A

Like your arguments that our sky bts they use like terabytes of memory or even more so uh fighting about one gigabyte, imperf dash, it actually doesn't make sense. If you really want to save memory, then we should optimize our scale test right. Yeah.

B

Yeah sure like so, I think the bottom line is that that looks that the the more growth is kind of expected so yeah. I I think that getting larger notes in this a cluster is a good good solution. I'm not sure.

A

B

How it works, but I can take action item on me too.

A

B

Push it like this, this pr is actually mine, so I can try to find some approval for that.

A

A

Nice, thank you mache. All right, xiaomi is that you, you login as a six-calibrity, so I assume it's you, but I don't know yeah yeah.

D

A

D

Think I forgot to change it back to my own personal id.

A

Yeah yeah um cool, so two more amounts announcements. Actually one is ai from the last meeting and this one I wanted to share that. There was like this effort to migrate our load test to do this cluster over two modules, so this is moving forward.

A

The work is striking this issue uh and one more that I had an ai last week to go through the hell bonded issues and to check whether there are any available and also like basically check because we had we had some issues where uh they were assigned to someone, but like nothing, was really going on there. So I I I went through like few issues on top of the list and basically pink them.

A

So if anyone is looking for anything to do in six credibility, just check this link with hell wanted and you should be able to find something there uh all right. That's it from my side. Do we have anything else to discuss today.

D

Sorry, hey there's one small thing um from my end, so I just shared an issue link with you, um so th. This recently came out in like uh one of the work that we're doing here. um So what we saw was for hcd um when there is a lot of heavy read requests right. There is like memory memory uh allocation. There are some unnecessary allocations happening uh around so yeah. It's it's a little bit funny, but you can see what the change was.

D

So as part of the request at cd was logging at one place, um the response size and previously it was calling uh range response, dot size function, sorry, proto, dot, size function, um which is, I think, calling creating a duplicate of that object, which was unnecessarily doubling the memory, um and this this was actually a significant uh impact on some of the um cluster. So uh okay ciao has joined the call, um so he was he was uh investigating into this issue and uh he improved that fix. uh Ciao. You wanna talk about.

D

uh Let's fix a little bit.

E

Yeah, so uh can you guys hear me yes.

A

E

Yep, um so basically, I'm like uh uh founding this issue, because one of our cloud cluster, the scd, is out of memory, um so market and technically we don't want like database to crash.

E

So uh we did some like customer case study and learned that um customer is issuing some big uh list pause costs across all namespaces without pagination, um and then we like reproduce it in our dev cluster and find out uh like and also do some like a profiling into the lcd and find out, uh there's um like unnecessary uh photo buffer like uh um to just to like compute the size using the portal size to show um like the whole response.

E

So we think it's um unnecessary and uh after we change it to um like a range response, size which doesn't take the whole like uh memory like a usage uh like the allocation, so uh we do observe a big uh memory used percent jobs, um so yeah. So that's the context.

D

Yeah, that's the graph you just had right. There.

D

So I think this is after the fix, like the the memory usage went down by almost 50 percent, um with like a lot of range, heavy requests.

E

Yeah, uh like another finding is uh actually uh the lcd um uh 3.3 and a street of uh four, but in the release branch they are built by like between go 112 and the goal 15 so um like starting from go 12. They are using some like a linux, mli system call which doesn't um like it. So basically it doesn't like a release memory.

E

uh From the monitoring perspective, um I I can like definitely link a uh golden community like issue um but like um so in our like a monitoring.

E

It's not like uh subtracting the lazy free process usage which in gold and like garbage collector.

E

um So if, if you are, if you are using m device um it will, the gc will release the memory, but the os will keep the best effort to uh to reduce the memory until on memory pressure.

E

So in one like a goal 116, they are like uh turn off this feature, and then we uh use uh previous um using the previous, like linux system call. So the monitoring can uh correctly calculate the memory use percent.

A

Right, thank you like. If you could link the the the second issue, you were mentioning that. That would be great. I will add it to the to the meeting.

D

Yeah yeah sure that comment you were, you just said.

D

A

Sorry, like could you repeat him like I.

D

Was saying it was the comment he was just on, like I think about about peter's comment.

A

uh Which one sorry ah here, okay,.

D

Yeah yeah, so I I think in in summary, what is happening is um so even though gc kicks in and memory is freed, it it still shows under the process. Memory like the memory is still with the process, and even though it is free, it is only apparently reclaimed back when the os is actually under memory pressure. So when the os actually needs some more memory which it is when it comes and takes it, and then we see a dip, so just because memory is high, it doesn't mean that it is actually using that much.

D

um So basically, this go debug that environment variable setting that will change this behavior. So.

A

Does it mean that if I understand correctly that, uh given this this, I don't want to call it back, but uh the way the golang works with with uh not releasing the memory immediately to the operating system? uh It will get even better like this graph or is my understanding wrong.

D

Yeah yeah yeah, instead of being at a flat line, um even though gc is happening it'll. Basically, uh it should be like drop as soon as it does gc. So that's what charles finding was.

A

Yeah I mean like it wouldn't be better person because also like baseline is affected by the same right. So it's likely that the baseline line is also more, although not likely, because if you said we are running out of memory or something like that, then probably yeah.

D

I mean you can think of it as if what is shown is always the it's. An upper bound yeah like.

E

D

Usage is at least that or or lesser.

A

I see uh cool so.

D

Like you said, we.

A

Had like similar issues with uh inside kubernetes, I assume like same story right when we were serializing some problems.

C

Nothing like that. I can't remember exactly it was like long time ago, but but yes, um I remember that, like the size of the protobuf were were causing us problems. That's.

A

Actually, funny right, you would assume that prototypes should be fast and optimized and because protein.

C

And it's not because, like it's going over the whole object, like I mean it's, it's going down recursively into like I see yeah yeah.

A

Nice all right cool, that's nice. I saw it was already backported to lcd34 right in the in the pr actually.

A

That's cool nice.

A

All right. Do we have anything else to discuss.

A

All right, if not, then I believe we can just take back the 12 minutes all right. So thank you. Everyone and hope to see you in two weeks take care.

D

A