Eclipse OMR Architecture, 31 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OMR Architecture Meeting 20220331

Description

Agenda:
* cgroup v2 support [ @babsingh ]

A

Welcome everyone to the march 31st omar architecture meeting um today we have uh one topic from abnet singh I'll be uh introducing the uh omrc group api so that whenever you're ready, please take it away.

B

Thanks daryl, sir, today I will be providing an update on the current state of uh cgr api and over what it does and the existing problems we are facing with this api and what steps we will be taking in resolving those problems.

B

So the main objective of this presentation is to start a discussion so that we can prevent breakage of this api in the future and before diving into that discussion, I will give a brief overview of what c-group is or what control group on linux is and what the omrc group api does.

B

So control group or seeger first shot provides a mechanism on linux, operating system to control and manage system resources.

B

Its implementation is hierarchical. That means it has a tree representation and it has ended entities which can be either called as a resource, controller and synonyms. Other synonyms are controllers or subsystems and for, for example, the cpu controller, regulates distribution of cpu cycles, bandwidth and scaling policy on the operating system.

B

The cpu set controller provides a mechanism to place cpu and memory nodes which becomes valuable on nema systems and the memory controller regulates distribution of memory and its usage. There are 12 controllers in total, and now there are also two variants of sega v1 and v2.

B

Currently, a user can choose which variant of c group it wants to use, and but there are ongoing efforts to adopt secret to across linux systems, which will probably take some time.

B

So, what's what are the differences between speaker, v1 and v2? So v2 was developed in order to counter the problems which were encountered with the secret v1 and, for example, in sega v1.

B

The users are given a lot of flexibility to do things, which leads to a lot of complexity and confusion and at the end it becomes very difficult to manage and account for system resources.

B

So in c group v2 things are more simplified and unified, and this helps in encountering most of the problems seen with seeker b1.

B

Some of the simplifications are seeker. V2 only allows a single hierarchy, which is not the case in seeker b1 in secret b1, each subsystem or each controller could have its own hierarchy, thus a complete just making the implementation more complex and another simplification is that processes can belong to a only belong to a single subgroup in v2, which is another step towards simplification, because in secret, b1 processors can choose to be part of multiple subgroups.

B

So overall seeger v2 aims to simplify and unify things in order to improve management of system resources.

B

So this is a brief overview of what c group or control group on linux is uh now moving to the omrc group api.

B

It's. It only reads: details about processes, resource information from the c group interface, for example. It gathers information about memory, stats limits, how much memory is being used and so on. It also needs information about cu, a cpu bandwidth, to determine our number of cpus available and for on numa systems.

B

I think it uses the cpu and memory node placement information, which can also be derived from c group from the c group interface, its usage and runtimes is for memory allocation thread management, and these stats can also be used to diagnose failures related to system failures, a system resources, because these things are the omer api, provides functions to print c group information in opengl through java cores, which is a.

B

Which is a core dump and contains all the information about the runtime.

B

So currently we are adding support for secret v2 and omar and omar issue. 1281 is being used to for the high level design discussion and as a global tracker for subtasks.

B

The main issue over here is that uh the missing secret v2 support was not caught by the omar testing through the pr build and diagnosing malfunctions of this api is challenging because the failures are not blocking the process keeps running and it can lead to performance issues, and these performance performance issues can go unnoticed if the purpose not being continuously monitored over different iterations of the runtime.

B

So we have seen breakages and downstream projects specifically opengl 9. runtime starts with incorrect memory. Runtime does not load the embedded aod code and the malfunctions can also lead to have a potential to cause more per issues, and in some cases it has also prevented customers to adopt open j9.

B

So this makes it critical that we avoid breakage of this api in the future.

B

And the first step in preventing breakage of this api is to enhance functional testing which will be pursued as part of omar issue 1281, but this would be sufficient on its own. We will also require uh infrastructure changes, since this api works differently on sega v1 and ziggura v2. In addition, its behavior changes whether we are running in a container or not.

B

So we need uh four configurations: a linux with seeker v1 linux, with seeker v2 and the same in and container container containerized environment.

B

So uh we can support these configurations without adding new pr bills. uh If we optimize existing linux pr wheels. There are, I think, seven pr builds on different linux uh platforms and if we run them with each of the above configurations, we should be able to support the required infrastructure to fully test dc group api without any new or more resources.

A

Can I ask a more fundamental question about.

B

A

um So how was the api breaking like what what happened to it that caused it to to break? I mean, yes, we weren't testing for it, but why did it become stale.

B

uh Because I think in some cases, uh some of the containers or some the newer linux operating systems are using secret v2 by default and on those systems. This api just gives incorrect information and the testing the current testing we have does not validate if it's not sufficient to validate. If this api works correctly,.

A

Okay, so we only supported v1 and that was causing some problems in systems that were expecting v2. We were giving incorrect, so the one of the solutions is to implement v2 support in omr.

B

And in containerized environments also, some of the functions this api relies upon was were malfunctioning, so we couldn't identify if we were running in a containerized environment. So since there were no tests and we were not running dpr bills in the containerized environment, so a lot of those uh malfunctions were also going unnoticed.

B

Okay, thank you so for discussion. I have a list of questions over here. We can go in order and see.

B

Or what others have what viewpoints others have so the first one is like: will there be side effects of modifying p orbitals in the manner I have suggested.

A

So what kind of testing do we currently have for the c group api? Even for v1? Are there actual port library tests that exercise that.

B

I only saw one test and it only exercised one function, which was to test the mem limit and a lot of other a lot.

C

B

No, that's not sufficient. It's like lacking a lot of other. There are a lot of other functions which have no tests so right so.

A

Is your proposal to round out the testing for v1 and v2 by providing tests for each of the api methods there.

B

Yep, that's what enhancing functional testing would cover.

A

um Okay and then in order to modify the pr builds. That presumably means that those tests need to be running uh in a container.

B

Or in a linux operating system bare metal, bare metal operating system with seeker v1, and then you would need another one with uh seeger v2 enabled and then you would also need to run secret, v1 and v2 in a container. So four configurations in total. But I've seen there are different container technologies and to testify.

A

B

A

So a vanilla linux installation like we would see from any of the nodes that we have on the test farm for omr. Do you have to do anything to enable even c group v1 or is it available by default?.

B

uh Both are available uh depending upon what os you're running, uh either c1, either v1 may be enabled, or if it's a newer operating system uh v2 may be enabled. So we don't know exactly so. You would need tags. You need to tag the machines by inspecting them.

A

But one or the other would be available.

B

Only one yeah you would, I think, both are not available simultaneously.

B

So you can only build one or the other.

A

And then you'd want to be testing both apis as well right.

B

uh Api stays the same. It needs to function correctly in both the environments.

A

So you need to have some nodes with v1 and some nodes of v2. Then.

B

And some nodes with container, which will run in linux, with secret, b1 and secret, another container containerized environment, which will run next with seeker v2.

A

A

Okay, now I understand the bottom part of your slide here. That's you've already found out which nodes have which.

B

No, I I I this is just a suggestion like we can like I'm assuming all of them currently only run secret v1, so we can modify these pi bills so that you know, for instance, linux x86 will only do linux with secret b1 in container x86 64 will do v2 in container and then linux and arm will only cover linux c group, b1 and so forth.

B

I haven't uh looked into uh what machines, what coverage or what secret version each machine has yet so that is still a to-do.

A

And then presumably you'd want this on power, linux and zed linux as well.

B

No, I think our linux operating system should function the same on all those architectures. We just need at least what it doesn't matter, what architecture it runs on. We just need a variation of this configuration running. It doesn't matter what architecture is chosen for the configuration.

C

A

Okay, well I mean, if you're just piggybacking on top of an existing either a linux installation. I don't know if we actually run uh do we do. I don't think we do container testing on omr yet, but um certainly running it on top of um whatever the bare metal or it's a virtual vm seems to be doable.

B

But then uh docker is no longer free as a container technology we'll we have to worry about all the things at some point.

B

So I was hoping to use docker for the container technology.

A

Do they have? I don't I don't recall all the restrictions on the other on the on the new license? What about do they give any sort of uh uh exception to open source projects, because that's.

B

A

B

Yes, there should be something like that, but I would have to double check, but because there are other container technologies as well, and uh we will I'm not sure if we need, we will probably need to make sure our functions will check whether we are running in a container works in those container technologies correctly, because the implementation may change different may change slightly depending upon the container technology.

A

Okay, so you'd want, I mean ideally you'd like a docker and you maybe want podman or something.

B

Yup, so we have some uh container variation in container technology, so we don't so we will know like like some, so I'm guessing podman and docker are the widely most widely used containers.

B

So we can probably go with those two. So.

C

A

What other questions did you have.

B

uh They're over here I just.

B

So, like I'm not sure like if we lose functional coverage for other things like gc engine, if we modify pr builds like this like, if we are running in a containerized environment, will you know there will be issues with gc or jet testing.

A

I mean we're not doing anything, I mean I'm thinking specifically about the compiler and the compiler in omr like this is not open, j9 testing. This is this is omr testing.

A

um I don't think the behavior will be any different.

A

Off the top of my head, okay,.

B

And the other question was like: is this sufficient to prevent breakages in future enhancing functional tests or and adding more infrastructure support, or do we need to observe.

A

Well, the fact that we're you're actually going to write tests for each of the apis. I mean that's that's way ahead of where we are right now it sounds and the fact that we even do v2 testing, where that's way ahead of where we are right now, so um it's definitely a lot more sufficient.

A

um You mentioned something earlier about performance problems. What's what's the origin of what? What would cause that? What what causes the performance issues.

B

uh For instance, if you uh allocate less memory, then what can be allocated, then you are spending a lot of time in gc, for instance, and which will affect the throughput of an application. Similarly, if you're not using aot code, then again your throughput will be impacted and all those things I think depend upon this api functioning.

A

Correctly so is this: is this sort of surprise behavior in that I'm asking for a certain chunk of memory? For my heap I don't and and somehow mysteriously I don't get it.

C

A

Tells me you don't have enough memory, you have to reduce the amount of heat and you have to restart it with a lower heap. Setting.

B

I think we've only seen startup behavior differences in startup behavior. The jvm starts with a lot less memory. For instance, I think it only used 512 megabytes where it could have used three gigs of memory.

B

But I think we have not identified any of those surprise issues where midway during runtime memory, usage fluctuates or memory. Behavior is different.

A

How long do the tests that you write? How long do they take to run? Are they just super simple unit tests or are there other.

B

They should be super simple.

A

Okay, so this isn't going to be a huge burden on the ci pipeline. Then.

B

No, this should be a fairly simple few seconds.

A

Okay, so what's the um what's the plan of attack here, um you need to configure the some of the nodes on the um on.

C

A

Ci farm with the um yeah with uh you have to re-image them, either with with with vt or sorry re-enable the enable v2 support on some of them. Some of them. You have to look into getting some container technology installed on there.

B

Yes and then I think the pr build scripts will need to be modified, which will you know, choose the right machine with the right configuration and, for instance, if it's running in containerize, I think some more commands will be added to the pr build scripts, so that will need to be updated.

B

But I don't have access to uh formatting the machine. So who would be the point of contact? Is it going to be adam joe.

A

Joe adam and joe have typically done that in the past. Yes, for us.

B

So I can uh create an outline for them and see if.

A

Yeah, I guess it would have been good to have adam here today to at least address some of the uh address some of those questions. But um if you have an issue that you create an omr for what you need, we can certainly tag the appropriate people there so that yep.

B

I will open an initial later, so, okay, we can have a discussion about it.

A

Okay, so that's the the infrastructure side that you need and then there's another. The other thing is to actually write the tests. Well, there's writing the tests and there's also, I guess, doing the implementation for v2.

A

Yes, and it's going to be part.

B

Of former issue, one two, eight one.

A

B

And I think eric is working on it and I'm also working on it and keith is also helping with reviews.

A

Just trying to think how that um the effects on the nodes, where you want to run container technology, that um that container technology is only you're just making it available on those nodes that doesn't mean that a test actually has to use container technology. Is that right? So, for example, just the tests that need to deploy in a container would use that, whereas we could still dispatch jobs to it that don't want to run in a container.

B

Correct, but at this point I think we run everything under a single command. So if you do jenkins build all, I think it's going to run c, make all tests under a single command. So we don't have the granularity to run certain tests in a different setting and certain tests are in the containers or.

A

B

A

Oh, I didn't mean it that granularity, I meant, um what do they mean.

B

But you can use the machine to either run the test fully on the bare metal side or you can choose to run yeah.

A

That's what I meant yes.

B

In a container environment, yep.

A

B

A

All right, um I don't know what, so you are confident that you don't need to be testing this on power or z,.

A

64, even I guess you had that you had arm on your um on your list, but.

B

I think once we do seeker re like just one configuration on any architecture, should be sufficient to test those apis. I think those apis do not function differently based upon architecture.

B

They are only linux specific, so we just need a linux operating system to test those apis.

B

Unless the linux implementation varies differently or on those architectures, but I don't think so. That's the case.

A

Okay, um not aware of this off the top of my head. Does this apply to mac os at all.

B

Cgr v1, I don't think it's only linux specific server, it's purely linux, okay,.

A

All right, okay, um any questions for babnet concerns.

A

So you're going to be driving all of this, then um the infrastructure side, the test development.

B

Yeah, I will cover everything.

C

Okay, so this support itself has been uh merged into omr. It's the testing that is being uh discussed here. Is that right or has the core v2 support not been merged yet.

B

It is actually ongoing at this point. We are, we have peers open, which are being reviewed.

C

Okay, but those pairs are not going to be held up by this. These broader questions is that right.

B

We are locally testing so so before merging, we are making sure it works locally in our environment,.

C

B

So, but in the future, because I think uh it will help us uh resolve those customer issues quickly if we can add v2 support as soon as possible. But in the future we will have tests which will automatically verify the functionality of this api.

C

Yeah, for sure I mean, as you said at the beginning, this is ensuring that it stays both v1 and v2 support, stay.

C

So don't break right, so that's certainly fine from a medium to long-term perspective, but in the short term, you're testing on your own machines and merging the pr continuing to push the support into the into the project. Right. Yes,.

B

That's correct, I think dpi bills will only test compilation failures, but uh behavioral testing is done locally and I think we are getting a confirmation, like things are being fixed in up in upstream or sorry in downstream projects such as open j9, as those fears get merged.

A

A

So when you're testing one of these api, like, for example, you you mentioned earlier an example about um the available memory- was not coming back correctly. How are you valid? How are you ver? How are you testing the api? Are you just making sure that it returns so that any kind of exception, or is there any attempt to currently.

C

That's what it says.

A

Make sense of the number that comes back it's just to see if it works like doesn't crash or anything or no error codes, or anything like that.

B

I think currently it only does uh the current test only checks if it's returning a valid value, it doesn't verify if that valid value is correct. So you would need system, information or the post information on uh what memory the host has and then you would need to compare it with what the api is returning.

B

That is missing, so you would need a further level of verification where you already know what the machine, what stats the machine has and then you will need you will compare with what the api is returning right now. I think uh the mem limit test only checks for a valid value or if there are any errors in running the api, so it can still return a valid value and it may still it may be incorrect.

A

Okay um sounds like this is.

C

A

Important uh important work for the container environments for sure and glad to see that there's more testing coming, that's good.

A

Any uh other questions for baby.

A

Okay, well, if not um thank you babneet. That was a good introduction to that. So look forward to seeing some work happening there.

A

uh Okay, that was our last uh uh our only topic for the for this week's meeting, so I've got the agendas created for the next two meetings. We've already got a topic for the next one in two weeks, but you know if it's a small topic, we can still squeeze one in there so by all means find those and propose topics on those issues. If you, if you want to bring something up, um if not, uh I guess that's all for today, we'll adjourn and uh we'll see everybody in two weeks, thanks thanks baby take.

C