Kubernetes SIG Node, 7 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Node Resource Management WG, 2022/03/07: Kubelet Plugin Discussion: CPU Topology management (PM)

Description

Meeting notes and Agenda:

https://docs.google.com/document/d/1ALxPqeHbEc0QOIzJ3rWWPpwRMRlYDzCv0mu2mR4odR8/edit#

A

Somehow, okay, let's see right, let me start uh Sergey uh and and Ed thanks for joining um yeah, so just to.

B

A

Some context where we are currently with the cap- I, don't know if you all are familiar with the attribute based API, um so we were discussing how to bring that um yeah in in kubernetes uh through through some sort of um if we could leverage some of the resource claim mechanisms available with gra or how we can extend that. This gives a short example which we used in the past, which kind of uh attribute based requests or resources.

A

Cpu resources can look like, um so you have standard kind of uh clean um kind of mechanism from theory in our kind of prototype which, for which I have a demo. Today um we started just with a simple Json format in the config Maps. um Basically, what you see is some sort of set. Of course, usually we with those setup course Define pools. Basically, it means um you want to request a small pool of two cores which is exclusive with some um basically also requiring that their end on the same siblings.

A

Maybe some frequency uh then another six exclusive cores, without additional configuration or poor shared course. Just to give an example how this is interpreted. uh This was our first iteration how such a Json thing can look like, and we worked in the last days to uh to give a little bit more details about possible API.

A

um So we were thinking uh to cap, basically, first, some sort of identifiers at the beginning with which you can easily identify the pools, usually similar to dra. You can use those in resource classes and um and and assign them to Containers quite easily.

A

um So, basically, you give some some IDs for each kind of pool. um Then the other thing what we were wondering, if it would make sense, is to add some sort of capacities. So we could so that we can relocate those pools in a static way, something similar to. If you think about the static policy, then we could predefine uh those pools and they will not change over time and basically, when you are claiming um some some set, of course uh the pools will get reduced over time and at some point, user will get error.

A

If you run out of resources, this was one possibility which we are considering. If we, if such kind of capacity would make sense- um because um we we are also thinking uh when we are pulling out resources, um there are several ways how to do it. We could do uh some sort of if you are in in um in the case that you want to pull exclusive resources. Usually those are pins, so um you will see if you are requesting an exclusive resource in most of the cases you don't want to.

A

um You want the full control over the core or The Logical course. That's why something quite guaranteed? Qs is very close to to that um kind of uh wish.

A

um So basically we could do the first two cases here are exclusive, um exactly two cores or let's say we want an exclusive with some sort of burst between six and eight. um But in that case the CPU sets what we want to allocate.

A

um We have several possibilities. We can um widen out the the CPU set to 20 cores and fix basically the CPU shares so that they fit this kind of range or we put in exclusive kind of uh um type. We we put limited to eight cores.

A

Basically, the CPU set is Max 8, of course, and then then, basically, it can burst between six and eight dependent on on the application, and the more interesting case is the shared uh where in that case, we again Define a range, um but there are multiple ways how you could uh deal with with shared again. This could be limited just to a CPU set of four cores, or we could limit it to the complete eight course and give CPU shares so that they can work on the 80 course.

A

The idea of the shared, who is basically, you, can put any kind of uh Bots which requests shared resource on that exclusive resource. You cannot put other resources on on it. They are not shared, so they they will be not allowed to overlap um and yeah that that was roughly what we were.

C

Thinking how do you know that this one to four is shared and six to eight is exclusive? The.

A

The order currently gives it the information, so we assume- or we are defining um the the Maps or the Json in a way that the truth have the same cardinality this list. Basically um it has always the cardinality of three uh all across the Json. So basically, if you look to the third place of the list is corresponds to the uh you can find out the CPU attributes for it it's basically matching shared. So the the position in the list gives us the calls all across uh the list more or less.

C

And I didn't quite understand what the pull means. uh Is it something that we expect machine to have or.

A

Basically, you are requesting a pool, um um or this was just a uh something which we wanted to throw to in the conversation today. If such kind of capacity would make sense.

A

um So originally we were thinking just to claim course based on the available resources on the machine, uh but we could also Define some sort of capacities, so we could pre-allocate the pool and then start pulling out of the boobs. uh But if there are not enough course um on a certain system to Kindle this pull request, it will fail right.

A

So the the pool is the upper limit, um so your shared course can be, or you can use a Max 80 cores that is shared. um Then you have 10 and 20 of the other two two cases.

C

And this is what port will Define.

C

A

Is part of the claim, so, basically it's defined in the in this Json uh configuration map which is attached to the resource, claim right.

C

A

um The other kind of uh aspects which we were considering to add to the spec is how to control Numa. We we have use cases where we need pneuma device, Affinity, uh basically Network, affinity and stuff, like that. So we were thinking to Define that, as some sort of close to a binary condition, um there's usually the Numa device will be picked by another Plug-In or by another system. Let's say device Plugin or a dra plugin.

A

um We put require this. This stuff will be executed after the the device was picked and we could require. Basically, we want the device Numa Affinity to the selected device. uh We we prefer. Let's say it's: it's good to have no more Affinity to the selected device, but it's not a must and then we don't care uh something like this, this kind of options- um and we we were thinking similar for memory, Affinity memory, Affinity, um basically pointing back to memory controllers. So basically we we were thinking to have in more like kernel, style definition.

A

uh Manner uh binds required means. Basically that's uh you are. You require to be uh on this to have to be on the same socket as as the memory controller. What you need so um bind preferred, so you try, but it's not a must, uh and then we have something uh which uh corresponds a little bit to today's um Numa spread options. If you think about topology manager, it's interleaving, basically, that you can interleave memory controllers.

A

Right um and then we might need additional parameters later for a huge page handling um um and and so on, right.

C

Sasha was telling me about this scenario when some Modern Hardware has this uh memory. That uh kind of bound to tune so I understand that we have some oversimplified model right now, with CPU manager, topology manager and.

D

C

Map and you trying to go one step forward here right so a little bit more uh involved like you you're trying to describe every single uh pull and like how much Auto School we take and how uh to find resources.

C

I wonder if it will cover all the hardware or like uh Next Step will be. We need completely different model because Hardware finished.

A

We might need to extend it over time. um I think the the idea is to cover at least Hardware, which is coming in the next um several years, um at least also. If you read further, we want to to deal with core siblings and cluster siblings, um um so they are right. We we will. We are thinking according to to trying to cover the needs of Hardware coming in the next I, don't know, let's say um three to five years: scope.

C

Okay, so there will be some version there.

A

Should be some versioning of that stuff, more stuff coming in over time, so I think we we have two versions.

C

um Just more details to be exposed to Cluster admin, so cluster admins will like. Is it something that can be Auto discovered out of the machine or it should be something the driver will know how to be configured.

C

uh I I still quite know like um this music is about like plugins for couplet I'm, like I, think you already made three steps forward and I wish I I still catching up, so maybe uh it will help if you can explain more like how this uh file being created like who is responsible for that.

A

This this uh yeah, so this is responsibility of the user, the only thing which we are not sure if it's a user responsibility is this capacity pool Pink So. Theoretically this this can be moved out where it's something which is not needed, um but all the other options. We see, there's responsibility of the user. Usually it's yeah!

A

um That user wants uh his application to be affinitized uh close with the device or he wants an exclusive core or a shared core right or even both in in a set of containers. Right.

C

Okay so and uh yeah so user will declare that yeah this capacity doesn't fit in my mental mode, like I was uh that's my.

A

Optional, currently that the reason for the capacity is the um if we want to um how we deal with this kind of burstable uh classes, more.

D

A

You have something quite burstable if you think about standard burstable and best efforts, Bots and containers. Currently in kubernetes, you get the cool set CPU set of available resources on the system um so um to to guarantee that we have an exclusive um course. We need something like Dynamic policy, more or less.

A

We will need to squeeze down uh over time this kind of um burstable and best efforts, um cases, and that's why we were thinking such kind of capacity can help us start small with with a static kind of rebooked resources, um but or the other way around this. If you have a shared pool and such kind of burstable uh definition we're limited to four this, this will work too.

A

So you will get the CPU set of four cores and you burst between one and four on that, so you can guarantee that your exclusive course will went on another CPU set, which will not overlap that that's the reason more.

D

A

Yeah we were also discussing internally. Maybe this doesn't fit in the user Model A little bit.

C

A

C

Don't know how many more slides do we have, but I I really want to test some scenario on this and if it hits this model, so um somebody came up with scenario saying uh it's really about sidecars. So if you have a job that is uh really CPU intensive, it wants exclusive course it wants some Numa Affinity it wants like it wants to be executed really fast and uh it knows what it wants.

C

But then you also want to run some Metric sidecar on the same port that will report that it's still alive and, like maybe CPU, like some metrics, like what? How many, how much things for protest and stuff like that, and you don't really care about this metrics thingy, because it's metrics container it can run on shared CPUs for all you know like you, you don't care about it like you, you just need some resources for it to run. Can it be expressed like this resource claim? Will look like uh I mean it will?

C

Can this resource claim be used to explain the scenario.

A

Yeah, so you you could basically your performance stuff. um Let's say if we want to I can write here so um basically yeah. Let's say your performance stuff, you you have two one for the sidecar and one for the main app. So your main app, let's say it, wants I, don't know how many cores, maybe two, um so you will get two pinned cores and then let's say you want. uh You have the sidecar, um maybe the the sidecar.

A

If it uses just one core, you define white one chord for the shared pool, um so it ends on a different CPU set um if you're a sidecar. It's it's some sort of burstable thing. Maybe you can Define one to five cores.

A

um Writes the the question now is we have several possibilities: how to deal with the burstable case. We can limit it through capacity or we can limit it um um based on the available CPUs on the platform. So right, uh if your platform, let's say had to originally 40 cores, uh we can take out two for your exclusive uh fast container and we can uh give the the the the rest of CPUs as a CPU set, basically for your sidecar and with with this kind of CPU share.

A

Okay, that's the idea more or less what we are going and.

C

How do you mark your containers to use those.

A

Questions the containers, usually in the um yeah- you you see here, you you have to specify in in your container what what you want to claim um the um okay most probably the names are a little bit wrong, um so you you can specify in in resources a claim, so this this is how you point to your claim: It's. Usually the the template uh has the name. So this this should be resource.

A

um So and then Additionally you have other options uh in in this stock. You can specify um so you see that it's per container um you have the claim, uh but there is also another option to inside the claims to specify resource class I think it was called it's part of the array, spec um resource class.

C

And it will be named, I think from next slide right. It.

A

Will be basically the ID, let's say your E2 is the fast thing you want this container to be the the fast uh stuff it.

B

A

Be and then your sidecar is basically uh S5, let's say just as an example right.

A

So the if this this uh was the okay in this case was uh a second container without claim. But if you get a container with a claiming again the resource stuff, you could specify, after that another resource class and, let's say basically the the shared resource class, and we will know to pull in to consider the basically the second element of the allocation kind of attribute list for it. For, for your.

D

C

Okay um and uh we'll cover direct scenario: Derek was uh asking about scenario when all system ports needs to be like uh system and static, Port needs to be assigned to specific set of cores and all the rest of the course needs to be assigned to like uh workload. uh Ports. Is it something that can be pressed here or it's? uh It doesn't cover the scenario.

A

The system resources we we were thinking to handle similar to how it they are handled today, so you have reserved CPUs, you define them by cubelet configuration um and and then they are removed out of the set. Basically, it's not non-available CPUs. So basically we will take the the standard mechanism, how you define system resources as you have it today and forward it to our manager and do and remove them from the setup available resources um just to maybe to give it as an illustration, I have this kind of picture.

A

Let's say: if you get available resources, um I will paint it a little bit. um So you you said when starting tablets I want two two cores um so two cores to be system: resources.

B

A

As we do it today in cubelet, um we will read this configuration through the new manager through the CCI manager and remove them, so what our manager knows at any time how many available resources we have.

A

So if you started with system resources, two cores so basically the initial set will be eight, um a chair of course, eight eight exclusive course basically depends you get four four pairs for for hyper threats or hyper trading pairs, bonus um that that's the at least the approach of what I was thinking to start with just similar to the current configuration. What we have that we we don't do it uh don't give the the it's more administrator kind of configuration, and we we make sure that we take it into account.

C

Okay and uh when, if you want to schedule ports on this reserved, CPUs.

C

Right I think that was a scenario that Derek wanted to investigate: I, think they're, doing it through crier right now. So cryo has this uh way to pin uh system ports on specific CPUs and.

A

uh Do you want to pin the, um but this this is.

D

A

Or they will be pinned more or less to um usually like yeah. If you specify reserves CPUs 2, we will give the first two cores, basically on the system or something um so um yeah Supply, which of course, to return us so yeah, I, think or I. Don't know, uh usually the way how it was done before it was just counts, but I don't know if you can, if it makes sense to configure, which course, maybe it makes sense to to to configure that you want them on both sockets or something but um yeah.

A

It is administrator kind of configuration we might think about it.

D

I think say: wait who owns that piece. So, whether or not we want to continue to do it through cryo or if Derek would like us to do it through these plugins and just send over the information. But the information is going may include, which Numa Zone and what type of core. So, if we're talking about cases where architectures are coming up with a combo of proficiency,.

A

And performance, but just just to understand, Derek, wants to run some bots in a later time frame, which you have to go to the system course is. Is this the request Morris or.

C

I I think and again uh I understand why I asking about it from my perspective, I think what dark scenario is that all these like? They run a lot of demons and uh some static pool, and they want to make sure that those static and demons only occupy specific pool of CPUs and uh they don't try to execute on any other CPU. So all other CPUs are allocated for workload so like there are, customers would know that uh those CPUs are not accounted for any workload and everything else is workload exclusive.

C

So no no system possible for running photos.

A

That that's the goal. We would also like basically to subtract them away from from from from the sets for available resources for applications.

A

um So we we have to yeah um figure out how if we still use the previous mechanisms to Define them to configure them or we we have rather another another one.

C

Yeah and I'm asking this question because I want to understand is this cap? Limiting to um this is what I want. uh Please figure out how to give me that uh versus, uh like you also specify like there is also component of it that you know how system looks like, and you know how your request will mop into uh actual system and maybe like uh scheduler, will even know like whether your workload will fit into the system or not fit into the system, and it can be done universally.

A

um User will not know the system, let's say, but user will know what he wants.

A

um White user will know that. Let's say my application is memory Bound. In that case it's good to spread across all memory controllers, um so I will user will specify that and let's say user knows that uh my application um needs to to be pinned um and then also yeah, the spreading of controllers. So it's more um not really knowing the hardware but knowing the nature of the application and then requesting requesting the right or giving a requests which shoots uh should find the hardware which fits fits. The needs right. Okay, yeah.

A

um Yeah that that's what we are after um so then uh we get. Also in the previous meeting we went a little bit through some architecture um decisions we got. Currently uh we are considering.

A

um Basically, um the idea is to have a new manager, the CCI manager, uh which can understand um the airway requests or basically claims um we, the we could we or we will reuse completely the controller components provided by dra. So we can recreate or use the same API to create controllers to handle the scheduling uh pieces.

A

um What and then um originally we we were thinking to have still um an old driver um which which can handles the reservation um of claims, uh but as discussed today, uh basically um Kevin was suggesting. We don't need that piece. We can actually um Implement everything in one component, so we could Implement um the reservation kind of um handlers uh inside the CCI Drive CCI Chandler, the CCI calendar is the one which the CCI manager will call. So basically, um our kind of.

D

A

uh For for allocating the the claims will be completely independent from the node driver. We will not need an old driver and we will just reuse the controller and the resource class. uh The resource class allows us basically um to um to pick um to pick to basically process spots which which kept claims. So we we are applying the driver on the on on the pots having a resource class with a driver name with the right driver name, but I will show that we we have it in the Prototype.

A

Basically, it's the the same mechanism like in theory where your your uh kind of uh resource claim uh it's basically um some called maps to a driver. So this is one-to-one basically similar for us, um then suggestion from Kevin from Nvidia was uh if we can get rid um or if we can handle or make our CCI components on the driver's side being able to handle the whole thing without additional node driver.

A

So he was suggesting that we have one single components: independent from the the array node driver, which implements the the logic which we need to handle the location uh or even the reservation of the resource requests, as they are a little bit different compared to the resource requests. What you have for devices. This will allow us to nicely split them away from from the original gra, which is more for devices.

C

The config file you showed, looks very generic like it. It looks at uh why do new specific driver for that? Do you have a sense like why why it's not like it looks like a policy for a CPU manager or like topology manager, more.

A

It's a remote policy, um the driver, actually it's uh it's helping that you could um handle a set, a set of pots, not all um so you could still run other Bots through the standard CCI manager without going through driver. This is one one of the benefits of the driver, but also another kind of benefit is you can do similar kind of claim or you can yeah? The the benefit of the driver is from scheduling perspective.

A

uh If you have this kind of complex or yeah attribute set, how do you schedule? It becomes a question because if you think about standard pots, you have the scheduler, which does runs a scoring algorithm.

A

um So if, if the podcast, uh basically a resource requests and limits, uh the scoring algorithm will start filtering nodes. So this is where the driver, um together with a controller component, takes over um on the scheduling side so or it helps on the scheduling side, determine the right node.

A

um The condition here is basically that the Bots, using these claims, um similar to the example, does not have any any kind of uh limits and, in you see in the resources, I'm, not adding resource limits and resource um requests, as they are considered by scheduler. Usually.

D

A

If I add those my the the scheduler will score the pots or according to requests and and the notes, um so we are currently, we could avoid that in Alpha version, by keeping them more or less best effort.

A

um Not having that, and basically a scheduling decision is taken by the controller.

A

So further kind of optimization on on the scheduler side- most probably it's possible, but uh we will think about it in literally in better Pace. If we can do some scheduler processing of claims and stuff like that. So this is uh still to be investigated, but I think for starts the we. We could um basically rely or require that that such kind of thoughts having a CPU Drive, the CPU request, claims um more or less they.

A

They don't should not Provo or if you provide uh request limits um and so on the then the scheduling can further limit the set of nodes well where, because, usually the array controller, if you think about the area controller, it's run after scheduling.

A

So um you know, if you did um the scoring before that, the controller might might have already less less nodes available to schedule. Two.

C

Yeah, it will be interesting how, like.

A

C

Isn't a lot of touring uh relying on requests and limits? So um it'll be interesting for the student to adapt uh to understand this uh resource claims.

C

D

Can you know yeah straight? If you can, can you get back to your spec on the cptk user.

A

Yep this one, you.

D

D

Yeah, it's very generic.

D

About versus uh efficiency course and stuff going forward. So when you look at the CPU attributes, I know right now we just have exclusive and shared, but because there's all the different cores coming out for sustainability reasons, we need to be able to handle those.

A

Another attribute we will have core types, most probably in the future,.

D

Because the other thing that you run into with users is that it's yours, but they don't necessarily know what they're running on. So, if you're, using a very large heterogeneous cluster and you're, just getting scheduled for Cores for like a 1.5 gigahertz, is going to be very different than gigahertz.

C

A

C

That uh those attributes would be driver specific because they'll be single driver covering everything you just want to develop. The driver faster.

D

D

Scheduled on on your chip, one types right and those drivers are only installed there and then your driver is scheduled. Also on you know your two types right, and so, when you do the scheduling it looks for so, if we were just doing exclusive and shared, we don't care, you got scheduled anywhere and then your driver handles it. Otherwise, you're going to have to get scheduled only to those nodes that have they available attributes.

C

Okay, yeah I, think I think I understand the reasoning. uh My comment still stands like. uh There are two link that looks at the requests and limits, um inclusion, monitoring and such, and it will be interesting how those ports will be handled and whether we need to do something to satisfy that. Maybe some calculated requests and limits will be needed.

D

As possible, okay, yeah.

A

Well, you mean also calculation on scheduler site for the future or.

C

D

Don't know yeah.

C

It's um I, don't know specific scenario. It may be um if customers just monitor like uh been packing, because there are nodes, they may have troubles not having a requests and limits on uh on specific ports, because they calculate this bin packing uh using those fields and not having these fields will require them to change the tooling or like their children, could be confused.

A

By the monitoring mechanisms.

C

Yeah yeah in parking of ports would be maybe affected. um Okay,.

C

A

So that that's more or less what we had um for from from slight perspective, uh we have 10 minutes, maybe for a short demo. If this is fine or some further questions.

B

Yeah I I have a question uh so how scheduler would know that uh it shouldn't schedule uh workloads that actually, like don't reference any claims to to that note, which is like under control of of this uh uh CCI manager.

B

A

The the notes or the the kind of um thoughts can be scheduled on that um on that note it can go, but what we should make sure is that, after that, if the um Bots goes and consumes resources, we have to take that into account. So we have to subtract the resources, but we we are not um rejecting um rejecting scheduling bots on that note, because it runs the CCI manager uh to the contrary. We allow it, but we just take that into account.

A

So basically, if you schedule a normal burstable container, it goes through, um but we we want to to make sure that later, if we put an exclusive container, they don't overlap right.

B

Yeah, so that that would require some kind of communication between CCI manager and CPU manager and memory manager.

A

Right not not really, at least in the Alpha version, the CPU manager is disabled. It's running on policy, so we take over.

A

um Basically, the the the CCI manager uh will will be responsible um to to basically Define the CPU set of this uh kind of um Bots incoming pods, um which are not Scandal by driver. Okay,.

B

So it will be handling like everything so the those spots that that actually require some CPU and memory sources and as well uh ports that reference claims right right, correct.

A

The the goal would be later in data if we can maximize call 3 use. So um there is a lot of code already available um to handle the standard cases, the the static, CPU management and so on. Maybe we can stage out some of the codes make it available also for the CCI manager, so that we can.

B

A

Just instantiate certain yeah kind of type which which does that for us, um we will be looking into that.

B

Okay, that's more or less clear! Thank you and the next question is: can you go back to the third Slide, the control, plane, development kit.

A

B

This one, no, no, no, not this one with the picture: fine, okay, yeah this one. So it's still not very clear to me like this red one and thread two. So can you.

A

Yeah collaborate.

B

A little bit yeah.

A

This was the original idea. Now we are changing that thread. 2 will disappear. uh We we were thinking to use the the controller we required the node driver. um Our thinking was wrong. After discussing that with Kevin, um he he suggested that we get rid of the first thread and basically implement the handling for all what we need to allocate the um the the claim inside the CCI Handler make it basically Standalone um right thread. One will go away honestly.

B

And how uh kublet would this team between two types of uh of PODS, those that reference, the array claims and those that reference, this CCI claims.

A

um Through through basically driver name, so um you hopefully you don't have the same driver.

B

A

There's the driver name and the second thing is: we use another socket, so we usually for Geary. You have bar lip d-array um kind of socket to do the registration. We will do a barlip CCI um to to handle the registration separately and they don't get mixed. So this is the.

B

Idea, yeah, okay, now I understood. Okay, thank you.

A

C

Does Kevin Envision any collaboration between the array and, like uh this driver, do we need to schedule a swimming post, I.

A

Saw from him some sort of future cap where he could allow multiple controllers somehow to communicate or to take Joint decisions, maybe it would be interesting in the future to make joint decisions um on the controller side if you have some devices into play, um so this is something which which I would we we here?

A

We might have also some use of it, uh because we will need the device Affinity informations, um so I think this future kept with multiple controllers um and- and uh he was pointing to that- that it could deal with some of the topology management challenges. What what are currently there in this, this future concept. So most probably there, there is some place for cooperation there. I guess.

C

And this collaboration is architecturally or in plug-in.

A

I think it will be, as he has a cap for that I assume. He needs to add something to cubelet before that: I'm, not completely deep in his his uh cap, how to support multiple controllers but I assume there is some. Some change still needed.

C

Okay, yeah I mean to be honest, uh completely honest about it. I I can give a lot of feedback on like how architecture they put like different components in different places. My biggest problem is I. Don't know all the scenarios that people may need with regards to like I want to schedule it this way, but then I want to do this like um energy, efficient, View and I want this device to be uh very close to me. So I don't know this scenarios like uh any help you can provide on highlighting those scenarios are important.

C

Those scenaries will not be covered, but we don't care. uh We really play a big role here, like I. I would really appreciate having this overview like what is supported, what we are cutting out of support explicitly.

C

um So um this will help understand whether, like uh like design choices, because I can point on like this may not be working for this kind of scenario, but I don't I have no idea whether how important the scenario I will definitely ask around on my side but like if you, if you already know that it's not important like you, can.

A

Leave oh yeah yeah. Maybe we can think about having some section in the cap listing some of the scenarios which which can can run with it and which are limited on home yeah.

D

um Because some of it has to do with the fact that we're looking at both performance and sustainability, they both come from HPC type backgrounds. So we've seen a lot of what those look like that we don't handle in kubernetes, necessarily that yeah that one of the- and maybe we should add in this data goals and still minimize pain on the user. As far as understanding system architecture, because HPC classically user has to understand here sure we don't want to do that in kubernetes.

C

Yeah, for instance, one problem is that somebody told me about is uh topology manager doing wrong decisions instead of like uh it picks up wrong number first and then it cannot schedule anything else because, like it's already took a half resources of like big pneumonauts and like nothing can be scheduled there any longer.

C

um So I don't think we address this scenario in any sense right now, because we don't know about future ports that will be scheduled, um so we're casting it out, but I do realize that we're addressing other scenarios and this scenario of proper scattering maybe delivers later.

A

hmm Yeah scheduling is here a little bit left to the controller writer. So um if the the way, how Lord? If also this future kept for for for the multiple controller stuff, is in place, um yeah I I think it's. It becomes important how the controllers handle this kind of decision making after that, so it's a little bit externalized. In that case, the decision about the scheduling.

C

Okay, thank you very much for our presentation. um I. Think I, don't think I will see anything new in German, so maybe I will just watch recording.

A

Okay, no no worries.

C

A

B

Bye. Thank you. Thank you. Thank.

A