Kubernetes SIG Scheduling, 7 Sep 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Scheduling Weekly Meeting for 20230907

Description

Kubernetes SIG Scheduling Weekly Meeting 2023-09-07T16:58:20Z

A

Okay, uh welcome everybody to another thing. Scheduling meeting today is September 7th. um As you all know, this meeting is recorded, uh so uh please be sure not to share anything that you want to share and please adhere to the cncf code of conduct.

A

We have uh two items in the agenda. uh The first one is by Kathy uh propose a cap proposal for a generic scalar cash extension.

A

um It's got here. I can see her.

B

A

All right, uh yeah take it away. uh Do you need to share your screen.

B

Okay, are you talking to me.

A

Yes, uh do you do you need to share your screen.

B

Yes, yes, I can share the screen. Okay, okay,.

A

Do you want me to proceed.

B

A

Yes, you should be able to to do that now: okay,.

B

Okay sounds good. um Let me um share my screen hold on a second okay yeah here: okay,.

B

um Can you see my screen.

A

Not yet still black, it's.

C

B

C

B

Get in there, okay, good, so I'm gonna, it will be so I'm going to go through on the generic schedule, cash expression for um for the schedule of plugging and why we need to do this. Okay, first side, so the motivations, um if I'm, not that you know um for each Auto, three scheduler plugin, um that plugin needs to save some uh store, some resources right in the cache and then, and especially if those resources involve some complicated specifications on accounting models.

B

And then you know each plugin has to develop a local cache for that, and also in addition to a develop.uk. It also involves Implement, uh independent, informal, informal and also event handling logic. To replicate I mean to read in every schedule: plugin has to implement that which is a duplicate, duplicated effort and also there's also another problem. The resource data, because there are two caches right: one is the core cash by the kubernetes scheduler the course scheduler.

B

Another is uh the cash by each Auto, Trade scheduler plugin, and then, if there is some, sometimes the resource data fetch.

B

um You know between the you know two type, one as I identified in the slide in the diagram on the on the right, so like the T1 tampon, a T1 and by the core, scheduler cache and also a type added. You know it could be another 10 point at T2, uh because these are different, uh different flow, different process.

B

um Then these two, if these two times may be different than the data fetched will be inconsistent, will be not the same like um at the same time point then they will could be inconsistent which could lead some scheduling. You know problem um or some preemption problem. um So this is a so two issues.

C

Okay hold on for a second so by inconsistent. So basically the inconsistency can happen even within the entry of schedule programming. That is why, in the beginning of every discussion cycle, we do a snapshot so I'm not fully understanding that, because the inconsistency, if you it's not recommended for you to but any time of your screen cycle, you go to the cache to fetch the latest fresh data. Nothing not recommended because we know this kind of inconsistency can happen. That is why we snapshot in the beginning and for out of the tree plugins.

C

If you use some CR objects, we will also recommend to do a snapshot on the CR resources and then use that they just also choose during this cycle and then go to the next cycle. You can do it. Another snapshot. Okay,.

B

Yeah yeah I think that that makes sense that yeah we understand that it could also happen. You know for even without this other tree, um but still uh I, think you know. If we can have you know one extended cash together, you know uh extend the current schedule of cash. This will reduce the inconsistency probability um reduce that, but the keep one is not that. The key point is you know: each scheduler plugin has to re-implement the you know, a local cache and also the event handling logic.

C

B

Thanks for that, you know, comment um so.

D

B

Yeah we we met this because when we developed tried to develop a decile, we're scheduling- and we found out that yeah we have to do this. You know we have to develop a cash. The story, it's all this data, yeah I, think this is just an example of that we have to you, know re-implement this virtual logic to implement the informa, and then we find out that inconsistency. Yeah problem um so I will hand it over to Marissa to go through. You know our proposal um with brief proposal.

A

One second: um can you actually spend more on the use case, uh like what kind of uh informers do you use uh if you use crds or not, because um before we do any significant refactoring? We wouldn't look at the use case. Whether the use case makes sense to just to make sure that the investment is is worth the complexity.

B

And so for the maybe Teresa can talk a lot more detail on that on the use. The use case is for the on for the dcio on So, currently in kubernetes. We know that we have. uh You know CPU memory, aware scheduling, uh maybe maybe some uh some large page and some storage, um but we do not have you know this IO aware scheduling and isolation. We do not have Network at all.

B

We also don't have any cash or any memory bandwidth um or we are scheduling and isolation, so we're thinking about to add those functionality to fill that Gap. So when we develop this I also, these are all aware, scheduling and with you know, just like follow the auto tree, scheduler plugin. We have to develop our you know uh to implement our own cache um so that to duplicate the logic, um does that make sense.

A

B

Decile Staggering this.

A

Island is another resource.

B

We got this requirement from customers saying you know they have no, it's a neighbor problem because there's no disable where scheduling and isolation- and you know the the workloads are scheduled onto the same note- would compete. But these are all resources and then there's those workloads service were impacted because of that by the neighbor problem.

B

um So that's why we we add this design over a scheduling, scheduling, plugin.

A

And where is this information coming from? Is here storing at CDE as well? Just here as a crd or what is it.

E

Yes, it's it's stored in the CR and we use an Informer to listen watch. Let's see how and update the latest resource information from from this CR and- and the idea comes from that- and every uh scheduled plugin needs to implement, need to add an Informer to watch, watch the resource, update and update, there's the latest info to the local cache, and why don't we use one Informer to take care of all of this.

E

A

E

A

Okay, so I have some questions. Maybe we can leave them after you uh go to the next slide, uh but one immediate question is: why does it matter because, first of all, the information are all shared, so it's still the same Informer um and again, even if you have a single cash, you can still have inconsistencies such as um that you might have a part, sorry or I. Guess you care about a node I'm, not sure.

A

Let's say you have a node before we, you have a corresponding IO CR and you could have a case with or without a share cash where the node exists in the cache, but the CR doesn't exist or vice versa. The CR exists and the node doesn't exist. So all of those things can still happen, even if you have a single cash.

A

So those will be my overarching questions. Maybe once you go through the yeah.

E

And then, as as there are two cache exists, coexist in in the current implementation, I mean between the entry, plugging and out of tree plugin. So there is a time interval between the time T1 and time T2 uh specified in the second page just now, and so uh I I think our proposal can reduce the likelihood of the scheduling failure in that time interval. So, although the probability is very low, but it's kind of uh we can help to reduce this. The probability in that time into yeah.

D

C

Yeah I think that's the key motivation and we should we have to address click. Clearly. That sounds very significant perspective.

C

Yeah, as Aldo said, if the inconsistency is coming from that to Total independent objects, distinction issues, then even with the native sharing cash for both entry call AP object as well as CR. Then this kind of issue cannot be resolved. For example, you have a nailed object. You have a node related, this I go CR, but they are not guaranteed to be always stick the status so yeah. That is that point we we should. We have to adjust clearly whether or not this can fundamental. This result.

A

I guess my my point, my point would be if we are just reducing the probability, I, don't think it's worth, um because we will be adding likely some complexity and we are not even going to solve the issue. We are just reducing the probability that to me doesn't sound like a good investment.

E

A

Let's go through the proposal, Maybe reference. All of this.

B

All right, so we cannot, um we won't I mean it will still exist. The inconsistency right just reducing the probability. It's not very strong reason, I, think you know the the other reason is um that you know we would like to reduce uh to increase the developer velocity to reduce uh you know, like you know, the um each developers uh so why we need to. If for each scheduler plugin, we need to develop a separate cache.

B

We need to repeat duplicate the effort of you know um creating that cash creating the handling, creating the event handling logic.

A

I'm not quite understanding that, because I guess you shouldn't build a cash for objects that are already cash like you, shouldn't have a cash for pots, for example, or nodes, and you would only be adding event handlers and a cash for all the other objects that are not already there in the existing cache.

A

So again doesn't sound that variable or that it would reduce the developer speed for one, uh but I think we are maybe just like thinking too. Theoretically, maybe just let's just go through the proposal and.

B

There's no need to add additional resources to the local cache. Is that what you mean.

A

Yeah I I'm I'm not seeing major benefits over just having your own cash for your own objects in the plugin, which it might be more, maybe a little bit more I'm, still even having a hard time thinking. That is a lot of effort, but even if it is complexity that doesn't affect the rest of the existing scalar code, which.

A

Yeah, it might slow down the development of the scheduler as a whole. Well it just optimizes for your use case of a of uh out of three plugin. So that would be my hesitation, but uh please please go ahead through The Proposal yeah.

D

B

Not just out nice for our plugin, it's for all the scheduler plugin. If you know we have a test extension so that each each schedule per view in the future, the new schedule project does not need to implement their own local cash yeah. But that way we can see you know the effort and the benefit whether it's a strong case, yeah, okay, Teresa, sorry, go ahead.

E

Okay, let me proceed with the proposal design and the core idea behind this proposal is to add and cash extension to the current scheduler cache, which is highlighted in the red in this diagram on the right hand, and as you can see that, and let's take the description where scheduling schedule plugin as an example, uh the available discount or support discount options, block size and and rewrite ratio. These attributes and their accounting method will be included in this extended cache and add the filter hook.

E

Point the entry plugin would fetch fetch the allocable resource and consumption resource from the scheduler cache, while the discount plugin will retrieve the discount available bandwidth from the extended cache, and in this case the scheduler will take care of watching the port and node events to update the event specific resource and and both resources in, in that case, the out of tree schedule. Plugin does not need to implement, uh do a use, another Informer to and maintain their own cash, and it will simplify the scheduled plugins development.

E

And, as a result, some new interfaces and apis's are required to be supposed to be introduced to describe the resource and customize the resource of accounting methods and also manage the life cycle of the extender source.

E

um Okay, so would you move to the next page?

E

Here is an uh reference design for our proposal, and this is a class diagram and the the cast diagram illustrates the modification to existing classes and interfaces in this current schedule, which is highlighted in blue and the new classes or interfaces introduced by our proposal, which is highlighted in green and some new classes that supposed to be implemented by vendors or the scheduled plugin developers, which is highlighted in Gray.

E

So from the left hand on the Node node in first chart is proposed to be extended to hold the extended resource. It is a map. The map's key is stands for the resource name and the value keeps the uh when the specific specific resource information and the extended resource interface uh is an interface that manages the resource.

E

Accounting method should add port and remove Port methods and the schedule plugin developers or the resource vendors can customize their resource attributes, and then there are accounting methods by implementing the extended resource interface and the extend resource handle is an interface to manage the extended resource and to keep the external resource up to date. The run method of the interface is to retrieve the resource info and and initialize this resource in the extended cache I mentioned the resource vendor here is because that it is, it can be in our scenario.

E

It can be an independent row different from the scheduled plugin developer, meaning that, because the resource vendor knows best about their products, the resource vendor can Define the can implement the extended resource interface and the external Source handle interface to define the resource attribute and the result how the source is accounted and the schedule plugin. In that case, the scheduler plugin developer can use the extended resource handle straight away without the need to know too much about how the resources counted for making their scheduling decision.

E

So finally, the scheduler plugin would, in instantiate the external Source, handle and retrieve the extended resource through this handle to get the information so overall, I think the chance to this current schedule is very limited, but it can largely simplify the scheduled plugins development and in the backup session, there's a reference, implementation and, and a simple example to demonstrate our our source for reference. If you have interest.

B

Yeah but probably will not go into that detail. Okay,.

E

So any questions.

C

So, by extended resource you mean CR objects right.

C

Now I just wanted to say distinguish with the the standard. Kubernetes term is standard resource because in kubernetes there is already a predefined name called external resource. Yes,.

E

E

It is an object stored in in the no in the new data structure we introduced from our proposal, so it is different from from the kubernetes turn. Okay,.

A

Where is the information coming from? Is it a different CR object.

A

In the pod, or in this case, I guess node, because it says no name for extended resource where.

E

A

The information coming from.

E

E

The though, will take care of how to take take care of it is because it will retrieve the information through and CR or from from a service or from for other for other source. So it it didn't, it is not fixed. It depends on how the vendor resource vendor defined it.

B

Yeah, so so you notice that we did not limit. Like you know the where the resource extended resource will come from I could could come from that I know, patient could come from. You know on telemetry collected by the you know uh by the um platform and feedback into the. You know that real-time resource information- it could come from that too.

B

We need to collect real-time um resource um information, Elementary information and do scheduling based on that information, because you know it's not like a fixed value it. So the the information depends on other um attributes, attribute characteristics of the workload like block size for this type of block size read, write ratio, something like that. So it's not like you know, uh fixed like it. Yeah.

C

I I just want maybe trying my best to to summarize the motivation. He may be just correct me from a while, so you seems to want to say the requirement is that once some senior object has very close direct relationship with the objects that is cached in the entry, not interested in the scheduling framework, which is you know, uh if, like you, have customer resources object that has closer relation with that. Then, if you define the different Handler in Alpha tree, then that can make is sort of inconsistent.

C

That is why you want to add a hook to the scheduling framework so that when the note a part gets updated, you can have immediately changed in one synchronizer call to update the cache. So everything for Notepad, as well as you'll, see objects, are always in sync, so I'm trying to assimilize, but not sure if it's the real motivation that you want to introduce, let's just scan your framework. Is that true.

C

um So that's that's.

B

One motivation but I think uh another motivation is you know on each schedule? Plugin developer does not need to read to implement. You know the event handling logic.

B

Another one for cash: you know we have to re-implement all this logic and if those logic, if we can leverage the existing arm even handling logic in the entry schedule, then we Do Not Duplicate. That logic.

C

Well, because I separated the email Handler. If there's no thinking issue, no inconsistent should sell some better design, comparing to have everything in the screen framework, because, for example, if you have totally independent three object, has nothing to do with node objects doesn't need to be, uh but with a node update, I, don't think that the piece of great benefits to put them all together in in the in the scheduling framework. So my my point is that this proposal, you have a strong case.

C

That is the inconsistent issues that blocks the out of trade plugins functionality.

C

That is my point.

F

I think we may I just have comments for that. Yeah.

F

Some comments about that yeah uh regarding to this diagram. Suppose, uh let's assume what a general out of Chip plugin will Implement and the first the uh in general. If the out of three is current plugging will handle additional results. That is also it will be a descale, so it will first, it will Implement a local cache to save the discount information and this information may get from the CR or we get from some other metronides. This is one thing the the current price is to be implemented then.

F

Secondly, the plugin needs to be incremented even 100 match new informal to to get to the notification for the node or the port change, so that if the new node added or new node deleted on the port ad deleted, then it can update the local cache and the two to to update the node capacities. So this is the second scene, the the plugging you to be done under the third thing is that the plugin used to implement the the filter or or the score machine like what is the other plugin needs to be done yeah.

F

uh This is the current implementation. If we want to implement a scheduling packet but suppose in this case and uh yeah as you just decide, uh because when we start a scoring cycle, we want to avoid the inconsistent issues so that, before the scheduling scheduling before a scoring cycle is start, we will create a snapshot. Then the snapshot will be great for both the entry plugin.

F

This will be used for the CPU membrane and the first auto-ship plugin, which will be used for the uh for the design, for example, in this case yeah, but in case we have a event like the positive it and the part list will get the the informal of the entry plugin for the CPU memory.

F

It gets the positive so that it released the CPU memory for the for the port for this node with results this part, but at the same time the auto chip plug in the informal does not get the notification for uh download notification, so it do not remove the disk IO information from the node. So at this point, I to the time t uh the the schedule of uh one scheduling cycle.

F

You start it creates a snapshot and, at this time the the snapshot quit time it actually with a contest, and this contest it is to see for CPU memory. It does not have the Pod, but for this aisle it has a port, so this maybe create some of the uh inconsistent issue. So this is what uh we see if we using different informal for the node part for both entry and the auto G plugin. This may cause some inconsistent just when you search the snapshot, yeah.

C

So the new implementation did you, you did you start another Informer Factory rather than reuse, the informed Factory that is already contained in the frame framework handle.

F

Yeah yeah well I think we will create another informal, we're.

C

F

The gold client, informal Factory, why.

C

You create a new one, instead of just reusing the the one that has passed past, the cheetah print framework handle.

F

uh Yeah I think in the implementation I we can get a informal uh informal Factory from the the framework we.

D

E

F

It to create uh yeah I.

E

Think we're using the the informal from the framework yeah.

C

Okay, yeah, then.

C

Okay, I think that is back to my previous point is that if you're using the same form of factory, then.

C

You can still you get the same copy, but you, uh your informal functions are invoked. Maybe at different times, then it costs you responded differently and hence have different cash. So that cost your function. That's what.

F

Yes, what I mean is the schedule will still work, but the the uh maybe this time is work. It will go to another scouting cycle, yeah, okay, so so, but if the time is different, for example, one is a T1, another is G2, but when you created a snapshot, it has a Time t under the t is between QR and the t2, so you will get a contest and this content will include the different, maybe the Pod information for the results for this node.

F

So in this time this may be called some inconsistent scheduling issues, I think uh if the node is not quite much yeah.

A

um We have another Topic in the agenda, so maybe we can work it here, I guess. Ultimately we have to go through the cap to the side, um make sure in your cap.

A

You describe the use cases as sources of information, because, if I think about, if I think about the case where the information is in another CR and then this doesn't fix anything um I think what you're trying to do is maybe getting some metrics. Perhaps maybe that's one use case um or one source of information to get to you know gather the you get the uh the event handler for pot for node, create, let's say, and then you go and do a query into uh into some some Metric framework.

A

Maybe that's one one of your use cases, I, don't know um what I'm trying to say is. Please describe what are the sources of information uh your the out of three plugin might use. So we understand why you need this extra cash, um because if it's in another CR, then for sure this doesn't doesn't help if it's somewhere else maybe makes sense. So please, please make sure, that's included in the proposal so that we can take a more informed decision.

B

Okay, yeah I think we will do that yeah, we will as a mesh as I said before we got the information from the metric from The Matrix yeah.

A

Okay, real-time.

B

Information and.

A

If you have any other sources of information, please include them. Okay,.

B

Okay in the cap right- okay, we'll add clarification on that part yeah. So we we submit a cap, um um so we also make a cap to the to the sixth scheduler repository right. It is.

A

Repo I think you already opened one. Oh.

E

Well: okay! Yes, it's.

B

Already there yeah- oh it's already there: okay.

A

Okay, so with that, we have a few minutes for the next topic.

A

um Patel are you here.

D

A

Do you need to share your screen or just.

D

Yeah I, don't know yeah just talk, so okay I do not have any slides as such.

D

um So what my question is, uh as of now, all the preferred plugins run serially and it, uh and at least from the default ones default pre-filter plugins uh I can see that if we parallelize, if we run the pre-filter plugins parallely I, don't think there will be. There will be any uh race condition as such.

D

So is the so so maybe just a question it has. This. Has this path of parallelizing all the prefetter runs bin uh explored previously or not? uh If, yes, why this has not been taken into account or why this has been rejected. If it has been.

A

um I can give a quick idea: I, don't think we have tested it, but uh in the current implementation, most of the I'm willing to say all of the pre-filter plugins they do parallel calculations Within.

A

um So, for example, in no resources, Plugin or the topology topology spreading plugin they as soon as they start. They opened some parallel computations for all the for all the nodes um and then some of them might actually have some locking to access a shared resource. So we already have long structure. We already have parallelism at that level within each of the pre-filter plugins and they are highly parallel. So I'm.

E

A

Sure, if paralyzing everything will overall help or actually it might even slow down um the calculations, as you know, maybe you don't even have more uh course to do more parallelism or maybe, if you do more of them more of those uh each of those routines were start colliding with each other because they want to get a lock do the same thing um so.

A

Worth exploring but I doubt I will bring uh uh benefit um Abdullah way. Do you have anything to add.

C

Yeah I I think the same way. That is, we only have a couple of usually we don't. We usually I just have a couple of pre-theater plugins and uh not everyone goes through other nose all right, so some of them are just to uh simple check so I'm, not sure paralyzed them as a more games.

C

Comparing these complexity, it brings in and also seen some Edge and then also in some edge cases. If some pre-pricator can real quick and then the federal parking is perfect, plugins can abort. So if you just in paralyzed Rundown, you have to another, maybe cancel function to pass that cancel thing you know to uh already running prefers running, so that is pretty small complexity. I, don't know so yeah and um I think unless we have a clear requirement and have clear metrics your scenario you can give us government burst.

C

In most cases we will adopt otherwise I think we will stick to the currencies methods.

D

Okay, understood is.

A

This coming for from a particular scenarios, how you run some tests or any.

D

uh No just a general question, not a particular scenario as such.

A

Okay, um if you're, if you're willing to explore by all means, um um but ideally we should have- we should run the benchmarks. There are some benchmarks already in the test: slash integration, perf. What's it called test, slash integration, slash scalar perf.

A

um Those might give some idea of whether the the change is worth but um yeah probably have to test in different sizes of of.

D

Machines, so is, it is the benchmark test set up in a separate repo or.

A

No, it's in the same report test in the test. Folder test, slash integration, slash scalar, perf, yep,.

D

Okay, I'll check thanks.

A

um Yeah, if you have some some numbers uh that justify the change, you can consider it, um but.

D

A

um With that we're one minute, we have one minute left: um we have about uh three or four open caps right now. We have some time until until the the cap freeze, so we have an assigned reviewers yet to each of the Caps um uh I think by the next meeting we might have maybe a a more complete list of of caps, and then we can do and run down on all of them in this meeting, um but yeah any last minute. Questions.

A

Right, thank you all for joining uh we'll see you in a couple of weeks, bye.

E

Thank you, bye-bye.