Kubernetes SIG Scheduling, 18 May 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Scheduling Weekly Meeting for 20230518

Description

Kubernetes SIG Scheduling Weekly Meeting 2023-05-18T16:58:56Z

A

All right, um hi everyone, so this is our sixth schedule meeting um for May uh 18th. um As you all know, this meeting is um schedule uh recorded I will be uploaded to YouTube. So please add here to kubernetes code of conduct during the meeting, um so we have three items on the agenda for today.

A

um Although you want to speak about the new uh contributing documentation that we have for six scheduling.

B

Yes, so, first of all, thank you to Kant and I can say that work on this, this document, but basically uh they have. They have collab collected a set of guidelines for contributing to this thing, specifically, of course, this Builds, on top of the guidelines for the kubernetes project, but we have a few details here about how we we are operating, how we are already operating or how we have been operating for the past few years.

B

um Things like getting a review from a reviewer before involving an approver.

B

And yeah and things like squashing the commits when ready to marriage, but not not before first add new commits when, when you're in review things like that, so I encourage everybody that wants to collaborate to contribute to to see scaling to to read through. Through these documents uh we might be adding uh an extra document a little bit more focused on on common common practices during during core reviews.

B

uh So you can expect that in the the next few weeks- and that's it- that's all I wanted to say about this. Thank you.

A

Thanks Aldo, um and we have.

A

To talk about the new product, dashboard I made you a co-host. If you want to present.

C

uh That would be great. Thank you. Yeah.

A

Yeah you're a co-host already.

C

Okay, uh let me share my screen now.

C

Let me know if you can uh see the document or.

D

C

Okay, so uh about the automating uh for the sex scheduling, dashboard I just wanted to you know, like I reach out to uh the scheduling, leads and um asked. If you know like, is there any way that I can help so one of them? One of the uh you know like few tests um uh was what there is. We have uh six scheduling um aboard, but we don't have any way to automate it. So it's all like a manual effort for now so uh I went ahead and I.

C

You know like tried to find what other sects are doing and uh basically they uh they saved us. They already have.

C

um You know, like a solution for how to you know, like uh you know, like get projects automated without having GitHub action and kubernetes uh organization, because it's not uh permitted yet uh we had actually this discussion with the um the you know like the GitHub admin group, but uh because of other security reasons, uh they don't prefer to have uh GitHub actions enabled for kubernetes uh Organization for now so because of this, uh take off actually went ahead and uh they created uh another uh or a separate trip in kubernetes sets and uh just right and go uh ago program to to do the exact same thing that GitHub actually can do which get the projects list.

C

All the issues by you know like specific label uh and just update the the dashboard. So uh basically, I did the same I just um uh forked at the cerebral uh and um instead of just you know like having this GitHub workflow working for specific, uh take off I, had it working for uh the six scheduling um and I already, you know like tested it, and it's working as you can see here so um once you know like this, this GitHub action can can be run um every couple of hours once it it's run.

C

Everything related to say, scheduling with lip and six scheduling would be a meteorage and we will just need to go ahead and you know like uh find whatever we want to add it to our backlog. And, of course, you know like uh I'm, trying to add more automation on you know like. If there's you know, like storage accepted, we should move it to in progress or something like that, but it's still not done yet, but at least we have. We have.

C

You know like this dashboard, automated somehow I'm working with the um kubernetes org now to add to either add this reboot as a new repo. So we can maintain it as a scheduling or I I had another proposal which thank you I'll do for that, so it was like if we can cooperate with the sick ass and have just one repo and a couple of other GitHub actions on Bob.

C

You know like under under you know, like the tooling uh repo, uh so uh you know like, instead of just having another repo to maintain, we will have one across all six, but we still have this discussion open. We didn't have this yet finalized, uh but again, uh I I went ahead created some views just to help people here, so the enhancement uh will have just a specific view, so you can go ahead and triage it once it's triage.

C

uh It will have like another section, just to be more focused on the enhancement for this specific uh Milestone and this. This is like the Milestone issue so far um yeah. So- and this is the issue this is the PRS abused, so.

B

C

Just you know like to help us. uh No, you know like to you know like what we have. What we need to you know like to get done and for new contributors to just you know like pick anything from the backlog and work on it, everything works yeah. That's it any questions.

A

Thank you very much. This is really great, so.

B

A

This is now like um released like is it? Is it functioning, or this is a prototype.

C

So this it's functioning from my uh from my Fork but I wanted to make it uh you know like um available for for the, for the sake sake, scheduling leads, uh so they can run this. You know like GitHub uh action whatever whenever they want, but.

D

C

This this is like my my Fork right now, because we don't have like final uh home yet for this. But again this is a current job, we'll run uh every day on 6PM 6 a.m. Pst- and this um you know like this, uh this dashboard would get uh will get updated uh for now. Yes, this is working, but from my work.

D

Thanks I think Hebrew is trying to propose that he she moves its own Fork to a sub project. Just like seek us that before we reach a agreement to have a unified repo to manage all the text to have a general or different GitHub action, but in the in the same repo yeah. So before we reach to that, maybe we can have a sub project to manage six scheduling, specifically I. Think there's a heat passport also.

C

Yeah, but for now it is working, but from my work, what I'm trying to get it done now is uh moving moving. This work to be either a repo, a separate triple for a sex scheduling or just another GitHub action on a unified repo for all uh things uh they wanted to to do the same.

A

Right so, basically like the dashboard is getting updated.

C

As long as exactly.

A

B

A

B

Has anybody from seek all said anything about this.

C

Not yet, but I can reach out. Who is, who is? uh You know, like the main, the main uh contributor to take off uh I I, know him um so I can reach out to him and see you know like how can we cooperate and just you know like rename this repo and at our GitHub action here and we will be done, but uh just uh I'm waiting for people to just you know like contribute to this issue. If there are any other opinion.

D

Or he might do a thing that it is possible to just add one GitHub action demo to their repo, so we can just contribute there.

C

Yeah, this is what I just uh mentioned, so this.

D

D

You need to sort of duplicated Ripple, just working for sick scheduling. We can just have a another yeah I. Think GitHub action. Has this separate yeah yeah.

C

This is what I have already. This is what I proposed. So instead of you know like having another reputu, uh uh you know like to take care of, but again I'm. Just you know like I. I will wait like cup. You know for today at least and then I will lead to a niche and ask if they are okay with you know, like renaming the the repo, so we can contribute back our GitHub action uh directly instead of having just a separately.

D

Yeah I think that can reduce any diverge developing developing efforts. Yeah I just have one unit of.

C

Fun and I believe other other segs. They might benefit from this as well, because.

D

C

They might need to have the same. The same automation as well.

B

Yeah, you might, you might also want to reach out to to contributor experience, because this is kind of uh if, if all things are supposed to be using this same Ripple then seek contributor seek contributed. Experience should have the ownership of the repo rather than seat off.

C

uh You're right, I will mention them and I will reach out to them as well.

B

Mm-Hmm yeah, okay, thank you. Yeah.

C

Thank you so much I'll stop sharing.

A

um um Can we put a link to that dashboard on uh in our last document? In our like, you know, a community page.

C

Sure I will add it right away.

A

A

uh Let's uh so.

D

A

Is the page I guess? Maybe we can add it here.

C

Yeah working on it right now, uh oh you wanted me to add it to the GitHub. Okay, sure.

A

Yeah I mean that would be I, think useful.

C

A

Don't know or maybe the contribute I think both Pages would work. Yeah.

C

I will add it to the agenda that the sake scheduling agenda on top of you know like this. um You know like the intro paragraph and I will work on adding it to um the GitHub pages.

B

It has to be the contributing file, because the readme is how to generate it.

C

uh Yeah yeah sure yeah. Thank you. Thank you.

A

All right, um although you want to bring up the recube extension point,.

B

Yes, uh mostly As FYI or anybody interested.

A

Was the original proposal updated to include the new right.

B

What one, second, uh okay! So um no actually the document was not updated, so everything is in code at the moment, uh but um so just to go, give a kiss to everybody.

B

um Basically in scheduler, uh uh whenever a pod is scheduled and it fails, it enters this intermediate state which we call unscalable um and then certain events in the cluster can bring that pot back into the into the scheduling queue to be retried.

B

So, for example, if you failed, if the post failed because of uh Paul at the affinity, then, for example, a port deletion could help could help the Pod schedule. So it's that that's one kind of event that could could make the pots get in the blogging so and then we have. um We have a few uh hard-coded.

B

A few hard-coded checks that filter this kind of events to decide whether a pot should go back into the scheduling queue so, and this proposal is uh doing two things. One is um optimizing or adding more control uh over over over these checks through a quick, heal extension point. So uh in this extension point let me just show a little bit of the code.

B

B

So here is the extension point: uh it accepts the pot that is being considered for retry. It accepts an event which describes what happened, for example, a port changed, a pot was created um or a conflict map. There are a few, a few uh static events that could happen, and then we have the old object of the event and the new object. So if it's a bot update this odd object will be a part.

B

This new object would be the new version of the pod, and with that, uh the plugin can decide whether this event would could potentially make the Pod schedule so going back to Infinity.

B

If there was a port Affinity failure whenever uh Whenever there is a new part that satisfies this Affinity, then the the original part will be recruited. So that's that's the the gist of the solution, and um so, and so this can be applied for existing plugins. We have many plugins that would benefit from this uh and the the effect is that, let's retry, less less Requiem means less retries, so better throughput, uh better, better latency for for every pod.

B

So that's that's one effect, and the other effect is that. Well, now any Uh custom scheduling, plugging or is out of three scaling plugging can Implement these mechanics on their own. So that's that's the the idea now.

B

um The contention here is that uh there are two proposals that kind of do the same. One is this free queue extension uh proposed by can say, and um the other proposal is by um yes.

B

Which is actually explained currently in two PRS I only had the chance to review the first one, um but these two PRS which they here they are called filter, callbacks uh and I- didn't have a chance to review this one yet, but uh at this point of in time they are very similar, uh so yeah I just wanted to bring it up to the attention of every anybody interested in the topic to uh review and comment.

B

um I, don't know if way or Abdullah, you had any chance to look at them. Yeah.

D

So yeah, basically this yeah to also here so by the last time, I tried to do a tldr and I think by that by my understanding by that time was like one and a half weeks ago, so I posted that in our sixth January select channel.

D

So if you haven't uh look into two solutions uh that tldr may give you a very high level uh fearing on some of these two solutions, a difference at but after that can say, may some Evolution changes to the especially to extract the original proposal, that is to mixing the pre-ink. You extending point to another ritual which I think the semantics is more clear, so yeah, so I need also to to catch up with his latest design, the req and to so.

D

Basically, there is some nuances here in two different solutions, so yeah I will try to squeeze some time this week to to catch up with this.

D

So uh did there any like Benchmark results or any strong opinions from either Patrick or can say the comparison of these two solutions.

B

um I think Patrick is okay with The requieu Proposal um I, not sure if there was a chance to run benchmarks. Okay,.

B

Personally, I had a better time reading through the Riku implementation So. Based on that um subjective feeling, I mean trying to to accept the requieu proposal. Yeah.

D

I will follow up with this because of the original wines like mixing some other semantics to pre and Q, which I don't really like, but right now, just extracting into a new Center Point looks better, but I need to look into the implementation details, especially when the Reiki plugin was invoked.

D

That is the primary details that I need to look into.

B

um Another thing I wanted to discuss or highlight is the fact that uh this proposal is exposing the req as an extension point mm-hmm, uh which is visible in the in the kubernetes. The cubeconf cube, scalar, config, uh configuration um and I, don't think. That's necessary. I think this could be a hidden extension boiler like an implementation detail of all the other plugins.

D

uh Can it be merged with the what's that yeah three in queue extension or something that there's an interface is an implicit contract? We have.

B

I, don't think it would work like that, because free Q can be triggered by filter or it could be triggered by permit.

D

B

Or it could be pre-filter, so it's actually kind of global. uh So.

C

B

Is it's a separate uh extension point, but at the same time it's like um it's not independent right. You wouldn't have a pre in sorry. You wouldn't have a a plugin that only implements req.

B

D

Would always say: that's a that's the same K, that's the same case for for in Q extension, because I think in the original design document of the enqueue extension I mentioned that it should be only implemented by some plugin that has the power or ability to playing a part, that's unscheduable, so that means a pre-filter can have can Implement that a filter can agreement that that score.

D

No, because you don't impact the schedulable results, so everything is forced into the same character, but anyway, I will check the details of the latest design of the req yeah, because if we want to extract into a new extension point that has a lot of efforts like API, etc, etc.

A

So I looked at the original proposal and I had the same pushback like it's a really um confuses things like with the with the pre-inq semantics, especially with the return types yes I didn't like at all. uh It was quite confusing. uh I, like the new Direction having a req I, do agree that, ideally we don't want this to be a new.

A

A new plugin uh I mean a new extension Point, explicit extension point and I do see it as a like an extension 2 filter like it should be fine, because that's where we would decide whether it makes us to really enqueue or not like that schedule, ability of the part um I wonder because, because like we would be implementing this for all filters right uh anyway,.

D

S I think I think so almost yeah, that's a same case for in-q extension, which almost all the plugins throughout the packings implement the events to register. So maybe we can use the this chance to combine them together, so one contract, at least for one interface, maybe have different functions, but in one interface will be much desirable.

D

A

Yeah I know that there is like subtle differences between this and the uh event, handling that we have right now- and this is my only concern- is my other concern here.

A

um I had hoped that we would be able to measure them somehow.

A

But yeah because they they in some sense so try to solve the same problem right.

D

B

Sorry, marriage, uh which two things.

A

We have already a way to try to fill like you know, avoid recuring.

B

uh It rechecks.

A

B

Yes, I agree. They they should be put back uh somewhere in the pr that is have to do to do that. To move all the pre-checks into.

A

B

Just like, like.

A

What the way it did right, like the the whole uh event, handling thing right.

B

The the filtering, the event filtering, yeah yeah same.

A

Like they're, not exactly the same but they're trying to solve the same problem like different parts of the problem, but at the higher level the problem is the same trying to avoid um you know unnecessary scheduling.

A

A

But yeah um is this being proposed for this cycle or.

B

Ideally, yes um at least some form of it, some reduced scope and that's another reason why we might not want an API so that we can iterate more internally. And if later we see the need for an API. We can expose it, but we can start without an API.

A

A

Especially that they are fairly like free, enqueue and re-que, also going to be confusing yeah.

C

A

For cluster admins.

D

Yeah, the real semantics is to specific to the people who really cares about the scheduling efficiency, especially, but for that means no, they they don't ship, hear too much about this. Actually.

B

Is this what you were referring to the events to register.

A

Yeah I mean like as a mechanism in general.

B

Yeah yeah, it's being put here.

B

My comment here was that uh it should probably be part of a framework instead of being pre-implemented here but yeah I guess um what I'm hearing is that we are leaning towards accepting these proposals or like one of these two uh but needs reviews.

B

um I think that's it from my side. I don't know, I, don't have any other questions.

A

All right, thank you. So much any other questions from the community last comments.

D

I, have one minor show if others don't have questions. I can grab issue to bring your attention.

D

Okay, so yeah I would just wait for other. If others have questions.

A

Please go ahead: okay,.

D

Let me share my screen.

D

So why is that, when I, when I look into some the plugin called volume Zoom, it's Oscar I came into this lines that so basically this plugins try to uh fill that out. If a past Associated PVC has been bound to a PV and that that PV carries some topology labels, like kubernetes.io slash house, name, slash Zone Etc. So if that TV carries those labels, we will check in the filter whether the node I also have those required labels and whether their values match, so that that is the intention of this plugin.

D

But when I look at this line, this means bit well check whether the node has any topology label like the topology or domain dot, kubernetes style Etc. So in some cases, if a node is not properly labeled, that means the filter will return success. So I'm wondering if this is a price which improve, because when the filter is hit, that means the prerequisite is that the part is really requesting a PVC and the PVC has about PV. Otherwise it won't return earlier here or the state here.

D

So that means, if the logic is which he is, that means that the part really has a PV Associated and we do need to check the the the topology labels presents.

D

So uh I'm not sure if we should disregard this so that if a node is not carrying the required PVS to project labels, we will return.

D

The pop cannot be scheduled through this note because I want you to know production environment some knows but somehow are not labeled properly, and then that the node was also considered as a scheduling candidate and somehow then the winds out in the scoring phase that then in one time well, that part cannot be running so that's the whole background, I'm tagging, uh Michelle and Jetson and as well to see whether they are issue for your information.

A

D

A

I mean sorry, I was just saying that we can't comment on the issue.

D

Yeah, that's funny. I, don't have any other questions or comments.

A

So it's not a bug right, like it's more of an efficiency.

D

Depending how low we look at it? Okay, yeah, it's like a move for the torrent to that means, configuration and now that will be you treat it as an enhancement that was it.

A

Right, thank you all and see you in two weeks and thank you all. Thank you.